Stan’s Obligatory Blog

The USGS Earthquake Notification Service, also known as My Pet Project, went online to the public on January 31, 2006. It all started back in about 2000, when I was talking to someone from Caltrans, and he was asking if we could set up something where they could put in lat/lon coordinates of key freeway bridges and interchanges, and then be notified any time there was an earthquake within some distance of any of them. At the time, we couldn’t do anything like that. But then, fate intervened.

We have occasional cookouts at the office, and in 2003, I thought it would be nice if I could set up a web form for people to fill out online so that I knew who was coming and what food they wanted. I thought this would be sort of like a gift registry, so I went on Sourceforge and found a little gift registry program that someone wrote. I downloaded it, and I hacked it to make an online signup for our office cookouts. In the process of doing this, I learned a bit of MySQL. And then, when I was riding my bike in one morning, I realized that a database like MySQL could do something like what the guy from Caltrans had asked for. So I whipped up a simple database with some rudimentary geographic information, I plugged in the worldwide earthquake feed, and it started sending me earthquakes from all over the world.

I recruited a few ‘guinea pigs’ around the office to set up accounts in it for testing. They suggested other things they would like it to do. At first, it could only define geographic regions as lat/lon points defining a box. People asked for circles, and then arbitrary polygons. Drawing a polygon on the map and figuring out if a given earthquake fell inside it kept me thinking for a while, but I worked out a reasonable way to do it. And while all this was happening, my little system was being shown around to everyone, until the National Earthquake Information Center saw it and decided that it should be an official product of the Earthquake Hazards Program.

We had a few old-style mailing lists that were open to the public at that time. One for worldwide quakes M5.5 and over, and two for California quakes. One for M3 and higher, and one for M4. Those mailing lists formed the initial subscriber base. I wrote some scripts to port the mailing lists over, creating an account for each person with notification rules that would give them the same earthquakes they had been getting before. All told, this made for about 100,000 initial subscribers.

Now it’s been ten years, and it now has about 400,000 subscribers. Over ten years, that’s an average of about 80 new subscribers every day. Most days get about 30-35 new subscribers, but this goes way up after big earthquakes. The largest jump was about 75,000 new subscribers in the two weeks after the 2011 M9 Tohoku Earthquake in Japan.

Because this all started from wanting to have an online signup form for the office cookouts, I thought we should have a cookout for the occasion. I made a cake, and we all gathered out on the patio behind the office. It was a nice time. And it’s still amazing to me that this thing I wrote that started out as a little Pet Project has turned into a thing. And that’s it’s used by 400,000 people. And in the end, I guess that’s the greatest satisfaction.

The Earthquake Notification Service, also known as My Pet Project, passed 400,000 subscribers over the weekend. It’s still hard to believe that something I built is being used by so many people.

I was tending to the Earthquake Notification Service admin email today, and I found this in the incoming mail queue:

To: ens@usgs.gov
From: Associate Email Administrator
Subject: Your email has been discarded.

Your email sent to [redacted]@us.aflac.com with subject line of: 2012-04-05 20:23:55 (Md 3.2) PUERTO RICO REGION 19.1 -65.1 (5121e), contains inappropriate content within the subject or body of the message and has triggered the Aflac Associate message filters. The message was discarded because of the inappropriate content.

So I went and had a look at the message, just to see what it was they might be complaining about. And here it is:

Subject: 2012-04-05 20:23:55 (Md 3.2) PUERTO RICO REGION 19.1 -65.1

== PRELIMINARY EARTHQUAKE REPORT ==

Region: PUERTO RICO REGION
Geographic coordinates: 19.142N, 65.070W
Magnitude: 3.2 Md
Depth: 90 km
Universal Time (UTC): 5 Apr 2012 20:23:55
Time near the Epicenter: 5 Apr 2012 16:23:55
Local standard time in your area: %localtime

Location with respect to nearby cities:
88 km (54 miles) NNW (334 degrees) of Little Harbour, Jost van Dyke, British Virgin Islands
89 km (55 miles) N (351 degrees) of CHARLOTTE AMALIE, US Virgin Islands
90 km (56 miles) NNW (328 degrees) of ROAD TOWN, British Virgin Islands
126 km (78 miles) NE (49 degrees) of Carolina, PR

Sheesh.

The Earthquake Notification Service, aka My Pet Project, just went over 300,000 subscribers. The small earthquakes in the San Francisco Bay area last Thursday had it close when I went home on Friday, and then the big earthquake in Turkey over the weekend pushed it over.

I still have a hard time believing that something I built is being used by so many people.

The M9.0 Tohoku Earthquake in Japan last Friday has caused a spike in interest in earthquakes. That’s pushed the USGS Earthquake Notification Service, also known as my Pet Project, past its latest milestone.

Yesterday, I saw that ENS had passed 250,000 subscribers. It still amazes me that something I invented is used by so many people worldwide.

After the M7.2 Sierra El Mayor Earthquake last year, it processed about 700 earthquakes and sent 4,600,000 messages. But at that time, that was enough that the system ground to a halt under the load. This time, the system ran fine the whole time. I checked the logs, and in the first 24 hours after the Tohoku Earthquake, it processed 308 events and sent about 4,500,000 messages about them. I’d done some re-architecture of the database last year to increase its performance, and the system ran fine this time.

Last summer, I was very pleased to see when the Earthquake Notification Service passed 150,000 subscribers. On average about 30-50 people sign up for it every day, and more after each large earthquake. And this week’s M7.2 Baja California earthquake pushed it over the top to more than 200,000 subscribers.

The earthquake also set a new record for the amount of mail sent in one day. The previous record of 2,326,000 messages was set after the M8.8 Chile earthquake in February. But this earthquake blew right past that record with 4,508,522 messages sent. And it would have been more if the database problems hadn’t slowed the system to a crawl. At the end of the first day, there were over 600,000 messages waiting to be sent, and over 500 earthquakes waiting to be processed.

Fortunately, I’ve found a workaround to keep the system from getting slow while processing large numbers of earthquakes, and I’m working on a re-architecture of the database to be able to handle heavier loads in the future.

My Pet Project got mentioned in xkcd today. As always, you have to read the mouse-over text on the cartoon to get the full effect:

Ordinarily, I’d be extremely excited by this. But right now I’m up at 4:54AM fixing the my Pet Project’s database. The M7.2 earthquake yesterday caused so much activity on the system that the database got swollen and filled up the disk on some of the servers. So I’m up rolfing it back into shape.

There have been a lot of earthquakes this afternoon. In particular, there have been these three big ones:

M7.3 2009/10/07 23:13:49 -13.145 166.297 33.3 VANUATU
M7.7 2009/10/07 22:18:26 -12.554 166.320 35.0 SANTA CRUZ ISLANDS
M7.8 2009/10/07 22:03:15 -13.052 166.187 35.0 VANUATU

In the Earthquake Notification Service, also known as My Pet Project, subscribers can pick their own location and magnitude criteria for notification, but the general rule is that the larger the earthquake, the more people want to know about it. And anything over M7 will generate a lot of mail. So I went and poked through the logs to see just how many messages it’s sent in the last few hours.

It’s currently not quite 7:00PM here in Los Angeles. I looked at logs back to 2:00PM, which is about when these big events started coming through the system. And the notification system has sent 642,590 messages in that time. That averages out to about 36 messages a second for five hours. No wonder some ISPs with automatic filters have blocked our mail servers as suspected spam sources.

Just for perspective, the system has sent about 775,000 messages in the past 24 hours. And this is not a record. The 24-hour record for the system is 943,833 messages in 24 hours, and that was set last Saturday.

Automated systems can be pretty stupid sometimes. There is some earthquake activity going on in the Gulf of California right now, and it’s generating a fair number of messages through the Earthquake Notification Service (aka, “My Pet Project“). In the messages, it gives the location of the earthquake relative to some local landmarks and cities in the area. And now I’m getting back nasty messages from various mail systems like this:

This email has violated the RACIAL DISCRIMINATION.
and Quarantine entire message has been taken on 8/4/2009 2:33:29 PM.
Message details:
Sender: ens@usgs.gov;
Recipient: redacted
Subject: 2009-08-03 18:40:50 REVISED: (Mw 6.2) GULF OF CALIFORNIA
29.4 -113.7

Or this:

A message sent from “ens@usgs.gov” to “redacted” with a subject of “2009-08-03 18:40:50 REVISED: (Mw 6.2) GULF OF CALIFORNIA 29.4 -113.7″ has been blocked as it contains profane language.

All this because the earthquake happened to be near the town of Guerrero Negro.

I haven’t seen anything like this since the last time we had a big earthquake near the Virgin Islands.

I checked this morning, and the subscriber list for the USGS Earthquake Notification Service passed 150,000 at about 07:34 GMT on Monday morning. This is a momentous occasion. I never imagined that something I invented could be so popular, useful, or entertaining. It’s come a long way from its beginnings as my Pet Project.