Slashdot Mirror
How Peer1 Survived Sandy
Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"
I would have thought that they barricaded the doors and windows with wicker baskets and throw pillows. Wait...
"A casual stroll through the lunatic asylum shows that faith does not prove anything." ~Friedrich Nietzsche
From what I hear, based on the StackExchange podcast, and the tweets that went out from SquareSpace and StackExchange during the whole idea is that Peer1 had a complete failure, and it was only due to the hard work of their customers (SE and SquareSpace) that the datacenter was able to remain operational. If your customers have to start carrying buckets of diesel up 17 flights of starirs, you, as a datacenter have failed. Peer1, left to their own devices would have just let the thing shutdown, and apparently head office wasn't aware of how bad things even were.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
http://xkcd.com/705/
From the story it looks like they specifically did NOT ask the employee to do it. He took it upon himself to find people to do it. Including customers. I would say make sure that guy gets a raise and part of the profits that he kept running.
In total darkness, up 17 flights of stairs, with a flooded basement? Sounds like a recipe for a potentially fatal fire. People's lives are more important than a freaking data center. Sorry, but I don't see this as a heroic story about people trying to keep critical infrastructure running, but as a desperate failure that could easily have turned into a disaster. They never should have gotten to the point where they're continually carrying fuel up stairs. It also sounds like they then decided to pump fuel up a pipe they installed in the stairwell. That doesn't sound terribly safe either, especially when done in a mad rush like I'm sure it was.
Gee.. couldn't have someone planned for this contingency rather than this sort of haphazard, dangerous sounding plan that was thrown together?
AccountKiller
----Original Message-----
From: PEER 1 Hosting NOC [mailto:@peer1.com]
Sent: Wednesday, December 05, 2012 4:04 PM
To: @peer1.com
Cc: @peer1.com
Subject: DAILY UPDATE - NYC DATA CENTER - December 5, 2012
Dear Customer,
Our facility engineers have identified an electrical explosion located in basement 2 that caused the building to flip to generator. Commercial power is available but the building advised that we stay on generator until it is safe to do so.
More updates to follow once available.
--
Network Operations Center
PEER 1 Hosting | Ping & People
1000-555 West Hastings Street
Vancouver, BC, Canada V6B 4N5
All I can say is you damn well better reward all your employees that helped. They kept you up and kept your revenue stream moving. You need to give them some kickass holiday bonuses or you're all major douche bags.
Oh please. That is about the most boneheaded thing he could have done. Stairwells full of open buckets of fuel being handed off to people. What could possible happen if a fire occurred? The guy should be fired for needless risk to not only the people, but the building in general. Just so he could keep his stupid uptime hours going after they had already notifed everyone things would be shutting down at any time.
"The guy should be fired" Further proof you cannot please everyone 100% of the time. Boneheaded, yes, dangerous, too. But it worked, didn't it? And now Peak10 gets all kinds of free publicity. The guy deserves recognition.
Did you seriously think they were carrying diesel in open buckets? They were almost certainly using proper fuel containers (ie gas cans). Bucket brigade is a figure of speech.
None of them can see the clouds; The polished wings don't care.
One would think, however when the specific question is asked all that is said is, "You've seen the pictures" or something like that. the only picture I saw seemed to imply they actually were open buckets of fuel.
Oh hell. I just looked at the picture in TFA. They were carrying it in buckets. Buckets with lids, but not fuel cans. You were right, I was wrong.
None of them can see the clouds; The polished wings don't care.
In the picture with the article, there are five gallon buckets. Not approved containers, although they do have lids. I would have rather had a pump and a lot of hose, myself.
Gamingmuseum.com: Give your 3D accelerator a rest.
Diesel fuel does not burn the way you think it does.
Why is it so hard to only have politicians for a few years, then have them go away?
The StackExchange Podcast had a excellent review of the events at Peer1.
http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-hit-by-a-hurricane/
http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/66762703-stack-exchange-stack-exchange-podcast-36.mp3
I don't find the bucket brigade thing that interesting. And I have little sympathy for a company that chooses to put a data center in a flood plane (and in very expensive real estate at that). What I find interesting is that the data center apparently was able to keep a connection to the rest of the world. I would have expected the power outages and the flooding to disconnect it, even if it could power itself.
I'm an American. I love this country and the freedoms that we used to have.
OSHA must be thrilled
Getting OSHA / union / bubblewrap parents involved means that those who are capable of helping are not allowed to because of the risk that some idiot gets hurt or damages something.
They have their place and time when things are normal to try and minimize the impact of a disaster, but once that disaster is in full swing, they need to sit down, shut up and let people self-mobilize to get the job done.
In the spring of '97 guys were working heavy equipment for days straight, often by the light of military flares, to build a dike that saved Winnipeg from one of the biggest spring floods in our history (often "stealing" clay/dirt from nearby farms to get the dike to the heights needed, dragging and dumping scrap cars, buses, anything they could find to shore up the water front side from erosion, etc.). Ignoring the union rules, safety rules, land procurement rules, etc. they got it done in time.
After the flood waters receded, then all the compensating processes kicked in to address the shortcomings.
I'm in my right mind and I have the answer to everything!
Having your employees stay in a emergency stricken zone that is flooded and carrying open canisters of diesel fuel to keep a data center running so that someone in California can share pictures of their cat is really not worth it IMHO.
I am sure some people were probably a little more worried about the lives of their families and themselves rather then some digital data.
I am not going to call someone a hero for this. At some point out there, people using cloud services and online storage are going to have to accept the fact that during emergency situations, their data just isn't accessible, period.
The basic fundamental problem I have about all this and what Sandy has highlighted is that the Internet was designed to be decentralized solely for the purpose of surviving natural or man-made disasters. Why is it then that a data center company creates a single centralized storage site instead of having an auxiliary site somewhere else, even on the other side of the country.
I think this is an epic fail in planning and execution. Anyone using Peer1 shouldn't be happy for putting people's lives in danger when common sense could have had them build in redundancy to their infrastructure allowing people to worry about their families more then your data.
Also, just like with Japan, don't build your backup generators at or below sea level.
I haven't thought of anything clever to put here, but then again most of you haven't either.
How many of these asinine data center advertisements are we going to get? This is at least the 3rd "How such and such data center survived Sandy!" I don't care... it's not news. You told your employees to stand in knee deep water in the middle of tons of electronic equipment and bail water? You're a god damned fool and lucky no-one got killed.
Perhaps you are unconsciously equating diesel fuel to gasoline? If a class alpha fire happened to break out somewhere enroute to the upstairs generator, they could likely have thrown the diesel fuel on the fire to put it out. Of course, that wouldn't be such a great idea for any other class of fire. Still - diesel fuel isn't so flammable as to cause any real hazard.
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
Haven't read the details of Peer1's trials and tribulations, but the situation reminds me of the Interdictor blog, about keeping DirectNIC running during Hurricane Katrina. That was one of the most thrilling blogs I've ever read.
load "windows7"
These were web servers, most of which apparently were already shut down. Not a city/town :)
In a stairwell, I'd be more worried about diesel's oily slipperiness than its flammability.
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
Yesterday, I squandered mod points that were going to expire. Today - I wish I had some. Screw the beauracrats, sometimes you just gotta do what's gotta be done!
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
Obligatory: http://xkcd.com/705/
The guy should be fired for needless risk to not only the people, but the building in general.
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
"USA, land of the restrained, home of the fearful"?
Conservatism: (n.) love of the existing evils. Liberalism: (n.) desire to substitute new evils for the existing ones.
They're lucky they're not in the middle of a lawsuit
In other words: when lawyers took over.
Conservatism: (n.) love of the existing evils. Liberalism: (n.) desire to substitute new evils for the existing ones.
Yes. Because what they were helping was someone's cat's blog stay up, whereas the risk they were taking was someone getting seriously ill from exposure to diesel fuel -- or burning the building down. (Ever dealt with spilled diesel fuel? Nasty, nasty stuff.)
Saving a web server from downtime is not comparable to saving a city. Turn the computers off and go home.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
Those look just like the buckets that I get transmission fluid in (from the tractor store). Transmission fluid is a petroleum product, and, in fact, will run a diesel engine just fine. I'd have no issues using those buckets to store/transfer diesel fluid.
Just another ignorant American.
...bucket brigade that carried diesel fuel up several flights of stairs...
Wow, their servers are diesel powered? Awesome!
Proverbs 21:19
Everybody likes video!
http://youtu.be/3LeRSPuA5Z4
Description: Gasoline is spilled on the ground and lit with a match. Then diesel is spilled on the ground and numerous attempts are made to light it with one then several matches.
When taking the decision to keep the emergency backup fuel pumps in the basement, did no one think of what would happen in the event of flooding.
AccountKiller
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
If it's my company, my money, my neck is on the line...
If I'm working for someone else--who would just as soon lay me off if it became cheap enough to do so--then I do what I'm paid to do. Carrying 5G buckets of diesel up 17 dark flights of steps while the lobby has 4' of water in it is NOT what I'm being paid for.
It has nothing to do with a lack of courage or fear. It has to do with understanding that it's not worth potentially DYING or becoming disabled in a situation where the company will almost certainly turn around distance itself by saying I was operating outside the scope of what they told me to do, possibly ending up in any insurance claims being denied, or even in my being personally litigated against...