Slashdot Mirror
How Peer1 Survived Sandy
Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"
I would have thought that they barricaded the doors and windows with wicker baskets and throw pillows. Wait...
"A casual stroll through the lunatic asylum shows that faith does not prove anything." ~Friedrich Nietzsche
From what I hear, based on the StackExchange podcast, and the tweets that went out from SquareSpace and StackExchange during the whole idea is that Peer1 had a complete failure, and it was only due to the hard work of their customers (SE and SquareSpace) that the datacenter was able to remain operational. If your customers have to start carrying buckets of diesel up 17 flights of starirs, you, as a datacenter have failed. Peer1, left to their own devices would have just let the thing shutdown, and apparently head office wasn't aware of how bad things even were.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
http://xkcd.com/705/
From the story it looks like they specifically did NOT ask the employee to do it. He took it upon himself to find people to do it. Including customers. I would say make sure that guy gets a raise and part of the profits that he kept running.
The myriad of regulations that were probably broken during all this should turn a watchful eye. Luckily no one was hurt and nothing bad happened, but it was just that, luck.
In total darkness, up 17 flights of stairs, with a flooded basement? Sounds like a recipe for a potentially fatal fire. People's lives are more important than a freaking data center. Sorry, but I don't see this as a heroic story about people trying to keep critical infrastructure running, but as a desperate failure that could easily have turned into a disaster. They never should have gotten to the point where they're continually carrying fuel up stairs. It also sounds like they then decided to pump fuel up a pipe they installed in the stairwell. That doesn't sound terribly safe either, especially when done in a mad rush like I'm sure it was.
Gee.. couldn't have someone planned for this contingency rather than this sort of haphazard, dangerous sounding plan that was thrown together?
AccountKiller
----Original Message-----
From: PEER 1 Hosting NOC [mailto:@peer1.com]
Sent: Wednesday, December 05, 2012 4:04 PM
To: @peer1.com
Cc: @peer1.com
Subject: DAILY UPDATE - NYC DATA CENTER - December 5, 2012
Dear Customer,
Our facility engineers have identified an electrical explosion located in basement 2 that caused the building to flip to generator. Commercial power is available but the building advised that we stay on generator until it is safe to do so.
More updates to follow once available.
--
Network Operations Center
PEER 1 Hosting | Ping & People
1000-555 West Hastings Street
Vancouver, BC, Canada V6B 4N5
All I can say is you damn well better reward all your employees that helped. They kept you up and kept your revenue stream moving. You need to give them some kickass holiday bonuses or you're all major douche bags.
Oh please. That is about the most boneheaded thing he could have done. Stairwells full of open buckets of fuel being handed off to people. What could possible happen if a fire occurred? The guy should be fired for needless risk to not only the people, but the building in general. Just so he could keep his stupid uptime hours going after they had already notifed everyone things would be shutting down at any time.
"The guy should be fired" Further proof you cannot please everyone 100% of the time. Boneheaded, yes, dangerous, too. But it worked, didn't it? And now Peak10 gets all kinds of free publicity. The guy deserves recognition.
My mistake, Peer1. Peak10 is a company here in my town :)
Did you seriously think they were carrying diesel in open buckets? They were almost certainly using proper fuel containers (ie gas cans). Bucket brigade is a figure of speech.
None of them can see the clouds; The polished wings don't care.
One would think, however when the specific question is asked all that is said is, "You've seen the pictures" or something like that. the only picture I saw seemed to imply they actually were open buckets of fuel.
Oh hell. I just looked at the picture in TFA. They were carrying it in buckets. Buckets with lids, but not fuel cans. You were right, I was wrong.
None of them can see the clouds; The polished wings don't care.
The original post sounds like a snippet from that Corey Doctorow end-of-the-world novel. Did they have to find parts to fashion a rudimentary lathe along the way? I applaud the efforts of the server team, but as one commenter stated, it sounds like a failure of the company's business continuity/disaster recovery plan. The cost of dealing with employees and customers in a burn ward should overshadow revenue flow.
If you consider the potential loss of a few days for being down compared to being sued into oblivion if an accident had occurred the answer as to whether it was the correct decision becomes obvious.
Idiots are +5 Insightful for lambasting hosting companies for not maintaining DR and remote site capabilities throughout Sandy.
Seriously Peer1's efforts are all one can ask for and I applaud their efforts to stay online during what has to be a worst case scenario for them aside from Pandemic.
---Up Up Down Down Left Right Left Right B A START
In the picture with the article, there are five gallon buckets. Not approved containers, although they do have lids. I would have rather had a pump and a lot of hose, myself.
Gamingmuseum.com: Give your 3D accelerator a rest.
Diesel fuel does not burn the way you think it does.
Why is it so hard to only have politicians for a few years, then have them go away?
The StackExchange Podcast had a excellent review of the events at Peer1.
http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-hit-by-a-hurricane/
http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/66762703-stack-exchange-stack-exchange-podcast-36.mp3
I don't find the bucket brigade thing that interesting. And I have little sympathy for a company that chooses to put a data center in a flood plane (and in very expensive real estate at that). What I find interesting is that the data center apparently was able to keep a connection to the rest of the world. I would have expected the power outages and the flooding to disconnect it, even if it could power itself.
I'm an American. I love this country and the freedoms that we used to have.
Having your employees stay in a emergency stricken zone that is flooded and carrying open canisters of diesel fuel to keep a data center running so that someone in California can share pictures of their cat is really not worth it IMHO.
I am sure some people were probably a little more worried about the lives of their families and themselves rather then some digital data.
I am not going to call someone a hero for this. At some point out there, people using cloud services and online storage are going to have to accept the fact that during emergency situations, their data just isn't accessible, period.
The basic fundamental problem I have about all this and what Sandy has highlighted is that the Internet was designed to be decentralized solely for the purpose of surviving natural or man-made disasters. Why is it then that a data center company creates a single centralized storage site instead of having an auxiliary site somewhere else, even on the other side of the country.
I think this is an epic fail in planning and execution. Anyone using Peer1 shouldn't be happy for putting people's lives in danger when common sense could have had them build in redundancy to their infrastructure allowing people to worry about their families more then your data.
Also, just like with Japan, don't build your backup generators at or below sea level.
I haven't thought of anything clever to put here, but then again most of you haven't either.
How many of these asinine data center advertisements are we going to get? This is at least the 3rd "How such and such data center survived Sandy!" I don't care... it's not news. You told your employees to stand in knee deep water in the middle of tons of electronic equipment and bail water? You're a god damned fool and lucky no-one got killed.
Perhaps you are unconsciously equating diesel fuel to gasoline? If a class alpha fire happened to break out somewhere enroute to the upstairs generator, they could likely have thrown the diesel fuel on the fire to put it out. Of course, that wouldn't be such a great idea for any other class of fire. Still - diesel fuel isn't so flammable as to cause any real hazard.
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
Haven't read the details of Peer1's trials and tribulations, but the situation reminds me of the Interdictor blog, about keeping DirectNIC running during Hurricane Katrina. That was one of the most thrilling blogs I've ever read.
load "windows7"
Indeed. It's pretty hard to get a fire started even if you throw a cigarette (or a lighter) right into a puddle of Diesel.
In a stairwell, I'd be more worried about diesel's oily slipperiness than its flammability.
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
If a class alpha fire happened to break out somewhere enroute to the upstairs generator, they could likely have thrown the diesel fuel on the fire to put it out
A class-a fire still puts out a lot of heat. Trying to put it out with diesel will give you a class-b fire to boot, I think.
Flash point of diesel fuel's 144F so it's not exactly something I'd throw on a fire (gasoline's flash point is -45F).
It's not something I'd want to handle around an open flame or anything, but it's pretty safe otherwise.
Don't anthropomorphize computers, they don't like it.
Obligatory: http://xkcd.com/705/
I am kind of interested to know what kind of fire hazard it would be. I would think the fumes and chemical part would be worse. When burning brush piles, we used diesel instead of gasoline because is burned slower and we had it on-hand from the tractor. It actually burned so slowly, it was hard to get it to light. It usually took a few news papers on fire and a couple of minutes before the diesel got hot enough to actually do something.
The guy should be fired for needless risk to not only the people, but the building in general.
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
"USA, land of the restrained, home of the fearful"?
Conservatism: (n.) love of the existing evils. Liberalism: (n.) desire to substitute new evils for the existing ones.
What kind of multistory buildings have you been in that can be taken down or even damaged by 5 gallons of diesel in the stairwells?
These stairwells are made to be fire resistant since they are supposed to be the safe way out in case of a fire, that means they don't burn very easily.
If they spilled the diesel and then somehow managed to fuck up and light it on fire it would burn until the fuel was gone and then they would have to replace parts of the stairwell, thats it.
True, but a spilled 5 gallon bucket of diesel fuel in a dark stairwell with bog-knows-what-else going on sounds really, really, really stupid. That's enough to basically burn the building down not to mention injuring people.
The risk-benefit ratio really doesn't come down on the side of the guy who suggested it.
You could be more wrong if you tried really hard.
But not much more.
How many people have you heard of that burned to death from a diesel fire?
Is there a reason you have not heard of such a thing?
Let me answer.
It is really hard to get diesel to burn. You have to try.
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
When it endangers the lives of my customers. Look, I'm all for stories about heroic efforts of uptime, but this company had no disaster plan. They're lucky they're not in the middle of a lawsuit for injury, or worse, wrongful death right now
I am sure he would have rathered it too. I bet he has already submited a request :)
That said, they may not be approved containers but, 5 gallon buckets even hold gas. Its what the scrap yard uses when draining cars for crushing. Boy do they get some gas doing that too. Saw a guy drain nearly two "full" buckets (probably about 4 gallons, really silly to fill them to the brim).
I dunno, if there was an "approved container" and a bucket, I would be happy to let someone else use the approved container. Those buckets are fine for the job. Kind of a bitch to pour, but, if you go slow or a few of the nicer pour spout lids that snap on tight, they are just fine.
"I opened my eyes, and everything went dark again"
They're lucky they're not in the middle of a lawsuit
In other words: when lawyers took over.
Conservatism: (n.) love of the existing evils. Liberalism: (n.) desire to substitute new evils for the existing ones.
What could possible happen if a fire occurred?
Have you ever tried to light diesel? It's much harder than you seem to think......
So it burns down in 4 hours not the expected 5, because you have no power to run a pump and cant get a fire truck to save your life? Surely all employees that go above and beyond the call of duty should be fired because they and a large lot of volunteers worked to keep your business running!
Good leaders run toward problems, bad leaders hide from them.
Those look just like the buckets that I get transmission fluid in (from the tractor store). Transmission fluid is a petroleum product, and, in fact, will run a diesel engine just fine. I'd have no issues using those buckets to store/transfer diesel fluid.
Just another ignorant American.
I'm sure the customers participated volunteered willingly. I doubt anyone will sue for spraining an ankle after having volunteered willingly to help in a crisis situation. They clearly all felt they had a lot more riding on this than just their own physical well-being.
Also, in response to some of the above posters; I think the middle of a hurricane, when your basements are flooded, you're highly unlikely to worry about someone accidentally lighting the building on fire.
...bucket brigade that carried diesel fuel up several flights of stairs...
Wow, their servers are diesel powered? Awesome!
Proverbs 21:19
Everybody likes video!
http://youtu.be/3LeRSPuA5Z4
Description: Gasoline is spilled on the ground and lit with a match. Then diesel is spilled on the ground and numerous attempts are made to light it with one then several matches.
Even more video! A guy tries to light a diesel spill with a blowtorch.
http://youtu.be/xMJQFl4az9I
When taking the decision to keep the emergency backup fuel pumps in the basement, did no one think of what would happen in the event of flooding.
AccountKiller
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
If it's my company, my money, my neck is on the line...
If I'm working for someone else--who would just as soon lay me off if it became cheap enough to do so--then I do what I'm paid to do. Carrying 5G buckets of diesel up 17 dark flights of steps while the lobby has 4' of water in it is NOT what I'm being paid for.
It has nothing to do with a lack of courage or fear. It has to do with understanding that it's not worth potentially DYING or becoming disabled in a situation where the company will almost certainly turn around distance itself by saying I was operating outside the scope of what they told me to do, possibly ending up in any insurance claims being denied, or even in my being personally litigated against...
if not for this version of "health and safety" thinking, lawyers would starve. we got all kinds don't we? NYC is not that special. Can-do attitudes exist in lots of places. When a "Sandy" smacks YOUR home town, you will eventually suffer or make out OK because there will be a few people who know what they are doing and who feel "making things work is my job". Try to be one of those people.
SLASHDOT: news for people who can't concentrate on work or have no life at all and got tired of yelling back at the TV.
If you consider the potential loss of a few days for being down compared to being sued into oblivion if an accident had occurred the answer as to whether it was the correct decision becomes obvious.
I hate this sort of argument. It creates a bogeyman about something that MIGHT happen, no matter how slim the chance, and completely ignores the fact that they did it and the worst case scenario did not happen. Unfortunately, it seems that a great deal of our lives are governed by this sort of thinking.
I agree that in a flat, well-lit environment they'd probably be okay for gas, and even safer for diesel, especially outdoors where plants and soil would cushion the fall and vapors would dissipate quickly. Falling down an enclosed concrete stairwell though... even if the bucket itself didn't break I seriously doubt the lid would stay on, and then you'd have a real problem on your hands.
--- Most topics have many sides worth arguing, allow me to take one opposite you.
most people killed in fires initially sucumb to smoke inhalation. and you're worried about the fucking stairwell materials.....
Plus, diesel is much less volatile than, say, gasoline.
"Probably someone thought about it, but decided that other potential hazards (e.g., a leak in the tank causing fuel to be soaked all through several floors of the building) were more important to deal with"
..
...
Specious logic, the position of the fuel pumps don't contribute to fuel tank leakages
"You can guess what the likelihood of each particular risk is, but that's definitely guesswork;
No need to guess, certain people are paid a lot of money to analyse the risks,
"the whole of New York really wasn't set up with this sort of storm surge in mind"
NYC Hurricane History: 1821,1893,1938,1954,1955,1960,1985,1995,1996,1999,1999,2011
Sep 2007: 4 Million Gallon Diesel Fuel Tank Fire
AccountKiller
What the SE crew did to keep their site up was amazing. They got no help from their site until way late in the game.
They counted on Peer1 to handle facilities... and they dropped the ball.
I have had multiple 72 hours outage due to local power going away. Having to plan to have fuel delivered to the backup gen-set, and having to ration power (I knew the burn rates and what equipment was non-essiental)... blah blah blah.
The bottom line is that the customers are not in the NOC, so if the organization cares about its customers... what needs to be done gets done.
Peer1 abandon ship for two days and left THIER customers to fend for themselves... and then treated them poorly when they came back to restore the services they failed to maintain during a crisis that while bad was not insurmountable.
Peer1 should get shutdown for failing to cover their end of the bargain.
Have you ever tried to light a bucket of diesel on fire? It isn't like in the movies.
HAL 7000, fewer features than the HAL 9000, but just as homicidal!
Thank god they did not spill that diesel oil on the stove in the stairway.