Slashdot Mirror
NYC Data Centers Struggle To Recover After Sandy
Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring."
I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."
When it's wet, the clouds go down
At 100 Williams Street, http://www.nyistatus.com/
My server and connections have been up non stop.
I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities.
I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.
-Malakai
A Dragon Lives in my Garage
Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).
Be seeing you...
People can be replaced. Uptime, on the other hand..
But, you see, without the Internet, it will be impossible to get all of those less essential services back up and running! After all, how is anyone going to obtain official updates from the Administration if they cannot access ready.gov, since that is apparently the only place to get them?
So, let's make sure you have the priorities straight in your head:
1) Internet
2) Power
3) Facebook Access
4) Starbucks
5) Running Water and Sewer
Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.
To close the sarcasm tag, here's also an obligatory xkcd.
On a side note, this reminded me of Cory Doctorow's "When Sysadmins Ruled The Earth".
Because below ground is convenient, the least expensive cost per square foot of floor space, and it's where all your crap you want to keep out of the way goes.
What, no cold Pizza slices? You inconsiderate clod!
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
You do your job. Many of these data centers are part of and important to communication, rescue operations, information. When you work there, you might not know how important that particular data center may be but -- you do your job.
Whatever part of the city you can keep operating is good.
Don't criticize what you know nothing about.
Why not move them to the roof? And while we're at it, do the same for all the nuke plants? A simple f*cking appliance that needs air and fuel to run and somehow they manage to spend life at the bottom of a potential indoor swimming pool.
Join the Slashcott! Feb 10 thru Feb 17!
Ahh: Diesel Control Protocol
Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?
Lets see how you feel with a few thousand gallons of highly flammable liquid suspended above your head, in a building with lots of electricity running through it, where an earthquake is more likely than flooding in the basement. And that is ignoring the possibility of deliberate sabotage. A building with fuel stored above ground level where something went wrong would turn rather quickly into a giant pillar of flame. If one of the tanks gets ruptured, all it takes is a single spark to kill hundreds or thousands.
Below ground, however, fire-fighters can deal with it relatively easily, and the flames won't descend to engulf the entire building in a matter of a few minutes.
"None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
You need to add a source of good coffee to this list.
No, Starbucks is NOT good coffee!
If I were God, wouldn't I protect my churches from acts of me?
Per the topic, the following locations are experiencing or have experienced outages:
if ($question !~ m/bb|[^b]{2}/i) { die(); }
The problem is that 'waterproofing' is a short-term guarantee. Water is insidious, it dissolves almost anything (although some things like metals very slowly), and it will eventually creep inside of any 'waterproof' container. That's why there's such a problem designing radioactive dumps like Yucca Mountain -- water would eventually eat its way into the vitrified radioactive cask.
Gas station underground tanks can survive for 10-20 years and still be waterproof. But most of the infrastructure in NYC is a hundred years old. There isn't anything waterproof in that city. Even brand new structures are probably permeable to water, if the designers just never thought it would be an issue.
Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
Fewer people died than the people who didn't die due to not being in a car crash due to not being able to drive to work. So on the life/death front it's a win!
The Starbucks is to test the Sewers with.
Like many other posters, my first question was why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious: fire codes. Duh. Thinking of how fuel is stored elsewhere, the only other option I can think of would be storing the fuel outside the building but above potential floodwaters. Not in a place like Manhattan. The price of real estate is much too high for tank farms on stilts. And the earthquake risk in New York is non-zero, so that solution might have the same problems as the current solution. So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure. Is relocating major data centers out of flood-prone areas of Manhattan (and other similarly risky areas) feasible? The potential of a major flood event in Manhattan has been well-known for a long time. Much of lower Manhattan is built on landfill. Did the builders of these data centers include basement flooding + extended power outage in their risk forecasts and just decide to deal with it if it happened?
It looks like Google was ahead of the curve after all with their idea for floating server farms.
The bigger problem is usually the pumps. You generally try to use turbine-type submersible pumps with the motor above the tank and the inlet down low to avoid problems with priming the pump. If the place where the pump motor floods, you are pretty much SOL.
If you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.
Ultimately, you have backups on backups in most data centers (and hospitals), but you often have a limited window to respond. We have an (illegal) 15-gallon gas can in one facility up by the generator. That can will give them about 9 minutes extra run-time if the day tank runs dry. There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?
Big enough problems need disaster recovery plans; you will go down, the issue is how quickly you can return to normal operations.
As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.
In*ter*es*ting - adj.
1. capable of holding one's attention.
2. arousing a feeling of interest.
3. oh God, oh God, we're all going to die.
/*Insert boring sig here*/
There are commercial generators which run on a variable mix of natural gas and diesel. With such a set, you can greatly extend your runtime by reducing the diesel percentage to a minimum when natural gas is available. Then if or when the natural gas goes out you can run them on 100% diesel and you're no worse off.
You are making a lot of assumptions there, starting with the assumption that what you are suggesting is even legal. Next is that the people in charge of the building are also the people in charge of the datacenter (unlikely), and that the people in charge of the building rank your datacenter as a higher priority than everything and everyone else in the building.
This is not some sort of heroic lifesaving operation where 'do anything possible' applies. It is just very bad disaster preparedness on the part of the data center operators. "Get a backup generator running onsite" should not be part of a disaster recovery plan, ever. What if the building is on fire? Is that still your plan?
If these datacenters are important enough where they are willing to put peoples lives at risk (hauling fuel up firewells or the outside of the building!) then they should have switched over to their alternate site (far from NYC) days ago. If they are no important enough to have alternate sites, then they should do what every other person and business impacted is doing: wait until it is safe, then start your cleanup.
Well, here in the real world, we have DR plans based upon the suggestions of the trade journals and then we have the actual implementation of the DR plan based upon the budget cuts of the past X years. Guess which one includes multiple redundant datacenters in locations of a significant distance away from each other and which one doesn't include those datacenters.
Those of you with systems up and running with Peer1 should be thankful to the data center folks as well as the Squarespace staff who're pitching in big-time! If you want to know what's really going on, along with some pictures, check out their status page: http://status.squarespace.com
Basically, to address other folks' various questions: Yes, the generators are on the roof (along with a small start-up + several hour supply of fuel), but most of the fuel supply is in the basement due to fire codes, etc.. They switched over smoothly to generator power but the pumps to get fuel up from the basement went offline. They've been doing the bucket brigade since, and they did have various backup plans in place (and even a basic plan for a flooded basement & no grid power), but everything is basically in worst-case-scenario-mode right now. That is, difficulty getting fuel trucks to re-supply (though that's OK now), no air-lift available for the refueling on the roof, accessible backup fuel pumps not powerful enough and the right equipment can't get to them because of conditions throughout the rest of the city, and they've been trying to clear the basement of water for a good while now but it appears that in addition to the flooding there may be a water main break that's pushing water back into their basement as fast as the pumps take it out.
Given the situation, they're doing a hell of a job!
Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.
Until it leaks out, saturates wood framing and other building materials, then comes into contact with an ignition source (like a pilot light, or candles that the residents are using since the power is out).
Then this hard to ignite fuel quickly turns an office into an inferno.
I can knock a candle over in my stationary cupboard and as long as I pick it up quickly, I wouldn't expect a fire. Soak that same cabinet and knock a candle over and it's a different story. Kerosene (very similar to diesel) makes a good fire starter to help get a wood fire started.
I work for the city. Happy to report that while we lost a lot of nodes and circuits downtown, none of our datacenters took a hit. We lost Mainframe DR replication for about 12 hours. No impact. 911 and 311 obviously experienced high call volume. 911 took no weather related hits, 311 saw about 2 minutes of weather related outage during a brief period where our two COs were flooded and we were in the process of switching to other trunks.
EXCELLENT work by the Cities workers throughout this event.
I was part of a crew installing and adding more battery backup for a firm in Wall St. Night job, had to disconnect batteries for a couple hours in order to move them to the floor below. Suits all around, doing nothing but wanting the hookup done ASAP, end of the world if power did fail. We had the job mostly complete, just needed another 15 minutes or less to do the final hookup. The brainiac suits on the floor above threw the breakers, they couldn't wait to activate. Meanwhile my co-worker had to grab live 220 volt cables that are shooting flames out of the ends, He looked like a 4th of July fireworks display! We finally got the suits on the phone, told them to turn off the damn breakers, 10 minutes they were back online with backup. My friend could've been killed because some idiot suits couldn't wait like they were told to. No one got hurt, but it easily could've turned into an office building fire. Trust in the pros, safety is paramount!
Your state office would in that case go offline roughly 90 minutes after virtually every disaster possible. Natural Gas, being a utility, is virtually guaranteed to be taken offline in cases where power goes out, ESPECIALLY when the power is deliberately cut off by the provider as a preventative against cataclysmic explosions where a gas pipeline explosion would be even worse.
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
That's nonsense. Do you even know what lockout/tagout is? How did those "suits" remove your coworker's lock? Did they cut it off or did he not put it on? Every person working on that circuit should have had their own lock and tag on that breaker.
I assume if they had cut his lock off you would have mentioned it since that's a much more serious offense than simply flipping a breaker that isn't locked/tagged.
Your coworker definitely should have been fired if he was working on an industrial power circuit without following lockout/tagout procedures.
BTW, for those not familiar the slash in this case means AND. It is always mandatory to use both a lock and a tag, it's not an either or choice.