Slashdot Mirror
NYC Data Centers Struggle To Recover After Sandy
Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring."
I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."
When it's wet, the clouds go down
At 100 Williams Street, http://www.nyistatus.com/
My server and connections have been up non stop.
I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities.
I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.
-Malakai
A Dragon Lives in my Garage
People can be replaced. Uptime, on the other hand..
Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.
You do your job. Many of these data centers are part of and important to communication, rescue operations, information. When you work there, you might not know how important that particular data center may be but -- you do your job.
Whatever part of the city you can keep operating is good.
Don't criticize what you know nothing about.
Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?
Lets see how you feel with a few thousand gallons of highly flammable liquid suspended above your head, in a building with lots of electricity running through it, where an earthquake is more likely than flooding in the basement. And that is ignoring the possibility of deliberate sabotage. A building with fuel stored above ground level where something went wrong would turn rather quickly into a giant pillar of flame. If one of the tanks gets ruptured, all it takes is a single spark to kill hundreds or thousands.
Below ground, however, fire-fighters can deal with it relatively easily, and the flames won't descend to engulf the entire building in a matter of a few minutes.
"None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
Per the topic, the following locations are experiencing or have experienced outages:
if ($question !~ m/bb|[^b]{2}/i) { die(); }
Depending on the generator size, there's the weight issue. There's also fire regulations. 500 gallons of Diesel on the roof + fire == cascading burning diesel fuel.
There are some people that if they don't know, you can't tell 'em.
Fewer people died than the people who didn't die due to not being in a car crash due to not being able to drive to work. So on the life/death front it's a win!
The Starbucks is to test the Sewers with.
Like many other posters, my first question was why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious: fire codes. Duh. Thinking of how fuel is stored elsewhere, the only other option I can think of would be storing the fuel outside the building but above potential floodwaters. Not in a place like Manhattan. The price of real estate is much too high for tank farms on stilts. And the earthquake risk in New York is non-zero, so that solution might have the same problems as the current solution. So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure. Is relocating major data centers out of flood-prone areas of Manhattan (and other similarly risky areas) feasible? The potential of a major flood event in Manhattan has been well-known for a long time. Much of lower Manhattan is built on landfill. Did the builders of these data centers include basement flooding + extended power outage in their risk forecasts and just decide to deal with it if it happened?
It looks like Google was ahead of the curve after all with their idea for floating server farms.
The bigger problem is usually the pumps. You generally try to use turbine-type submersible pumps with the motor above the tank and the inlet down low to avoid problems with priming the pump. If the place where the pump motor floods, you are pretty much SOL.
If you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.
Ultimately, you have backups on backups in most data centers (and hospitals), but you often have a limited window to respond. We have an (illegal) 15-gallon gas can in one facility up by the generator. That can will give them about 9 minutes extra run-time if the day tank runs dry. There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?
Big enough problems need disaster recovery plans; you will go down, the issue is how quickly you can return to normal operations.
Well, here in the real world, we have DR plans based upon the suggestions of the trade journals and then we have the actual implementation of the DR plan based upon the budget cuts of the past X years. Guess which one includes multiple redundant datacenters in locations of a significant distance away from each other and which one doesn't include those datacenters.