Slashdot Mirror
NYC Data Centers Struggle To Recover After Sandy
Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring."
I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."
Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).
Maybe you'd understand better if you learned where to use a fucking apostrophe.
Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.
Why not move them to the roof? And while we're at it, do the same for all the nuke plants? A simple f*cking appliance that needs air and fuel to run and somehow they manage to spend life at the bottom of a potential indoor swimming pool.
Join the Slashcott! Feb 10 thru Feb 17!
Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?
Lets see how you feel with a few thousand gallons of highly flammable liquid suspended above your head, in a building with lots of electricity running through it, where an earthquake is more likely than flooding in the basement. And that is ignoring the possibility of deliberate sabotage. A building with fuel stored above ground level where something went wrong would turn rather quickly into a giant pillar of flame. If one of the tanks gets ruptured, all it takes is a single spark to kill hundreds or thousands.
Below ground, however, fire-fighters can deal with it relatively easily, and the flames won't descend to engulf the entire building in a matter of a few minutes.
"None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
Like many other posters, my first question was why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious: fire codes. Duh. Thinking of how fuel is stored elsewhere, the only other option I can think of would be storing the fuel outside the building but above potential floodwaters. Not in a place like Manhattan. The price of real estate is much too high for tank farms on stilts. And the earthquake risk in New York is non-zero, so that solution might have the same problems as the current solution. So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure. Is relocating major data centers out of flood-prone areas of Manhattan (and other similarly risky areas) feasible? The potential of a major flood event in Manhattan has been well-known for a long time. Much of lower Manhattan is built on landfill. Did the builders of these data centers include basement flooding + extended power outage in their risk forecasts and just decide to deal with it if it happened?
It looks like Google was ahead of the curve after all with their idea for floating server farms.
Poor planning, plain and simple.
I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat. What, were we just "lucky?" I don't think so.
Starting a week ago we had disaster crisis centers setup.
* Every few hours all East coast facilities reported in any issues
* Inspection and testing of all critical systems ahead of time
* Stockpiles of supplies on hand
* Prefail over to DR where possible
* All hands on deck to respond
Sadly, if you want to be prepared, you can be. If tons of money is on the line, then the price of being prepared is well worth it. We test our systems continuously year round. We have disaster recovery drills at all facilities multiple times a year. Departments' rating depend on how well prepared they are for things like this.
And don't throw that "1888," "worth storm ever" crap around. This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even! If you did not prepare for this including flooding and sealed underground tanks and sandbag walls, it was your own fault.
HTFU
Well, here in the real world, we have DR plans based upon the suggestions of the trade journals and then we have the actual implementation of the DR plan based upon the budget cuts of the past X years. Guess which one includes multiple redundant datacenters in locations of a significant distance away from each other and which one doesn't include those datacenters.
Thank you! I thought the same thing about all the hospitals. I worked for a hospital for a while and believe me, they don't do disaster planning or even equipment life cycle planning. You would think with all the lip service paid to 'patient care' that continuity and disaster planning would be more of a priority, but no. Usually these aren't real businesses and they have no incentive to behave like one. As another poster mentioned, evacuation is usually their continuity plan.
This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
Those of you with systems up and running with Peer1 should be thankful to the data center folks as well as the Squarespace staff who're pitching in big-time! If you want to know what's really going on, along with some pictures, check out their status page: http://status.squarespace.com
Basically, to address other folks' various questions: Yes, the generators are on the roof (along with a small start-up + several hour supply of fuel), but most of the fuel supply is in the basement due to fire codes, etc.. They switched over smoothly to generator power but the pumps to get fuel up from the basement went offline. They've been doing the bucket brigade since, and they did have various backup plans in place (and even a basic plan for a flooded basement & no grid power), but everything is basically in worst-case-scenario-mode right now. That is, difficulty getting fuel trucks to re-supply (though that's OK now), no air-lift available for the refueling on the roof, accessible backup fuel pumps not powerful enough and the right equipment can't get to them because of conditions throughout the rest of the city, and they've been trying to clear the basement of water for a good while now but it appears that in addition to the flooding there may be a water main break that's pushing water back into their basement as fast as the pumps take it out.
Given the situation, they're doing a hell of a job!
Well, it's a fact. If you can't deal in facts, you shouldn't be working where you claim to be.
You're lucky enough to be able to afford to have a complex defense in depth - this isn't true of everyone.
So take your "I'm big bad dude" attitude, and stuff it. You're only big because daddy has money.
To be fair, if conditions are that bad at the hospital, evacuation is probably the best plan anyway.
This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.