Slashdot Mirror
NYC Data Centers Struggle To Recover After Sandy
Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring."
I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."
At 100 Williams Street, http://www.nyistatus.com/
My server and connections have been up non stop.
I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities.
I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.
-Malakai
A Dragon Lives in my Garage
Per the topic, the following locations are experiencing or have experienced outages:
if ($question !~ m/bb|[^b]{2}/i) { die(); }
The bigger problem is usually the pumps. You generally try to use turbine-type submersible pumps with the motor above the tank and the inlet down low to avoid problems with priming the pump. If the place where the pump motor floods, you are pretty much SOL.
If you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.
Ultimately, you have backups on backups in most data centers (and hospitals), but you often have a limited window to respond. We have an (illegal) 15-gallon gas can in one facility up by the generator. That can will give them about 9 minutes extra run-time if the day tank runs dry. There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?
Big enough problems need disaster recovery plans; you will go down, the issue is how quickly you can return to normal operations.
There are commercial generators which run on a variable mix of natural gas and diesel. With such a set, you can greatly extend your runtime by reducing the diesel percentage to a minimum when natural gas is available. Then if or when the natural gas goes out you can run them on 100% diesel and you're no worse off.
They don't have a choice. Fire codes require the fuel tanks to be underground. It doesn't have to make sense, it's the law.
I work for the city. Happy to report that while we lost a lot of nodes and circuits downtown, none of our datacenters took a hit. We lost Mainframe DR replication for about 12 hours. No impact. 911 and 311 obviously experienced high call volume. 911 took no weather related hits, 311 saw about 2 minutes of weather related outage during a brief period where our two COs were flooded and we were in the process of switching to other trunks.
EXCELLENT work by the Cities workers throughout this event.