NYC Data Centers Struggle To Recover After Sandy

Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring." I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."

Comments

Add to that, NYI...

[malakai]

At 100 Williams Street, http://www.nyistatus.com/
My server and connections have been up non stop.

I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities.

I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.

Re:Add to that, NYI...

[rcamera]

for residential, that's true. for commercial and/or municipal, they stick to diesel.

in the case that con-ed shuts off nat gas flow due to fires (which is happening all over the place right now...), do you want your generator to choke out?

that leaves lp. diesel packs about a 50% higher energy density than lp. which would you prefer to use? further, diesel is much more commoditized than lp, and is more readily available during a crisis.

Re:Add to that, NYI...

[acoustix]

why diesel? any generator worth it's salt is running on Natural Gas or Propane.

Obvious answer: Natural gas lines are usually cut during emergencies like flooding, fires, etc. Storing a liquid fuel like diesel allows you use the generator when external energy sources have been severed.

Status of datacenters affected

[Samuraid]

There's a detailed list of downed datacenters as well as a good discussion of status over at webhostingtalk: http://www.webhostingtalk.com/showthread.php?t=1205042
Per the topic, the following locations are experiencing or have experienced outages:

• 75 Broad Street
• 33 Whitehall
• nLayer at 882 3rd Ave
• Voxel/Internap at 111 8th Avenue
• XO, nLayer, Cogent, Verizon, Sidera Networks and AT&T at 882 3rd Ave
• 121 Varick

Re:Waterproof...

[aaarrrgggh]

The bigger problem is usually the pumps. You generally try to use turbine-type submersible pumps with the motor above the tank and the inlet down low to avoid problems with priming the pump. If the place where the pump motor floods, you are pretty much SOL.

If you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.

Ultimately, you have backups on backups in most data centers (and hospitals), but you often have a limited window to respond. We have an (illegal) 15-gallon gas can in one facility up by the generator. That can will give them about 9 minutes extra run-time if the day tank runs dry. There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?

Big enough problems need disaster recovery plans; you will go down, the issue is how quickly you can return to normal operations.