NYC Data Centers Struggle To Recover After Sandy

← Back to Stories (view on slashdot.org)

NYC Data Centers Struggle To Recover After Sandy

Posted by Unknown on Wednesday October 31, 2012 @05:01AM from the bucket-brigade dept.

Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring." I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."

8 of 231 comments (clear)

Min score:

Reason:

Sort:

The Cloud by azalin · 2012-10-31 05:04 · Score: 5, Funny

When it's wet, the clouds go down
Add to that, NYI... by malakai · 2012-10-31 05:08 · Score: 5, Informative

At 100 Williams Street, http://www.nyistatus.com/
My server and connections have been up non stop.
I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities.
I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.

--
-Malakai
A Dragon Lives in my Garage
1. Re:Add to that, NYI... by afidel · 2012-10-31 05:20 · Score: 5, Interesting
  
  NYU's generators were fine, it was their fuel supply that got fouled. Fire regs don't allow them to have thousands of gallons of diesel anywhere but underground holding tanks and those were overcome by seawater. Bellevue lost two of their primary generators due to water in the basement but was still running on another on the 13th floor but they had the same limited fuel problem Peer1 is running into. They considered having the national guard bucket brigade fuel up to the 13th floor but after some analysis it was decided it would be better to transfer folks to other hospitals (I'm not sure how many generators were on the 13th floor but it was probably only a single one and so they were down to a SPOF so better to transfer people in an orderly manner while you still have working facilities than to try it after the generator went down).
  
  --
  There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
2. Re:Add to that, NYI... by twisted_pare · 2012-10-31 06:25 · Score: 5, Insightful
  
  Poor planning, plain and simple.
  
  I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat. What, were we just "lucky?" I don't think so.
  
  Starting a week ago we had disaster crisis centers setup.
  * Every few hours all East coast facilities reported in any issues
  * Inspection and testing of all critical systems ahead of time
  * Stockpiles of supplies on hand
  * Prefail over to DR where possible
  * All hands on deck to respond
  
  Sadly, if you want to be prepared, you can be. If tons of money is on the line, then the price of being prepared is well worth it. We test our systems continuously year round. We have disaster recovery drills at all facilities multiple times a year. Departments' rating depend on how well prepared they are for things like this.
  
  And don't throw that "1888," "worth storm ever" crap around. This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even! If you did not prepare for this including flooding and sealed underground tanks and sandbag walls, it was your own fault.
  
  --
  HTFU
Re:Your priorities are all messed up!! by kav2k · 2012-10-31 05:14 · Score: 5, Funny

People can be replaced. Uptime, on the other hand..
Re:Poor Planning? by Anonymous Coward · 2012-10-31 05:23 · Score: 5, Insightful

Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.
Status of datacenters affected by Samuraid · 2012-10-31 05:41 · Score: 5, Informative
There's a detailed list of downed datacenters as well as a good discussion of status over at webhostingtalk: http://www.webhostingtalk.com/showthread.php?t=1205042
Per the topic, the following locations are experiencing or have experienced outages:
- 75 Broad Street
- 33 Whitehall
- nLayer at 882 3rd Ave
- Voxel/Internap at 111 8th Avenue
- XO, nLayer, Cogent, Verizon, Sidera Networks and AT&T at 882 3rd Ave
- 121 Varick
--
if ($question !~ m/bb|[^b]{2}/i) { die(); }
Re:Your priorities are all messed up!! by nedlohs · 2012-10-31 05:49 · Score: 5, Interesting

Fewer people died than the people who didn't die due to not being in a car crash due to not being able to drive to work. So on the life/death front it's a win!