How CoreSite Survived Sandy

Nerval's Lobster writes "When Hurricane Sandy hit the East Coast, the combination of high winds, rain, and storm surges wreaked havoc on homes and businesses alike. With a data center on the Avenue of the Americas, CoreSite Realty escaped the worst the storm had to offer. But was it coincidence or careful planning? Slashdot sat down for an interview with Billie Haggard, CoreSite's senior vice president of data centers. He's responsible for the design, construction, maintenance, facilities staffing and uptime, reliability and energy efficiency of CoreSite's data centers. He described what it took to weather the worst weather to hit New York City in decades."

Comments

RTFA

[mcgrew]

I found it interesting, especially the part about renting hotel rooms, only to have the staff sleep in cots on-site because the hotels had no electricity, food, or water.

Excellent!

[sunking2]

"So eight hours in, we already had fuel trucks running. And every 24 hours, we had fuel, even though we didn’t need to."
I'm sure the people in shelters and waiting in line for 6 hours to fill their own generators so they can keep their family warm at night are happy for you.

Re:Excellent!

[mcgrew]

If the people in line had paid in advance like CoreSite did, they wouldn't have run out of fuel, either. Sheesh, these folks have foresight and you bash them because others didn't? And what does being in a shelter because your house was demoloshed have to do with fuel?

Re:Excellent!

[TheSpoom]

I'm not getting your point here. Are you arguing that they should have simply sent all their fuel to shelters and gone down for the entirety of the storm? In what way was it their job to supply the needs of the shelter?

Your argument is similar to parents who say that kids should always finish their food because people are starving in China. One has little to do with the other. I'm betting there's a named logical fallacy here but I can't identify it.

Re:Excellent!

[hawguy]

It's not about whether they could do it, but how smug he comes across in patting himself on the back for having an excess of fuel while those around do without. Because his internet hosting is by far the most important thing in the world. And shelters would take people who still have a house, yet have no means of heating it.

I was struck by his smugness at having prepaid for fuel, but then a few paragraphs later he pointed out that the carrier that prepaid for a 4 hour generator delivery had their generator confiscated by the police. He was lucky they didn't need to refuel the confiscated generator by confiscating his fuel.

In a big disaster, fuel contracts mean very little - if the government decides that a hospital or police station (or the mayor's mistress's apartment building) needs the fuel more than you do, they will take it.

Re:Excellent!

[egamma]

"So eight hours in, we already had fuel trucks running. And every 24 hours, we had fuel, even though we didn’t need to." I'm sure the people in shelters and waiting in line for 6 hours to fill their own generators so they can keep their family warm at night are happy for you.

It is not the responsibility of CoreSite to provide fuel for everyone in New York City. Billie Haggard did his job well, and he deserves kudos for that.

Perhaps those people should have heeded the mandatory evacuation warnings and moved further away from the coast--100 miles inland would have made a world of difference to them, come Monday morning. Your house being without power doesn't matter if you're two hours away from it. There's no good reason for putting your children at risk by staying put when a hurricane is coming at you, and you had several days warning.

Re:Excellent!

[hawguy]

Generators for data-centers usually run on propane. It's not going to do the locals much good.

I've never seen a propane fueled datacenter class generator. I've seen big natural gas generators, but never propane. They may exist somewhere where the long lifetime of stored propane is a bonus (plus the easy cutover between natural gas and propane). Nearly all of the datacenter generators I've seen are diesel fueled - the generator referenced in the article is diesel fueled.

They likely still have natural gas service without interruption, but their furnaces wont run without electricity. They too could have purchased propane or natural gas generators for their homes for just a couple of hundred dollars.

You're not thinking the problem through if you're suggesting that apartment dwellers should just buy $200 propane powered generators for use in a disaster - where they run them? In the hallway? Where will they store the propane? Having millions of people storing propane tanks in the bedroom closet doesn't sound very safe (and maybe not even legal).

If they are going to use propane to run a generator to run the furnace, they may as well just use the propane for heat and ignore the furnace. Which they may not even have access to if the entire building is served from a single source of heat.

Maybe you're suggesting that the building management should have installed generators (and many large buildings do have generators to run critical equipment), but when building codes dictate where you can house your generator and fuel supply, you may find that it's underwater in a flood.

Gasoline powered generators for emergencies are about the stupidest thing you can buy. It's expensive and runs out REALLY fast when there's any sort or service interruption.

And propane is better? A 5 gallon propane tank will run a 3500W generator at half load for about 10 hours. About the same as 5 gallons of gasoline. Diesel generators tend to run longer, you might get 12 - 16 hours of of a 5 gallon tank of diesel.

Propane is attractive because it stores forever, but gasoline is nice because most people already have 10 - 20 gallons of gasoline in their car (or more if they have more than one car) that can be siphoned out and used to provide power. That's probably not as true in an urban area like NYC where many people don't have a car.

But I wouldn't store significant quantities of gasoline, diesel, or propane in my apartment.

Re:Excellent!

[couchslug]

The people in shelters had ample time to prepare or prepare to evacuate.

The people with generators had plenty of time to fill their vehicles and fill reserve fuel containers for their gensets. (They had time to buy generators. Fuel cans are cheap enough.)

I always fill my vehicles before major storms, and always have reserve fuel for my small generator. (G.I. fuel cans and STA-BIL fuel additive work very well. Fuel small engines off your reserve or burn it in your vehicle to keep it fresh.)

The best tool in such times is a clue. If you have almost nothing you can still fill a "bug out bag". Take a train somewhere while they still run, crash at a train station if you have no other option.

I'm not a "prepper", BTW. My lifestyle seamlessly incorporates a high degree of preparation because it "fits" with many other activities at trivial cost. Gas heat means I can cook when a storm (or more commonly, a drunk driver) whacks local power lines. I always have enough cold weather clothing that I don't need heating for survival, just to keep water lines from freezing. (Own hiking/combat BOOTS for all weathers, break them in before you need them!)

While my home is very well prepared by default, if the shit really hit the fan I'd load my pickup truck and Get The Fuck Out. That's what homeowners insurance is for! If I'm guaranteed to be fucked for staying, the possibility of reduced fucking makes evacuation look good.

Nice photo of the building

[timeOday]

I guess Kodak is still selling $5 disposable cameras after all.

Re:First off,

[volxdragon]

First rule : don't build a data center at a location that gets week-long power outages.

Building down in that area is pretty much for one reason only: length of the fiber run to wall street. In a world of nanosecond trading, every mile you are away from Wall Street means that much more of an advantage someone else has over you. Is it extremely risky/expensive to build in these locations? Hell yes. Is it likely financially worth doing so? Hell yes. There are costs to doing business, this is just one you have to factor in and see if the overall risk/reward equation works out (I'm betting it does for these folks)

Re:Police confiscation

[hawguy]

Can anyone elaborate on what radio system he was talking about? I contributed to a DR plan several years ago, but my concerns about cell phone reliability were shot down.

Probably a regional UHF business radio network provider like Fisher Wireless. I think Fisher is a regional provider that only covers California and AZ, but I know national coverage is available. I think all of the interconnects between their radio sites run over UHF links, so they are completely independent of the cellular and wired network (but I'm not 100% sure about that).

Talk to any business radio provider and they can hook you up. Prices are reasonable, for around $20 - $50/month you can get unlimited talktime on a regional net (I think you pay more for more regions). You just add a channel onto your existing business radios to talk on the regional network.

However, I wouldn't count on it working in a large disaster - all communications systems have capacity constraints, whether radio, cellular or satellite, and when everyone tries to use the same backup form of communication, it's likely that they will run out of capacity. But for normal communications it works well.