How Data Center Operator IPR Survived Sandy

Nerval's Lobster writes "At the end of October, Hurricane Sandy struck the eastern seaboard of the United States, leaving massive amounts of property damage in its wake. Data center operators in Sandy's path were forced to take extreme measures to keep their systems up and running. While flooding and winds knocked some of them out of commission, others managed to keep their infrastructure online until the crisis passed. In our previous interview, we spoke with CoreSite, a Manhattan-based data center that endured even as much of New York City went without power. For this installment, Slashdot Datacenter sat down with executives from IPR, which operates two data centers—in Wilmington, Delaware and Reading, Pennsylvania—close to Sandy's track as it made landfall over New Jersey and pushed northwest."

Comments

PPPPP

[Anonymous Coward]

Proper Planning Prevents Poor Performance

Re:PPPPP

[jesseck]

I remember seeing this in the past... but we used the 7 P's: "Proper Prior Planning Prevents Piss-Poor Performance"

Duh, just outsource it to India

[crazyjj]

My friend Rahul and Sameer will take care of your needs, and are to be speaking excellent English, most also.

Ducked the most important question

[joeflies]

In his response to the question "So you suffered no downtime at all?", the business development manager provided a non-answer to a yes/no question. The interviewer should have followed that question up to clarify.

tl;dr version

[kiite]

"We didn't do anything special; our power never went out."

Re:tl;dr version

[mcgrew]

Of course their power never went out, they had three separate electric companies wired in, if one went down a second kicked in, if that went down one of their two generators kicked in.

Re:tl;dr version

[kiite]

Right. And, according to TFA, none of their supplies ever went out. I live in NYC. A lot of the city lost power, sure. The transit system was knocked out, sure. There was a lot of flooding in fringe areas, where most data centers weren't. This guy is talking about NYC like it got demolished by the storm. Slashdot already did a piece on how a NYC data center mitigated power loss; reading TFS, I was hoping for a point of view from a more heavily battered standpoint. Instead, I got, "We had back-ups, and we think they work because we test them regularly, but we didn't actually have to do anything."

Re:tl;dr version

[hawguy]

Of course their power never went out, they had three separate electric companies wired in, if one went down a second kicked in, if that went down one of their two generators kicked in.

I wonder if that's true - I can believe that they bring in power from several different substations, but if there were a widespread grid outage, it seems like that would have taking out all of their substations.

I don't understand this comment:

We potentially have two power grids. We actually have three. The third we would never go to, I don’t think.

Why would they connect to a power source that they'd never go to?

Re:tl;dr version

[Rich0]

Yeah, Reading PA. Go look it up on a map. They weren't going to be having multiple substation outages that far inland. My own workplace didn't lose power and is about 20 miles further East. It doesn't hurt that they're about 200 yards from a substation and that both the substation and the plant site are fed by transmission lines on steel towards that stand WAY above the height of nearby trees.

Most of the outages for Sandy were due to flooding or downed trees. The former was only a problem along the coast or near rivers, and really a big problem for NYC where they have transmission equipment underground. Trees are horrible for the last mile of power delivery, but aren't an issue for the major substations, since if you drive by one of them you'll note that the transmission lines are WAY up in the air, and the trees are trimmed back a huge distance on either side of them anyway. The towers themselves are steel and on concrete foundations - they're not going to fall unless they're hit by something like a tornado.

The reason so many lost power wasn't because of transmission being cut, but by a bazillion downed trees taking out every other telephone pole in the region. If you want an IT analogy imagine if all your big network feeds and datacenter are intact, but some vandal walks around your building and sticks a firecracker next to every single network port.

For an inland location like Reading PA, this was just a matter of having either good power connectivity, or generators. Wilmington is next to the Delaware Bay and would be at more risk, but as long as you're at reasonable elevation and above-ground you'd be fine.

Re:tl;dr version

[Vancorps]

While you are right, the big selling point for a lot of data centers is physical location. IO Data here in Scottsdale for instance prides itself on the fact that there really is no severe weather in the area. Historically the area is geologically stable, not prone to flooding, no where near any forest fires. So their location is their first defense against disaster with N+3 redundancy as additional defenses.

Disaster planning is hard, some things you take for granted during normal times simply aren't available during a disaster. Think diesel fuel delivery using trucks. That's why a lot of data centers rely on pipelines for fuel delivery with trucks as a standby.

My Datacenter Had Perfect Uptime

[Anonymous Coward]

I had perfect 100% uptime during Sandy. No packet loss, no adverse effects, no fuss, no muss.

Please write a Slashdot article about me. Also, please conveniently ignore the fact that my datacenter is in Kansas City, MO.

Datacenter catastrophe checklist

[sl4shd0rk]

1) On-site Diesel to power ops for 48hours
2) Tanker of Diesel pump->doorstep within 12hours
3) Generators
4) Backup generators
5) 48hours worth of food for staff + repair guys
6) nearby lodging reservations staff + repair guys

Re:generators

[ATestR]

I'd say that locating outside of the hurricane's path was the better choice

If it isn't a hurricane, it's an Earthquake. If not that, then a nasty Blizzard. Or a Tornado. You can't avoid them all, and its best to prepare as best you can for the events possible at the location that you choose. No one can prevent all disasters, but you can mitigate the risk.

Me too

[sjames]

I did just fine during Sandy as well. I have a laptop with a good battery and I can always run it from a cigarette lighter adapter in the car, but I never lost grid power. Of course, I live in Ga. but that's beside the point.

Re:generators

[LoRdTAW]

I work next to a Verizon data center out here in Farmingdale, Long Island (supposedly all the Verizon cell phone traffic for long island). They recently built extensions to the building and had two large diesel generators installed, a 15,000 gallon fuel tank along with two large cooling systems. Turns out they needed it.

During the aftermath they didn't run out of diesel because they bought in an additional on site 15,000 gallon fuel tank (in a 40 foot container). Plus they had semi trucks with sleeper tractors from out of state with trailers full of diesel ready to fill the tanks back up on site 24/7. Armed guards manned the premises 24/7 and lived out of a mobile home. They also bought in generator powered flood lights to keep the surrounding property lit up like it was day light. Those generators sounded like a pair of locomotives running, and probably because they use engines of similar size. My manager found out they burn 4000 gallons of diesel a day keeping the building going. They ran the generators until the 7th or 8th. So they burned something like 40,000+ gallons of fuel in that time frame.

If you have the money and the right infrastructure, you can keep the power going as long as you need.