Slashdot Mirror
Datagram Recovers From 'Apocalyptic' Flooding During Sandy
1sockchuck writes "During SuperStorm Sandy, few data centers faced a bigger challenge than the Datagram facility in lower Manhattan. The storm surge from Sandy flooded its basement, disabling critical pumps. 'It was apocalyptic,' said CEO Alex Reppen. 'It was like a tidal wave over lower Manhattan.' While companies like CoreSite dealt primarily with the loss of ConEd power, the Datagram team sought to recover operations in an active flood zone. Why was mission-critical equipment in the basement? Because city officials restrict placing fuel tanks on rooftops and upper floors, citing concerns about diesel emerging from the 9-11 attacks."
Everyone knows that flying airplanes into the tops of buildings happens more often than floods in the basement. Gotta keep the priorities straight.
If I have been able to see further than others, it is because I bought a pair of binoculars.
If only there was some sort of technology that allowed us to keep fuel in one place and the generators somewhere else...
Well, if the ordnance pertains to fuel tanks specifically, why not put waterproof tanks in the basement, and run sealed lines (including venting) up from there, locating the pumps somewhat higher. Obviously you're limited as to how much higher the pumps can be, but you can draw fuel a fair bit upwards on vacuum (maybe 20 feet?). If you're allowed to send pressurized air down the vent you could put the pumps up higher - I'm not sure what the laws are around that. If there are concerns with pressurized air mixing with fuel, another option might be a tank with a rubber bladder inside where the space between can be pressurized with either air or fluid - that's how they do it in liquid fueled rockets. As long as the tank and lines are waterproof you could keep it in the basement and operate indefinitely - but you'd need to work out all the details (like refueling - if the tank has to operate under pressure then you need to have pressure on the fueling lines as well, and suitable couplings and all that, unless you have more than one tank and can operate on one while fueling another).
All of that entails certain hazards - you'd want well-trained personnel to operate it - you're starting to resemble operations on a jet or spacecraft...
If you questioned why and you still placed your companies life in that data center you should be fired for stupidity.
It's the same reason I won't place my companies data at a DC in a crowded downtown area. Sporting events, politcial events, terrorist events.
If you say you don't have a choice then you haven't thought of alternative means. Cloud, managed hosting, or a more weather stable state.
Lower Manhattan is pretty much land filled area and 911 showed how vulnerable the WTC was below ground. They were extremely concenred about the Hudson flooding lower Manhattan.
Again if you placed your companies data at a DC in lower Manhattan you should be fired.
If shit is going to hit the fan, the company (Datagram) that advertises "highly secure and fully redundant Disaster Recovery Solutions (DRS) ensure your company’s information is safe, duplicated and available immediately and at any time." probably should have its personal Disaster Recovery Solution in another state, not in the basement OR upstairs. Seriously. Why didn't they have as an insurance/contingency plan a relationship with another server network in a state that doesn't share the same power grid as NY? Instead, they went with: "As several of its best known customer sites went dark, Datagram began a week-long struggle to bring its storm-ravaged infrastructure back online."
Am I the only one who finds it irritating that everyone is calling it "Superstorm Sandy"? It was a HURRICANE. Let's call it what it really was!
Fuel
Some cars and a lot of gas stations use in tank pumps, just put the buster pumps high. Your problems then become leakage in to the tanks and the problem of a tank trying to float when under water.
Backup sites:
Some bussinessed can not justify full sites as backup, some do not need full sites as backup, some would be happy with just a few web pages or files. Using slashdot as an example, "We are closed/down due to ______ check back here for updates" is all that is needed on web. One server at backup site could easly do this for a lot of servers at main site.
They might be the leaders, but they don't know jack about how technology works nor what is needed for backup power.
It won't stop though because you have id10ts voting in id10ts...
People are more apt to believe a smooth-talking politician than a techno-weenie. More the good-looking the politician, the more believable they are.
So somebody with a financial motive blamed putting essential equipment in the basement on somebody else. Why should we believe them?
Citing 9/11 is interesting in light of the NIST report:
Did fuel oil systems in WTC 7 contribute to its collapse?
No. The building had three separate emergency power systems, all of which ran on diesel fuel. The worst-case scenarios associated with fires being fed by ruptured fuel lines-or from fuel stored in day tanks on the lower floors-could not have been sustained long enough, could not have generated sufficient heat to weaken critical interior columns, and/or would have produced large amounts of visible smoke from the lower floors, which were not observed.
As background information, the three systems contained two 12,000 gallon fuel tanks, and two 6,000 gallon tanks beneath the building's loading docks, and a single 6,000 gallon tank on the 1st floor. In addition one system used a 275 gallon tank on the 5th floor, a 275 gallon tank on the 8th floor, and a 50 gallon tank on the 9th floor. Another system used a 275 gallon day tank on the 7th floor.
Several months after the WTC 7 collapse, a contractor recovered an estimated 23,000 gallons of fuel from these tanks. NIST estimated that the unaccounted fuel totaled 1,000 ±1,000 gallons of fuel (in other words, somewhere between 0 and 2,000 gallons, with 1,000 gallons the most likely figure). The fate of the fuel in the day tanks was unknown, so NIST assumed the worst-case scenario, namely that they were full on Sept. 11, 2001. The fate of the fuel of two 6,000 gallon tanks was also unknown. Therefore, NIST also assumed the worst-case scenario for these tanks, namely that all of the fuel would have been available to feed fires either at ground level or on the 5th floor.
...who, after reading the headline, thought that I was about to read the incredible story of a lone UDP packet that bounced around hurricane-flooded routers and switches for many hours, to finally, and miraculously, reach its intended destination, with a TTL of negative one-billion?!
Now *that* would be epic! :-)
(Edit: My captcha is "transmit"!)
"superstorm" crap. its a junk term coined by news outlets to either gin up ratings or avoid brining up climate change by calling it an unusually destructive hurricane..
Good people go to bed earlier.
So, why do they keep on putting data centers where they KNOW it WILL be flooded. And no, not all data centers in that area are for the stock exchange.
This is criminal incompetence.
How nobody can make a waterproof fuel container...
Except pretty much everyone who trys..
Seal that fucker up. vent it with checkvalves and traps out the roof. ez peezy no water in your gas.
Maybe their fuel containers were designed by bp?
Has anyone ever tried to set fire to diesel? It must be one of the most impossible substances to ignite ever!
The problem with these datacenters is not that the fuel tanks were in the basements. It's that the generators / starter motors / AC distribution panels were in the basement. The fuel tank is a sealed, enclosed metal container. It needs a vent pipe, which can easily be run up to the roof (or even the second or third floor). The generators, which are not as tightly regulated as fuel storage, can be placed just about anywhere -- like the roof.
Putting the fuel tanks on the roof is a laughable notion at best. Go ahead and put the fuel on the roof and keep your generators / starters / AC panels in the basement. Be sure to come back and tell us how well that works the next time it floods.
The datacenters got caught with poor engineering. They took the cheap way out, and they paid for it later. That's fair, but blaming the city is ridiculous. Don't wank in my ear and tell me it's raining.
Should have planned and executed better. Something like moving your data center out of an area that doesn't meet your business needs. Your customers should go to someone that does.
A (former) customer of mine had dumped their in-house server farm in the Midwest for a cloud-only strategy, making rude noises to their long-time server vendor in the process. They 'discovered' during Sandy that both of their cloud hosting vendors were in the NYC area, both out of power and that they were SOL. Theirs was a business that *really* depended on 7x24 uptime and they're royally screwed now. Their CTO (the guy who made the rude noises) is now unemployed.
Lessons? Some in-house capacity might be a good thing. Geographic diversity is probably worth the cost. Outsourcing everything inevitably means some loss of control. Don't believe everything the hosting vendor tells you. Long-term vendor relationships can really matter when things hit the fan.
I still blame the city ... just because it is too low and too near the sea. I'd have things inland more. Geographic diversity is still a good thing, too. But lower Manhattan is not a candidate for such things.
now we need to go OSS in diesel cars
Don't build data centers in a coastal city under sea level. Just like you don't build nuclear power plants in tsunami prone regions where the backup generator is next to the sea. We can all either fear global warming and its effects, or we can adapt to it and start thinking about how to plan for future technological advancement that doesn't leave us victims of bad weather.
There are no natural disasters, only disasters in human arrogance.
I haven't thought of anything clever to put here, but then again most of you haven't either.
Every time I hear a sales pitch about a data center or hear of someone wanting to add generator power to their building, they talk about diesel powered generators. Usually after they show off their huge generators they talk about how often they run them for test purposes.
When you start to ask about fuel capacity they get kind of hinky and immediately start emphasizing their multi-supplier, prepaid, penalties-for-non-delivery fuel delivery contracts.
Which is all well and good if you have a simple issue, like a down power line or some other small utility problem. But what happens when have you a significant issue, like the hurricane where there is serious demand for fuel and limited ability to delivery it? No delivery company in the world would accept a contract that held them liable when governments issue emergency edicts, rationing or just plain commandeer the fuel supply (and there may already be legal exemptions in those cases anyway that would nullify contract provisions).
So I always think "Why not natural gas?" It's less fuel efficient, but its gotten cheaper lately and in backup power applications its not meant to be a form of primary generation anyway. There's seldom a supply issue as most gas piping is done underground and is seldom affected by disasters except by earthquakes.
This means while everyone else is hoping for diesel deliveries, paying pollution surcharges and paying fees for on-premise fuel storage, you're just burning the near infinite natural gas supply.
Storage, for one. What happens when your gas pipeline is disrupted?
It doesn't mean much now, it's built for the future.
the biggest thing I would say is that I personally had no idea about the changes in policy regarding diesel fuel on higher floors and that would have affected my decision certainly. However, one of the main reasons we picked them is because we are based in NYC and ran fiber there for internet-connect between sites, internet and access to colo gear. Would I have liked to have MPLS Fiber to a datacenter 400 miles away with a managed/cloud service? sure, that is quite a bit more expensive (at least at was at the time). We have backup/DR in place but it wasn't designed for the unfortunate scenario Datagram experienced (I wish they would mirror their generator saga somewhere)
The real question is why anyone would locate any data center in NYC. While Wall Street may still be there, there is no reason it's or any businesses data center needs to be there. Moving the trading servers to higher ground might be a good idea.
If your trading needs so little latency that you have a problem with the latency added by the speed of light over a fiber cable to somewhere maybe a couple miles away in NYC on higher ground, of 10-20 miles away on mainland NY, NJ, CT, etc, maybe you're abusing the stock market, and we shouldn't bend over to help you ruin our financial system.
You might want to invest in some 1U rack mount umbrellas.