Slashdot Mirror


New York Data Centers Battle Floods, Utility Outages

miller60 writes "At least three data center buildings in lower Manhattan are struggling with power problems amid widespread flooding and utility outages caused by Hurricane Sandy. Flooded basements at two sites took out diesel fuel pumps, leaving them unable to refuel generators on higher levels. One of these was Datagram, which knocked out Buzzfeed and the Gawker network of sites. At 111 8th Avenue, some tenants lost power when Equinix briefly experienced generator problems." The NY Times has a running list of Sandy-related problems, including 5,700 more flight cancellations, 6 million people without power, rising water levels at a nuclear plant, official disaster declarations from President Obama, and a death toll of 38. On the upside, and despite the high water levels, the Nuclear Energy Institute was quick to point out that all 34 nuclear facilities in Sandy's path made it through without problems.

22 of 186 comments (clear)

  1. Look at the bright side by AuMatar · · Score: 5, Interesting

    If we're really lucky, it'll take out all the high frequency traders systems for a few days and we can have an actual market without parasites.

    Nah, who am I kidding. If that actually happened they'd keep Wall Street closed.

    --
    I still have more fans than freaks. WTF is wrong with you people?
    1. Re:Look at the bright side by jessehager · · Score: 3, Informative

      Most of the traders try to get as close to the old Western Union Building at 60 Hudson St. as possible. If not inside it, in a building adjacent to it. That's the central network hub for the financial district.

    2. Re:Look at the bright side by tlhIngan · · Score: 3, Interesting

      HFT systems are located as close to the exchange's servers as they physically can be, because all the marketeers think it's bad juju to have a ping time above .01ms.

      Needless to say, if the HFT systems go down, then the market's exchange servers 2 feet away will probably be down, as well.

      If you're within the datacenter, there's a chance the trading computers are a bit further away from that actually. What happens is that the exchange actually ends up finding the longest cable they need to reach from the trading computer to the farthest rack, then they ensure that every cable from the trading computer any rack is that length. That way all the HFT traders who pay to be physically close in the data center aren't getting any advantage - there's just a huge coil of cable above the rack to ensure every rack gets the same latency. One could argue the coil of cable adds to latency by being an inductor, I suppose...

  2. Where are the mid-American datacenters by blueforce · · Score: 4, Interesting

    Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.? Surely the threat from Tornados could be mitigated and the electrical infrastructure built out more cheaply than the losses due to coastal disasters, no?

    --
    If you do what you always did, you get what you always got.
    1. Re:Where are the mid-American datacenters by 54mc · · Score: 4, Insightful

      Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?

      If you're paying the premium to host in Manhattan, you're doing so because latency is a big deal to you.

      --
      Joy! Beautiful spark of the gods!
    2. Re:Where are the mid-American datacenters by OverlordQ · · Score: 5, Informative

      > Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?

      There are, you just dont hear about them as often because they generally dont have anything newsworthy to report about them.

      --
      Your hair look like poop, Bob! - Wanker.
    3. Re:Where are the mid-American datacenters by Revotron · · Score: 3, Informative

      That's what Chicago is for. Far west enough to avoid most eastern seaboard troubles, far east enough to avoid the earthquakes, but central enough to provide good connectivity and ping times to both coasts.

      There are a few datacenters in Omaha, Nebraska, but they're either lights-out carrier-grade (Level3) or Fortune 500 warm-site backup grade. (CoSentry). They're also ungodly expensive because they're the only players in a 250 mile radius.

      Kansas City, MO has a good selection of datacenters for non-mission-critical systems, but most of the "data fortress" type places are built and run by the business that needs them.

      This is all completely ignoring the issue of latency, though - when you're doing financial transactions there's no better seat in the house than the heart of Wall Street. Every millisecond counts, I've been told.

    4. Re:Where are the mid-American datacenters by bluefoxlucid · · Score: 5, Funny

      One bowl of Colon Blow has more fiber.

    5. Re:Where are the mid-American datacenters by Shatrat · · Score: 5, Informative

      5 microseconds per kilometer tends to be a pretty good approximation, depending on the transport gear.
      Things like FEC, EFEC, dispersion compensation modules (non-bragg grating type), frequent OEO regens can add up and make it worse.

      That would give you a ballpark of 11ms for a 1450 mile circuit.

      --
      09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
    6. Re:Where are the mid-American datacenters by Anonymous Coward · · Score: 4, Informative

      I work for a daatcenter in Kansas. I can tell you first hand that there are a lot of datacenters around here (Kansas City metro). If you aren't doing high speed trading, then you have no reason to not have a datacenter out here. Power is cheaper, space is cheaper, cost of living is cheaper, you don't have to worry about hurricanes or snow, most DCs are tornado proof, and connectivity here is WAY better than people know (and growing faster than any other area in the country). If latency is your primary concern though, then close to the source is your only option.

  3. WHERE ARE THE MELTDOWNS!!?!?!?! by CajunArson · · Score: 3, Funny

    I was promised a NUKULAR OPOCALIPSE yesterday, and here we are with satellite images that don't even show the entire East Coast as a glowing radioactive wasteland.

    I think that this complete lack of NUKULAR meltdown is 100% absolute irrefutable proof of two key concepts:
    1. Capitalism is an abject failure and we need U.N. control of everything and everyone right now.
    2. NUKULAR power is obviously far too dangerous and should be banned right now before somebody doesn't get killed again JUST LIKE FUKUSHIMA.

    --
    AntiFA: An abbreviation for Anti First Amendment.
  4. Disaster Plan Fail by OverlordQ · · Score: 4, Insightful

    [...] work hard to assess the situation and our recovery plans.

    How about not putting mission critical equipment susceptible to water damage in the one place all water will go.

    --
    Your hair look like poop, Bob! - Wanker.
  5. Re:No big deal. It was a cat1 storm by sglewis100 · · Score: 4, Insightful

    This is as strong as a case forever to stop paying attention to the media.

    Sounds like you already stopped paying attention. Had you even checked the one link in the article, or even read the summary, you'd know it was a catastophy, and New York will not function "perfectly tomorrow like nothing ever happened."

    Schools closed. Subways closed 4-5 days. 38 people dead. Market closed. Fire in Queens destroys dozens of homes. Power outages for millions. 7% of the US population in fact without power. Tunnels flooded (subway and car). NYU Tisch hospital evacuated due to flood related generator failures, including premature babies on ventilators. Just a small summary, of just one city.

    I'm a former NYer. Have spoken to many friends and family. None expect normal life tomorrow. Some have considerable property damage. None lost a life, thankfully. I live in South Florida, incidentally, and rather well understand how damaging hurricanes can be. Wilma damaged my car, and cancelled my wedding day as the roof caved in on the place we were getting married in that Friday.

    This is a rare strong case to NOT stop paying attention to the media.

  6. Re:Oblig. XKCD by Revotron · · Score: 3, Insightful

    I don't think you're doing that right.

  7. Re:No big deal. It was a cat1 storm by NatasRevol · · Score: 3, Informative

    No 'might' about it.

    Death rate of NYC is around 1,000/week. or 140 per day.

    --
    There are two types of people in the world: Those who crave closure
  8. Fukushima and Sandy show by circletimessquare · · Score: 3, Insightful

    that flooding ruins backup generators, pumps, fuel storage. I hope that disaster mitigation plans are reviewed.

    I also hate people who judge negatively from hindsight, but disaster planning is about considering the most probable of the improbable. Flooding looms most threatening and probable of the improbable.

    Perhaps putting all the backup infrastructure on a higher floor makes it harder to maintain, access, and/ or protect from mischief/ terrorism. However plain old flooding seems to be an issue time and time again in disaster scenarios and really needs highest priority in disaster plans.

    --
    intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
  9. Re:No big deal. It was a cat1 storm by bws111 · · Score: 3, Insightful

    "Be better prepared" - than what? The storm surge that flooded lower Manhattan beat the previous record by 2 feet, and that record was set almost 200 years ago. The fact that there were only 38 people killed in the entire region shows just how well prepared they were. Nobody was drowned in the subways, because they stopped the subways before they got flooded. Nobody was stuck in elevators, because they turned off the elevators before the power was shut down.

    What is it with all these people saying 'it was not that bad a storm'. It was that bad a storm for the area. It was record flooding. From what I understand, Category 5 hurricanes are 'not that bad a storm' compared to the storms on Jupiter - pretty meaningless comparison, isn't it?

  10. Re:I'm waiting for the calls... by DerekLyons · · Score: 3, Insightful

    But this was a weak storm, and did about normal damage as happens to places with a storm like this hitting a coastal area.

    I don't know where you live - but here on Planet Earth, nobody rational calls a storm with 100mph winds and an 11 foot storm surge, "weak". Not to mention, this storm was considerably more severe than is "normal" for that area.
     

    Sorry if it sounds like sour grapes...

    No, you don't sound like sour grapes - you sound like an ignorant jackass.
     
    Catch a clue.

  11. Re:Oblig. XKCD by oodaloop · · Score: 4, Funny

    Reminds me of one of xkcd's funniest comics:

    http://xkcd.com/404/

    --
    Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
  12. Re:I'm waiting for the calls... by bws111 · · Score: 5, Insightful

    Getting several feet of snow in one day is not all that unusual in Buffalo. What do you think would happen to New Orleans if that happened there? An average cold winter's day where I live has a low temp of about 0F. Think New Orleans could take that for a few weeks at a time? The tidal change in the Bay of Fundy is something like 40ft - think New Orleans could take it? Sure, you get winds over 100MPH in New Orleans, but Mt Washington, NH recorded over 230MPH. Think you could take it?

    Comparing things like wind speed and storm surge and temperature between different regions is a fools game. What really matters is deviation from normal, and this was a very large deviation from normal. Yes, the storm surge was 'only' 13 feet, but the last time it was that high in NY was - unknown. The previous recorded max was in 1830 something, and this beat it. No, 70MPH winds are not that high in absolute terms, but tell that to the trees that couldn't take it (because normally they are only subjected to 50MPH winds).

    In short, get over yourself. The fact that you have experienced similar absolute numbers without devastation does not in any way mean that the same conditions are not devastating elsewhere, or that they shouldn't be devastating. No matter where you live, someone else is living with conditions that you would consider devastating.

  13. Re:No big deal. It was a cat1 storm by AuMatar · · Score: 3, Insightful

    The navy has much bigger, more modern ships. There's also national security risks if the navy boats are damaged. No such problems for a movie prop.

    And like I said- it may be safer for the boat. It isn't safer for the crew. This wasn't a surprise storm, this had been forecast for a week. Tie up the boat, stay in a hotel (preferably about 100 miles or so inland), and then repair the boat as needed. The health and safety of the crew is far more important than the ship. The owners deserve to be sued into oblivion for even asking. It is NOT acceptable to risk 16 lives to save money on repairs. Hell, take it to a dry dock if you're that scared.

    --
    I still have more fans than freaks. WTF is wrong with you people?
  14. From my VPS provider by Anonymous Coward · · Score: 3, Informative

    All Linux VPS located in New York City, NY VPS are currently offline. They are located in Internap's LGA6 facility in 111 8th Avenue.

    Please be advised that Internap's LGA11 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level.

    Thankfully, our NYC server nodes are not directly located in LGA11's facility, but rather LGA6. The cause of this temporary outage is that LGA6 routes certain parts of the network's backbones through LGA11 which is currently offline as explained above. URPad's downtown NYC facility, located at 111 8th Avenue, is currently experiencing a network-only outage. The datacenter is not located in the storm surge zone, and is not suffering from any flooding. All URPad hardware and assets are safe and remain powered on. Engineers are aware of the network outage and all efforts will be made to restore network connectivity as soon as possible.

    Internap & URPad will continue to work hard to assess the situation and our recovery plans, and will communicate those plans as soon as possible. Thank you for your patience and understanding during this crisis. Please trust that we are doing everything we can to bring your services back online as soon as possible.