Slashdot Mirror


Con Ed Says NYC Datacenters Should Get Power Saturday

Nerval's Lobster writes "The local utility serving most of the New York City area, Con Edison, reported that it should begin supplying utility power to midtown and lower Manhattan by Saturday evening, returning the island's data centers and citizens to some semblance of normalcy. In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power. That should come to an end, for the most part, when utility power is restored. In a possibly worrying note, Verizon warned late on Nov. 1 that its services to business customers could be impacted due to lack of fuel."

8 of 107 comments (clear)

  1. Fuel logistics by hawguy · · Score: 5, Insightful

    In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power.

    Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work. Few inner city datacenters have a week or more of fuel on-site - most have only days of fuel, and they count on fuel contracts from suppliers to keep them running. And the supplier may not be able to honor the contract in a disaster.

    Suburban and rural datacenters have the space (and less conflict with fire codes since the fuel is not stored in or near an office building) to keep weeks of fuel on hand. The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable. They had a spare generator that can be trucked in from that other facility as well. (and this facility could send fuel and a generator to that facility if needed)

    1. Re:Fuel logistics by hguorbray · · Score: 4, Funny

      And on the plus side, the IT guys who spent the week bucket brigading now have arms like Hulk....

      -I'm just sayin'

    2. Re:Fuel logistics by Anonymous Coward · · Score: 5, Informative

      True, but you can't legally use the fuel that's meant for the data center in your truck to get it there. We did that after Hugo hit the SC coast in Sept 1989 and got caught. The generator fuel has dye in it that will stain the fuel filter. It took nearly six months and several tens of thousands in legal fees to get our truck back. Meanwhile, the servers in our data center in Goose Creek, SC ran out of fuel and nearly put the company out of business. There were dozens of other trucks that got caught at the weight stations over the next few years that also used fuel not meant for use on the road that were also fined and/or confiscated. My father-in-law owns a towing company so they got a lot of towing and storage business from that. Don't underestimate the US government's desire to screw over the little guy and their desire to put companies out of business.

    3. Re:Fuel logistics by hawguy · · Score: 4, Insightful

      Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work.

      I don't think any data center manager had a line item in the disaster recovery plan that included having all transportation access cut to the entire island due to flooding of the tunnel and closure of the bridges, for over a week. Everyone is having a problem getting fuel into the city; even mission-critical services like emergency services, hospitals, and telecommunications facilities.

      What kind of disaster plan is it that doesn't account for a likely disaster? I can guarantee that every sizable datacenter had exactly this scenario in the DR plans. You don't run a $10M facility in a coastal city on an island without including flooding in your DR plan. This was a 100 year event, so it was definitely on everyone's horizon. Having it in the plan doesn't mean that you have a good solution - living in SF means Earthquakes are a big part of our DR plans, but we have no expectation that our facility will survive a 7.5 earthquake intact, or even that our employees will be motivated to come to work when they are worried about their own survival.

      As to your comment that "suburban and rural datacenters have the space", sure... but where's the fiber optic cable hookups and the telecommunications infrastructure located? I'll give you a hint: Not in a barn. Those data centers are located downtown because that's where everything else is.

      Just follow the train lines to find out where the major telecommunication lines are -- I have access to more carriers down on the Peninsula outside of San Francisco than I do downtown. You may be surprised at how much bandwidth runs through Colorado and even Missouri.

      Not only that, but most of the data centers on the island are there because that's where Wall St. is, and milliseconds matter when it comes to high volume trading and financial transactions.

      Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.

      Commercial real estate is at a premium in New York. Actually, all real estate is, leading to the old joke that when a New Yorker hears someone has died, the first question they ask is, "Is their apartment for rent?"

      I'm not sure what your point is? Datacenters have to be built in the city because that's where the carriers are, but commercial real estate is expensive so don't build your datacenter in the city?

      I think it's more likely to assume you've made an error in your reasoning, writing opinions from the comfort of an armchair, than people being paid over six figures who's job depends on balancing everything out exactly and to the nearest penny an hour.

      You're obviously not in NYC if you think "over six figures" means someone is highly paid. Part of my job is planning our IT DR strategy. Fortunately, our Facilities dept is in charge of the generator so I don't need to worry about fuel contracts or keeping it running, but I do need to make sure our data is safe no matter what happens to the building and that we can continue to operate as a business. The only "disaster" that we plan on riding out on the building generator is a localized power outage when we know we'll be able to get fuel once our 3 - 5 days of fuel runs out. If there's a widespread power outage or disaster, our plan is to transition to the remote site since we know we may not be able to keep the generator fueled.

  2. Re:Get Some Priorities! by CastrTroy · · Score: 4, Insightful

    Depends what's running in the datacenter. Banks, grocery stores, public utilities, repair contractors all need datacenters to get their jobs done. Also, many people work in data centers, or have jobs which rely on the datacenters being up. For them, the datacenter does feed their family

    --

    Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
  3. Re:Get Some Priorities! by hawguy · · Score: 4, Insightful

    Dozens are dead, billions of dollars of property destroyed and businesses decimated, millions still in darkness and cold, and you are talking about data centers??

    Data centers cannot feed my family or heat my cold, flooded house.

      Where is your decency sir? It's a long week and weekend for the millions affected and the first responders. The glibness of this post is shocking and disgusting.

    This is Slashdot, there are plenty of other sites that cover human misery and suffering, this site is for geek news, and geeks care about datacenters.

    Datacenters can't feed your family or heat your cold, flooded house (unless you live next to a datacenter that recovers waste heat for residential heating), but the datacenter can help your utility get power back online, it can help your local merchant process credit card transactions and use their POS system so you can buy replacement goods, it can let you post your family's status so your mom can stop worrying about you, it can help your bank conduct online transactions efficiently to let you receive your insurance money, and of course, you're using multiple datacenters right now to read and post to Slashdot.

    If you think datacenters are unimportant, try going a week without using any good or service that wasn't produced or delivered to you without the aid of datacenters.

  4. Re:Get Some Priorities! by hawguy · · Score: 4, Insightful

    As little as I like agreeing with ACs, this was my exact thought upon reading TFS. I know this is News For Nerds, but let's not pretend getting data centers back up is more important than rebuilding an area that's been severely decimated (and not in the Roman 10% way).

    Take a look at this:

    https://www.google.com/search?tbm=nws&sclient=psy-ab&q=+new+york+city+recovery+hurricane+sandy&oq=+new+york+city+recovery+hurricane+sandy

    Over 200,000 articles and 3600 news sources covering NYC's recovery efforts. Surely there's enough space left on the Internet for a News for Nerds site to cover news for nerds?

  5. Re:New York will re-build. by Animats · · Score: 4, Informative

    Obviously not, because they are going to re-build.

    Manhattan barely has to rebuild anything. Building codes are tough there - everything has to be brick, concrete, or steel. Building foundations go down to bedrock. Few Manhattan buildings were damaged by the hurricane. One three-story slum had the front facade collapse; the walls and floors held and no one was injured. One construction crane had its boom broken by the wind, but the safety cables held and it didn't fall. That was about it for Manhattan.

    Yes, there was about a half billion gallons of water in subway, railroad, and road tunnels. Was. The MTA has big pumps. They have pumping trains made from old subway cars which they pushed up to the water with small Diesel locomotives. Half the East River tunnels are already pumped out and some lines under the river are operating. Limited subway service between Manhattan and Brooklyn should resume tomorrow.

    Power never failed for Manhattan above 34th St, and it's back on now for most of lower Manhattan. Even when flooded, underground power lines can be restored rapidly. That will speed up the remaining pumping work. With power back on, New York City's gasoline pipeline is running again, and gas stations are reopening.

    The areas that are severely damaged are single-family residential frame structures in coastal communities. Some of them are totally wiped out. People in the outer boroughs and the Jersey shore are getting cold and hungry. The first supermarket in Far Rockaway reopens at 11 AM Saturday. In Manhattan, as soon as the infrastructure came back up, the city was ready to go. Not so in the 'burbs.

    The idiots who stayed on Fire Island despite a mandatory evacuation order were finally rescued, with great difficulty. The first group of rescuers had to themselves be rescued; they were cut off when water cut all the way across the island. Now the people who built expensive frame houses facing the Atlantic Ocean only a few feet above sea level are whining for Government funding to rebuild.