Slashdot Mirror


Con Ed Says NYC Datacenters Should Get Power Saturday

Nerval's Lobster writes "The local utility serving most of the New York City area, Con Edison, reported that it should begin supplying utility power to midtown and lower Manhattan by Saturday evening, returning the island's data centers and citizens to some semblance of normalcy. In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power. That should come to an end, for the most part, when utility power is restored. In a possibly worrying note, Verizon warned late on Nov. 1 that its services to business customers could be impacted due to lack of fuel."

24 of 107 comments (clear)

  1. Fuel logistics by hawguy · · Score: 5, Insightful

    In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power.

    Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work. Few inner city datacenters have a week or more of fuel on-site - most have only days of fuel, and they count on fuel contracts from suppliers to keep them running. And the supplier may not be able to honor the contract in a disaster.

    Suburban and rural datacenters have the space (and less conflict with fire codes since the fuel is not stored in or near an office building) to keep weeks of fuel on hand. The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable. They had a spare generator that can be trucked in from that other facility as well. (and this facility could send fuel and a generator to that facility if needed)

    1. Re:Fuel logistics by hguorbray · · Score: 4, Funny

      And on the plus side, the IT guys who spent the week bucket brigading now have arms like Hulk....

      -I'm just sayin'

    2. Re:Fuel logistics by PolygamousRanchKid+ · · Score: 2

      Well, given that folks are already drawing firearms in gas lines, http://www.cbsnews.com/8301-201_162-57544187/new-yorker-sean-bailey-accused-of-pulling-gun-in-gas-line/ , part of the plan had better be to contract Mad Max to bring in a tanker of precious juice past The Humungous, Wez, and their pals.

      --
      Schroedinger's Brexit: The UK is both in and out of the EU at the same time!
    3. Re:Fuel logistics by hawguy · · Score: 2

      Well, given that folks are already drawing firearms in gas lines, http://www.cbsnews.com/8301-201_162-57544187/new-yorker-sean-bailey-accused-of-pulling-gun-in-gas-line/ , part of the plan had better be to contract Mad Max to bring in a tanker of precious juice past The Humungous, Wez, and their pals.

      Well the good news is that few drivers in the USA can use the diesel that's needed to feed datacenter generators in their cars, so hopefully the trucks will get through.

    4. Re:Fuel logistics by girlintraining · · Score: 3

      Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work.

      I don't think any data center manager had a line item in the disaster recovery plan that included having all transportation access cut to the entire island due to flooding of the tunnel and closure of the bridges, for over a week. Everyone is having a problem getting fuel into the city; even mission-critical services like emergency services, hospitals, and telecommunications facilities.

      As to your comment that "suburban and rural datacenters have the space", sure... but where's the fiber optic cable hookups and the telecommunications infrastructure located? I'll give you a hint: Not in a barn. Those data centers are located downtown because that's where everything else is. Not only that, but most of the data centers on the island are there because that's where Wall St. is, and milliseconds matter when it comes to high volume trading and financial transactions. Commercial real estate is at a premium in New York. Actually, all real estate is, leading to the old joke that when a New Yorker hears someone has died, the first question they ask is, "Is their apartment for rent?"

      I think it's more likely to assume you've made an error in your reasoning, writing opinions from the comfort of an armchair, than people being paid over six figures who's job depends on balancing everything out exactly and to the nearest penny an hour.

      --
      #fuckbeta #iamslashdot #dicemustdie
    5. Re:Fuel logistics by hawguy · · Score: 2

      The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable.

      How often do they rotate/burn through that supply?

      As has been mentioned in a few articles that I've read, diesel fuel doesn't last forever, and so if you tried to use it after it's been sitting too long you may have problems. Hopefully as part of regular maintenance / testing they burn through it of the course of a year and replenish it with fresh stuff.

      I don't remember the details on their rotation schedule, but I imagine that having their own fuel trailers made it easier to swap out fuel with some other site that needed it. Maybe they sold 9 month old fuel to an industrial site at a discount.

      I do remember that they had their generator vendor bring in a load bank large enough to let them do full-load testing every month. And once a quarter they did a live cutover from utility power to generator. This seemed a little risky at first, but it makes sense - would you rather find out during a disaster that some component of the emergency power system wasn't working because it was not adequately tested, or would you rather find out when the generator vendor and electricians are standing by and utility power is available to fail back from the generator.

      That facility weathered several power outages without a glitch, but the longest I can remember was around 48 hours.

    6. Re:Fuel logistics by Anonymous Coward · · Score: 5, Informative

      True, but you can't legally use the fuel that's meant for the data center in your truck to get it there. We did that after Hugo hit the SC coast in Sept 1989 and got caught. The generator fuel has dye in it that will stain the fuel filter. It took nearly six months and several tens of thousands in legal fees to get our truck back. Meanwhile, the servers in our data center in Goose Creek, SC ran out of fuel and nearly put the company out of business. There were dozens of other trucks that got caught at the weight stations over the next few years that also used fuel not meant for use on the road that were also fined and/or confiscated. My father-in-law owns a towing company so they got a lot of towing and storage business from that. Don't underestimate the US government's desire to screw over the little guy and their desire to put companies out of business.

    7. Re:Fuel logistics by girlintraining · · Score: 3

      Diesel fuel lasts about a year if kept dry, around 68F, and in a sealed container. It can be prolonged with additives but this is not usually done because they can clog the fuel filter. Typically a data center keeps 2--5 days of fuel on site in an urban area; Not due to the cost of the fuel (which is a pitance) but the cost of storage, which requires commercial real estate. In most metropolitan areas, that price is $40-200 per square foot. Do the math.

      --
      #fuckbeta #iamslashdot #dicemustdie
    8. Re:Fuel logistics by hawguy · · Score: 4, Insightful

      Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work.

      I don't think any data center manager had a line item in the disaster recovery plan that included having all transportation access cut to the entire island due to flooding of the tunnel and closure of the bridges, for over a week. Everyone is having a problem getting fuel into the city; even mission-critical services like emergency services, hospitals, and telecommunications facilities.

      What kind of disaster plan is it that doesn't account for a likely disaster? I can guarantee that every sizable datacenter had exactly this scenario in the DR plans. You don't run a $10M facility in a coastal city on an island without including flooding in your DR plan. This was a 100 year event, so it was definitely on everyone's horizon. Having it in the plan doesn't mean that you have a good solution - living in SF means Earthquakes are a big part of our DR plans, but we have no expectation that our facility will survive a 7.5 earthquake intact, or even that our employees will be motivated to come to work when they are worried about their own survival.

      As to your comment that "suburban and rural datacenters have the space", sure... but where's the fiber optic cable hookups and the telecommunications infrastructure located? I'll give you a hint: Not in a barn. Those data centers are located downtown because that's where everything else is.

      Just follow the train lines to find out where the major telecommunication lines are -- I have access to more carriers down on the Peninsula outside of San Francisco than I do downtown. You may be surprised at how much bandwidth runs through Colorado and even Missouri.

      Not only that, but most of the data centers on the island are there because that's where Wall St. is, and milliseconds matter when it comes to high volume trading and financial transactions.

      Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.

      Commercial real estate is at a premium in New York. Actually, all real estate is, leading to the old joke that when a New Yorker hears someone has died, the first question they ask is, "Is their apartment for rent?"

      I'm not sure what your point is? Datacenters have to be built in the city because that's where the carriers are, but commercial real estate is expensive so don't build your datacenter in the city?

      I think it's more likely to assume you've made an error in your reasoning, writing opinions from the comfort of an armchair, than people being paid over six figures who's job depends on balancing everything out exactly and to the nearest penny an hour.

      You're obviously not in NYC if you think "over six figures" means someone is highly paid. Part of my job is planning our IT DR strategy. Fortunately, our Facilities dept is in charge of the generator so I don't need to worry about fuel contracts or keeping it running, but I do need to make sure our data is safe no matter what happens to the building and that we can continue to operate as a business. The only "disaster" that we plan on riding out on the building generator is a localized power outage when we know we'll be able to get fuel once our 3 - 5 days of fuel runs out. If there's a widespread power outage or disaster, our plan is to transition to the remote site since we know we may not be able to keep the generator fueled.

    9. Re:Fuel logistics by kriston · · Score: 2

      The community of Great Falls, VA, experienced an odd situation during Hurricanes Irene, Isabel, and the 2012 Derecho. The local natural gas utility was unable to provide the bottle pressure required to maintain service in these areas due to the widespread installation of residential generators running on natural gas fuel.

      I don't live in Great Falls but I have a propane-powered generator only because of the lack of natural gas service. One thing that's nice about propane is that it is so stable that its storage life is virtually unlimited.

      --

      Kriston

    10. Re:Fuel logistics by kriston · · Score: 3, Insightful

      The government is not "screw[ing] over the little guy" as you so eloquently put it. The non-road diesel fuel is dyed specifically because that fuel's tariff does not carry road tax. Road tax pays for the roads. This is what the situation is really about. So many violators were using non-road fuel that they had to take steps.

      All licensed truck drivers implicitly understand that you do not put non-road/farm fuel into a road vehicle if you intend to keep that vehicle in revenue service. It's intuitively obvious to everyone, but it's too bad your drivers were either ignorant of the law or chose not to follow it.

      --

      Kriston

    11. Re:Fuel logistics by dgatwood · · Score: 2

      Many compressor stations do run on gas, but not all.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    12. Re:Fuel logistics by hawguy · · Score: 3, Insightful

      People who vote for big government get everything that comes with it... like dyes added to products for no reason other than to enable government to control you (by regulating how you use things and by taxing things with complex lobbyist-influenced tax codes).

      The Fuel tax is one of the more fair taxes - it taxes you (roughly) based on your usage of the resource it's supposed to be paying for. The thing I like least about fuel taxes is that they are not high enough to pay for road maintenance.

      Roving looters who cannot be stopped by the disarmed citizens

      If very citizen had a gun, then those packs of roving looters would be well armed, so there would be wild-west style shootouts in the street between the good guys and the bad guys. But in a disaster, the lines are blurred and it's not always clear who the good guy is and who the bad guy is. Is the guy walking down the street with a few bottles of water for his elderly mother a good guy or a bad guy when he refuses to share it with the young mother and child begging for it. What if the mother had a gun and took the water by force? Or what if the man had a gun and shot the mother when she tried to take the water by force?

      Lack of fuel (nobody has their own reserves. Most would not bother but there are plenty of people and businesses who would if they were not going to be taxed-and-regulated to death for trying)

      People are bemoaning the lack of fresh water and food. Both are legal to stockpile both by citizens and private businesses, yet there seem to be few businesses willing to stock up on thousands of bottles of water just in case disaster strikes. Why do you think they'd be more likely to stock up on a perishable and expensive to store fuel?

      Lack of sufficient private help (Americans used to have a huge number of private charitable orgs and "mutual aid" orgs that average people used to join that rushed in to help after disasters, but now nobody bothers because they've been taught to pay taxes and then count on government)

      The Red Cross and other private, charitable organizations always offer disaster relief. But some disasters require more manpower, machinery, legal law enforcement powers, or weapons than can be reasonably mustered by an organization funded by donations, so that's where the US government can step in. How many $10M helicopters and $100M heavy lift aircraft do you think the Red Cross should own and maintain in order to shuttle people and supplies to a disaster zone?

      Hopefully this very sad event will finally hammer into peoples' heads what Katrina should have (but apparently failed to) ... that in the end, when the going gets tough, you are responsible for being prepared for disasters and taking care of your family first, your neighbors next, and the larger community if you can; no government jerk from a far-away city will care as much as you do.

      That's definitely true and a good lesson to be learned from any disaster - be prepared to be on your own for 3 - 7 days. But few city dwellers can provide enough food, water, heat, medications (some need refrigeration), and sanitation (with water and possibly sewage not working in a big highrise) to stick it out for more than week or so in a disaster.

      If the people and businesses of NY and NJ were not burdened by crazy regulations and threats of lawsuits they'd probably get things back up and runninga on their own twice as fast as they are going to (locals would commit acts of common sense on behalf of themselves and their neighbors).

      Not all regulations are driven by government greed or thirst for power, many regulations are safety based. If you try to fill the back f your pickup with twenty 5 gallon buckets of fuel and try to sell it to the highest bidder on the streets, you're (hopefully) going to get stopped on the street. Likewise, if you set up your own s

    13. Re:Fuel logistics by drinkypoo · · Score: 2

      You're both just wrong. Diesel starts to spoil after about six months without a stabilizer; with one you could keep it a maximum of about two years without significant degradation. But you'll need to keep it sealed, because water in fuel is a very bad thing. The water will react with the fuel and create acids if left over time, which won't be removed by the water separator when there even is one in the system, which isn't the case with many diesels, e.g. my 1982 300SD. And more notably, diesel will grow mold, so for any kind of fuel storage, it is always standard to add a biocide to the fuel. Finally, most conventional diesels are bad at running on anything other than diesel fuel. Getting the fuel into the injection pump at the correct viscosity (as that can actually affect operation) and getting the fuel to the injector at the correct viscosity are both difficult. Running veg results in varnishing that must be removed by running biodiesel. Most fuels need to be adjusted to have not just similar viscosity but also similar cetane, or most engines won't run them correctly. Some vehicles (notably older GM IDIs) even have a fuel quality sensor and if you feed them too-dark a fuel they refuse to run.

      If you don't have a classic, indirectly-injected diesel with a non-rotary pump (e.g. the bosch inline design) then you should stay the hell away from any altfuels but biodiesel or well-filtered WMO thinned with RUG, which will work well in the International IDI. A sufficient filtering regimen is difficult, though. If you have a TDI, you probably can get away with B100 with a lubricity additive like Diesel Kleen, they have a bio-specific blend. But they are well known to crap on altfuels given sufficient time. The sad thing is that it seems like everything is cool right up until the expensive repair work.

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  2. Great News by ohnocitizen · · Score: 2

    Power to mid and lower Manhattan means a lot of people who have been without power will finally have it (and all the modern conveniences like refrigeration we rely so heavily on). It also means subway service between Manhattan and Brooklyn will come back sooner, which will be huge.

  3. Re:Get Some Priorities! by CastrTroy · · Score: 4, Insightful

    Depends what's running in the datacenter. Banks, grocery stores, public utilities, repair contractors all need datacenters to get their jobs done. Also, many people work in data centers, or have jobs which rely on the datacenters being up. For them, the datacenter does feed their family

    --

    Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
  4. Re:Get Some Priorities! by hawguy · · Score: 4, Insightful

    Dozens are dead, billions of dollars of property destroyed and businesses decimated, millions still in darkness and cold, and you are talking about data centers??

    Data centers cannot feed my family or heat my cold, flooded house.

      Where is your decency sir? It's a long week and weekend for the millions affected and the first responders. The glibness of this post is shocking and disgusting.

    This is Slashdot, there are plenty of other sites that cover human misery and suffering, this site is for geek news, and geeks care about datacenters.

    Datacenters can't feed your family or heat your cold, flooded house (unless you live next to a datacenter that recovers waste heat for residential heating), but the datacenter can help your utility get power back online, it can help your local merchant process credit card transactions and use their POS system so you can buy replacement goods, it can let you post your family's status so your mom can stop worrying about you, it can help your bank conduct online transactions efficiently to let you receive your insurance money, and of course, you're using multiple datacenters right now to read and post to Slashdot.

    If you think datacenters are unimportant, try going a week without using any good or service that wasn't produced or delivered to you without the aid of datacenters.

  5. Re:Get Some Priorities! by hawguy · · Score: 4, Insightful

    As little as I like agreeing with ACs, this was my exact thought upon reading TFS. I know this is News For Nerds, but let's not pretend getting data centers back up is more important than rebuilding an area that's been severely decimated (and not in the Roman 10% way).

    Take a look at this:

    https://www.google.com/search?tbm=nws&sclient=psy-ab&q=+new+york+city+recovery+hurricane+sandy&oq=+new+york+city+recovery+hurricane+sandy

    Over 200,000 articles and 3600 news sources covering NYC's recovery efforts. Surely there's enough space left on the Internet for a News for Nerds site to cover news for nerds?

  6. Our datacenter is in lower Manhattan.. by HerculesMO · · Score: 2

    And in the brilliance of the building engineers, the generator is in the basement.

    Which is now filled with 13 feet of water.

    It's going to be fun cleaning up.

    --
    The price is always right if someone else is paying.
  7. Re:New York will re-build. by drinkypoo · · Score: 2

    First 9/11 and now this. Have they not suffered enough?

    Obviously not, because they are going to re-build.

    By far the smartest thing to do would be to re-build somewhere else. Preferably someplace with more altitude. Unless you really don't believe in AGW, in which case, many have beachfront land to sell you.

    --
    "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  8. Dice, get better authors by girlintraining · · Score: 3

    You guys recently bought slashdot, and let me say, the first few "sponsored links" have been a real disappointment. TFA has a picture that's a screenshot of the ConEd website, and poorly cropped. The information is almost 10 hours out of date at time of posting, and most of the article consists of direct quotes from articles previously submitted to slashdot! Where's the originality? Where's the reporting on why this matters? Journalism includes an analysis of the facts, not just a compilation of them.

    --
    #fuckbeta #iamslashdot #dicemustdie
  9. Re:Get Some Priorities! by MrNaz · · Score: 2

    While I agree with you on your overall sentiments, your general tone is offensive. Many who died may have prepared, but the means to do more may have been out of their capacity. What would you do to prepare for a storm? Board the windows? Brace the doors? Put everything sharp in a closed, low height cupboard? How would that help if Sandy dropped a few cars on your house, levelling it? And even if you did survive that, remember, the biggest killer in natural disasters is the disease and sepsis that follows.

    Off your high horse, buddy.

    --
    I hate printers.
  10. Re:Get Some Priorities! by onyxruby · · Score: 2

    Datacenters can't feed your family or heat your cold, flooded house

    I would really have to argue that as data centers are often what drive the engines of commerce. The modern engines of commerce that send out deliveries of food, keep the heat running, operate the financial system that allows commerce and every other thing that our modern society depends on are completely dependent upon data centers.

    The only question is whether or not the data center that needs the electricity happens to be the one that drives one of the particular above functions. Even if it isn't, you can rest assured that those functions are wholly dependent on a data center /somewhere/.

  11. Re:New York will re-build. by Animats · · Score: 4, Informative

    Obviously not, because they are going to re-build.

    Manhattan barely has to rebuild anything. Building codes are tough there - everything has to be brick, concrete, or steel. Building foundations go down to bedrock. Few Manhattan buildings were damaged by the hurricane. One three-story slum had the front facade collapse; the walls and floors held and no one was injured. One construction crane had its boom broken by the wind, but the safety cables held and it didn't fall. That was about it for Manhattan.

    Yes, there was about a half billion gallons of water in subway, railroad, and road tunnels. Was. The MTA has big pumps. They have pumping trains made from old subway cars which they pushed up to the water with small Diesel locomotives. Half the East River tunnels are already pumped out and some lines under the river are operating. Limited subway service between Manhattan and Brooklyn should resume tomorrow.

    Power never failed for Manhattan above 34th St, and it's back on now for most of lower Manhattan. Even when flooded, underground power lines can be restored rapidly. That will speed up the remaining pumping work. With power back on, New York City's gasoline pipeline is running again, and gas stations are reopening.

    The areas that are severely damaged are single-family residential frame structures in coastal communities. Some of them are totally wiped out. People in the outer boroughs and the Jersey shore are getting cold and hungry. The first supermarket in Far Rockaway reopens at 11 AM Saturday. In Manhattan, as soon as the infrastructure came back up, the city was ready to go. Not so in the 'burbs.

    The idiots who stayed on Fire Island despite a mandatory evacuation order were finally rescued, with great difficulty. The first group of rescuers had to themselves be rescued; they were cut off when water cut all the way across the island. Now the people who built expensive frame houses facing the Atlantic Ocean only a few feet above sea level are whining for Government funding to rebuild.