Slashdot Mirror


NYC Data Centers Struggle To Recover After Sandy

Nerval's Lobster writes "Problems in New York's data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements. The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It's also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services. The following is a list of some of the data centers and services in the area, and how they're faring." I'm responsible for a few servers at Peer1, and their efforts are interesting: "Peer1’s operations at 75 Broad are operating on sheer manpower: a bucket brigade. According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs."

231 comments

  1. citibank.com by colin_faber · · Score: 0

    I figured they just went out of business. I guess it's this hurricane thing this time..

  2. The Cloud by azalin · · Score: 5, Funny

    When it's wet, the clouds go down

    1. Re:The Cloud by girlintraining · · Score: 0

      When it's wet, the clouds go down

      I fail to see how this is informative; It's patently obvious that electricity and water doesn't mix, cloud or no cloud.

      --
      #fuckbeta #iamslashdot #dicemustdie
    2. Re:The Cloud by Anonymous Coward · · Score: 0

      When the storm is over, the clouds are gone.

    3. Re:The Cloud by firewrought · · Score: 1

      I fail to see how this is informative; It's patently obvious that electricity and water doesn't mix, cloud or no cloud.

      Comment #41832161, meet comment #41832323.

      When it's wet, the clouds go down even where it's dry.

      --
      -1, Too Many Layers Of Abstraction
    4. Re:The Cloud by muon-catalyzed · · Score: 2

      On my netbook, I sinced all my work
      to the Cloud, the promised best way
      No storms over here, only sunny pleasant day
      my work is gone anyway.

    5. Re:The Cloud by tlhIngan · · Score: 2

      On my netbook, I sinced all my work
        to the Cloud, the promised best way
        No storms over here, only sunny pleasant day
        my work is gone anyway.

      There's your problem. A sunny pleasant day doesn't have clouds! So by definition your cloud is gone.

      Or put another way - cloud gone due to stretch of good weather.

    6. Re:The Cloud by Anonymous Coward · · Score: 0

      All I want to know is if a Google data center went down and suffered unrecoverable data loss.

    7. Re:The Cloud by Ol+Biscuitbarrel · · Score: 1

      Sandy blows through
      Generators in flooded basement
      The Cloud

  3. Add to that, NYI... by malakai · · Score: 5, Informative

    At 100 Williams Street, http://www.nyistatus.com/
    My server and connections have been up non stop.

    I know it's cynical of me, but I find it a bit sad that we can better plan data centers then medical factilities.

    I know all the colocation facilities I've been to in Manhattan have generators above the 6th floor ( sometimes in addition to generators in the basement). A few had them on the roof with some special setup that allows fuel to be flown by helicopter for worse case scenarios.

    1. Re:Add to that, NYI... by afidel · · Score: 5, Interesting

      NYU's generators were fine, it was their fuel supply that got fouled. Fire regs don't allow them to have thousands of gallons of diesel anywhere but underground holding tanks and those were overcome by seawater. Bellevue lost two of their primary generators due to water in the basement but was still running on another on the 13th floor but they had the same limited fuel problem Peer1 is running into. They considered having the national guard bucket brigade fuel up to the 13th floor but after some analysis it was decided it would be better to transfer folks to other hospitals (I'm not sure how many generators were on the 13th floor but it was probably only a single one and so they were down to a SPOF so better to transfer people in an orderly manner while you still have working facilities than to try it after the generator went down).

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    2. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      We can plan medical facilities that are as disaster-resistant as data centers...
      Most medical facilities are quite a bit older than most data centers.
      The new hospital just built in my county is quite a bit better prepared for coastal storms (southeast North Carolina coast) than the previous hospital.

      Yes.. that's an assumption on my part based on my own anecdotal evidence, but it passes the common-sense test for me.
      Perhaps someone 'in the know' has better data to agree or disagree with my hypothesis.

    3. Re:Add to that, NYI... by aaarrrgggh · · Score: 0

      There is a lesson learned that is rarely applied though-- keep a fuel polisher upstairs at the generator. I might also consider a compressed air displacement pump at this point as well, although that is gawdawfully inefficient. But, for either of these to work, you need enough time on your day tanks to allow for some manual operations and time to allow for the water to be removed from the fuel. I'm trying to think of how you could pipe in a pneumatic displacement pump on the fly to get things restarted, but it is pretty hard unless you had access to an unused port on the tank.

      Most data centers I've worked on have a hand pump in the basement as a backup to pump failure, but it is almost as hard to use as the bucket brigade, especially for a tall building. Your general hope is that you can get the generators running long enough to get the pumps operating to clear the water... but you have to have a way to stop more water from coming in.

    4. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      What boggles my mind is... backup generators in the basement, or on the main floor.

      Have we learned ABSOLUTELY NOTHING from Fukushima, or any of the other multitude of less high-profile problems caused by generators below flood water levels?

      YOU'RE ON THE COAST! PUT YOUR BACKUP POWER WHERE WATER WON'T REACH EASILY!

      How is it that this isn't the absolute first thing taken into account in building plans out there? Where I live (Winnipeg, Manitoba), one of the first things taken into account is flood concerns. ie: no basements unless you live on a hill. That's not to say it's never done... people are still stupid (or have plans to sell a year after they get the house... the basement means they can charge more whether the buyer wants it or not, and they just roll the dice that it doesn't flood that year. Not like buyers have enough options to choose other houses), but generally speaking if you don't want your shit fucked up, you don't have it below ground, or at ground level. One of the first things you look into with regards to a new house - if any part of it is below ground... how new is the sump pump.

      So seriously... the cities on the coast get hit damn near every other year with a storm. Why the goddamn hell aren't you people learning from this? "Oh no, it'll cost extra to support the weight of fuel and generators on a platform one or two floors high". Well, hope those savings back then were enough to rebuild everything.

    5. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      why diesel? any generator worth it's salt is running on Natural Gas or Propane.

    6. Re:Add to that, NYI... by Bronster · · Score: 2

      Been really impressed with NYI - we haven't had a single glitch at FastMail either.

      We have an emergency backup plan (Iceland) - but it's nice not to have to use it.

    7. Re:Add to that, NYI... by nedlohs · · Score: 2

      Good luck convincing the fire department you meet the fire codes with your five thousand gallon tank of fuel on floor 45.

    8. Re:Add to that, NYI... by twisted_pare · · Score: 5, Insightful

      Poor planning, plain and simple.

      I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat. What, were we just "lucky?" I don't think so.

      Starting a week ago we had disaster crisis centers setup.
      * Every few hours all East coast facilities reported in any issues
      * Inspection and testing of all critical systems ahead of time
      * Stockpiles of supplies on hand
      * Prefail over to DR where possible
      * All hands on deck to respond


      Sadly, if you want to be prepared, you can be. If tons of money is on the line, then the price of being prepared is well worth it. We test our systems continuously year round. We have disaster recovery drills at all facilities multiple times a year. Departments' rating depend on how well prepared they are for things like this.

      And don't throw that "1888," "worth storm ever" crap around. This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even! If you did not prepare for this including flooding and sealed underground tanks and sandbag walls, it was your own fault.

      --
      HTFU
    9. Re:Add to that, NYI... by rcamera · · Score: 4, Informative

      for residential, that's true. for commercial and/or municipal, they stick to diesel.

      in the case that con-ed shuts off nat gas flow due to fires (which is happening all over the place right now...), do you want your generator to choke out?

      that leaves lp. diesel packs about a 50% higher energy density than lp. which would you prefer to use? further, diesel is much more commoditized than lp, and is more readily available during a crisis.

      --
      Wave upon wave of demented avengers March cheerfully out of obscurity into the dream
    10. Re:Add to that, NYI... by acoustix · · Score: 4, Informative

      why diesel? any generator worth it's salt is running on Natural Gas or Propane.

      Obvious answer: Natural gas lines are usually cut during emergencies like flooding, fires, etc. Storing a liquid fuel like diesel allows you use the generator when external energy sources have been severed.

      --
      "A plan fiendishly clever in its intricacies"- Homer Simpson
    11. Re:Add to that, NYI... by afidel · · Score: 1

      Because during a disaster utilities including natural gas aren't guaranteed to be up, plus if diesel is considered a fire hazard then natural gas is probably verboten. Oh, and high pressure natural gas of the type needed for 100kw+ generators isn't available everywhere. Diesel generators also tend to be cheaper per rated and true kw.

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    12. Re:Add to that, NYI... by tibit · · Score: 1

      Hand pump? What kind of an idiot thought that up? You need a pump with a foot crank drive. A.k.a. a pickup off an exercise bike or somesuch.

      --
      A successful API design takes a mixture of software design and pedagogy.
    13. Re:Add to that, NYI... by tibit · · Score: 1

      Ta-ta-ta-tah! We have a winner :) Couldn't agree more.

      --
      A successful API design takes a mixture of software design and pedagogy.
    14. Re:Add to that, NYI... by alphatel · · Score: 1

      100 William and 111 Eighth Ave have both been planned for the worst quite appropriately. All the equipment at Telx and Level3 datacenters in both buildings are operational, though I did see some minor outages during the storm as systems switched over.

      --
      When the foot seeks the place of the head, the line is crossed. Know your place. Keep your place. Be a shoe.
    15. Re:Add to that, NYI... by jbmartin6 · · Score: 4, Insightful

      Thank you! I thought the same thing about all the hospitals. I worked for a hospital for a while and believe me, they don't do disaster planning or even equipment life cycle planning. You would think with all the lip service paid to 'patient care' that continuity and disaster planning would be more of a priority, but no. Usually these aren't real businesses and they have no incentive to behave like one. As another poster mentioned, evacuation is usually their continuity plan.

      --
      This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
    16. Re:Add to that, NYI... by DerekLyons · · Score: 4, Insightful

      And don't throw that "1888," "worth storm ever" crap around.

      Well, it's a fact. If you can't deal in facts, you shouldn't be working where you claim to be.
       

      I work for a major financial institution on the street. Various facilities were swamped, and we never missed a beat.

      You're lucky enough to be able to afford to have a complex defense in depth - this isn't true of everyone.

      So take your "I'm big bad dude" attitude, and stuff it. You're only big because daddy has money.

    17. Re:Add to that, NYI... by Ltap · · Score: 1, Troll

      Well, it's a fact. If you can't deal in facts, you shouldn't be working where you claim to be.

      I disagree. Dealing with facts would probably be an impediment to his job.

      --
      Yet Another Tech Blog
      (but so much more, including game and movie reviews)
      http://yanteb.peasantoid.org
    18. Re:Add to that, NYI... by artg · · Score: 1

      This is Wall Street. Manhattan. Terrorists have tried to blow it off the map multiple times. Several hurricanes have hit this spit of land that sits a mere few feet above sea level in the last decades. A hurricane hit and flooded parts last year even!

      And that's the best site you can find for a datacentre ? Doesn't give me much confidence in your financial organisation's acumen.

    19. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      I agree with being prepared, but you were lucky.

      We were prepared, even had a smaller backup datacentre but the main one almost went down anyway. When you have to deal with 2 megalitre per second flood water flows (that actually may have been 2 gigalitre ... it was a shit tonne of water), preparation doesn't really help. Nature at its finest will bring down the best laid plans. We were saved because it stopped raining. You don't believe it until you see it.

      This was in the major floods that happened in Queensland, Australia in January 2011.

    20. Re:Add to that, NYI... by AK+Marc · · Score: 1

      Natural gas would never be considered because it's a utility. Anything that cuts power could cut NG. Propane is easier/cheaper than compressed natural gas in tanks. So the choice comes down to propane or diesel. Most of the rules for diesel apply to propane (storage locations and such) and propane is harder to deal with. Oddly, for that, the harder to deal with issues will generally allow it to remain pure when the tank is submerged. If they made the diesel tanks completely airtight and pressurized, as propane is, then there'd be no problem. Diesel is long-lasting, cheap, and easy to get, even in national emergency conditions, as long as you are willing to pay for it.

    21. Re:Add to that, NYI... by dkuntz · · Score: 2

      Because diesel generators last up to 10 times longer than propane generators. In addition, they cost more to purchase, and cost about 3x as much to operate.

      Oh, lets not forget the fact that diesel is a lot safer than propane or natural gas.

      http://www.generatorjoe.net/html/genfuel.html

      --
      OMG... I have a sig?
    22. Re:Add to that, NYI... by AK+Marc · · Score: 2

      They are heavy. Also fire code won't let them put the fuel up high. Some of the failures were generators up high starved of fuel from the tainted fuel in a groud-level tank, the only legal option. Personally I'd put a tank on the roof for "helicopter delivery" and pump the fuel from the ground up there when a flood is predicted. Sometimes the law is wrong.

    23. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      Agreed. Losers can take a hike, we need producers not excusers.

    24. Re:Add to that, NYI... by jbmartin6 · · Score: 3, Insightful

      To be fair, if conditions are that bad at the hospital, evacuation is probably the best plan anyway.

      --
      This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
    25. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      No glitches except that you broke it with your "upgrade" ...

    26. Re:Add to that, NYI... by cheater512 · · Score: 1

      I'm not entirely sure why they can't shunt generator power to the elevators.
      Fill the cargo elevator with diesel and your bucket brigade just became an awful lot more efficient.

    27. Re:Add to that, NYI... by StevisF · · Score: 1

      Or, hospitals don't have the billions of dollars of profit at their disposal that financial companies do. I think this is a sad reflection of our society's priorities, not on hospitals' planning efforts.

    28. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      Im on the street as well, and surprisingly the ONLY problems we had were from the issues we had with the incumbent financial market data / social networking provider in the world, Bloomberg. All my other providers were beyond prepared from application to layer 1 and 2.

    29. Re:Add to that, NYI... by aaarrrgggh · · Score: 2

      You are never allowed to carry fuel in an elevator. The fire department would red tag the building if you were caught doing it.

      Spilling fuel in an elevator shaft makes for a very effective chimney.

    30. Re:Add to that, NYI... by aaarrrgggh · · Score: 1

      The pumps are kind of like a hydraulic floor jack. You have quite a bit of head pressure to overcome.

    31. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      We use propane for our generator. I would prefer diesel, but some of our research is on particulates in the air, and the concern is that soot particles from a diesel generator could impact the measurements. And, it's not a small unit; 100 kW, three phase. And, the tanks are huge. I don't agree that diesel is more readily available. It's all delivered by trucks. There are eight propane delivery companies in a 20 miles radius, probably the same number of bulk fuel companies.

    32. Re:Add to that, NYI... by Macgrrl · · Score: 2

      My experience of multiple contracting gigs for hospitals is that unless the expenditure can be linked directly to day-to-day patient care you pretty much have no chance of getting funding approved for any initiative, even when it will reduce operational expenses.

      --
      Sara
      Designer, Gamer, Macgrrl in an XP World
    33. Re:Add to that, NYI... by toddestan · · Score: 2

      So instead you risk spilling it down a stairwell?

    34. Re:Add to that, NYI... by Cyberax · · Score: 1

      Propane can be a liquid at room temperature. My parents have a house with natural-gas fueled generator and heating. They also have a small tank of liquid propane buried in their backyard as a backup gas supply.

    35. Re:Add to that, NYI... by timbo234 · · Score: 1

      Good to hear fastmail has an emergency plan, it's been very, very reliable for the last couple of years and I hope it stays that way :)
      I'm begginning to realise how reliant I am on email - today I got an invoice by email that if I hadn't seen it straight away would've costed over 100 pounds extra due to not cancelling something early enough.

      BTW the new Fastmail interface is great!

      --
      Pre-canned Evolution Links for all those Slashdot holy wars.
    36. Re:Add to that, NYI... by tibit · · Score: 1

      Sure, but that's what gearing is for. You'd want to set up such a system to require about 150W to pump. That's what a semi-fit human should be able to cope with.

      --
      A successful API design takes a mixture of software design and pedagogy.
    37. Re:Add to that, NYI... by aaarrrgggh · · Score: 1

      Let's try my math:
      1 Gallon Diesel = 7.15#
      HP = # x ft / (33,000 x min) => Gallons x 7.15 x rise /min = HP =>GPM = HP/( 7.15 x Rise)
      150W/746 = 0.2HP

      Assuming no friction losses in the pipe and perfect mechanical efficiency, with a 100 ft rise you get:
      GPM = .2 x 33,000 / (7.15 x 100) = 0.92
      So, that puts us at 55GPH, or enough to support a 750kW generator. Realistically, with efficiency, you are down to ~300kW

    38. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      stairwells have plentiful access to for both humans and airflow. You can clean a stairwell of a fuel spill a hell of a lot faster and safer than you ever could an elevator shaft.

    39. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      Goldman? Good for you

    40. Re:Add to that, NYI... by metrometro · · Score: 1

      Make your own obvious point about allocation of resources in late-stage capitalism.

    41. Re:Add to that, NYI... by Anubis+IV · · Score: 1

      That makes sense for your parent's house, since they're already relying on natural gas on a regular basis, which means that propane has a huge advantage when you're considering which fuel to use during emergencies. But data centers rely on the electrical grid for their power, meaning that propane has no such advantage, and when you compare it against diesel, you'll see that diesel has about 65% more BTUs per unit of volume, making it easier to pack more energy away for a rainy day. And when you're talking about generators with tanks that take thousands of gallons of fuel (as some of these are), energy/unit volume is something you definitely need to be considering, since it factors into your costs to store the fuel.

    42. Re:Add to that, NYI... by UnknownSoldier · · Score: 1

      Nice summary of the various pros/cons of gas, diesel, propane, natural gas, and bio-diesel !

      Thanks!

    43. Re:Add to that, NYI... by dgatwood · · Score: 1

      If the fire code bans that, then the fire code is broken. If anything, a fire in the basement is worse than a fire on an upper floor because the whole building becomes a chimney, including the stairwells and elevator shafts. By contrast, the only added risks from having tanks on a higher floor are the risk of leakage (which is easily detected and prevented through proper multi-hull tank design) and the risk posed by having pipes filled with fuel in your walls (which is far less dangerous than the natural gas pipes you already have in those walls).

      Okay, so there's a small additional risk if a fire and massive leakage somehow occur simultaneously (catastrophic tank failure combined with an outside ignition source, e.g. a terrorist bombing), but even then, the risk is primarily to folks close to the source of fuel, which is still a lot better than a fire in the basement that puts the entire building in immediate danger and makes exiting the building difficult for everyone on every floor.

      What we have here is a situation where regulations created to solve a theoretical problem that is better solved in other ways prevents solving a very real problem that can't be solved in any other way. Such regulations need to be rethought.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    44. Re:Add to that, NYI... by Ledgem · · Score: 1

      I don't think this has anything to do with behaving like a business. Most hospitals that I'm familiar with aren't exactly awash with money. Plenty struggle to maintain their daily operations. I agree that they should be prepared for disasters, but it makes little sense for them to put money into staving off a potential future disaster when they're battling against one major disaster every day: their bankruptcy and shutdown.

    45. Re:Add to that, NYI... by Anonymous Coward · · Score: 0

      why diesel? any generator worth it's salt is running on Natural Gas or Propane, said the master of ignorance.

    46. Re:Add to that, NYI... by tibit · · Score: 1

      Yup. And with a hand pump you can crank out 150W for about 90 seconds (see research papers from the Gamera II HPH project) :)

      150W from legs is something I can do reasonably indefinitely, and I'm hardly very fit (I hike a lot about once a year and that's it).

      So for a 1MW or similarly sized generator, you'd need a team of 4 people going in 50 minute shifts with 10 minute breaks. Or 3 fit people which should be in ample supply in NY I'd hope :)

      --
      A successful API design takes a mixture of software design and pedagogy.
    47. Re:Add to that, NYI... by CAIMLAS · · Score: 1

      Yes, but then you run into the problem of relying on gravity to keep the diesel inside the buckets.

      --
      ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
    48. Re:Add to that, NYI... by CaptSlaq · · Score: 1

      I would assume that the reason why the DC is there is for latency reasons. Probably peered to or on the same network as whatever major market their institution deals with.

      And just in case you think I'm blowing smoke, let me remind you of a previous article: http://tech.slashdot.org/story/11/09/13/0258217/300m-to-save-6-milliseconds

  4. extraordinary effort = extraordinary cost? by prgrmr · · Score: 1

    How much in fines from OSHA or the NY EPA are these companies looking at for the bucket brigade?

    1. Re:extraordinary effort = extraordinary cost? by vlm · · Score: 1

      Usually the epic fail of "fuel in non-fuel rated areas" is the fire chief flips his lid, especially if the sprinkler system is down and/or you're transporting slippery oil via the emergency evac route.

      This is just "no cans of gasoline (for scooters, mopeds, etc) allowed in the dorms" writ large.

      --
      "Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
    2. Re:extraordinary effort = extraordinary cost? by Anonymous Coward · · Score: 0

      As soon as I saw buckets, I thought the same thing. Jesus. Buckets of fuel, on the stairs...

    3. Re:extraordinary effort = extraordinary cost? by sunking2 · · Score: 1

      Stop being so literal, it says (or cans)

    4. Re:extraordinary effort = extraordinary cost? by CastrTroy · · Score: 1

      I was wondering why they don't carry it up the side of a the building using a winch or window washing lift.

      --

      Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
    5. Re:extraordinary effort = extraordinary cost? by bws111 · · Score: 1

      A winch or window washing lift powered by what?

    6. Re:extraordinary effort = extraordinary cost? by houghi · · Score: 1

      human power, a rope and http://en.wikipedia.org/wiki/Pulley

      --
      Don't fight for your country, if your country does not fight for you.
    7. Re:extraordinary effort = extraordinary cost? by CastrTroy · · Score: 2

      As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.

      --

      Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
    8. Re:extraordinary effort = extraordinary cost? by hawguy · · Score: 1

      As somebody else already said, human power. There's also the option of tapping into the generator. I don't know the specifics, but if it takes less than a gallon of fuel to lift a gallon of fuel up the side of the building, you could just tap into the generator you're trying to refuel. The other option is to have a second smaller generator on the ground that powers just the lift/winch. This could be easily refueled. Seems like bucket brigade was the first thing that came to mind, and once they had that going, people stopped thinking of better ideas.

      Or maybe they've seen enough Bugs Bunny cartoons to know what happens when a makeshift winch fails - whatever it's carrying falls to the ground in a big splat, flattening whoever it falls onto.

      Hoisting a 30 pound gas can 200 feet in the air during a disaster is not the time for amateurs to jury rig something together.

    9. Re:extraordinary effort = extraordinary cost? by bws111 · · Score: 2

      You are making a lot of assumptions there, starting with the assumption that what you are suggesting is even legal. Next is that the people in charge of the building are also the people in charge of the datacenter (unlikely), and that the people in charge of the building rank your datacenter as a higher priority than everything and everyone else in the building.

      This is not some sort of heroic lifesaving operation where 'do anything possible' applies. It is just very bad disaster preparedness on the part of the data center operators. "Get a backup generator running onsite" should not be part of a disaster recovery plan, ever. What if the building is on fire? Is that still your plan?

      If these datacenters are important enough where they are willing to put peoples lives at risk (hauling fuel up firewells or the outside of the building!) then they should have switched over to their alternate site (far from NYC) days ago. If they are no important enough to have alternate sites, then they should do what every other person and business impacted is doing: wait until it is safe, then start your cleanup.

    10. Re:extraordinary effort = extraordinary cost? by Budgreen · · Score: 1

      Why not just haul the drum up?? a 55gal. drum of diesel weighs about 400# and i'm sure 3 or 4 guys and a dolly could get it (slowly) up the stairs in about the same time as it takes to fill cans and have 11 guys to haul up 5gal each.

      --
      The greatest right given is the right to be wrong...
    11. Re:extraordinary effort = extraordinary cost? by ottothecow · · Score: 1
      I don't know if you have ever driven a car, but you can move your gallon of fuel through many stories of elevation change (along with literally tons of steel) for maybe a teaspoon or two of combusted fuel.

      This is why everything still uses combustion engines (including series-hybrid trains which are driven by purely electric motors connected to diesel generators).

      --
      Bottles.
    12. Re:extraordinary effort = extraordinary cost? by AK+Marc · · Score: 1

      Human powered up the stairs is moving 250 lbs up for every 5 gallons. A can hoisted up a rope/winch would be human power lifting 40 lbs or so per 5 gallons. And you are complaining about the power for moving 40 lbs, but not 250 lbs?

  5. Your priorities are all messed up!! by Anonymous Coward · · Score: 0

    Don;t you all understand!? People _died_ during this storm. Lots of Newyorkers are still without power and water and here your all woried about data centers!? Get a grip nerds!

    1. Re:Your priorities are all messed up!! by kav2k · · Score: 5, Funny

      People can be replaced. Uptime, on the other hand..

    2. Re:Your priorities are all messed up!! by Anonymous Coward · · Score: 0

      You think a bunch of sysadmins are going to get the power back on, fix the plumbing, or bring people back to life?

    3. Re:Your priorities are all messed up!! by EmagGeek · · Score: 3, Funny

      But, you see, without the Internet, it will be impossible to get all of those less essential services back up and running! After all, how is anyone going to obtain official updates from the Administration if they cannot access ready.gov, since that is apparently the only place to get them?

      So, let's make sure you have the priorities straight in your head:

      1) Internet
      2) Power
      3) Facebook Access
      4) Starbucks
      5) Running Water and Sewer

    4. Re:Your priorities are all messed up!! by kav2k · · Score: 3, Funny

      To close the sarcasm tag, here's also an obligatory xkcd.

      On a side note, this reminded me of Cory Doctorow's "When Sysadmins Ruled The Earth".

    5. Re:Your priorities are all messed up!! by Anonymous Coward · · Score: 0

      Call me if the population safety rating drops below five nines.

      (I'm kidding of course.. maybe)

    6. Re:Your priorities are all messed up!! by war4peace · · Score: 2

      What, no cold Pizza slices? You inconsiderate clod!

      --
      ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
    7. Re:Your priorities are all messed up!! by Anonymous Coward · · Score: 0

      When something like this happens, being interested in one particular aspect of the event and how people are dealing with it doesn't mean you don't care about the rest.

      This is a geek "news" site, where we often discuss things like disaster preparedness for data centers. If you're offended that we're actually doing that, please feel free to gtfo.

      Your illogical holier-than-thou rant isn't appreciated.

    8. Re:Your priorities are all messed up!! by Capt.DrumkenBum · · Score: 2

      You need to add a source of good coffee to this list.
      No, Starbucks is NOT good coffee!

      --
      If I were God, wouldn't I protect my churches from acts of me?
    9. Re:Your priorities are all messed up!! by nedlohs · · Score: 5, Interesting

      Fewer people died than the people who didn't die due to not being in a car crash due to not being able to drive to work. So on the life/death front it's a win!

    10. Re:Your priorities are all messed up!! by ciderbrew · · Score: 4, Funny

      The Starbucks is to test the Sewers with.

    11. Re:Your priorities are all messed up!! by malakai · · Score: 1

      I don't think anyone died in NYC. I heard someone got electrocuted in Queens, but haven't heard of any fatalities in the city proper.

      It really was just a flood event. The wind didn't do much to the infrastructure ( other than increase the surge ). I saw a pretty large tree in half out back of the Natural History Museum, but not much else around me in the city.

    12. Re:Your priorities are all messed up!! by CanHasDIY · · Score: 1

      Don;t you all understand!? People _died_ during this storm.

      Yup, a damn shame, especially the avoidable deaths (not sure that there were any, but you know how people can be when they panic).

      Lots of Newyorkers are still without power and water and here your all woried about data centers!? Get a grip nerds!

      We are not paramedics; we are not firemen, or police, or any other sort of emergency personnel. What would you have us do, other than get in the way of the professionals?

      We let the emergency personnel do their jobs by doing our jobs and getting the systems back up and running. Believe me, you wouldn't want a bunch of sysadmins doing triage anymore than you would want a bunch of cops and medics rebuilding a data center.

      --
      An enigma, wrapped in a riddle, shrouded in bacon and cheese
    13. Re:Your priorities are all messed up!! by EmagGeek · · Score: 1

      Manhattaners wouldn't know good coffee if they scalded their crotches with it. I know, I used to be one.

      No need to waste good coffee on them! :p

    14. Re:Your priorities are all messed up!! by Anonymous Coward · · Score: 0

      I don't think anyone died in NYC. I heard someone got electrocuted in Queens, but haven't heard of any fatalities in the city proper.

      It really was just a flood event. The wind didn't do much to the infrastructure ( other than increase the surge ). I saw a pretty large tree in half out back of the Natural History Museum, but not much else around me in the city.

      Since win wasn't Queens one of the city proper 5 boroughs? Or do you only count Manhattan as NYC?

    15. Re:Your priorities are all messed up!! by SolitaryMan · · Score: 1

      You pretty much ruined the joke with your second sentence.

      --
      May Peace Prevail On Earth
    16. Re:Your priorities are all messed up!! by AK+Marc · · Score: 1

      People _died_ during this storm.

      They should have had a backup.

      Lots of Newyorkers are still without power and water and here your all woried about data centers!?

      Bah, we couldn't care less about the New Yorkers or their datacenters. We care about ours, generally built to similar standards. If theirs failed, we need to know why and how so we can avoid the problem for ours (or, more likely, because we know the problem, but the senior management doesn't want to pay for it, ans watching someone else fail in your predicted way turns it from "fiscally responsible" to "negligent"), so now we get to point to these failures for more funding to buy ourselves new stuff.

    17. Re:Your priorities are all messed up!! by couchslug · · Score: 1

      Every able adult can learn about and plan for disasters.

      If their priorities were different, sorry but "tough shit". Don't live in storm surge areas or be rich enough to walk away. Have at least a "bug out bag" of enough food and other supplies to feed you for a few days. Take proper shelter during storms and stay there.

      --
      "This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
    18. Re:Your priorities are all messed up!! by SpzToid · · Score: 1

      obligatory XCID: https://xkcd.com/705

      --
      You can't be ahead of the curve, if you're stuck in a loop.
    19. Re:Your priorities are all messed up!! by SternisheFan · · Score: 2

      I was part of a crew installing and adding more battery backup for a firm in Wall St. Night job, had to disconnect batteries for a couple hours in order to move them to the floor below. Suits all around, doing nothing but wanting the hookup done ASAP, end of the world if power did fail. We had the job mostly complete, just needed another 15 minutes or less to do the final hookup. The brainiac suits on the floor above threw the breakers, they couldn't wait to activate. Meanwhile my co-worker had to grab live 220 volt cables that are shooting flames out of the ends, He looked like a 4th of July fireworks display! We finally got the suits on the phone, told them to turn off the damn breakers, 10 minutes they were back online with backup. My friend could've been killed because some idiot suits couldn't wait like they were told to. No one got hurt, but it easily could've turned into an office building fire. Trust in the pros, safety is paramount!

    20. Re:Your priorities are all messed up!! by EmagGeek · · Score: 1

      Those breakers should have been locked out/tagged out prior to commencement of work. That they were not locked out is a pretty grievous violation.

      Was your co-worker even wearing NFPA 70 PPE?

    21. Re:Your priorities are all messed up!! by SternisheFan · · Score: 1

      Those breakers should have been locked out/tagged out prior to commencement of work. That they were not locked out is a pretty grievous violation.

      Was your co-worker even wearing NFPA 70 PPE?

      Hell no, he was holding that 220 line bare handed, he had to grab it to keep it from burning the equipment. You just don't think about the possibility that people can be so stupid. That's how accidents happen sometimes.

    22. Re:Your priorities are all messed up!! by EmagGeek · · Score: 1

      Well, this accident happened because he did not follow proper lockout/tagout procedures, and didn't wear the prescribed PPE for the work he was doing. I would be amazed if he still had a job after that, because he should have been fired immediately for what happened.

    23. Re:Your priorities are all messed up!! by SternisheFan · · Score: 1

      I forgot to add that the suits who flipped that breaker were the guys who were in charge of making sure it didn't get flipped prematurely. But they were know it all types, masters of the universe, more worried about the (very small) chance of the power going out. I'm sorry, but having a college education does not give people common sense, that's what I've seen many times in life.

    24. Re:Your priorities are all messed up!! by SternisheFan · · Score: 1
      I agree, but shit happens in life. In retrospect, monday morning quarterbacking, yeah, procedures should have been in place. The job did get done, no one got hurt, that's most important. I don't think some of the people posting flip comments here truly understand the dangers involved in keeping our infrastructure up and going, or the risks ordinary people take to ensure it's all working, they just want their internet to work.

      That's why I posted. Some slashdotter's goofing on the bucket brigade method used to keep generators going. Those people were doing what needed to be done, when it had to be done! The individual cells used in backup power weigh upwards of 400 pounds apiece. For a full unit to be complete it uses 24 cells. 8 units used for the backup job in my story, Hydraulic lifting equipment needs to be used for these jobs. It's damn dangerous work involving high-power electric, battery cells containing battery acid, etc. I just wanted to tell this story here (and it's just one story of millions) so that people reading /. might be better informed on the subject, some of the ignorant comments just got to me, I suppose.

    25. Re:Your priorities are all messed up!! by Paul+Carver · · Score: 2

      That's nonsense. Do you even know what lockout/tagout is? How did those "suits" remove your coworker's lock? Did they cut it off or did he not put it on? Every person working on that circuit should have had their own lock and tag on that breaker.

      I assume if they had cut his lock off you would have mentioned it since that's a much more serious offense than simply flipping a breaker that isn't locked/tagged.

      Your coworker definitely should have been fired if he was working on an industrial power circuit without following lockout/tagout procedures.

      BTW, for those not familiar the slash in this case means AND. It is always mandatory to use both a lock and a tag, it's not an either or choice.

    26. Re:Your priorities are all messed up!! by Anonymous Coward · · Score: 0

      Newyorkers is derogatory. It's New Yorkers. Have some respect.

    27. Re:Your priorities are all messed up!! by CaptSlaq · · Score: 1

      Don;t you all understand!? People _died_ during this storm.

      Yup, a damn shame, especially the avoidable deaths (not sure that there were any, but you know how people can be when they panic).

      Lots of Newyorkers are still without power and water and here your all woried about data centers!? Get a grip nerds!

      We are not paramedics; we are not firemen, or police, or any other sort of emergency personnel. What would you have us do, other than get in the way of the professionals? We let the emergency personnel do their jobs by doing our jobs and getting the systems back up and running. Believe me, you wouldn't want a bunch of sysadmins doing triage anymore than you would want a bunch of cops and medics rebuilding a data center.

      Relevant: http://xkcd.com/898/

  6. generators in basements, smart or not? by Nyder · · Score: 2

    Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).

    --
    Be seeing you...
    1. Re:generators in basements, smart or not? by Anonymous Coward · · Score: 0, Insightful

      Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).

      Maybe you'd understand better if you learned where to use a fucking apostrophe.

    2. Re:generators in basements, smart or not? by Anonymous Coward · · Score: 0

      Maybe if you werent (yes that was intentional) such a pedantic jackass you could contribute something useful to the conversation.

    3. Re:generators in basements, smart or not? by Anonymous Coward · · Score: 0

      Generators need lots of fuel. Putting generators higher up means putting the fuel tanks higher up too, and that has a whole host of dangers and problems that are far worse than simply losing power when the basement floods.

    4. Re:generators in basements, smart or not? by war4peace · · Score: 3, Funny

      Volcano's what?

      --
      ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
    5. Re:generators in basements, smart or not? by Anonymous Coward · · Score: 0

      He must mean the volcano's ).

    6. Re:generators in basements, smart or not? by aaarrrgggh · · Score: 1

      If I recall correctly, NYC requires emergency generators to be at roof level so that a mid-rise fire would not cut off power to the upper floors. Most generators in flood prone areas are well protected-- we jumped through some odd little hoops to make generators work in Florida.

      HOWEVER, there is a limit to what you can economically protect against. Usually, you are looking at 100-year events as a basis, not 500 year events or 500 years plus 10%.

    7. Re:generators in basements, smart or not? by Nyder · · Score: 1

      Seems sort of stupid to me to put generators in a basement, considering that your on the coast, surrounded by water, and hurricanes like to come thru every now and then. Maybe this doesn't happen all the time, I don't know. I live on the west coast. I just have to worry about volcano's. (and I don't worry about volcano's).

      Maybe you'd understand better if you learned where to use a fucking apostrophe.

      I did it to bother you.

      --
      Be seeing you...
    8. Re:generators in basements, smart or not? by Miamicanes · · Score: 2

      > Seems sort of stupid to me to put generators in a basement

      "Up north", basements are PRECISELY where you store things like gunky, messy generators (not necessarily with the full blessing of local officials) if you're in a big city like New York, in an old building that was built before elevators were mandatory and people still used coal for heat. They can't go on the roof, because they'd get damaged by the wind and rain. They can't go on the top floor, because it's the expensive penthouse. A newer building might have an integrated parking structure, in which case it would get tucked under a ramp or something... but a smaller building that doesn't occupy most of a block doesn't have that luxury.

      My own dream peripheral for South Florida residents who live in tall buildings: a unit that combines a polyphase propane-fueled generator combined with 5,000 (give or take) BTU mini-split air conditioner. The outdoor part sits on the balcony, along with the 50-100 pound propane tank (both stored indoors during the storm itself). It contains the generator and air conditioner compressor. The indoor part is connected by a nonremovable 25-foot umbilical, and contains the controls, air conditioner blower, inverter/battery charger, power outlets, and connections for the lead-acid batteries. The umbilical contains the refrigerant flex pipes (well-insulated), wires for the polyphase A/C, and power wires for the compressor (it uses generator power to run, but takes advantage of battery power for extra reserve current when starting up).

      It's propane, because you can safely store propane indoors & it doesn't go bad the way gasoline does. The exhaust is also cleaner and safer... an important point to consider in light of its proximity to living spaces. The outdoor part goes on the balcony (skyscraper) or on the ground outside a window, and the umbilical is oval-shaped so you can mostly close the sliding glass door or window and block the remaining gap with foam or a towel. The 25-foot umbilical length allows you to run it through the living room and enjoy A/C in an adjacent bedroom (it's quieter this way, too). It also allows you to put the outdoor unit on the ground, and put the indoor unit on a table just inside a second-story widow on a single-family home or townhouse. The generator runs when the air conditioner does, when there's a significant (> 250 watts) load, and/or the batteries are charging. On a cool night, the generator might shut down occasionally to conserve fuel. Whenever the load exceeds the generator's capacity, or slightly exceeds its 50% load capacity, it draws current from the batteries.

      Imagine having a concrete-bunker warehouse full of them somewhere in western Broward County (near I-75/I-595/Sawgrass) and Orlando, and leasing them in a "fractional ownership" kind of way (with full service maintenance between storms) for $125/day (5-day minimum per year, 3-day minimum per storm, reduced to 1 day if you return it the next day unused so it can be put right back in the warehouse). For an optional nonrefundable $250 per year, you can have one set aside for you and guaranteed to be available, and get a discount of $25/day and have the minimum rental periods waived (since you're basically pre-paying 2 days and gambling on having at least one hurricane during the year). For another optional $100 per storm, you can get it delivered (voluntarily paying an additional $50 or $100 moves you to the front of the delivery queue, or lets you cut to the middle or front of the line if you go pick it up in person ).

      Propane purchase is optional, but not included. This is a MAJOR potential profit, because bulk LPG is dirt cheap, and there are VERY FEW places in the urban parts of South Florida where you can fill your own tank at non-ripoff prices. The rest is all Blue Rhino LPG being sold in 2/3 full tanks at *criminal* prices, like $25 for a 20-pound tank with ~16 pounds of actual gas and proprietary valves that prevent you from ever refilling any tank you swap at a cheaper place going forward

    9. Re:generators in basements, smart or not? by bobcat7677 · · Score: 3, Informative

      They don't have a choice. Fire codes require the fuel tanks to be underground. It doesn't have to make sense, it's the law.

    10. Re:generators in basements, smart or not? by mcgrew · · Score: 1

      Well, one of two things:
      1. Volcano's hot ash and lava, or
      2. He's a semi-literate who hasn't graduated junior high yet.

    11. Re:generators in basements, smart or not? by Anonymous Coward · · Score: 0

      Why would you use a generator to power the air conditioner when you have propane already? Most RVs and camper trailers have this sort of refrigerator.

      http://en.wikipedia.org/wiki/Absorption_refrigerator

      Also:

      http://www.altfuelprices.com/index.php?location=florida&lat=27.6648274&lng=-81.5157535

      If you can fill up a car there, you can certainly get cylinders refilled. :) That being said, I've found a LOT of inaccuracies in the US with the altfuelprices map. Seems a lot of idiots running propane stations don't know you can fill a car with propane (or that you can even run one on it) and so list themselves on that site when they don't refill cars, because they think it's for people filling BBQ tanks. It isn't. Drove me nuts on my last trip because every single place I called wouldn't fill a car, so I ended up knocking half of the NY and MA stations off the map and ended up filling up at U-Haul (Yes, U-Haul is smarter than Suburban Propane and other "real" propane suppliers. Very weird...)

    12. Re:generators in basements, smart or not? by AK+Marc · · Score: 1

      Where should they put them? For weight reasons, many must be on the lower floors. For fire reasons, the fuel must be on the lower floors. The building/fire codes don't care about uptime.

      But yes, at Fukishima where they could have made them their own enclosures, not putting them 20 feet in the air on stilts was stupid. There's no reason they couldn't have gotten a cheap water tower and filled it with fuel. More than they need, and tsunami-proof. Though it all would have toppled over in the earthquake.

    13. Re:generators in basements, smart or not? by war4peace · · Score: 1

      Not to mention the amazing "your on the coast" thingie he spewed.

      --
      ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
    14. Re:generators in basements, smart or not? by war4peace · · Score: 1

      And considering the two mistakes combined, I'd pick option #2.

      --
      ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
    15. Re:generators in basements, smart or not? by AK+Marc · · Score: 1

      The outdoor part sits on the balcony, along with the 50-100 pound propane tank (both stored indoors during the storm itself).

      Stored indoors? That implies moving them. Wouldn't it make more sense to have them perminent, then you could have more propane if you wanted, and have the balcony with a pull-down metal garage door, like some more secure storefronts have? In a storm, enclose the balcony, rather than relocating the equipment.

    16. Re:generators in basements, smart or not? by Anonymous Coward · · Score: 0

      I presume that I can't interest you in Volcano Insurance then?

    17. Re:generators in basements, smart or not? by Anonymous Coward · · Score: 0

      > sort of stupid to me to put generators in a basement

      "Up north", basements are PRECISELY where you store things like gunky, messy generators (not necessarily with the full blessing of local officials) if you're in a big city like New York, in an old building that was built before elevators were mandatory and people still used coal for heat. They can't go on the roof, because they'd get damaged by the wind and rain. They can't go on the top floor, because it's the expensive penthouse. A newer building might have an integrated parking structure, in which case it would get tucked under a ramp or something... but a smaller building that doesn't occupy most of a block doesn't have that luxury.

      For residential maybe, but data centres are dedicated buildings. They have no penthouse to worry about and the generators are their own installed specifically for disaster recovery.

  7. Poor Planning? by cruff · · Score: 1

    Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?

    1. Re:Poor Planning? by nitehawk214 · · Score: 1

      Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?

      Yeah, when you can put them on the roof... where... there is rain and wind...

      I suppose there is no winning here.

      --
      I'm a good cook. I'm a fantastic eater. - Steven Brust
    2. Re:Poor Planning? by Zeromous · · Score: 1

      I'll field this one, slashdot.

      Managers. Lots of them.

      --
      ---Up Up Down Down Left Right Left Right B A START
    3. Re:Poor Planning? by Anonymous Coward · · Score: 0

      http://www.despair.com/med24x30prin.html

    4. Re:Poor Planning? by Anonymous Coward · · Score: 5, Insightful

      Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either? Lightning...wind...spills...leaks...fires... all probably more statistically relevant than major flooding, and the consequences of failure far more disastrous than simply losing power in a flood. Even storing the tanks underground and the generators above ground has 2 problems: 1) you need power to pump the fuel up to the generator, which kind of defeats the purpose, and 2) high pressure fuel lines running through a building isn't exactly safe or desirable either.

    5. Re:Poor Planning? by berashith · · Score: 1

      I am hoping that the "hasn't happened since the 1880's" bit is partly to blame, but Japan's little issue last year definitely comes to mind. I thought the same thing. I also question hauling tanks of diesel up the fire escape. That should have been better thought out. I guess the firewalls work both ways as long as you dont plan on leaving.

    6. Re:Poor Planning? by Anonymous Coward · · Score: 1

      Managers + Politicians = plenty of hot air to dry out those basements!

    7. Re:Poor Planning? by sunking2 · · Score: 2

      Because below ground is convenient, the least expensive cost per square foot of floor space, and it's where all your crap you want to keep out of the way goes.

    8. Re:Poor Planning? by Zeromous · · Score: 1

      And to bungle risk assessments.

      Seems to me Manhattan has been a true flood risk, forever.

      Now, what happens when we balance that assessment with other risks, such as fire risk of storing a generator on the 2nd floor. Say the risk survives to the next phase, where Fire Risk A and Flood Risk A are both on the table.

      Then you get a room full of managers or politicians that say: We have X dollars and Risk A costs Y to mitigate and risk B costs Y+1 to mitigate. X is already Y. And that's how you have your generators underground and your datacenter wherever it will fit.

      --
      ---Up Up Down Down Left Right Left Right B A START
    9. Re:Poor Planning? by Baloroth · · Score: 4, Insightful

      Why would anyone in their right mind place generators and tanks below ground where flooding would be an issue?

      Lets see how you feel with a few thousand gallons of highly flammable liquid suspended above your head, in a building with lots of electricity running through it, where an earthquake is more likely than flooding in the basement. And that is ignoring the possibility of deliberate sabotage. A building with fuel stored above ground level where something went wrong would turn rather quickly into a giant pillar of flame. If one of the tanks gets ruptured, all it takes is a single spark to kill hundreds or thousands.

      Below ground, however, fire-fighters can deal with it relatively easily, and the flames won't descend to engulf the entire building in a matter of a few minutes.

      --
      "None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton
    10. Re:Poor Planning? by Anonymous Coward · · Score: 0

      Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either?

      Can you imagine the headline? "Thunderstorm punctures diesel storage tank on hospital roof. Lightning ignites 2,000 gallons spilling down side of hospital"

    11. Re:Poor Planning? by firewrought · · Score: 1

      Because having large tanks of diesel fuel dozens of stories above ground isn't a good solution either?

      Can you imagine the headline? "Thunderstorm punctures diesel storage tank on hospital roof. Lightning ignites 2,000 gallons spilling down side of hospital"

      And, to top of it off, the very same parade of internet know-it-alls would be asking "Why would anyone in their right mind place generators and tanks above ground where lightning would be an issue?"

      --
      -1, Too Many Layers Of Abstraction
    12. Re:Poor Planning? by Anonymous Coward · · Score: 2, Interesting

      Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.

    13. Re:Poor Planning? by Buelldozer · · Score: 1

      Is an earthquake a more likely event than a flood in Manhattan?

    14. Re:Poor Planning? by Zeromous · · Score: 1

      Nope, once again +5 Insightful is meaningless.

      Generators should not be below ground, and at least 2-12 ft above ground if possible (or higher depending on risk).

      They should not be too high, lest you have trouble fueling them.,

      As with anything this likely came down to a retrofit risk assessment. The risk of fire or not making code is far higher than floods in manhattan, but being an island and subject to the ocean, still a high risk and should have been planned for. To me, it comes down to money at the end of the day. Cheaper is often better sadly.

      --
      ---Up Up Down Down Left Right Left Right B A START
    15. Re:Poor Planning? by barc0001 · · Score: 1

      Because they're required to by building code? That's what happened at the hospital that lost power. Tanks underground, water covers them, fuel gets contaminated, and poof, no more power.

    16. Re:Poor Planning? by Anonymous Coward · · Score: 0

      People who like to comply with the fire code.

    17. Re:Poor Planning? by bobcat7677 · · Score: 1

      The tanks are required to be below ground by the fire code. I think the better question is why put big data centers on a low lying coastline/island and/or a city with a giant target painted on it by every anti-american, anti-establishment, anti-whatever whacko the world over. Data centers belong in places with low risk of natural disaster, war, terrorist attack, riot or really anything that brings the police out.

    18. Re:Poor Planning? by hawguy · · Score: 2

      Since when has Diesel been highly flammable? You can actually use it to put fires out. It takes quite some heat to get it going so really poses less risk than you average stationary cupboard.

      Until it leaks out, saturates wood framing and other building materials, then comes into contact with an ignition source (like a pilot light, or candles that the residents are using since the power is out).

      Then this hard to ignite fuel quickly turns an office into an inferno.

      I can knock a candle over in my stationary cupboard and as long as I pick it up quickly, I wouldn't expect a fire. Soak that same cabinet and knock a candle over and it's a different story. Kerosene (very similar to diesel) makes a good fire starter to help get a wood fire started.

    19. Re:Poor Planning? by hawguy · · Score: 1

      The tanks are required to be below ground by the fire code. I think the better question is why put big data centers on a low lying coastline/island and/or a city with a giant target painted on it by every anti-american, anti-establishment, anti-whatever whacko the world over. Data centers belong in places with low risk of natural disaster, war, terrorist attack, riot or really anything that brings the police out.

      Because some companies want to be physically close to their datacenter. Virtualization and cloud computing is changing this... slowly.

    20. Re:Poor Planning? by AK+Marc · · Score: 1

      Because the fire codes wouldn't allow the fuel to be stored off the ground anyway. Multiple failures in this storm were where the generator was fine, but the fuel was fouled.

    21. Re:Poor Planning? by Kalriath · · Score: 1

      No, it's because below ground is the only legal place to put it, and the only safe place where the fuel source igniting in a disaster won't blow up the upper floors and fling shrapnel for 100 feet in every direction.

      --
      For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
    22. Re:Poor Planning? by Kalriath · · Score: 1

      Wall Street. It's a big deal for them to be literally right next to their data centres, as to them it is critically important that their trade order makes it over the wire in 0.001 seconds, rather than 0.006 seconds.

      --
      For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
    23. Re:Poor Planning? by tftp · · Score: 1

      it is critically important that their trade order makes it over the wire in 0.001 seconds, rather than 0.006 seconds

      If a lot worth of $100M falls down to $50M within a minute, each additional millisecond costs you about a thousand dollars. As more and more trading is done directly by computers that are loosely guided by traders, the prices can swing up or down as fast as those computers can generate trades. The firm with the fastest computer will have an advantage. If several computers are participating in that $100M to $50M event and the price drops not in one minute but in 1 second then each millisecond will cost you $50K. A firm with a reasonably fast computer can just sit on the stock exchange all day and collect money from every move in prices before that move is even noticed by humans. Buy 1M shares for $100.0 each and sell them five seconds later for $100.001 ... you just earned $1,000 in five seconds by sitting next to your computer and doing nothing. Since computer can do this to all stocks that exhibit interesting trends, you can make money hand over fist. And they do.

    24. Re:Poor Planning? by Anonymous Coward · · Score: 0

      Sorry dear, but as far as the rest of the world is concerned, you'd have to be wacko NOT to be so-called "anti-american".

    25. Re:Poor Planning? by Anonymous Coward · · Score: 0

      By "terrorist attack" do you mean: "attack by the USA or Israel"?

    26. Re:Poor Planning? by CAIMLAS · · Score: 1

      * The EPA requires all above ground storage containers to have a secondary containment tank; my recollection is that it has to be 2/3 the total capacity of the actual tanks, though it may be 1.5 to account for eg. rainwater accumulation. This is probably why they don't put tanks with gensets on the roof.
      * A good quality generator measures efficiency in hours per generator (depending on kwh rating and load, of course). We're not talking about a high pressure flow rate, we're talking about the passive volumetric pressure from the bottom of the tank being able to feed a motor many stories above without issue (assuming its a sizable tank).
      * It's much easier to clean up spilled fuel at ground level (or above it, as the case may be ) than it is to clean up spilled fuel from an underground spill (ie, you really can't, especially when there's high "groundwater" levels as the case is here).

      Try again. This is just a case of stupid gov't regulations contradicting other stupid gov't regulations, with one taking local precedence due to the local Fire Marshall enforcing more rigorously than the EPA. :)

      --
      ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
  8. Re:Your priorities are NOT all messed up!! by miltonw · · Score: 4, Interesting

    You do your job. Many of these data centers are part of and important to communication, rescue operations, information. When you work there, you might not know how important that particular data center may be but -- you do your job.

    Whatever part of the city you can keep operating is good.

    Don't criticize what you know nothing about.

  9. NYC should sue the Koch brothers for damages by WOOFYGOOFY · · Score: 0

    Denier == terrorist.

  10. Why are generators in the basement? by sl4shd0rk · · Score: 3, Insightful

    Why not move them to the roof? And while we're at it, do the same for all the nuke plants? A simple f*cking appliance that needs air and fuel to run and somehow they manage to spend life at the bottom of a potential indoor swimming pool.

    --
    Join the Slashcott! Feb 10 thru Feb 17!
    1. Re:Why are generators in the basement? by Anonymous Coward · · Score: 0

      Because the very real risk of fire and death from some jackass deciding that the roof is a good place for fuel far outweighs the inconvenience of your precious datacenter being offline for a few days.

    2. Re:Why are generators in the basement? by Darth_brooks · · Score: 4, Insightful

      Depending on the generator size, there's the weight issue. There's also fire regulations. 500 gallons of Diesel on the roof + fire == cascading burning diesel fuel.

      --
      There are some people that if they don't know, you can't tell 'em.
    3. Re:Why are generators in the basement? by Anonymous Coward · · Score: 0

      NYU had generators on the roof. They didn't have any fuel on the roof.

    4. Re:Why are generators in the basement? by Solandri · · Score: 1

      The generators are on the roof or in the middle of the building close to the data center.

      The fuel is stored underground due to fire safety regulations. The problem is fuel pumps don't work very well when submerged underwater.

    5. Re:Why are generators in the basement? by RogerWilco · · Score: 1

      I think in places like NYC it has to do with real estate prices. Everything above ground is probably expensive as humans can live/work there. It's the underground places where no sunlight penetrates that gets the machines and utilities as nobody cares if they have any windows. Also these machines are usually heavy so it's easy to put them into basements as no lifting is required and means the structure of the building can be lighter.

      It's not a good idea in case of disasters, but it makes a lot of economic sense in the short term for a lot of reasons. The problem is that especially the USA isn't very good at having their infrastructure and buildings robust against disasters. To an outsider it's obvious that the choice in the US seems to be that it's cheaper to rebuild than to make things hurricane/flooding/tornado/earthquake-proof. The best example is Katrina. Areas in the Netherlands have separate organisations for managing the water. It's not part of government and has it's own taxes and separate elections. A lot of these go back to medieval times. They care for the waterways, dikes, levies, shelters, disaster relief and similar structures. It's their sole responsibility and absolute priority. So there is never a trade-off against healthcare, military, education, transportation or any other government function. It's like if you would make FEMA and the corps of engineers into completely independent organisations with their own taxes and elections.

      --
      RogerWilco the Adventurous Janitor
    6. Re:Why are generators in the basement? by Phillip+Birmingham · · Score: 1

      "Cascading burning diesel fuel..." Stop it, you're makin' me hot.

      --
      Make me aerodynamic in the evening air
  11. Bucket brigade??? by kevin_j_morse · · Score: 1

    Why on earth would you put generators on the 18th floor if you don't have a fuel line running up to them... That seems like quite a basic oversight???

    1. Re:Bucket brigade??? by sys_mast · · Score: 1

      Just a guess, but there is likely a fuel line...but the input is currently under water. Other comments have said that the fuel tanks must be at/below ground per city regulations.

      If you say put the redundant input higher, how high? the 17th floor?

      Slightly above water level is the correct location, but that might not be easy to figure out until flooding occurs.

      --
      Those who can, do.
    2. Re:Bucket brigade??? by CastrTroy · · Score: 1

      There is fuel running up to them. The problem is that the pumps to the fuel are in the basement, which is flooded. So the pumps aren't working.

      --

      Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
    3. Re:Bucket brigade??? by CityZen · · Score: 1

      If we can figure out how to put a pump into a well full of water deep underground, it should not be too difficult to figure out how to do something similar with a fuel pump, relatively speaking. To have not done so is a fail.

    4. Re:Bucket brigade??? by hawguy · · Score: 1

      If we can figure out how to put a pump into a well full of water deep underground, it should not be too difficult to figure out how to do something similar with a fuel pump, relatively speaking. To have not done so is a fail.

      There's a lot that *could* be done, but usually the price tag precludes *everything* from being done. If the electric panel that feeds your underwater pump is underwater itself, then the pump isn't going to do any good.

    5. Re:Bucket brigade??? by CityZen · · Score: 1

      If the whole purpose of the backup system is to provide electrical power, then you'd think they'd consider the power
      sources needed for it.

      We generally assume that the purpose of a backup system is to actually operate when the main system fails.
      Providing assurance that it will is an engineering job that requires enumerating the possible failure modes and
      providing contingencies for them. The customer should presumably have that assessment done, then decide which
      of the contingencies that they'll pay for (hopefully based on some cost/benefit/likelihood balancing strategy).

      In this case, either they didn't do the right assessment, or else they didn't think this mode of failure was likely.
      Either way, you hope they learn something.

    6. Re:Bucket brigade??? by hawguy · · Score: 1

      If the whole purpose of the backup system is to provide electrical power, then you'd think they'd consider the power
      sources needed for it.

      We generally assume that the purpose of a backup system is to actually operate when the main system fails.
      Providing assurance that it will is an engineering job that requires enumerating the possible failure modes and
      providing contingencies for them. The customer should presumably have that assessment done, then decide which
      of the contingencies that they'll pay for (hopefully based on some cost/benefit/likelihood balancing strategy).

      In this case, either they didn't do the right assessment, or else they didn't think this mode of failure was likely.
      Either way, you hope they learn something.

      It's not that they didn't account for having power to the fuel pump, they didn't account for having the basement, the fuel tank, fuel pump and associated electrical panels to be submerged in water.

      After a certain point, you stop designing redundancy and resilience into your current facility and use the money to build a completely separate, geographically distant facility. When building in an existing office building, it may not be possible to design in basic safeguards that you'd have in a purpose built datacenter. In a suburban datacenter, the backup generators and fuel tanks will be outside on skids, raised several feet above ground level - in a city office building, you rarely have that luxury and local building and fire codes further restrict what you can do.

      The biggest takeaway is to not count on a single datacenter since there's always some disaster that can take it down.

  12. Interdictor blog by mistapotta · · Score: 1

    I'm reminded of the Intedictor LiveJournal account during Hurricane Katrina. You do whatever you have to do to get the job done. http://en.wikipedia.org/wiki/Interdictor_(blog)

  13. DCP by ThatsNotPudding · · Score: 2

    According to a blog post from Fog Creek Software, one of the clients at the building, about 30 customers are lifting buckets (or cans) of diesel fuel up 18 flights of stairs.

    Ahh: Diesel Control Protocol

  14. Continent-wide VOIP provider by Lwood_at_COG · · Score: 1

    I'm nowhere near NYC, but my voip provider was taken down by these floods and was very slow to failover to their other, lower-capacity sites. Still not working right. Complaints were especially vocal from people on the west coast, who were quite surprised to have their business lines knocked out.

    --
    "Beware of all enterprises that require new clothes." --Henry David Thoreau
    1. Re:Continent-wide VOIP provider by silas_moeckel · · Score: 1

      Think that falls under did you actually get a business class SLA that's backed up by a deep pocket insurance company? Nope you got the cheap guy that works fine 99% of the time. I work primarily with the hosting industry and a decent subset of that is voip hosting the vast majority of these guys talk a good redundancy plan but fail to test it and/or execute it. Being that a huge chunk of the industry's costs are gear not people having to have n+1 gear and the tech to do site to site redundancy makes huge differences in bottom line.

      --
      No sir I dont like it.
  15. Waterproof... by David_Hart · · Score: 1

    Gas stations bury their gas tanks underground with few problems of water seepage. Granted, they aren't surrounded by water and there is little water pressure (i.e. covered by water). However, you would think that they would have at least waterproofed the fuel tanks.

    1. Re:Waterproof... by h4rr4r · · Score: 1

      Even if you did, if the top of the tank is under water, how do you refuel it?

      You have X dollars and Y risks, you can never be prepared for everything.

    2. Re:Waterproof... by Remus+Shepherd · · Score: 3, Interesting

      The problem is that 'waterproofing' is a short-term guarantee. Water is insidious, it dissolves almost anything (although some things like metals very slowly), and it will eventually creep inside of any 'waterproof' container. That's why there's such a problem designing radioactive dumps like Yucca Mountain -- water would eventually eat its way into the vitrified radioactive cask.

      Gas station underground tanks can survive for 10-20 years and still be waterproof. But most of the infrastructure in NYC is a hundred years old. There isn't anything waterproof in that city. Even brand new structures are probably permeable to water, if the designers just never thought it would be an issue.

      --
      Genocide Man -- Life is funny. Death is funnier. Mass murder can be hilarious.
    3. Re:Waterproof... by aaarrrgggh · · Score: 4, Informative

      The bigger problem is usually the pumps. You generally try to use turbine-type submersible pumps with the motor above the tank and the inlet down low to avoid problems with priming the pump. If the place where the pump motor floods, you are pretty much SOL.

      If you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.

      Ultimately, you have backups on backups in most data centers (and hospitals), but you often have a limited window to respond. We have an (illegal) 15-gallon gas can in one facility up by the generator. That can will give them about 9 minutes extra run-time if the day tank runs dry. There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?

      Big enough problems need disaster recovery plans; you will go down, the issue is how quickly you can return to normal operations.

    4. Re:Waterproof... by Anonymous Coward · · Score: 0

      There is a hand pump in the basement that can be used to manually pump the fuel up 50' to the generators, but if the room it is in is flooded what can you do?

      Hire ninja scuba divers. Even if it is flooded, fuel will get to the tanks, and you wouldn't even know it was them.

    5. Re:Waterproof... by Ironhandx · · Score: 1

      That ones fairly easily dealt with after the fact. In fact many modern refueling trucks come equipped with air-tight locking mechanisms on the refueling lines. Diesel generators are inherently pretty fucking tough so if a gallon or two of water gets into a sealed tank while they're getting the refueling line hooked up, who cares? A few gallons won't make it out of the fuel tank all at once anyways and with the compression ratio on diesel turbines and engines it'll probably just cause some sputtering and vanish into steam.

      On the second go-around you could even set up a self-sealed electronic gate inside the tank that you have to push a button to release. Presto, no more than 1 litre of water to X hundred or thousand gallons of diesel. If you leave the truck hooked up you can even refill the existing connected truck from another truck and never break the seal till the facilities tanks are full.

      What we're talking about here is at most $1000/tank solution.

    6. Re:Waterproof... by Medievalist · · Score: 1

      Gas stations bury their gas tanks underground with few problems of water seepage.

      Are you sure? Because when I managed a gas station 20 years ago, I had to check the tanks daily for water, and generally had to pump the water out at least twice a month before seepage and condensation built up enough to reach the uptake pipe. Using a hand pump.

      And that's with gasoline, which floats on water - diesel absorbs water so you can't just pump it off the bottom of the tank. Look up "hygroscopic" and "diesel fuel dryer"...

      Granted, they aren't surrounded by water and there is little water pressure (i.e. covered by water). However, you would think that they would have at least waterproofed the fuel tanks.

      This is totally doable, and well within the capabilities of a New York hospital or large data center, except it would cost so much the doctors and CEOs would have to stop lighting their cigars with hundred dollar bills, and we can't let that happen in Obamney's America! It would be wrong to let emergency preparedness rob the hardworking rich people of their bailout money, wrong I tell you.

      All kidding aside, the return on investment is not going to be very attractive. Evacuation is probably cheaper.

    7. Re:Waterproof... by Anonymous Coward · · Score: 0

      The problem is that 'waterproofing' is a short-term guarantee. Water is insidious, it dissolves almost anything (although some things like metals very slowly), and it will eventually creep inside of any 'waterproof' container.

      The all destructive power of water... Where do you come up with such asinine opinions? Plenty of things will waterproof an object, area or structure for hundreds, thousands or even millions of years.

    8. Re:Waterproof... by Anonymous Coward · · Score: 0

      TIf you place a suction pump 25' above the bottom of the tank to avoid flooding risks, you have the problem of priming the thing and maintaining suction. You could do a submersible pump with a really long shaft so the motor is high enough... but that would look really stupid.

      It would look moronically stupid. Until your facility is the only one around with power. Then it would look fucking brilliant.

    9. Re:Waterproof... by CityZen · · Score: 1

      The (Bosch) fuel pump on my old Saab was inside the gas tank. Gas ran through the motor to keep it cool. I suppose that since gas isn't conductive, a completely submersible fuel pump is even easier to make than a submersible water pump (as used in deep wells).

    10. Re:Waterproof... by Anonymous Coward · · Score: 0

      The problem is that 'waterproofing' is a short-term guarantee. Water is insidious, it dissolves almost anything (although some things like metals very slowly), and it will eventually creep inside of any 'waterproof' container.

      This is Just Plain Wrong. Submarines have no issue with staying at sea for over a year. Some of them even have diesel engines and batteries, just like a data center.

      That's why there's such a problem designing radioactive dumps like Yucca Mountain -- water would eventually eat its way into the vitrified radioactive cask.

      Are you seriously equating a maintenance-free container that needs to last 10,000 years with a 24/7 staffed building designed to last less than 200?

    11. Re:Waterproof... by CAIMLAS · · Score: 1

      10-20 years?

      I see where you got your numbers from, but my experience says otherwise. Most common non-residential underground tanks will last 30+ years without a problem (though they need to be regularly inspected and maintained, of course). Unlike in a residential heating oil setting, underground tanks (incl. fill stations but mainly gas stations) get regularly (think: weekly) inspected for water. This is also true for smaller operations: in diesel specifically, water content will encourage bacteria growth and make the fuel all but useless in a couple of weeks, destroying tens of thousands of reserve energy.

      Not sure where you get the "infrastructure is 100 years old" idea. This isn't a movie with Italian gangsters lurking in the shadows, smoking cigars. Even though there's certainly infrastructure that old within the city, we're not talking about gas or electric mains. If we're talking about government in NYC, they regularly dispose of and replace perfectly good materials because they're simply old (we got close to 10k worth of terracotta roofing tile for a couple hundred dollars at auction). If we're talking about business, they're the richest businesses in the world and there is constant change going on with the buildings which hold their facilities. Above ground or below, it doesn't matter - they've got several of the highest quality generators with new tanks available to them.

      --
      ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
  16. Status of datacenters affected by Samuraid · · Score: 5, Informative
    There's a detailed list of downed datacenters as well as a good discussion of status over at webhostingtalk: http://www.webhostingtalk.com/showthread.php?t=1205042
    Per the topic, the following locations are experiencing or have experienced outages:
    • 75 Broad Street
    • 33 Whitehall
    • nLayer at 882 3rd Ave
    • Voxel/Internap at 111 8th Avenue
    • XO, nLayer, Cogent, Verizon, Sidera Networks and AT&T at 882 3rd Ave
    • 121 Varick
    --
    if ($question !~ m/bb|[^b]{2}/i) { die(); }
    1. Re:Status of datacenters affected by ender- · · Score: 1

      I can only speak to 111 8th Ave. The affects in that building appear to be company specific. There's no power in the building, but our datacenter [NTT/Verio] there has successfully been running on generator since 8am Monday.
      I guess we got lucky on tank/pump placement? Either that or our frequent generator testing has been a huge success. :)
      We've already had one fuel delivery and are expecting another today.

      Our other datacenters in the area [NY, NJ, CT] have also managed without outages. Many of them already have commercial power back on again.

      NOTICE: The above information is unofficial and may be wildly inaccurate as I am located in another region.

    2. Re:Status of datacenters affected by bastion_xx · · Score: 1

      111 8th apparently has common fuel from the landlord (no individual tanks). This morning a valve failure starved out some gen sets.

      It is still wild and wooly in downtown....

  17. Geez by Anonymous Coward · · Score: 1, Funny

    This would never happen at an Amish data center.

  18. Peer 1 by Anonymous Coward · · Score: 0

    What a waist of potential energy. Just push the generator out the window.

    1. Re:Peer 1 by Anonymous Coward · · Score: 0

      I have a waist of massive potential energy.

  19. Manhattan unsuitable for data centers? by GeoGreg · · Score: 4, Insightful

    Like many other posters, my first question was why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious: fire codes. Duh. Thinking of how fuel is stored elsewhere, the only other option I can think of would be storing the fuel outside the building but above potential floodwaters. Not in a place like Manhattan. The price of real estate is much too high for tank farms on stilts. And the earthquake risk in New York is non-zero, so that solution might have the same problems as the current solution. So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure. Is relocating major data centers out of flood-prone areas of Manhattan (and other similarly risky areas) feasible? The potential of a major flood event in Manhattan has been well-known for a long time. Much of lower Manhattan is built on landfill. Did the builders of these data centers include basement flooding + extended power outage in their risk forecasts and just decide to deal with it if it happened?

    1. Re:Manhattan unsuitable for data centers? by thesandbender · · Score: 2

      The data centers are located downtown because that's where the banks and exchanges are. The banks and the exchanges originally built their data center close to them (this started in the 70's). Customers wanted to be as close to the servers as possible (and still do - high frequency trading) and it just kind of organically grew into what it is now. It also didn't hurt that AT&T and Verizon both have massive switch stations downtown and when these things were being built out high speed connections were not as easy to get as they are now.

    2. Re:Manhattan unsuitable for data centers? by bastion_xx · · Score: 1

      They are downtown because all the fiber terminates there (111 8th, 60 Hudson, 75 Broad). They originate from the teletype days. Closest point to the main meeting places is a good place to build data centers. Hence Ashburn, London, Frankfurt, etc.

    3. Re:Manhattan unsuitable for data centers? by tibit · · Score: 1

      So maybe the answer is that flood-prone urban areas are just not a good place for critical data infrastructure.

      That's aggrandizing the problem, really. The solutions are quite simple.The fuel pumps and other fuel delivery and refill systems for the generators should be in waterproof enclosures and rated to operate while the basement is submerged. Alternatively, there should be provisions for feeding fuel from a tanker parked out on the street to a riser going to the upper floor generator(s). That's all there's to it. Why nobody thought of it, I wouldn't know.

      --
      A successful API design takes a mixture of software design and pedagogy.
    4. Re:Manhattan unsuitable for data centers? by Guppy06 · · Score: 1

      why were the generators on upper floors but fuel (and pumps) in the basement? And as soon as I read the answer, it was completely obvious:

      Because diesel fuel weighs around 55 lb/ft^3?

    5. Re:Manhattan unsuitable for data centers? by GeoGreg · · Score: 1

      I wonder if waterproofing everything is feasible? Containing and excluding water are both hard. I do like the tanker idea. That seems pretty straightforward.

    6. Re:Manhattan unsuitable for data centers? by GeoGreg · · Score: 1

      Good point. Floor strength/floor space are not trivial issues.

    7. Re:Manhattan unsuitable for data centers? by Nivag064 · · Score: 1

      Earthquake risk is _NEVER_ non-zero!

      In Christchurch in New Zealand, a fault line moved that had not been known about.

    8. Re:Manhattan unsuitable for data centers? by Kalriath · · Score: 1

      You do realise you just said that Earthquake Risk is ALWAYS zero right?

      This is why double negatives are bad.

      --
      For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
    9. Re:Manhattan unsuitable for data centers? by Nivag064 · · Score: 1

      (-: :-)

    10. Re:Manhattan unsuitable for data centers? by Kevin+Stevens · · Score: 1

      About the history and feasability of moving data centers out of Manhattan:

      I am not an expert on this, but Wall St has been a datacenter hub because it has always been an information hub. There has always been a huge telecom demand down there, because there has always been a huge information demand down there. From telegrams, ticker tapes, masses of traders on phones, to bloomberg terminals, to exchanges sending quotes out, there has always been a big demand for data lines down there- I am using data lines generically, because we are talking technology spanning back to the 1900s. So when the first data centers were being built, lower Manhattan was a natural location. It actually made financial sense for a time, as running lines used to be really expensive, and there were major telco facilities based in lower Manhattan.

      Of course things have changed, connectivity has gotten way cheaper, to the point where proximity is no longer very important, especially from an economic standpoint. On top of that, 9/11, and to a greater extent the blackout of 2003, made companies realize that having geographical diversity is really important.

      So yeah, not only is it feasible to move datacenters out of Manhattan, it is happening, and preferred. Its much cheaper and easier to build out in suburbs and exurbs. However, partly because these facilities already exist in Manhattan, and partly because a good Business Continuity Plan also has data centers in close proximity to your main offices, there are still data centers downtown. My firm is running at reduced capacity for some specific processes due to a data center under water, and at least as of yesterday was running some on generator power, but we are fully functioning. I am sure there is an infra guy right now who is regretting on a micro level the usage of this particular facility due to its placement and vulnerability (though I know this building houses machines from many firms, and I am not sure how easy it is to find data center space in Manhattan), but having a facility in Manhattan is part of the BCP.

      High Frequency Trading doesn't really play a part in this btw. The stuff running in our data centers in Manhattan is mundane stuff. HFT requires you to be close to the exchange's matching engines, which in the case of NYSE are actually in mahwah NJ, near the NY border.

    11. Re:Manhattan unsuitable for data centers? by Anonymous Coward · · Score: 0

      telegrams, ticker tapes...we are talking technology spanning back to the 1900s.

      I assume you meant "back to the 1800's"? Or perhaps "back to the 19th century"?

    12. Re:Manhattan unsuitable for data centers? by CAIMLAS · · Score: 1

      Absolutely.

      It confounds me why people would put a high-area-requirement industrial facility in one of the most expensive locations in the world (same goes for San Francisco). Do you not see the value of moving it off the 3rd floor of an elaborate building in the Financial district? I know of many companies that do this and just have to 'wtf' when they obviously have the $$$ to locate it somewhere more sensible.

      Sure, cost per server difference might be relatively negligible, but there are costs there which you wouldn't have elsewhere - travel, accessibility, facility cost and accessibility cost can all collude against you.

      Hell, even the Facebook place up in the arctic is, in some ways, more accessible than somewhere like this: you're not going to have flooding, rioting, earthquakes, rioting, etc. from preventing your engineers/technicians from getting in. They'll already be there, and you've got enough actual land to stow 50k gallons of reserve fuel. In the grander scheme of things there are many other options that make more sense for a datacenter; the small-minded, culturally-centric types who run these things just aren't really able to see the obvious. :)

      --
      ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
  20. Linode NJ datacenter use power gen by e065c8515d206cb0e190 · · Score: 1

    source: http://twitter.com/linode

    Let's hope power is restored soon...

  21. Floating server farms by concealment · · Score: 4, Insightful

    It looks like Google was ahead of the curve after all with their idea for floating server farms.

  22. Re:NYC should sue the Koch brothers for damages by firewrought · · Score: 1

    Denier == terrorist.

    And um, yeah, potato==squirrel. Just because the GOP has been taken over by ideologically-crazed individuals is no excuse for the rest of to start doing it too.

    --
    -1, Too Many Layers Of Abstraction
  23. Re:Your priorities are NOT all messed up!! by bws111 · · Score: 1

    If any of these data centers are really as important as you claim, they should have been switched to their alternate sites (far from NYC) days ago. If these sites are not important enough to have real disaster recovery plans including alternate sites then they simply are not critical. Just being a datacenter does not make them any more important than the thousands of other businesses (stores, restaurants, you name it) that are also 'offline' because they are flooded.

  24. "interesting" bucket brigade by 3nails4aFalseProphet · · Score: 2

    In*ter*es*ting - adj.
    1. capable of holding one's attention.
    2. arousing a feeling of interest.
    3. oh God, oh God, we're all going to die.

    --
    /*Insert boring sig here*/
  25. My favorite generator by Beryllium+Sphere(tm) · · Score: 3, Informative

    There are commercial generators which run on a variable mix of natural gas and diesel. With such a set, you can greatly extend your runtime by reducing the diesel percentage to a minimum when natural gas is available. Then if or when the natural gas goes out you can run them on 100% diesel and you're no worse off.

  26. Re:Your priorities are NOT all messed up!! by grommit · · Score: 4, Insightful

    Well, here in the real world, we have DR plans based upon the suggestions of the trade journals and then we have the actual implementation of the DR plan based upon the budget cuts of the past X years. Guess which one includes multiple redundant datacenters in locations of a significant distance away from each other and which one doesn't include those datacenters.

  27. Re:Your priorities are NOT all messed up!! by Anonymous Coward · · Score: 0

    (stores, restaurants, you name it)

    I'm sorry, but a burrito and a new pair of Doc Martens does not take priority over networking and communications infrastructure in the middle of a natural disaster.

    Do you think your cell phone service is powered by magic sky fairies who just fly your words off to the person on the other end of the line? It's not. When you dial 911, your local cell tower pipes the data down a fiber optic backhaul to the nearest data center where it hooks into the outside network and routes your call to your local 911 dispatch, which (surprise surprise) relies on telecommunications infrastructure. There's a pretty good reason why most big data centers are #2 in line for generator fuel behind hospitals - they need it because they provide essential services that are crucial in the wake of a disaster.

  28. Re:NYC should sue the Koch brothers for damages by WOOFYGOOFY · · Score: 1

    Definition of "ideologically crazed"- a group of people who systematically deny reality. In this case, human civilization threatening reality.

    What reality are they denying? The reality that either we will take world-wide coordinated action against carbon-induced climate change or we will extinguish human civilization.

    That's pretty much the definition of a terrorist. Think the al Queda true believers didn't think they had a point to what they were doing? Who cares WHY a terrorist acts the way a terrorist acts? Who cares if ,in their view , they're terrorists or that what they're doing qualifies as terrorism?

    Charles Manson will give you a long elaborate argument about the rightness of his actions too. So what.

    What you can't accept is that about 25% of this nation is taking action which will surely result in the destruction of the earth and human civilization. That's not hyperbole, that's not a melodramatic characterization of some social issue; that's a plain fact.

    You can't reconcile that your crazy FoxNews - Limbaugh listening uncle Larry is going to be directly responsible for the end of human civilization. It's too big a jump. He's just harmless old man. But the fact is that there is nothing stopping us from taking the action we need to take EXCEPT the uncle Larrys of the world and the financiers and think tanks and media personalities that stoke their false beliefs.

    The wretches at the CATO Institute and the Koch Brothers and psychotic, sociopathic scum at the Heritage Foundation and FreedomWorks and Fortune Magazine and the WSJ and Grover Norquist and Limbaugh and all the other professional deniers and merchants of genocide and their mouthpieces will end their lives on the business side of a hemp rope provided courtesy of the very United States Government they so loathe. They're the dead enders, people whose fates are sealed and whose recorded voices and writings are all anyone needs to convict them.

    But it's more than the professional subhuman filth that is involved. It's civil war we're hurtling towards, thanks to your uncle Larry. When the world wide crop failures come, when the terrorists start their recruiting in earnest around American's denial of climate change and anti-American antipathy becomes a global tidal wave manifesting as humans from every corner of the earth wanting one thing and one thing only, to destroy this nation, when the full hell is unleashed because of fools like your uncle Larry and his jackassian evaluation of his own scientific credentials, then you'll see that people like me who you always regarded as over the top were in fact just telling you a truth you didn't want to hear.

    Your uncle Larry is a genocidal maniac whether he believes that about himself, whether he intends that to be true about himself, or not.

    The bloodiest most vicious wars in history are civil wars. Each side has exhausted sympathy with the other and every attempt at rational reconciliation has been played out, ad infinitum. Generally people can't wait for the killing to start and it's completely merciless when it does. That's what we're hurtling towards. You hate me and hate my voice and you hate what I'm saying and the way I'm saying it but has something else worked so far? How's that new scientific study influencing the other side ? It's me and not you who is actually provoking people to feel the fear they need to feel in order to act, who's painting the picture of our shared and certain future which we are ignoring because it makes us feel unhappy inside.

    If rational argument was going to work, then it would have worked already. The rational debate is now 40 years old. Meanwhile, about 50% of the electorate is gearing up to vote a denier into the office of the Presidency from whence he will begin to build out the Keystone pipeline and seal all our fates absent the event of some technological miracle.

    Deniers need to hear what the deal is. The deal is reality is going

  29. A First-Hand, On-Site Ongoing Account... by Anonymous Coward · · Score: 2, Insightful

    Those of you with systems up and running with Peer1 should be thankful to the data center folks as well as the Squarespace staff who're pitching in big-time! If you want to know what's really going on, along with some pictures, check out their status page: http://status.squarespace.com

    Basically, to address other folks' various questions: Yes, the generators are on the roof (along with a small start-up + several hour supply of fuel), but most of the fuel supply is in the basement due to fire codes, etc.. They switched over smoothly to generator power but the pumps to get fuel up from the basement went offline. They've been doing the bucket brigade since, and they did have various backup plans in place (and even a basic plan for a flooded basement & no grid power), but everything is basically in worst-case-scenario-mode right now. That is, difficulty getting fuel trucks to re-supply (though that's OK now), no air-lift available for the refueling on the roof, accessible backup fuel pumps not powerful enough and the right equipment can't get to them because of conditions throughout the rest of the city, and they've been trying to clear the basement of water for a good while now but it appears that in addition to the flooding there may be a water main break that's pushing water back into their basement as fast as the pumps take it out.

    Given the situation, they're doing a hell of a job!

    1. Re:A First-Hand, On-Site Ongoing Account... by couchslug · · Score: 2

      If you are involved in improving systems after the recovery, consider using pumps driven by hydrostatic (hydraulic) drive which would allow you to place pump drive motors well above flood zones. No electric motors below water level means nothing to short out. Hydraulic driven fuel pumps are often located IN aircraft fuel tanks, and it may be an aircraft pump or pumps would solve your problem. They move a LOT of fuel. The parts would be easy to get (tough hydrostatic pumps and motors have been standard farm equipment products for decades) and hydraulic drives mean the pumps could be located where most effective. Pumps are most efficient when pushing fluid.

      Dunno if they have enough horsepower to elevate fuel to your desired height, but aircraft refuel trucks have adjustable output pressure and their pumps are driven by the truck engine. If you have the proper fitting for their hose and connect it to your fuel plumbing, you may be able to pump direct to the roof using the refuel truck pump. Basically
      roll up, unreel the hose, connect it, open the appropriate building valve and you could feed the roof tank or the storage tank as you wish.

      Diesel engines will run on jet fuel, which is basically diesel with different additives. (USAF diesel trucks etc are required to run on JP-8). If all you have is a jet fuel tanker available because all others are busy, it might save the day.

      --
      "This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
  30. Re:Your priorities are NOT all messed up!! by bws111 · · Score: 1

    Certainly there are criticial datacenters that have been impacted. Exactly what criticial services are on Peer1 (which is where they are carrying fuel through emergency stairwells)?

  31. Re:NYC should sue the Koch brothers for damages by Attila+Dimedici · · Score: 1

    What reality are they denying? The reality that either we will take world-wide coordinated action against carbon-induced climate change or ...

    Except that reality is that mankind will not take world-wide coordinated action against carbon-induced climate change. That has nothing to do with "deniers". It has everything to do with the fact that India and China (and many other developing nations) have other priorities at this time. So, you had better hope that the "deniers" are more right than you are because that is the reality. There are several major nations that are only too happy to make deals with the industrialized nations so that the industrialized nations will cripple themselves, but those other nations have no intention of limiting their CO2 emissions if it means slowing down their own industrialization.
    I will repeat, there will be no "world-wide coordinated action". There never has been, there never will be. The closest the world has ever come to "world-wide coordinated action" was when the British Navy imposed the end of the slave trade on the world with the reluctant cooperation of the rest of the world's powers (who only cooperated because the Royal Navy was in a position to pretty much effect that result whether they cooperated or not by making the slave trade too expensive to be profitable).

    --
    The truth is that all men having power ought to be mistrusted. James Madison
  32. Can anyone tell me by kilodelta · · Score: 1

    How prevalent natural gas service is in NYC highrises? And another, was natural gas service disrupted in the latest conflagration?

    If so - why the hell are those generators powered by diesel fuel? Back in the mid 2000's I spec'd out redundant power for a state office and their I.T. unit. a 125kW natural gas fired generator, full auto transfer switch, generator exercises monthly, etc. That was backed up by an APC Symmetra system that could keep the entire server room and I.T. unit powered for 90 minutes.

    1. Re:Can anyone tell me by Kalriath · · Score: 2

      Your state office would in that case go offline roughly 90 minutes after virtually every disaster possible. Natural Gas, being a utility, is virtually guaranteed to be taken offline in cases where power goes out, ESPECIALLY when the power is deliberately cut off by the provider as a preventative against cataclysmic explosions where a gas pipeline explosion would be even worse.

      --
      For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
    2. Re:Can anyone tell me by kilodelta · · Score: 1

      We understood this but we studied events of the great hurricanes that struck in 1938 and 1954. Seems natural gas service stayed on during BOTH those events. So we went on that.

      And 90 minutes with tie-ins to ALL the servers meant they could shut down gracefully. So that puts your scenario into stark relief.

    3. Re:Can anyone tell me by CAIMLAS · · Score: 1

      It's government we're talking about here, quite obviously. "Let's spend a lot more than we need to spend on something which won't be half as resilient as the common, industry accepted approach".

      There's a reason why even nuclear power facilities use diesel generators. They're a good fall-back.

      --
      ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
  33. Re:Your priorities are NOT all messed up!! by Anonymous Coward · · Score: 3, Informative

    I work for the city. Happy to report that while we lost a lot of nodes and circuits downtown, none of our datacenters took a hit. We lost Mainframe DR replication for about 12 hours. No impact. 911 and 311 obviously experienced high call volume. 911 took no weather related hits, 311 saw about 2 minutes of weather related outage during a brief period where our two COs were flooded and we were in the process of switching to other trunks.

    EXCELLENT work by the Cities workers throughout this event.

  34. Re:NYC should sue the Koch brothers for damages by WOOFYGOOFY · · Score: 1

    . That has nothing to do with "deniers". It has everything to do with the fact that India and China (and many other developing nations) have other priorities at this time.

    Yeah what's going on is that China and India made a reasonable argument that they're not responsible for the C02 currently in the atmosphere, which is historically true, and wanted those responsible- Europe and America to take the lead cleaning up the mess they made with themselves making a smaller contribution. This makes sense since the US hs the most resources with which to implement reductions and cap and trade - a formerly Republican idea- was a viable economically palatable method through which the required reductions could be achieved. The rest of the world agreed. The US is now the sole non-signatory to Kyoto, thanks t the Republican opposition to cap and trade which they call "big government" and "socialism" .

    In Copenhagen we have a world-wide agreement with definite targets but it is non-binding. This is what deal making looks like, it's what deal making has always looked like. It's not to my taste and I find it distressing however, it is what it is nad the process does lead to progress and agreements.

    The question is not "will the world agree"?" The question is "will conservatives in America forestall necessary action for so long that the world spirals down into an unrecoverable abyss through some mechanism we either do or do not understand as of today.

    So it's not true that China and India are "going their own way" or anything else. If you want to see what Kyoto and Copenhagen required of the signatories- of which China and India are both members "

    It's a nonsensical argument anyway. If everyone is throwing grenades up in the air endangering everyone including themselves, the first thing you do is stop throwing grenades yourself irrespective of what others do.

  35. Re:NYC should sue the Koch brothers for damages by WOOFYGOOFY · · Score: 1

    I meant to say you should consult Wikipedia if you want to see what Kyoto and Copenhagen require of their signatories.

    Just to put some more meat on the bones of this conversation, here's what 4 degree centigrade means to the earth- and remember, that is a number LESS than the one we're on track for today.

    http://www.guardian.co.uk/books/2007/apr/23/scienceandnature.climatechange

    http://grist.org/climate-energy/what-would-it-mean-to-treat-climate-change-like-a-security-threat/

  36. Re:NYC should sue the Koch brothers for damages by Attila+Dimedici · · Score: 1

    Wow, just wow. You really believe that. The European Union countries, which were theoretically all on board for the Kyoto protocols were further from meeting their Kyoto obligation than the U.S., which never signed on...You remember the Kyoto protocols, the ones that Clinton never bothered to submit to the U.S. Senate for ratification, but which the U.S. Senate vote 97-0 to reject?

    --
    The truth is that all men having power ought to be mistrusted. James Madison
  37. Curious fact about Broad St by gelfling · · Score: 1

    Broad St is the only street in NYC where the #'s go UP as you go south.

    1. Re:Curious fact about Broad St by Morpork · · Score: 1

      I've heard that's common for broads everywhere [da boom CHING!]

      --
      -- Always borrow money from a pessimist; they don't expect to be paid back.
  38. Re:Your priorities are NOT all messed up!! by Unknown+Lamer · · Score: 1

    How about ... people pay them for the servers never to go down, even in an emergency. I'm sure the 150 people in my hosting cooperative are appreciating their efforts. Also, given that peer1 makes money from colocation, it's in their business interest to keep it running. TFA notes that customers have joined the bucket brigade, so I'm sure those folks think their servers are criticial.

    --

    HAL 7000, fewer features than the HAL 9000, but just as homicidal!
  39. Re:NYC should sue the Koch brothers for damages by WOOFYGOOFY · · Score: 1

    Yeah do you ever bother to think that something you say might get fact checked? For the EU-15 from http://www.eea.europa.eu/publications/ghg-trends-and-projections-2012/

    Average 2008â"2011 emissions in EU-15 were 11 % lower than the base-year level, below the Kyoto target of -8 % for the period 2008â"2012. In the sectors not covered by the EU ETS, emissions were lower than their respective target, by an amount equivalent to 1.7 % of base-year emissions.

    LULUCF activities are expected to decrease net emissions by an annual amount equivalent to 1.4 % of base-year level emissions. EU-15 intends to use the flexible mechanisms at government level by acquiring an amount of Kyoto units equivalent to 2 % of base-year emissions per year.

    Taking all these effects into account, average emissions in the sectors not covered by the EU ETS in EU-15 were standing below their target level, by a gap representing 4.9 % of the base-year emissions.

    The EU-15 was therefore on track towards its Kyoto target by the end of 2011. However, to ensure that the EU-15 reaches its common target, all of its Member States must achieve their respective burden-sharing target. Excess Kyoto units resulting from overachievement by some countries might not be available to the EU-15 for achieving compliance.

    Oh I get it, you're not a "word and concept type " person. You need pictograms. Well, fortunately I have some handy:

    http://www.guardian.co.uk/environment/interactive/2009/nov/30/copenhagen-summit-world-carbon-emissions

    That pretty much says , well, golly if it doesn't say EXACTLY THE OPPOSITE of what you were saying.

    Do you ever get sick of being just plain wrong. I mean I know we're all writing in persistent-anonymous identities and what not but ... aren't you ashamed to show your persistent identity on line after you get caught out telling whale-size, exact-opposite-from-reality whoppers like that?

    Just asking to get a datapoint for my little self-imposed project to try to understand the psychology behind denialism, especially the "I'm not a denier" type of denialism.

  40. Fail in the pumping department... by Anonymous Coward · · Score: 0

    The pump problem is so easy to solve...
    Use a freaking oil powered pump inside the fuel thank, attach another oil pump to the generator shaft and done, generator pumps it own fuel, and you can have all the water you want in the basement(given that the tank is sealed)..

    So much for smart people...

  41. Re:NYC should sue the Koch brothers for damages by firewrought · · Score: 1

    What reality are they denying? The reality that either we will take world-wide coordinated action against carbon-induced climate change or we will extinguish human civilization. That's pretty much the definition of a terrorist.

    Nope. Terrorism is the use of mass murder, industrial sabotage, and threats of same to achieve political ends, usually calculated to promote fear in a target population. We can refine it further, distinguishing the term from "war", "assassin", "murder", "freedom fighter", "mass shooter", etc., and enjoy the rich contradictions and ambiguities we would undoubtedly discover.

    What we can't do is stretch the term to apply to everything that we disagree with... at that point, you're just trying to manipulate people in the same dirty way that FOX and other pundits do.

    Here's a different idea: tell the truth. Viz., "think tanks and marketing firms that concoct lies to discredit global warming ARE NOT engaged in terrorism, they are engaged in something MUCH WORSE". Maybe it's not as catchy, but it still has a little bit of a hook while remaining honest.

    Even better--in my view of these matter--would be not to focus your attack on tearing down the opposition, but on building a positive case. That tends to "play" better to undecided audiences, and too much vitriol (even if deserved) makes you look like the unreasonable party. Even better if you can inject humor, like this famous cartoon did. There's no shortage of emphatic opinions on the internet; honest, insightful wit on the other hand...

    --
    -1, Too Many Layers Of Abstraction
  42. Location, location, location... by Anonymous Coward · · Score: 0

    What moron builds a data center near the ocean? You know, oceans, big bodies of water that on occasion invade the land and swamp out buildings that have never had so much as a drop in their basements, where we put the generators... Cyclones and tsunami waves are the reason you don't build data centers near the ocean or Gulf of Mexico. Hello! Any kid in a 6th grade geography class can list the dangers of living so close to the water.

    Yes every location has it's potential hazards. If it's not storm surge waves it'll be something else, but seriously, oceans and data centers do NOT mix! You know why people put a data center in NYC? Three reasons; One - the allure of having your data in one of the biggest cities in the world (read: source of pride and arrogance) Two - that's where the company is and having the data center at arms reach of the senior management belies their fundamental misunderstanding of the power of networks, and Three - That's where the big internet backbones originate, so we got to be there.

    As for point #2 a few companies experienced something similar when the WTC buildings were hit and they lost either their entire data center of one of the fail over centers used for live backups or network fail over. You don't put data centers in tall buildings for most of the same reasons above.We have data centers in the plains states and if the expense can be justified, we bury the facilities that would be little affected by a tornado to keep them safe or minimize the impact the weather can have on the date center. So in summary I have no sympathy for the data centers that have been lost, destroyed, damaged. Idiotic management of such valuable resources breaks my "give a damn"

  43. Re:NYC should sue the Koch brothers for damages by WOOFYGOOFY · · Score: 1

    " Terrorism is the use of mass murder, industrial sabotage, and threats of same to achieve political ends, usually calculated to promote fear in a target population."

    Murder? Check- the wanton deliberate and reckless taking of human life, in this case to further political and economic ends. The only part of this that anyone can dispute is "deliberate" .

    It's not really that relevant whether they *mean* to kill hundreds of millions or not btw. Doing so is still manslaughter.

    But I hold that it IS deliberate. I hold that inside themselves these individuals believe on balance that what they are promulgating is *wrong*. These are people with variously, oppositional-defiant personality disorder and extremist political and racial views . It's not a coincidence that global warming deniers are also that segment of the population who are right wing and racist. For both cases, the death of large numbers of other people is a trifling thing. For oppostional defiant personality types, the ONLY thing that matters is defying authority. That is the unconscious, ultimate good and nothing else matters, everything else is rationalized.

    For the extremist political reactionaries at FOX and Forbes, the death of a lot of undesirable people in Africa and elsewhere is not something to particularly worry about and even more specifically if the US government can be overthrown through the device of bankrupting it or undermining general social order, then that's a good way to achieve their ends- the reordering of all of society to their liking.

    It's not insignificant that Koch when he was running on the Libertarian ticket for VP espoused the disbanding of the CIA the FBI and nearly all of national security. He means to topple this nation as it is through any device he can. He gave up on politics (libertarians get about 2% of the vote in any general election) and has has happened across a lucky fact- if we do nothing about climate change, human civilization as we know it will simply unravel. In the resulting "what's left" , he'll be able to build a new world. This is how insane psychopaths reason. This is what goes on in their imaginations. Just as completely ordinary mainstream you work next to them Christians long for the Apocalypse and the wanton destruction and mass death that event entails, so also do other delusionists imagine that at the end of civilization its THEY and THEIR cronies who will be imposing order on THEIR new world.

    Ditto FoxNews and Forbes. They know they're lying, they are consciously or barely-unconsciously aware they're acting in bad faith. People doing this should note with fear - a near future lie detector will definitely nail. Lying to yourself is not an undetectable, no-cost / no signal activity. We really want to get there. This is very very heavily funded. We're are getting there. It's coming.

    Don't kid yourself, these people are deliberately building a bomb called "catastrophic climate change" and they mean to set it off.

    That my friend is murder and if we have to expand the definition of murder to include it, we 'll do that.

    Criminals think they're cleverer than the rest of society. Criminals believe they have a loophole in the law through which they can smuggle their criminal activity. The thing is, laws go wherever criminals lead. If criminals try to do X to harm society, then the law will go there. If they try Y, then the law will go there. It's criminals who ultimately decide through their chosen conduct what laws are created by society and what is counted as a criminal act, not visa-versa. The law is not static, and can even e retroactive if needed because the law is there to serve a purpose- to stop and punish anti-social behavior however it manifests itself. And we will do that, have no doubt.

    The other idea of "being positive" has exactly zero effect on people like Koch Fox Murdoch Forbes and those who would follow them. What you're saying is get more people on your side by sounding positive. Has that worked y

  44. And once again by Sean · · Score: 1

    Peer1 fails to take adequate measures to keep the power on.

    1. Re:And once again by jmkrtyuio · · Score: 1

      Who else in the building hasnt lost power yet?

  45. Re:NYC should sue the Koch brothers for damages by Anonymous Coward · · Score: 0

    Except that the new-hippie delusional tree huggers like yourself are loath to own or touch guns and the people you are uselessly frothing from the mouth at are not. You will also jealously cling to your ipad and iphone while apple continues to rape and pillage via "Intellectual Property"(really, please pick one you can't have both). But please, don't let the real reality (which is somewhere not at the middle or either extreme) get in the way of your delusions( i.e the y2k bug called, it wants you to stop copying its histeria meme)

  46. archimedes screw diesel pump by Anonymous Coward · · Score: 0

    and done in one

  47. Dewar tank diesel storage by Anonymous Coward · · Score: 0

    lets see:

    Dewar tank diesel storage (twin metal, not metal/glass)
    Archimedes screw diesel pump

  48. Re:NYC should sue the Koch brothers for damages by WOOFYGOOFY · · Score: 1

    So wrong on all counts and every *single* thing you said regarding "people like me" , what my opinion of Apple and patents is and the non-delusional reality of climate change.

    Your post distinguishes itself by consisting entirely of false statements.

    But he fact that I am not the least bit of what anyone would call unfamiliar with weapons and how to use them means nothing in this context since no one, especially me is considering taking the law into our own hands. We don't need to. We are the law. That's what democracy is about. There is zero doubt that the subhuman filth I referred to will, if they have the grave misfortune to live another 20 years , end their lives at the end of a hemp rope courtesy of a due process and jury of their peers.

    An orderly society conducting itself within the confines of due process is these guy's worst nightmare. They're betting big on chaos. We are society; we are the rational truth telling children and inheritors of The Enlightenment and they are the enemies of civil society, or greater order of rationality itself.

    Our power lies entirely in the orderliness of society, in the well functioning of civil society and the well being of its citizens. Why do you think they attack the public good ? Why do you think Koch wants to do away with the FBI and the CIA , a state he calls "freedom"? Why do you think he wants to suppress the vote? Why do you think he wants to devolve us into individual fiefdoms - The Confederacy called it "States Rights" and declared war on the United States over it. Why do you think they pushed fro Citizens United?

    Because all that is the enemy of representative democracy , a system of government The Koch's and Paul Ryan's heroine Ayn Rand had nothing but contempt for. They hate representative democracy, they hate the Untied States of America they hate our system of government and the power it gives to :"ordinary" citizens, including the power to find them guilty of Crimes Against Humanity and to put a noose around their necks.

    Social order, due process, facts, a reality-based populace able to make their voice heard all these things are what our power base consists of and as long as those things hold together , they're fucked.

    Why do you think the Governor of Texas is always talking about secession? Why do you think Norquist et. al. are always bashing the government? Why was it a Republican operative who whipped up a frenzy over provably non-existant "voter fraud" and used that to repress votes ? Why do you think Norquist has stated that taxes are worse than the Holocaust and he wants to take the federal government and drown it in a bathtub? Because a government by for and of the people is the worst thing that ever happened to this kind of scum criminal. Law and order and reality based thinking are the sworn enemy of this filth, and they know it .

    Please. The last thing anyone on our side is going to do is start to take the law into their own hands. We'll leave that to them, to their "2nd Amendment solutions ",.....and then destroy them utterly, completely and finally when they do.

    Yep, there' s more ways than one to tear someone's head off and shit down their neck and throw their lifeless worthless corpse in the fucking gutter.

  49. NG is *NOT* always cut off when Electricity is cut by Anonymous Coward · · Score: 0

    Well during the big afternoon and part of the evening blackout event of San Diego county, ALL power was cut off in San Diego, but the natural gas was still flowing at my house... that's how I cooked a pasta dinner for myself. Also all my co-workers' gas was still working... although some of them didn't have matches or lighters to ignite the gas since the electric igniters on the stoves were not working.