Slashdot Mirror


Lightning Wipes Storage Disks At Google Data Center

An anonymous reader writes: Lightning struck a Google data center in Belgium four times in rapid succession last week, permanently erasing a small amount of users' data from the cloud. The affected disks were part of Google Computer Engine (GCE), a utility that lets people run virtual computers in the cloud on Google's servers. Despite the uncontrollable nature of the incident, Google has accepted full responsibility for the blackout and promises to upgrade its data center storage hardware, increasing its resilience against power outages.

32 of 141 comments (clear)

  1. Whaaa? by SpankiMonki · · Score: 4, Funny

    Permanently erased? How can this be? Doesn't Google keep an off-site backup of my pr0n on tape or DVDs or sumpthin? So much for best practices, I guess.

    1. Re:Whaaa? by GigaplexNZ · · Score: 5, Interesting

      The affected service was Google Computer Engine, meaning that data may be changing. Replication isn't instantaneous, so I'd imagine the lost data was pending modifications.

    2. Re:Whaaa? by Anonymous Coward · · Score: 5, Interesting

      From what I read elsewhere it was new/current data, not even an hour old, and the lightening may have caused things to run off batteries for a bit too long due to the multiple strikes. Seems not unreasonable as an explanation, might be entirely wrong though. Articles implied that users can also backup on their own sites to ensure that they are not behoved to anyone.

    3. Re:Whaaa? by sexconker · · Score: 4, Insightful

      All datacenter class storage devices should be backed by battery units with enough capacity to flush all pending writes to disk.
      I have never bought a server that didn't have battery-backed hardware RAID.

      Google, however, runs the cheapest, commodity parts, often refurbished / purchased used, and relies on software RAID and massive replication schemes. Such schemes don't work for new data, as they've found out.

      I wouldn't blame them if their shit got directly hit by lightning and that caused damage (you can't expect anything to survive that), but if we're saying the extended power outage caused data loss, then it's absolutely Google's fault.

    4. Re:Whaaa? by Hognoxious · · Score: 5, Funny

      lightening may have caused things to run off batteries for a bit too long

      Darkening reduces the output of solar panels, so you can't win either way.

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    5. Re: Whaaa? by Anonymous Coward · · Score: 3, Informative

      In this case it was disks the customer had specifically requested to be un replicated for performance. So all the data could be gone in a flood and Google would still be within its obligations.

    6. Re:Whaaa? by DNS-and-BIND · · Score: 2

      The purpose of a battery backup isn't to let things run when the power goes out. The purpose of a battery backup is to allow an orderly shutdown when the power goes out, instead of a sudden outage that results in data loss and damaged systems.

      --
      Shutting down free speech with violence isn't fighting fascism. It IS fascism!
    7. Re:Whaaa? by TheCarp · · Score: 2

      Except you don't have to blame them, they took responsibility.

      Pretty sure what they have 'found out' is that paying for the fallout from the occasional freak occurrence and minor data loss is cheaper in the long run than buying more expensive hardware to gaurd against occurrences so rare that they end up on news sites.

      --
      "I opened my eyes, and everything went dark again"
  2. Re:Cannot be trusted by Anonymous Coward · · Score: 5, Interesting

    Just use Amazon like everyone else. Google cannot be trusted, and I have said that many times. They 1) frequently decide to shut down services users rely on. One of the persistence mechanisms we depended on recently got the head shot, costing us so much money that we decided to move to Amazon, which has a standardized stack, and 2) data loss, and 3) non-existant customer service. Try contacting Google with a pressing issue.... you'll eventually give up.

  3. Oh realy? by SeaFox · · Score: 5, Interesting

    Lightning struck the same place not twice, but four times?

    1. Re:Oh realy? by Anonymous Coward · · Score: 5, Funny

      Cloud to Cloud lightning is about 3 times more common than cloud to ground, so its not that crazy.

    2. Re:Oh realy? by Anonymous Coward · · Score: 5, Informative

      Contrary to popular belief, it's common that ligthning strikes several times at the same place. Something that was attracting lightning, if not destructed by the first strike, will still be a major target standing for the next ones.

    3. Re:Oh realy? by Anonymous Coward · · Score: 4, Funny

      It's cloud computing after all. Of course there will be frequent lighting.

    4. Re:Oh realy? by Anonymous Coward · · Score: 5, Funny

      Another reason not to store your important data in the cloud.

    5. Re:Oh realy? by DigiShaman · · Score: 4, Interesting

      How rapid? I'm of the understanding that the first bolt ionizes the pathway in the air thereby reducing the resistance for the subsequent strikes to follow. All of this occurring within a few seconds.

      --
      Life is not for the lazy.
    6. Re:Oh realy? by slimshady76 · · Score: 2, Funny

      Another reason not to store your important data in the cloud.

      Arrrgh, somebody please mod this up!!! You made me snort my coffee.

    7. Re:Oh realy? by pubwvj · · Score: 4, Interesting

      Lightning striking in the same spot repeatedly is a lot more likely than people think. The reason lightning may have struck a spot is due to there being a good path. Thus lightning is likely to strike that easy path again.

      We have that. We live on a mountain where there is a large copper vein running under us. I have watched lightning strike repeatedly in the same spot.

      There are videos of lightning repeatedly striking tall buildings during a single storm.

      More over, lightning does not need to be very close to do a lot of damage. In a recent storm we had nine nearby strikes - not all in the same spot but spread out over at least a square mile of our land. We lost many miles of wire because of the EMP that the lightning strikes generated got picked up by the wires and overloaded them causing the wires to melt. Some sections of fence wire simply vanished. Google could have had a few nearby strikes that did that. This happens.

      See:
      http://sugarmtnfarm.com/2015/0...
      and
      http://sugarmtnfarm.com/2015/0...

  4. In other news... by LatePaul · · Score: 5, Funny

    ... Alphabet's new "personal re-vivification" project is making good progress. The project leader, V.Frankenstein was unavailable for comment however.

  5. Re:Cannot be trusted by Anonymous Coward · · Score: 5, Informative

    The announcement is about Google Cloud Engine. Not about Google's own services (gmail, search, photos, docs, that sort of things). AFAIK, none of Google's own service announced any loss - presumably because they don't rely on a single location.

    From the post:

    > In particular, it was possible at all times to recreate new Persistent Disks from existing snapshots.

    i.e. snapshots were fine.

    > This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data.

    So, if some poor users of GCE thought a single geographical location can withstand disasters, they now know.

  6. Another Flash Vulnerability ? by Dave+Whiteside · · Score: 5, Funny

    n/t

    --
    who where what when now?
  7. Re:What do you mean permenantly erased...only 1 DC by Anonymous Coward · · Score: 2, Informative

    If you RTFA you'll see they mention it only affected "recently written data" that had not yet made it to persistent storage. So probably only a few hours old at most.

  8. If my calculations are correct... by h33t+l4x0r · · Score: 3, Funny

    That comes to 4.84 Jiggawatts! No wonder there was outage.

  9. Re:What do you mean permenantly erased...only 1 DC by Viol8 · · Score: 2

    Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.

  10. Re:What do you mean permenantly erased...only 1 DC by bledri · · Score: 5, Insightful

    Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.

    So faster than the speed of light using the infinitely-wide infinite improbability data bus?

    --
    Some privacy policy Slashdot.
  11. Re:What do you mean permenantly erased...only 1 DC by Skapare · · Score: 2

    hours to replicate data? why so long? are they still running __________? (insert your most hated system/language)

    --
    now we need to go OSS in diesel cars
  12. Headline Failure by smallfries · · Score: 2

    Come on people. This has the potential to be legend... ary. What a complete failure.

    Even just form a quick punt we could glimpse such lyrical word play as:
    "Lightning strike inside Cloud"
    "Cloud damaged by lightning"
    "Cloud not lightning-proof"

    Please read the fucking Register until you gets it.

    --
    Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
  13. Re: Cannot be trusted by Lord+Bitman · · Score: 4, Insightful

    I know that's what I base my critical data storage choices on - how fast a tangentially-related service's static front page loaded 15 years ago on dialup.

    --
    -- 'The' Lord and Master Bitman On High, Master Of All
  14. Re: lightning strike by MasterOfGoingFaster · · Score: 5, Insightful

    Bullshit. I used to design high voltage connections, and tested using a 300kV impulse generator. I've seen a lot of crazy stuff analyzing field failures. You can greatly reduce the risk, but you cannot remove all risk in an above ground facility, as a practical matter.

    I do see lots of silly stuff done, based on myth and lack of knowledge.

    --
    Place nail here >+
  15. Re:What do you mean permenantly erased...only 1 DC by Viol8 · · Score: 3, Informative

    *GUFFAW*

    You're wasted here, you should do stand up.

  16. Re:What do you mean permenantly erased...only 1 DC by ScentCone · · Score: 2

    Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.

    You can have that services if you want to pay for it. You get that, right?

    --
    Don't disappoint your bird dog. Go to the range.
  17. So... by drunk_punk · · Score: 2

    If I understand this correctly, to get your personal data removed from Google search engines it requires 4 lightning strikes to the exact same location?

    Must have missed that part in the EULA...

  18. instead, check out polyphaser.com by swschrad · · Score: 2

    you left out about 99-44/100 percent of the technology and art of lightning protection.

    --
    if this is supposed to be a new economy, how come they still want my old fashioned money?