Slashdot Mirror


Lightning Wipes Storage Disks At Google Data Center

An anonymous reader writes: Lightning struck a Google data center in Belgium four times in rapid succession last week, permanently erasing a small amount of users' data from the cloud. The affected disks were part of Google Computer Engine (GCE), a utility that lets people run virtual computers in the cloud on Google's servers. Despite the uncontrollable nature of the incident, Google has accepted full responsibility for the blackout and promises to upgrade its data center storage hardware, increasing its resilience against power outages.

141 comments

  1. Whaaa? by SpankiMonki · · Score: 4, Funny

    Permanently erased? How can this be? Doesn't Google keep an off-site backup of my pr0n on tape or DVDs or sumpthin? So much for best practices, I guess.

    1. Re:Whaaa? by Anonymous Coward · · Score: 0

      I wonder if this is a case of data not being replicated straight away - you upload something to the cloud but it sits in a queue for 4 hours before it's copied around the world

    2. Re:Whaaa? by GigaplexNZ · · Score: 5, Interesting

      The affected service was Google Computer Engine, meaning that data may be changing. Replication isn't instantaneous, so I'd imagine the lost data was pending modifications.

    3. Re:Whaaa? by Anonymous Coward · · Score: 0

      No, the problem is that replication in GCE only occurs within a data center. For natural disasters like this, affecting much or all of a facility, replication doesn't help.

      I'm stunned to learn that Google doesn't duplicate data across data centers.

    4. Re:Whaaa? by Anonymous Coward · · Score: 0

      Or the controlled shutdown while on battery power involved systemd of a certain version. https://www.debian.org/releases/stable/amd64/release-notes/ch-information.html#systemd-sigkill-regression

    5. Re:Whaaa? by Anonymous Coward · · Score: 5, Interesting

      From what I read elsewhere it was new/current data, not even an hour old, and the lightening may have caused things to run off batteries for a bit too long due to the multiple strikes. Seems not unreasonable as an explanation, might be entirely wrong though. Articles implied that users can also backup on their own sites to ensure that they are not behoved to anyone.

    6. Re:Whaaa? by Anonymous Coward · · Score: 0

      The affected service was Google Computer Engine...

      Drop the "r", dude. The second one.

    7. Re:Whaaa? by Dog-Cow · · Score: 1, Troll

      No, the problem is that replication isn't instantaneous. If you can't accept reality, please kill yourself.

    8. Re:Whaaa? by sexconker · · Score: 4, Insightful

      All datacenter class storage devices should be backed by battery units with enough capacity to flush all pending writes to disk.
      I have never bought a server that didn't have battery-backed hardware RAID.

      Google, however, runs the cheapest, commodity parts, often refurbished / purchased used, and relies on software RAID and massive replication schemes. Such schemes don't work for new data, as they've found out.

      I wouldn't blame them if their shit got directly hit by lightning and that caused damage (you can't expect anything to survive that), but if we're saying the extended power outage caused data loss, then it's absolutely Google's fault.

    9. Re:Whaaa? by Hognoxious · · Score: 5, Funny

      lightening may have caused things to run off batteries for a bit too long

      Darkening reduces the output of solar panels, so you can't win either way.

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    10. Re: Whaaa? by Anonymous Coward · · Score: 3, Informative

      In this case it was disks the customer had specifically requested to be un replicated for performance. So all the data could be gone in a flood and Google would still be within its obligations.

    11. Re:Whaaa? by Anonymous Coward · · Score: 0

      Why is this modded down?
      Computers need time to do work.
      Expecting all of your crap to be instantly replicated across the globe is ridiculous.

    12. Re: Whaaa? by Anonymous Coward · · Score: 0

      You have just won a stupid internet.

    13. Re: Whaaa? by Anonymous Coward · · Score: 0

      Hey Dogcow, just a reminder that 2600 doesn't miss you ;)

    14. Re:Whaaa? by Anonymous Coward · · Score: 1

      It was on the VAXcluster I administered in the '80s. The "reality" is that until the replication has happened, the data shouldn't in principle be regarded as ready for processing. In practice, a few seconds' write caching to ALL nodes is the limit of acceptability.

      I've come to learn that these days most people doing stuff "with computers" are doing nothing of value, and coincidentally this means that the fucking atrocious reliability and security of outsourcing are considered good enough.

    15. Re:Whaaa? by HxBro · · Score: 1

      I demand my data is backed up before I've sent it to google!

    16. Re: Whaaa? by Anonymous Coward · · Score: 0

      Calling a lighting a natural disaster is a bit.... too much isn't it?

    17. Re:Whaaa? by Anonymous Coward · · Score: 0

      Do you keep your .obj files in source control or just your .c files? And have you modified your compiler so that the .obj files are sent to source control, and confirmation is returned, before the link phase begins? Or do you understand that all data isn't equal and sometimes you want more performance in return for less reliability on data that can easily be recreated?

      Perhaps you should name a complainant before claiming that they're doing it wrong, then it will look more like wisdom than reactionary bluster.

    18. Re:Whaaa? by Anonymous Coward · · Score: 0

      I don't know what possible confusion of design can lead to needing to modify a compiler to ensure replication in a filesystem or block driver.

    19. Re:Whaaa? by Anonymous Coward · · Score: 0

      From what I read elsewhere it was new/current data, not even an hour old, and the lightening may have caused things to run off batteries for a bit too long due to the multiple strikes. Seems not unreasonable as an explanation, might be entirely wrong though. Articles implied that users can also backup on their own sites to ensure that they are not behoved to anyone.

      Just for shits and grins, let's take a look at TFA:

      "Although automatic auxiliary systems restored power quickly, and the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain."

      Translation? Google fucked up with an inconsistent resiliency design across part of their storage systems. The overall power loss itself was reported to be "brief".

    20. Re:Whaaa? by DNS-and-BIND · · Score: 2

      The purpose of a battery backup isn't to let things run when the power goes out. The purpose of a battery backup is to allow an orderly shutdown when the power goes out, instead of a sudden outage that results in data loss and damaged systems.

      --
      Shutting down free speech with violence isn't fighting fascism. It IS fascism!
    21. Re:Whaaa? by TheCarp · · Score: 2

      Except you don't have to blame them, they took responsibility.

      Pretty sure what they have 'found out' is that paying for the fallout from the occasional freak occurrence and minor data loss is cheaper in the long run than buying more expensive hardware to gaurd against occurrences so rare that they end up on news sites.

      --
      "I opened my eyes, and everything went dark again"
    22. Re:Whaaa? by Anonymous Coward · · Score: 0

      As usual the summary is very wrong. How can you tell? (Other than the fact that it is a slashdot summary?)

      Lightning does not cause permanent erasure of magnetic media, by far the most commonly used storage medium for digital data. A lightning strike can induce a strong power surge and that can cause all manner of problems (some mention of this in the article), but "permanently erasing" data is not one of them. For example, the power surge might burn out a disk controller, thus rendering the disk unusable and the data irretrievable (at least, without replacing the controller).

      If you read the article, it is clear that data was *lost* not *erased*. This happened because some intermediate storage had inadequate backup power (they mention repeatedly draining battery power -- most likely power went out, battery was used, power came up so system powered up, power went out, repeat until failure). This is a fairly typical scenario with electrical storms and no real surprise and not particularly newsworthy. But /google/ /permanently erase/ and whatever else.

    23. Re:Whaaa? by Anonymous Coward · · Score: 0

      the catch comes in turning equipment back on when power returns. If you want to maintain service uptime you want to have equipment power on ASAP. The problem comes up if battery power was nearly drained in an attempt to maintain uptime with the first outage, and when power comes back on it doesn't last long enough for systems to finish booting -- and there isn't enough battery power to allow a clean shutdown.

      So in any enterprise setup you have to manage this. In general, boot-when-power-applied is bad -- you only want to boot when there's enough battery to allow full boot and clean shutdown. Generators help a lot with this because battery just isn't enough. So any place that is halfway serious will use battery as a stop-gap until the generator kicks in. The only remaining problems are theory-and-practice. Batteries fail (inexplicably -- if you have very many than you will have some that fail in any given outage). Generators fail (to start, provide stable voltage). Power supplies fail. Redundant power supplies fail.

      In the end, if you have a lot of equipment, in any given power anomaly event you *will* have failures. It is just a matter of what and to what extent.

    24. Re: Whaaa? by Anonymous Coward · · Score: 1

      Our datacenter generator fail to start at the last powermoutage we had, the culprit was a squirel

    25. Re:Whaaa? by gweihir · · Score: 1

      Very likely. Cloud providers are routinely lying about their performance, or you need to dig very deep to find what the actual technical assurances are.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    26. Re:Whaaa? by bobbied · · Score: 1

      Brief is apparently long enough to cause data loss. But power disruptions to running systems don't have to be very long to shut them down.

      However, Google apparently hasn't really engineered a proper solution here. In the data center, batteries are there only to bridge you over until the onsite power generator can come on line or until you can perform a safe and orderly shutdown. Plus, you *DON'T* shutdown the generator or reboot shutdown systems after power comes back until you have enough battery capacity and fuel for the generator built back up to survive the next power transient.

      Few people think beyond the first event, but Google should know better.

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    27. Re:Whaaa? by gstoddart · · Score: 1

      But but ... it's the cloud. Why would you need a backup in the cloud?

      You mean we've been lied to?

      --
      Lost at C:>. Found at C.
    28. Re: Whaaa? by Anonymous Coward · · Score: 0

      What else would you call a natural event that just so happened to occur in an area causing damage?
      We don't call hurricanes that never land and cause destruction a natural disaster just because nothing happened. That's how we usually treat lightning. But when it actually destroys something that is a kind of disaster.

    29. Re: Whaaa? by Anonymous Coward · · Score: 0

      If the data is still stuck in the cache lightning will easily erase it permanently.

    30. Re:Whaaa? by Darinbob · · Score: 1

      It's a good thing that users routinely make local backups before uploading to the cloud.

    31. Re: Whaaa? by Anonymous Coward · · Score: 0

      Did your data center conduct monthly generator tests?

    32. Re: Whaaa? by Chexsum · · Score: 1

      its my understanding that google backs up their backups. queue backup song

      --
      Pixels keep you awake!
  2. Cannot be trusted by Anonymous Coward · · Score: 0

    So Google doesn't distribute my data geographically?? I have been mislead.

    Which PaaS provider distributes geographically, if any?

    Thanks.

    1. Re:Cannot be trusted by Anonymous Coward · · Score: 5, Interesting

      Just use Amazon like everyone else. Google cannot be trusted, and I have said that many times. They 1) frequently decide to shut down services users rely on. One of the persistence mechanisms we depended on recently got the head shot, costing us so much money that we decided to move to Amazon, which has a standardized stack, and 2) data loss, and 3) non-existant customer service. Try contacting Google with a pressing issue.... you'll eventually give up.

    2. Re:Cannot be trusted by Anonymous Coward · · Score: 5, Informative

      The announcement is about Google Cloud Engine. Not about Google's own services (gmail, search, photos, docs, that sort of things). AFAIK, none of Google's own service announced any loss - presumably because they don't rely on a single location.

      From the post:

      > In particular, it was possible at all times to recreate new Persistent Disks from existing snapshots.

      i.e. snapshots were fine.

      > This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data.

      So, if some poor users of GCE thought a single geographical location can withstand disasters, they now know.

    3. Re: Cannot be trusted by Anonymous Coward · · Score: 0

      As I have said countless times before - GOOGLE IS SHIT. They built their company on smoke and mirrors and managed to fool quite a large number of people.

    4. Re: Cannot be trusted by Anonymous Coward · · Score: 0, Offtopic

      Google was and is built on it's search engine. Maybe you weren't alive when it first came out, but it was anything but shit. Even ignoring the search results, their homepage was the fastest loading out of every other engine at the time. At slow dial-up speeds, that mattered. That mattered a lot.

    5. Re: Cannot be trusted by Lord+Bitman · · Score: 4, Insightful

      I know that's what I base my critical data storage choices on - how fast a tangentially-related service's static front page loaded 15 years ago on dialup.

      --
      -- 'The' Lord and Master Bitman On High, Master Of All
    6. Re: Cannot be trusted by Antique+Geekmeister · · Score: 1

      I'm afraid that had nothing to do with write speed for storage. It had a great deal to do with deliberately keeping the design very light, using very effective proxies for the very light and consistent images, and keeping tight reins on web designers who might want to front load the Google pages with exciting content that had no relationship to the service.

      I'm confused why you mention it, unless you think that the wise practices and designs that led to such effective and quick interfaces affected their storage designs, as well.

    7. Re: Cannot be trusted by Coren22 · · Score: 1

      I base all my decisions on ACs responding to themselves. Splitting a single post up into three posts all acting like different people agreeing makes an AC look that much more trustworthy.

      --
      APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
    8. Re:Cannot be trusted by Anonymous Coward · · Score: 0

      Any cloud service is going to lose data if you not only have all of your instances in the same physical datacenter but also haven't bothered to snapshot and archive in hours.

      You are correct about the lack of customer service though.

    9. Re: Cannot be trusted by coolsnowmen · · Score: 1

      Oh Lord Bitman. One person said ALL of google is shit based on shit The responder said, you must be too young to remember what google gave us, and how it was better than everything. How its search engine set the bar, and how its free email was a gift to many many people. And you say...non-sequitur in this case. My critical storage needs!!!

  3. Oh realy? by SeaFox · · Score: 5, Interesting

    Lightning struck the same place not twice, but four times?

    1. Re:Oh realy? by Anonymous Coward · · Score: 5, Funny

      Cloud to Cloud lightning is about 3 times more common than cloud to ground, so its not that crazy.

    2. Re:Oh realy? by Anonymous Coward · · Score: 5, Informative

      Contrary to popular belief, it's common that ligthning strikes several times at the same place. Something that was attracting lightning, if not destructed by the first strike, will still be a major target standing for the next ones.

    3. Re:Oh realy? by Anonymous Coward · · Score: 4, Funny

      It's cloud computing after all. Of course there will be frequent lighting.

    4. Re:Oh realy? by Anonymous Coward · · Score: 5, Funny

      Another reason not to store your important data in the cloud.

    5. Re:Oh realy? by Anonymous Coward · · Score: 1

      I hate it when I miss an obvious pun. I shall now steal this for every argument against cloud computing, along with all the serious ones.

    6. Re:Oh realy? by Anonymous Coward · · Score: 1

      Agreed ... I have a tree that my city wont let me cut down because it is an old growth tree and it has been hit 3x and every time it happens the EMP from the bold nukes half of the running electronics in that side of the house.

    7. Re:Oh realy? by DigiShaman · · Score: 4, Interesting

      How rapid? I'm of the understanding that the first bolt ionizes the pathway in the air thereby reducing the resistance for the subsequent strikes to follow. All of this occurring within a few seconds.

      --
      Life is not for the lazy.
    8. Re:Oh realy? by Big+Hairy+Ian · · Score: 1

      Somebody was on the roof wearing armour and yelling "Gods a bastard!"

      --

      Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.

    9. Re:Oh realy? by slimshady76 · · Score: 2, Funny

      Another reason not to store your important data in the cloud.

      Arrrgh, somebody please mod this up!!! You made me snort my coffee.

    10. Re:Oh realy? by Anonymous Coward · · Score: 0

      We had a radio tower struck 14 times within 2 hours last week. It was one of the most active electrical storms I have seen. Massive amounts of cloud-to-cloud looked incredible. I could see the air above the tower from my house, crazy light show.

      Also, not a single thing damaged on that tower and it has some very sensitive radio gear and computers connected to it. You know why? Proper grounding.

    11. Re:Oh realy? by pubwvj · · Score: 4, Interesting

      Lightning striking in the same spot repeatedly is a lot more likely than people think. The reason lightning may have struck a spot is due to there being a good path. Thus lightning is likely to strike that easy path again.

      We have that. We live on a mountain where there is a large copper vein running under us. I have watched lightning strike repeatedly in the same spot.

      There are videos of lightning repeatedly striking tall buildings during a single storm.

      More over, lightning does not need to be very close to do a lot of damage. In a recent storm we had nine nearby strikes - not all in the same spot but spread out over at least a square mile of our land. We lost many miles of wire because of the EMP that the lightning strikes generated got picked up by the wires and overloaded them causing the wires to melt. Some sections of fence wire simply vanished. Google could have had a few nearby strikes that did that. This happens.

      See:
      http://sugarmtnfarm.com/2015/0...
      and
      http://sugarmtnfarm.com/2015/0...

    12. Re:Oh realy? by bobbied · · Score: 1

      If you don't get directly hit by lightning, it can still do a lot of damage to things... There are two ways this happens...

      1. Induced currents into wires close to the strike - This is where the huge amounts of current flowing into the ground from the lighting strike induces currents in other wires. It's basically cross talk, but with a really large impulse input in one wire, you get some pretty impressive signals in parallel wires. This is what fries your corded phones and electronics which are plugged in when lighting hits the tree down the street.

      2. Disruption of the local "ground" reference - Lighting pumps a boat load of charge into the ground at one point and it takes a bit of time for this to dissipate. What this causes is a very large voltage potential to exist between points on the ground. So if you have multiple ground reference points spread around, any device that bridges the gap between two ground references can experience huge voltages. A common example is an electric fence charger which has a ground rod. It will see the voltage differential between it's local ground reference and the power company ground which may be feet apart and see hundreds of thousands volts between the "common" and it's local ground and arc over.

      You avoid lighting damage by first keeping yourself from being hit directly and then being very careful to lower your exposure to the above two risks.

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    13. Re:Oh realy? by Anonymous Coward · · Score: 0

      Before the next lightning strike, store several (how many depends on size of tree) bottles of 90% isopropol alcohol in the tree. Lightning hit tree, alcohol burns, FD puts out fire, what's left of tree gets cut down.

    14. Re:Oh realy? by Anonymous Coward · · Score: 0

      Lightning hits lightning conductor multiple times shock. Who'da thunk it.

  4. Almost expected by Anonymous Coward · · Score: 0

    Google is always wider than deep.

  5. lightning strike by Anonymous Coward · · Score: 0

    If you don't build your data center to be totally safe from lightning strike, it is your fault. What is this rubbish about uncontrollable nature and taking responsibility? This is just plain negligence.

    1. Re: lightning strike by MasterOfGoingFaster · · Score: 5, Insightful

      Bullshit. I used to design high voltage connections, and tested using a 300kV impulse generator. I've seen a lot of crazy stuff analyzing field failures. You can greatly reduce the risk, but you cannot remove all risk in an above ground facility, as a practical matter.

      I do see lots of silly stuff done, based on myth and lack of knowledge.

      --
      Place nail here >+
    2. Re: lightning strike by Anonymous Coward · · Score: 0

      Woosh.

    3. Re: lightning strike by gx5000 · · Score: 1

      Place Lightning Rod "A" in area "B", repeat until coverage and grounding complete.
      Or maybe they should start digging, copy the NSA and have everything underground would solve most issues no ?

      --
      End of Line.
  6. In other news... by LatePaul · · Score: 5, Funny

    ... Alphabet's new "personal re-vivification" project is making good progress. The project leader, V.Frankenstein was unavailable for comment however.

  7. How about a lightning conductor? by Anonymous Coward · · Score: 0

    Morans.

  8. What do you mean permenantly erased...only 1 DC? by AbRASiON · · Score: 0

    You mean to tell me users data is stored in a SINGLE DC and not redundant elsewhere?

    I find that easy to believe for most "cloud" providers but google? Really?
    Ummmmmmmm?
    So presumably if someone took out the right, big big DC in a huge fasion, many users would be fucked by it?

  9. Another Flash Vulnerability ? by Dave+Whiteside · · Score: 5, Funny

    n/t

    --
    who where what when now?
    1. Re:Another Flash Vulnerability ? by Anonymous Coward · · Score: 1

      I bet that sent a shockwave

    2. Re:Another Flash Vulnerability ? by Carewolf · · Score: 1

      No I think this was using Thunderbolt

    3. Re:Another Flash Vulnerability ? by Bongo · · Score: 1

      Oh wait, I never realised my Lightning connector was for charging my iPhone outdoors.

    4. Re:Another Flash Vulnerability ? by Anonymous Coward · · Score: 1

      Witness said "it was like a silver light"

  10. Re:What do you mean permenantly erased...only 1 DC by Anonymous Coward · · Score: 2, Informative

    If you RTFA you'll see they mention it only affected "recently written data" that had not yet made it to persistent storage. So probably only a few hours old at most.

  11. Re:What do you mean permenantly erased...only 1 DC by Anonymous Coward · · Score: 1

    GCE is not a backup solution.

    Google cloud data is geographically replicated.

  12. Containment Failure by Anonymous Coward · · Score: 0

    Probably from the Church of the Broken God...

  13. If my calculations are correct... by h33t+l4x0r · · Score: 3, Funny

    That comes to 4.84 Jiggawatts! No wonder there was outage.

    1. Re:If my calculations are correct... by Anonymous Coward · · Score: 0

      That comes to 4.84 Jiggawatts! No wonder there was outage.

      Wouldn't that be Gigawatts? I believe Christopher Lloyd was trapped in the gif vs.jif pronunciation of GIF.

    2. Re:If my calculations are correct... by binarylarry · · Score: 1

      Great Scott!

      --
      Mod me down, my New Earth Global Warmingist friends!
  14. But google said... by Chrisq · · Score: 1

    But google said that that it "....replicates data three times for redundancy. It can afford to be cavalier about hardware failures. So a drive fails. Log it, switch queries on that data to a replica and move on. It's all pretty instant".

    1. Re:But google said... by F.Ultra · · Score: 1

      That article is not about the Google Compute Cloud though.

    2. Re:But google said... by Anonymous Coward · · Score: 0

      Correct, and Amazon Web Services has had this same problem many times. Its not unexpected and you don't store UBER IMPORTANT DATA!1!111 on your web application running on the compute engine or AWS or anything. If you want that redundancy and guarantee, you use their backend storage services.

      if it can't be rebuilt with puppet/chef or some sort of script, don't host it on AWS or GCE or anything like this.

    3. Re:But google said... by F.Ultra · · Score: 1

      Exactly, I've had Amazon take down EC2 instances and then all data stored there went to /dev/null so if the data is important you have to do backup. That people are shocked about that is very strange.

  15. Re:What do you mean permenantly erased...only 1 DC by Viol8 · · Score: 2

    Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.

  16. Re:No lightning can wipe this haiku from memory by Anonymous Coward · · Score: 0

    You forgot to post anonymously, TheHaikuLover. Now we know who's been posting all this shit here in the past few months.

  17. After Trump Wins by Anonymous Coward · · Score: 0

    Google will have to accept full responsibility for bringing those data centers back to American soil.

  18. Re:What do you mean permenantly erased...only 1 DC by bledri · · Score: 5, Insightful

    Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.

    So faster than the speed of light using the infinitely-wide infinite improbability data bus?

    --
    Some privacy policy Slashdot.
  19. Also, adding a couple of lightning rods by Rinikusu · · Score: 1

    And a mad scientist with a cobbled together corpse.

    --
    If you were me, you'd be good lookin'. - six string samurai
  20. "Resilience"... remember that word... by Anonymous Coward · · Score: 0

    You'll be sick of hearing it soon... just like the bullshit word 'sustainability'...

    All part of the Jews' plan to totally destroy the West.

    Germany to take 800,000 'asylum seekers' over the next year. What joy. How 'democratic'.

  21. Re:Lightning is for cows. by Anonymous Coward · · Score: 0

    Sheep joke

  22. Strike one for the cloud! by Alumoi · · Score: 1

    Nuf said.

    1. Re:Strike one for the cloud! by Zak3056 · · Score: 1

      From the summary, it was at least strike four...

      --
      What part of "shall not be infringed" is so hard to understand?
  23. Full Responsibility by Anonymous Coward · · Score: 0

    That's not what "full responsibility" means. Accepting full responsibility would mean replacing any lost data with replacement data of equal value* to the dataholder, or with equivalent monetary compensation.

    *Those who have other backups and haven't lost data, would, of course be entitled to no compensation, as other than some unwanted downtime, no costs were incurred.

    *Those whose data was not valuable would also be entitled to no compensation.

    *Those whose data was valuable, but should have known better than to store it in the cloud with no backup, are guilty of negligence, and are also entitled not only to no compensation, but, in some cases, are also entitled to spend some time in jail.

  24. Re:Lightning is for cows. by Anonymous Coward · · Score: 0

    Moooooo *twitch* moo moo moo *twitch* *thud*

  25. location location location by Skapare · · Score: 1

    location location location ... all data should be replicated in at least THREE locations around the world.

    --
    now we need to go OSS in diesel cars
  26. Re:What do you mean permenantly erased...only 1 DC by Skapare · · Score: 2

    hours to replicate data? why so long? are they still running __________? (insert your most hated system/language)

    --
    now we need to go OSS in diesel cars
  27. Headline Failure by smallfries · · Score: 2

    Come on people. This has the potential to be legend... ary. What a complete failure.

    Even just form a quick punt we could glimpse such lyrical word play as:
    "Lightning strike inside Cloud"
    "Cloud damaged by lightning"
    "Cloud not lightning-proof"

    Please read the fucking Register until you gets it.

    --
    Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
  28. Re: What do you mean permenantly erased...only 1 D by Anonymous Coward · · Score: 0

    Even Google doesn't have unlimited bandwidth.

  29. Offsite backups by Meneth · · Score: 1

    This is why you need offsite backups, preferably on hardware under your own direct control.

    1. Re:Offsite backups by Antique+Geekmeister · · Score: 0

      Administered by a skilled storage, hardware, and network team. Training up such a team and keeping them employed, without managerial or technical errors destroying the whole system, for years, is far beyond the skills of most IT departments.

      I've built such offsite storage: I've helped people recover from the _inevitable_ sitewide disasters that overwhelm reasonable protections. Much like running your own mail server, Google has consistently outperformed and outlasted single company systems. Bulk data storage and handling is one of their core technologies, and this was a genuinely exceptional failure which very, very few home brew systems would have survived.

  30. Re:What do you mean permenantly erased...only 1 DC by Viol8 · · Score: 3, Informative

    *GUFFAW*

    You're wasted here, you should do stand up.

  31. Re:No lightning can wipe this haiku from memory by Anonymous Coward · · Score: 0

    According to posting history, this is the only thing the account has posted. Repeatedly.

  32. What?!! by Anonymous Coward · · Score: 0

    You mean The Cloud needs actual hardware to run?! That can't be right.

  33. Re:How about learning to spell? by Anonymous Coward · · Score: 0

    How about learning to meme?

    http://ct.weirdnutdaily.com/ol...

    Fool.

  34. Re:No lightning can wipe this haiku from memory by Anonymous Coward · · Score: 0

    At least his user name is appropriate...

  35. What did you expect? by johnsnails · · Score: 1

    That's why you don't put things in the cloud, put them in the aether.

  36. Re:What do you mean permenantly erased...only 1 DC by ScentCone · · Score: 2

    Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.

    You can have that services if you want to pay for it. You get that, right?

    --
    Don't disappoint your bird dog. Go to the range.
  37. Did you wipe the server clean? by Muntzsky · · Score: 1

    'What, like with a cloth or something?"

  38. Lightening is wierd stuff by Anonymous Coward · · Score: 0

    The telco's have quite a bit of history in protecting from it.
        They had a hard problem with all that outside copper entering the CO's.
          Now days with fiber, much easier, but still non-trivial.

    Sound like big G has a few lessons to learn about routing fault currents.
        I wonder if they have a published mitigation strategy?

    1. Re:Lightening is wierd stuff by bobbied · · Score: 1

      Actually, this seems to be a problem with their power redundancy systems not really a lighting protection problem.

      After three lighting induced power outages, the UPS ran out of reserve capacity so on the fourth power outage the UPS dropped the system's power.

      Really, what they have is a process failure... They should have their standby AC generator running until the UPS batteries where charged enough to safely handle another fail over process. Either that, or they should have quickly moved the processing out of the facility until full redundancy could be restored.

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
  39. HIllary by sycodon · · Score: 0

    ...is now requiring all of her email servers be hosted by Google.

    --
    When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
  40. Re:How about learning to spell? by Anonymous Coward · · Score: 0

    meme
    noun \mm\
    Definition of MEME
    : an idea, behavior, style, or usage that spreads from person to person within a culture

    You can't "meme", Moran.

  41. Lighting Ha? by Anonymous Coward · · Score: 0

    it was the NSA or CCHQ what better way to cover stealing the information than making people blieve it was wiped?

  42. Oxymoron by Dcnjoe60 · · Score: 1

    Despite the uncontrollable nature of the incident, Google has accepted full responsibility for the blackout and promises to upgrade its data center storage hardware, increasing its resilience against power outages.

    If it is uncontrollable, then any changes Google makes won't matter. On the other hand, if using other equipment, hardening the system, installing better grounding, etc. would have kept the loss from happening, then it is controllable. Maybe what they meant to say was unpredictable. Of course, then they would have had to explain why they didn't plan for the possibility.

    1. Re:Oxymoron by bobbied · · Score: 1

      No, not really. As I understand this, what happened is they experienced multiple failures of the AC power coming into the facility which happened to be lighting induced. When the power went out the first few times, the UPS's switched to battery power and everything kept going as the batteries provided the necessary power. Eventually, after repeated dependence on the UPS batteries during the multiple failures, the batteries had no more power left and the UPS output power stopped.

      This is really a process problem. Apparently nobody recognized that they had lost their redundant power system's battery capacity and took the necessary steps to move the processing to a system where full redundancy still remained. Their process failed them, although it apparently took *four* power failures to expose the issue.

      --
      "File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
    2. Re:Oxymoron by Anonymous Coward · · Score: 0

      Yeah, if that's what happened it's quite surprising. The UPS for our dinky systems here automatically keeps the gens in hot standby while battery energy is insufficient to allow cut over to cold gen with a big margin.

  43. like lightning on your wedding day by Thud457 · · Score: 1

    Wait...
    Are you saying Google users lost their data from the cloud due to lightning?
    Whoever scripts your reality down there is the most inept of hacks.

    --

    the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff

  44. Re:What do you mean permenantly erased...only 1 DC by Anonymous Coward · · Score: 0

    Yes. We were low on unobtanium, hence running on batteries.

  45. Re:What do you mean permenantly erased...only 1 DC by Anonymous Coward · · Score: 0

    Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.

    Wow, it's just not possible to keep people like you happy, is it?

    The customer explicitly and directly specifies NOT to distribute their data outside of the single data center, and you bitch and moan that when that single data center goes down they lose data.

    The customer explicitly and directly specifies NOT to distribute their data outside of the single data center, and Google does so anyway to meet your current seconds demand, and you will bitch and moan that Google doesn't honor user demands when explicitly made.

    Then you wonder why no company wants you as a customer!

  46. Re:How about learning to spell? by Coren22 · · Score: 1

    Around here, there is a guy that runs for sheriff (never really cared if he won) named Moran, it is entirely possible that the poster is a joke.

    http://www.washingtonpost.com/...

    All his signs say "Moran for sheriff", I always read it as Moron for sheriff and wondered why anyone would vote for a moron for sheriff.

    --
    APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
  47. So... by drunk_punk · · Score: 2

    If I understand this correctly, to get your personal data removed from Google search engines it requires 4 lightning strikes to the exact same location?

    Must have missed that part in the EULA...

  48. How much data loss? by Anonymous Coward · · Score: 0

    [I know I shouldn't do this here, but I've got a serious question.]

    Google's announcement about the damage includes the statement that "less than 0.000001% of PD space in europe-west1-b" was irrecoverably lost (PD is "persistent disk"). That's a very small fraction (~1GB out of 100PB, for those of us who commonly think in terms of storage size), but it obscures the actual number.

    [Here goes...get ready...prepare yourself...a serious question]
                          Does anyone know the actual storage capacity for GCE in europe-west1-b?
    [OK, done, now we can resume rampant speculation and juvenile gossip]

    Of course, loosing a small number of bytes may correspond to a much larger amount of data loss...if, for example, a few bytes of the encrypted Snowden archives or compressed Ashley Madison dump got flipped, the whole data set would be lost.

  49. no generators? by swschrad · · Score: 1

    come ON, google eyes... slack-writing disks is always a danger. you don't have generator backup for the battery backup, and you're slopping data all over the floor as a result?

    losers.

    --
    if this is supposed to be a new economy, how come they still want my old fashioned money?
  50. first bump should have taken the center offline by swschrad · · Score: 1

    and all data flushed to disk, and resources examined, before they opened for input again.

    --
    if this is supposed to be a new economy, how come they still want my old fashioned money?
  51. boom by Anonymous Coward · · Score: 0

    The electricity hit 4 times in rapid succession? The planet actually felt the presence of the localized power in the googleplex, such that the voltage struck there to 'equalize' the potential between it an the atmosphere? Cue random sci-fi electric jacob's ladder effects over all the blade runner landscapes..

  52. you've never been near high places, then, son by swschrad · · Score: 1

    at some point in your life, have you ever listened to AM radio or broadcast television? their stuff gets hit with lightning all the time. if a really good thunderstorm happens to float over their tower, AM radio tends to sound like "Weather Desk Radar shows the //SPLATbzzHummm// county, with //SPLATbzzHummm// t's the latest repor //SPLATbzzHummm//" with the transmitter knocked off the air every few seconds by a direct hit. in NTSC television, the return would be about 3-4 seconds of variable contrast and variable quality sound all over the place.

    many are the films and photos of multiple lightning hits on the Empire State Building, because it's tall enough and prominent enough that folks single it out for their photo assignments.

    --
    if this is supposed to be a new economy, how come they still want my old fashioned money?
  53. instead, check out polyphaser.com by swschrad · · Score: 2

    you left out about 99-44/100 percent of the technology and art of lightning protection.

    --
    if this is supposed to be a new economy, how come they still want my old fashioned money?
  54. depends on who lost data in slack-write centers by swschrad · · Score: 1

    if it's yours, I could give a shit.

    if it's mine, it's war.

    --
    if this is supposed to be a new economy, how come they still want my old fashioned money?
  55. Re:What do you mean permenantly erased...only 1 DC by sysrammer · · Score: 1

    You're wasted here, you should do stand up.

    And get hit by lightning?

    --
    His ignorance covered the whole earth like a blanket, and there was hardly a hole in it anywhere. - Mark Twain
  56. Re:How about learning to spell? by dcw3 · · Score: 1

    There's a Senator, and Representative by the same name. Speaks volumes for Congress.

    --
    Just another day in Paradise
  57. Just ask the NSA for it by Anonymous Coward · · Score: 0

    they have a copy

  58. Re:No lightning can wipe this haiku from memory by Anonymous Coward · · Score: 0

    Novelty accounts: not just for reddit!

  59. AC could be Jeff Bezos by Anonymous Coward · · Score: 0

    AC possibly works for Amazon, could even be Jeff.

  60. Really? 4 times? Court ordered data? by jimbob6 · · Score: 1

    Well that what Google gets for building a data center at the top of castle Frankenstein.

  61. The new excuse for dodgy politicians everywhere by Anonymous Coward · · Score: 0

    the lightning wiped my data.