Slashdot Mirror
Lightning Wipes Storage Disks At Google Data Center
An anonymous reader writes: Lightning struck a Google data center in Belgium four times in rapid succession last week, permanently erasing a small amount of users' data from the cloud. The affected disks were part of Google Computer Engine (GCE), a utility that lets people run virtual computers in the cloud on Google's servers. Despite the uncontrollable nature of the incident, Google has accepted full responsibility for the blackout and promises to upgrade its data center storage hardware, increasing its resilience against power outages.
Permanently erased? How can this be? Doesn't Google keep an off-site backup of my pr0n on tape or DVDs or sumpthin? So much for best practices, I guess.
Just use Amazon like everyone else. Google cannot be trusted, and I have said that many times. They 1) frequently decide to shut down services users rely on. One of the persistence mechanisms we depended on recently got the head shot, costing us so much money that we decided to move to Amazon, which has a standardized stack, and 2) data loss, and 3) non-existant customer service. Try contacting Google with a pressing issue.... you'll eventually give up.
Lightning struck the same place not twice, but four times?
... Alphabet's new "personal re-vivification" project is making good progress. The project leader, V.Frankenstein was unavailable for comment however.
The announcement is about Google Cloud Engine. Not about Google's own services (gmail, search, photos, docs, that sort of things). AFAIK, none of Google's own service announced any loss - presumably because they don't rely on a single location.
From the post:
> In particular, it was possible at all times to recreate new Persistent Disks from existing snapshots.
i.e. snapshots were fine.
> This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data.
So, if some poor users of GCE thought a single geographical location can withstand disasters, they now know.
n/t
who where what when now?
If you RTFA you'll see they mention it only affected "recently written data" that had not yet made it to persistent storage. So probably only a few hours old at most.
GCE is not a backup solution.
Google cloud data is geographically replicated.
That comes to 4.84 Jiggawatts! No wonder there was outage.
But google said that that it "....replicates data three times for redundancy. It can afford to be cavalier about hardware failures. So a drive fails. Log it, switch queries on that data to a replica and move on. It's all pretty instant".
Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.
Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.
So faster than the speed of light using the infinitely-wide infinite improbability data bus?
Some privacy policy Slashdot.
And a mad scientist with a cobbled together corpse.
If you were me, you'd be good lookin'. - six string samurai
Nuf said.
location location location ... all data should be replicated in at least THREE locations around the world.
now we need to go OSS in diesel cars
hours to replicate data? why so long? are they still running __________? (insert your most hated system/language)
now we need to go OSS in diesel cars
Come on people. This has the potential to be legend... ary. What a complete failure.
Even just form a quick punt we could glimpse such lyrical word play as:
"Lightning strike inside Cloud"
"Cloud damaged by lightning"
"Cloud not lightning-proof"
Please read the fucking Register until you gets it.
Slashdot: where don knuth is an idiot because he cant grasp the awesome power of php
I know that's what I base my critical data storage choices on - how fast a tangentially-related service's static front page loaded 15 years ago on dialup.
-- 'The' Lord and Master Bitman On High, Master Of All
This is why you need offsite backups, preferably on hardware under your own direct control.
Bullshit. I used to design high voltage connections, and tested using a 300kV impulse generator. I've seen a lot of crazy stuff analyzing field failures. You can greatly reduce the risk, but you cannot remove all risk in an above ground facility, as a practical matter.
I do see lots of silly stuff done, based on myth and lack of knowledge.
Place nail here >+
*GUFFAW*
You're wasted here, you should do stand up.
I'm afraid that had nothing to do with write speed for storage. It had a great deal to do with deliberately keeping the design very light, using very effective proxies for the very light and consistent images, and keeping tight reins on web designers who might want to front load the Google pages with exciting content that had no relationship to the service.
I'm confused why you mention it, unless you think that the wise practices and designs that led to such effective and quick interfaces affected their storage designs, as well.
That's why you don't put things in the cloud, put them in the aether.
Thats no excuse. It should be distributed amongst seperate machines in seperate centres instantaniously.
You can have that services if you want to pay for it. You get that, right?
Don't disappoint your bird dog. Go to the range.
'What, like with a cloth or something?"
Place Lightning Rod "A" in area "B", repeat until coverage and grounding complete.
Or maybe they should start digging, copy the NSA and have everything underground would solve most issues no ?
End of Line.
Despite the uncontrollable nature of the incident, Google has accepted full responsibility for the blackout and promises to upgrade its data center storage hardware, increasing its resilience against power outages.
If it is uncontrollable, then any changes Google makes won't matter. On the other hand, if using other equipment, hardening the system, installing better grounding, etc. would have kept the loss from happening, then it is controllable. Maybe what they meant to say was unpredictable. Of course, then they would have had to explain why they didn't plan for the possibility.
Wait...
Are you saying Google users lost their data from the cloud due to lightning?
Whoever scripts your reality down there is the most inept of hacks.
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
I base all my decisions on ACs responding to themselves. Splitting a single post up into three posts all acting like different people agreeing makes an AC look that much more trustworthy.
APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
Around here, there is a guy that runs for sheriff (never really cared if he won) named Moran, it is entirely possible that the poster is a joke.
http://www.washingtonpost.com/...
All his signs say "Moran for sheriff", I always read it as Moron for sheriff and wondered why anyone would vote for a moron for sheriff.
APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
Actually, this seems to be a problem with their power redundancy systems not really a lighting protection problem.
After three lighting induced power outages, the UPS ran out of reserve capacity so on the fourth power outage the UPS dropped the system's power.
Really, what they have is a process failure... They should have their standby AC generator running until the UPS batteries where charged enough to safely handle another fail over process. Either that, or they should have quickly moved the processing out of the facility until full redundancy could be restored.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
If I understand this correctly, to get your personal data removed from Google search engines it requires 4 lightning strikes to the exact same location?
Must have missed that part in the EULA...
come ON, google eyes... slack-writing disks is always a danger. you don't have generator backup for the battery backup, and you're slopping data all over the floor as a result?
losers.
if this is supposed to be a new economy, how come they still want my old fashioned money?
and all data flushed to disk, and resources examined, before they opened for input again.
if this is supposed to be a new economy, how come they still want my old fashioned money?
at some point in your life, have you ever listened to AM radio or broadcast television? their stuff gets hit with lightning all the time. if a really good thunderstorm happens to float over their tower, AM radio tends to sound like "Weather Desk Radar shows the //SPLATbzzHummm// county, with //SPLATbzzHummm// t's the latest repor //SPLATbzzHummm//" with the transmitter knocked off the air every few seconds by a direct hit. in NTSC television, the return would be about 3-4 seconds of variable contrast and variable quality sound all over the place.
many are the films and photos of multiple lightning hits on the Empire State Building, because it's tall enough and prominent enough that folks single it out for their photo assignments.
if this is supposed to be a new economy, how come they still want my old fashioned money?
you left out about 99-44/100 percent of the technology and art of lightning protection.
if this is supposed to be a new economy, how come they still want my old fashioned money?
if it's yours, I could give a shit.
if it's mine, it's war.
if this is supposed to be a new economy, how come they still want my old fashioned money?
You're wasted here, you should do stand up.
And get hit by lightning?
His ignorance covered the whole earth like a blanket, and there was hardly a hole in it anywhere. - Mark Twain
There's a Senator, and Representative by the same name. Speaks volumes for Congress.
Just another day in Paradise
Well that what Google gets for building a data center at the top of castle Frankenstein.
Oh Lord Bitman. One person said ALL of google is shit based on shit The responder said, you must be too young to remember what google gave us, and how it was better than everything. How its search engine set the bar, and how its free email was a gift to many many people. And you say...non-sequitur in this case. My critical storage needs!!!