Slashdot Mirror
Google Blames Gmail Troubles On Maintenance Goof
Slatterz writes "Google has apologised for the two-and-a-half-hour Gmail outage on Tuesday morning, and admitted that the cause was down to data center maintenance. 'Lots of people around the world who rely on Gmail were disrupted during their waking and working hours, and we are very sorry. We did everything we could to restore access as soon as possible, and the issue is now resolved,' said Gmail site reliability manager Acacio Cruz in a blog post. Google had been testing new code designed to keep data geographically closer to its owner, which brought about disruption when maintenance in one data center caused another facility to be overloaded. This had a cascade effect, according to Google, and it took the company an hour to get it back under control."
Gmail = Goofmail
Sometimes it feels like the whole world is pissed at the outages my maintenance and site updates cause.
At least Google has the money to afford a proper load balanced / never down system...
Maybe it's related to this but I noticed this past weekend that the Jabber server running on my Linux machine no longer can get presence information for people on GMail/GTalk. From the logs I can see my server attempting to make a connection but nothing happens after 20 seconds and my server gives up for the time being. I haven't changed anything on my side but I'm unsure who to contact about issues like these.
the cloud can breakdown? WTF? I thought cloud computing fixed any conceived computer problem out there.
damn marketing bs...
greed@All_Evils:~#
So you're saying I *didn't* need to throw my iPhone out the car window the other day? I hit some poor lady right on the noggin with it.
Punch drunk, and without bail.
I mean, sure, if the janitor brought down the service, that's pretty bad, but it seems a bit harsh to start calling him a "maintenance goof" ...
(tip your bartenders and waiters)
.. Gmail is Beta or something.
Nobody complain about that silly beta label anymore.
"As the stunned world slowly recovers from 2.5 hours of complete hibernation, digging through wreckage, restarting life support systems we all came to depend on, re-animating accidentally dead and restoring their brains from backups (provided backups are available and reasonably error-free), Google has apologised for causing 'the disruption' and blamed it on a maintenance goof in the Google Cloud, said GCloud site reliability manager Acacio Cruz IV v10.0.013 in a BrainTwitter post. We can only envy our ancestors who used to just lose access to their electronic mail via primitive personal computers when Google was having a glitch."
They were hacked.
This was Google's way of telling the world that if they are going to host your company's email work load, please consider paying for it!
Makes sense to me with them rolling out their hosting plans soon.
The first thing that came to mind when reading the article is, "They were 'testing' code in a fscking production environment?!" Then I realized that Gmail is still a beta app. I think these things are to be expected from beta software. What I'm curious about is whether or not corporate users who are paying for Gmail were effected as well. If so, then Google better get their ducks in a row, and fast. It's one thing to play around with your servers when people aren't paying you for uptime. It's another thing entirely to test code on a production network.
for their lame layout - should give people a way to avoid (or change) the styled buttons, not all of us can easily read them now.
If Google really cared they would fix Android Chrome to reflow text, instead of discriminating
Sorry, My bad.
First off, it's free, it gives you 7 Gigs of mail storage and it's accessible from any where or any device with an Internet connection.
It searches through my 4 years of e-mail faster than Outlook ( in cached Exchange mode) can search
the last week. They keep adding features - for free;
have no annoying Flash ads and the ones they do have are off on the extremes of the page.
If you don't like it, stop using them - I promise you there won't be any pesky cancellation fees.
Hotmail and Yahoo await you and we'll miss you all - maybe.
Pain is merely failure leaving the body
I don't understand people who rely on Gmail, or any other free webmail, as their primary and business-critical point of contact. There is no SLA, no contractual obligation, no guarantee of anything. Anything can happen to your email and there's absolutely nothing you can do about it.
The logic is quite simple: if you can't live without something, then get a guarantee in writing, and pay the premium for that extra service. In Gmail's case, there is no premium service, so you'd better start looking elsewhere.
It's cute that the Google folks are apologetic, but what they should have said is "Shut the fuck up, we don't owe you a damn thing. Do you know how much of a pain it is to run a public mail service ? Be grateful we give you free email at all, you whiny little attention whores.". I wish!
-Billco, Fnarg.com
Not sure how wide spread this is, but I use OpenDNS both at home and at the office as my resolving name servers. Recently some ass hat apparently set gmail.com on OpenDNS's filters. Labeled it as a Webmail client. So, for the past 2 days I couldn't get logged on to my Gmail account while at the office, kept saying login failure. But at home it would work fine. I changed to the company's internal DNS servers for resolving and suddenly my Gmail would connect... So, anyone using OpenDNS and still not able to connect might look into that. I have sent OpenDNS admins a request to re-check that filter... It's kinda pointless to just block everything that someone *thinks* should be blocked.
The day Microsoft creates a product that doesn't suck, it will be known as the Microsoft Vaccuum Cleaner!
They want people to use gmail, which is of course the reason they offer it in the first place. They make a significant profit off it, and would lose money if they drove away users.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
It's alot worse than you seem to know.
I've been having problems with gmail for 4 days now. My mail STILL isn't being delivered.
I have sent two emails a day (morning and evening) to my Yahoo account over the last 4 days.
None have been delivered. This still isn't fixed.
Yes it's damn annoying when email or some other part of your critical infrastructure goes out, but this really should have been planned for in advance. Not by google but by you.
Things happen. Things that are out of our control but we still have to deal with them. This outage was quite short for most people. A day at the most from what I hear, but what if the outage had been longer? A week? A month? How would you have dealt with it?
I always keep a few lists of things to do, people to call, things to write should a business disturbing event occur.
So should any business that want to continue living do too. And so should probably you!
You are not entitled to your opinion. You are entitled to your informed opinion. -- Harlan Ellison
a failure in a piece leads to a global failure.
Oh!, I think we were talking about economics here.
Too bad they don't tell more details. Their software can withstand lots of problems: network partitions, data center outages, failing routers, etc. This time, a new piece of of algorithm apparently did not do a very good job at redistributing data at the time of the data center failure. I'd like to know what it tried to do? Did it try to push too much data to one single location, causing that location to become unresponsive, in turn causing it to start redistributing data as well? I'm glad they didn't loose my email. But I think now is a good time to start making backups using IMAP.
Maybe if there was some way to contact Google when a problem arises?
And no, forums do not count.
As much money as they have, they should have a HUGE customer support staff working 24/7.
Am I the only one who noticed it should be apologized not apologised?
I don't have the tech know-how to know why, but I do note that since the blackout, a cascade of spam has sputtered off to nearly zero. Were they shutting a big hole somewhere?
It can pretty much kick your ass and fuck your mum.
Just like your mum, Gmail supports millions of users.
"I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)
If only all global problems could be fixed in an hour or two.
In the same datacentres as the production servers.
Or something like that.
IANAL but write like a drunk one.
Except Gmail is NOT free if you are a PAID user and they were just as affected as everyone else.