Google Blames Gmail Troubles On Maintenance Goof

Slatterz writes "Google has apologised for the two-and-a-half-hour Gmail outage on Tuesday morning, and admitted that the cause was down to data center maintenance. 'Lots of people around the world who rely on Gmail were disrupted during their waking and working hours, and we are very sorry. We did everything we could to restore access as soon as possible, and the issue is now resolved,' said Gmail site reliability manager Acacio Cruz in a blog post. Google had been testing new code designed to keep data geographically closer to its owner, which brought about disruption when maintenance in one data center caused another facility to be overloaded. This had a cascade effect, according to Google, and it took the company an hour to get it back under control."

Comments

Mail time happy time.

[orsty3001]

Gmail = Goofmail

Problems with Jabber connections to GMail users

[Jon_Hanson]

Maybe it's related to this but I noticed this past weekend that the Jabber server running on my Linux machine no longer can get presence information for people on GMail/GTalk. From the logs I can see my server attempting to make a connection but nothing happens after 20 seconds and my server gives up for the time being. I haven't changed anything on my side but I'm unsure who to contact about issues like these.

Re:Problems with Jabber connections to GMail users

[Achromatic1978]

Not sure, but it's not just you. I use a couple of different clients for work communication (the dozen people in my company telecommute) - Gtalk, Trillian, Adium, and fring on my cellphone. My boss got snarky one day recently because he said my status said available, but I was unresponsive all day. After investigating, I saw the same thing. If I set my status away on any of the non-official clients, it wouldn't propogate out. So it's a two way issue, not just one.

Re:Problems with Jabber connections to GMail users

[emag]

Been this way since at least last Thursday (Feb 18) for me. I have several contacts ($grandboss, $director (who's out sick), and $wife among others) that insist on using GMail/GTalk, all of them went "remote-server-not-found" last week, with no changes on my end. As a lark, I restarted my XMPP server, without it making a difference. If I had to guess, server federation was deactivated on the Google end, out that's just a WAG on my part.

Re:Problems with Jabber connections to GMail users

[SCPRedMage]

see what i did there? i mixed some code in appropriately (= instead of is saves me one character)

Which you promptly wasted by explaining what a jackass you are...

So you're saying...

[Narnie]

the cloud can breakdown? WTF? I thought cloud computing fixed any conceived computer problem out there.

damn marketing bs...

Thanks a LOT Google...

[Tarmus]

So you're saying I *didn't* need to throw my iPhone out the car window the other day? I hit some poor lady right on the noggin with it.

Re:Thanks a LOT Google...

[Anonymous Coward]

it's ok, some poor lady got an iphone that should would have never been able to afford otherwise. cause she's poor.

"Maintenance goof?"

[mea37]

I mean, sure, if the janitor brought down the service, that's pretty bad, but it seems a bit harsh to start calling him a "maintenance goof" ...

(tip your bartenders and waiters)

It's almost like..

[Anonymous Coward]

.. Gmail is Beta or something.

This is why we're still beta.

[buddyglass]

Nobody complain about that silly beta label anymore.

Fast-forward 100 years...

[glebd]

"As the stunned world slowly recovers from 2.5 hours of complete hibernation, digging through wreckage, restarting life support systems we all came to depend on, re-animating accidentally dead and restoring their brains from backups (provided backups are available and reasonably error-free), Google has apologised for causing 'the disruption' and blamed it on a maintenance goof in the Google Cloud, said GCloud site reliability manager Acacio Cruz IV v10.0.013 in a BrainTwitter post. We can only envy our ancestors who used to just lose access to their electronic mail via primitive personal computers when Google was having a glitch."

Re:Fast-forward 100 years...

[jnuzzo]

It's a FREE service. I don't have a problem with an outage when the service is free. It's when I pay for a premium service, they can't keep it stable, and finally raise my rate to cover their idiocy that p*sses me off.

Re:Fast-forward 100 years...

[Idiomatick]

Google has a better uptime than any pay email service I know of anyways. I'm pegging Gmail at around 4 9s and the search engine to around 5~6. Most ISPs get 2 9s and business ISPs 3, 4 if you hit the ISP jackpot (even then i'd be shocked). So how are people so up in arms about this? Sheesh if you saved it up you'd get a day off once every 30 years. Which reminds me, nevermind ISPs. You get less uptime for ELECTRICITY in north america than you do google, going just on the continent wide outage a few years back.

Re:Fast-forward 100 years...

[Anonymous Coward]

How did you calculate 4 nine's for gMail? 4 9's is 52 minutes of downtime per year, while this outage was over 2 hours.

And this isn't their first outage. The last one I remember was April of 2008.

Is it even possible to measure 6 9's of downtime for an internet service? 6 9's is just 30 seconds of downtime per year -- less than 3 seconds per month -- 100 msec/day. Can you honestly say that you never experience 100 msec of additional latency once a day? Maybe once a month they have a hard disk timeout that makes a query take 3 seconds instead of a fraction of a second. Can you tell if this is a problem with that service, or somewhere on the general internet? Even 5 nines is less than one second per day.

And measuring availability of a search engine is tricky anyway since the search engine database is constantly changing. If you are connected to a google server in that just happens to be having an indexing problem and is 12 hours out of date, how would you even know that is the case. Your search may fail to find the recently updated website that you're looking for but you don't know it's because there's an indexing problem. So you may say that the search engine is up, but in reality it's not able to give you the result you're looking for so.

Re:Fast-forward 100 years...

[QuantumRiff]

It was the eastern seaboard. Most east coasters think the Pacific ocean is just past Chicago, hence why they call it the "mid-west" instead of "Almost-Central"

Re:Fast-forward 100 years...

[Anonymous Coward]

And this isn't their first outage. The last one I remember was April of 2008.

Having seen forum threads around the internet discussing gmail downtimes in the past a general trend is, that only one or two persons see an outage, everybody else can access gmail just fine. That makes me think the majority of their downtimes only affect a tiny fraction of their users. If you count all outages even though they affected maybe just 1% of users, then you are not giving correct availability figures. If 1% of the time there is an outage for 1% of the users, the availability isn't 0.99, it is 0.9999. This latest outage was an exception, I don't think gmail ever before had an outage affecting the same number of users.

Re:I know the feeling...

[ani23]

well about the never down part

Beta = Test Environment

[dave562]

The first thing that came to mind when reading the article is, "They were 'testing' code in a fscking production environment?!" Then I realized that Gmail is still a beta app. I think these things are to be expected from beta software. What I'm curious about is whether or not corporate users who are paying for Gmail were effected as well. If so, then Google better get their ducks in a row, and fast. It's one thing to play around with your servers when people aren't paying you for uptime. It's another thing entirely to test code on a production network.

Re:Beta = Test Environment

[sloth jr]

There are all manner of tests, and sooner or later, you do have to test in production. It's important to know that in cloud computing, there are certain kinds of tests that are only possible in production; production load is the surest way to characterize your application and platform. Who knows where in the deployment lifecycle this happened? Someone at Google, certainly, but not us.

Re:Beta = Test Environment

[PotatoFarmer]

The fact that they have corporate accounts paying for access to the service should preclude the 'beta' label. I like a lot of what Google has done, but sometimes it seems like the whole beta thing is just a convenient excuse for failure, or as a free pass for iffy behavior like testing in production.

Re:Beta = Test Environment

[jadin]

I don't think they are testing it on their corporate users. My domain is signed up for google apps which includes email, but not the pay for premium version. When I read on slashdot that gmail was finally adding an option for 'always use https connection', I looked in the options where people said it would be, and found nothing. Logging into the "official" gmail I was able to find it right away. It took some time before it showed up in my domain's gmail client.

My conclusion is they test all the code on the official gmail users to make sure it's stable enough before updating the corporate clients etc.

Re:Beta = Test Environment

[BikeHelmet]

It wasn't 2.5 hours for me - it was more like 14-15 hours.

It stopped working at night time, around ~9PM (this is when Gmail Notifier failed to login, and curious, I tried to login manually). It wasn't working yet at 2AM in the morning. I went to sleep, woke up, and it was still broken. It finally came back online some time after lunch.

This would be quite irritating if I were a business. As it was, I did have some important emails to send off, but waiting a day didn't kill me.

Re:Beta = Test Environment

[lefiz]

Google is an innovative company, and innovation often includes trial and error as well as improvements to an original idea. No one makes you use their products, and in this case, gmail is only one of many email providers. If you would prefer slightly more reliability from a corporation providing a product guarantee, feel free to look elsewhere. I like the way gmail works much more than any other email app I've tried, and am happy to accept the occasional issue, especially for all of the positive developments that have come from continued work on the project. Remember not so long ago when you couldn't chat in gmail?

Re:Beta = Test Environment

[Strake]

Be thankful that, at least, Google calls their testing versions "beta", not "Sevice Pack n" | n < 2.

Re:Beta = Test Environment

[digitalchinky]

You strike me as fitting the stereotype I have of the typical user at my place of work. A service can be running for months (or even years) on end without a single glitch, yet the thing goes down for a little bit and they act like the previous aeons of total reliability never happened. Ever. A good many even go so far as actually saying "insert some service or other" never fucking works, it's always broken! This lie filters up the chain and two days later you're in the big office explaining shit you should never have to explain.

I really doubt more reliable service providers actually exist in any great number for it to matter to you - in a nutshell - you put all your eggs in one basket and you suffered for 24 hours or so. For me the downtime really was just 2 hours 30 minutes, and I'm inclined to believe Google are telling the truth on this - I suspect some of your problems lay elsewhere, I'd start with your ISP.

My bad.

[Maintenance Goof]

Sorry, My bad.

Re:My bad.

[Maintenance Goof]

Yep, I could not resist, 12 years lurking, ruined in an instant.

Stop complaining people, it wasn't that big a deal

[haruchai]

First off, it's free, it gives you 7 Gigs of mail storage and it's accessible from any where or any device with an Internet connection.
It searches through my 4 years of e-mail faster than Outlook ( in cached Exchange mode) can search
the last week. They keep adding features - for free;
have no annoying Flash ads and the ones they do have are off on the extremes of the page.

If you don't like it, stop using them - I promise you there won't be any pesky cancellation fees.
Hotmail and Yahoo await you and we'll miss you all - maybe.

Actually another problem may exist...

[Puppet Master]

Not sure how wide spread this is, but I use OpenDNS both at home and at the office as my resolving name servers. Recently some ass hat apparently set gmail.com on OpenDNS's filters. Labeled it as a Webmail client. So, for the past 2 days I couldn't get logged on to my Gmail account while at the office, kept saying login failure. But at home it would work fine. I changed to the company's internal DNS servers for resolving and suddenly my Gmail would connect... So, anyone using OpenDNS and still not able to connect might look into that. I have sent OpenDNS admins a request to re-check that filter... It's kinda pointless to just block everything that someone *thinks* should be blocked.

there's a reason they didn't say that

[Trepidity]

They want people to use gmail, which is of course the reason they offer it in the first place. They make a significant profit off it, and would lose money if they drove away users.

Re:Gmail

[MobileTatsu-NJG]

It can pretty much kick your ass and fuck your mum.

Just like your mum, Gmail supports millions of users.