Confirmed Gmail / Google App Outage

mbone writes "Earlier today there was a confirmed Google outage which got a lot of attention from network operators. From a post to NANOG after everything calmed down: 'Google ack'd a maintenance on their core network did not go as planned-Forced traffic to one peer link that was unable to handle all the traffic. Maintenance has been rolled back. Issue has been restored.' This is exactly what makes me nervous about cloud computing and data storage. It's bad enough when I screw up a config and it takes down my mail, but what about when it happens to the entire globe at once?" Several readers also point to CNET's coverage of the outage. Update: 05/14 19:25 GMT by T : CWmike adds this: "Steven J. Vaughan-Nichols writes that what may be happening is a massive DDoS attack. Based on the size of the attack that would be needed to interfere with Google, I believe that it's quite likely to be the result of an attack from the controllers of the Windows worm, Conficker. Another theory that has been put about — that the problem was due to AT&T NOC routing problems — does not appear to hold water, writes Steven." Update: 05/14 21:01 GMT by T : Google's put up a low-detail explanation on their blog that says "An error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam. As a result, about 14% of our users experienced slow services or even interruptions."

Comments

comments from google

[JWSmythe]

    In comments from Google Admins, they said "oops." :)

Re:comments from google

[TommydCat]

We apologise for the fault in the comments. Those responsible have been sacked.

We have to take an deep breath...

[WarwickRyan]

...and take an stroll to the great big place known as "outside".

e-mails down?

[Anonymous Coward]

call me....

Hey Chicken Little

[llZENll]

And yet somehow miraculously we are all still alive. The sky is not falling!

Mail Servers

[Aladrin]

Having run my own mail server, and used mail servers run by companies I work for, I'll -gladly- take GMail's track record for reliability. Even with no 'guarantee', it's been a hell of a lot better than anything else I've experienced.

And what's -really- the difference between a server going down locally that affects you and a server going down globally that affects you? Nothing.

Re:Mail Servers

[ACMENEWSLLC]

>>And what's -really- the difference between a server going down locally that affects you and a server going down globally that affects you? Nothing.

Actually, I disagree. There is a difference. If it's local and I own it, I have to fix it. If it's outsourced and Google owns it, I sit back and let Google fix it. Which is nice.

ThePlanet.com had a bad switch install a few days ago which brought down part of our cloud. Our website was down, as was our access to Google DNS gave an IP down there for Google. If you look at the last year, the cloud solution has had a better uptime than what I was providing computing in planned maintenance, patching, updates and all.

It was nice to leave at 5pm, knowing ThePlanet would fix the switch and get us back up. And they did. It's a lot easier to gripe about the cloud being down and sit back, than to manage and fix your own local servers switches and such. When you get to managing hundreds of servers, it becomes time to know what to outsource.

Re:Mail Servers

[Cro Magnon]

The flip side is, if it's local and important to me, I'll make fixing it a priority. If it's important to me and I DON'T have control, I just have to hope that it's reasonably important to whoever can fix it, or I'm screwed.

Re:Google vs. Twitter...Once in an e-lifetime.

[SBrach]

Using a Pentium I see.

Re:Google Voice Issues

[MickyTheIdiot]

I noticed that when I did an images.google.com search for "hot anime chicks" it showed me a picture of Cowboy Neal.

So yeah, it did cause issues on all of Google's services. It also soiled the inside of my trash can with puke.

Re:The news is not that Google went down

[sharkey]

It'll get better once it's out of Beta.