GMail Experiences Serious Outage

JacobSteelsmith was one of many readers to note an ongoing problem with Gmail: "As I type this, GMail is experiencing a major outage. The application status page says there is a problem with GMail affecting a majority of its users. It states a resolution is expected within the next 1.2 hours (no, not a typo on my part). However, email can still be accessed via POP or IMAP, but not, it appears, through an Android device such as the G1." It's also affecting corporate users: Reader David Lechnyr writes "We run a hosted Google Apps system and have been receiving 502 Server Error responses for the past hour. The unusual thing about this is that our Google phone support rep (which paid accounts get) indicated that this outage is also affecting Google employees as well, making it difficult to coordinate."

Comments

Indeed

[pablodiazgutierrez]

So much for handing your email over to Google because it's more reliable than hosting locally...

Re:Indeed

[DSW-128]

I dunno - I've been using G-Mail and Google Apps since each was introduced, and this is the first time one of their outages has impacted me, or anyone else that I talk to (true, that's not a lot of people, but...).

Re:Indeed

[tsotha]

Yeah, except that it has been extremely reliable. "Reliable" not being the same thing as "perfect".

Re:Indeed

[ryanvm]

That was dumb. I have handed over our email because it's more reliable than hosting locally. This was the first time we've been affected in over a year and it was for a little more than an hour. That's an order of magnitude better uptime than we had before.

Can you beat Google's uptime? I doubt it. Sure, it's not impossible, but you won't be doing it for less than $50/user?

Re:Indeed

[ryanvm]

No problem.

Hi Boss -

I'm the guy that switched our email service to Google. See, it only costs us $50/year/user and this has been the first outage in over year. We used to pay a full time sysadmin to manage the mail server and would average about 12-20 hours of total downtime per year (maintenance, outages, etc.).

Obviously, the switch to Google has been much better for the corporate bottom line. Not to mention that we also get calendaring, wiki/sites, docs, and chat for the same price.

Ah, I'm glad you understand. You have a nice evening too.

Re:Indeed

[betterunixthanunix]

Except that instead of affecting one organization, this outage is affecting hundreds at a time.

Re:Indeed

[Anonymous Coward]

365 * 24 = 8760hrs/year "99.99% uptime" = 8759.124

So to meet 99.99% uptime, you have to have less than 52 minutes of downtime, total, planned and unplanned, in a year. That's really hard. Really. Think about it, few enterprise systems can rarely do that (Peoplesoft update in 50 minutes? HA!). But here, a 1.2hr outage puts them firmly out of the four nines club.

Not surprising

[Caustic Soda]

I don't know that this is actually news-worthy. I have never worked for a company which has not suffered email outages, no matter how their email is supported. Granted, GMail has a large list of client companies, but you are a fool of the highest order if you think the name will protect you from outages.

Re:Not surprising

[TheRealFixer]

I think the real story here is that it outlines the downside to moving everything to The Cloud, as a lot of people are trying to promote these days. As you said, email outages are pretty common even at large enterprises. The difference is, CIOs like to be able to go and yell at someone in their office for an outage, and know that it's being worked on in some measurable fashion. They don't like it when your answer is, "I don't know what's going on. Ask Google."

The Cloud is great, as long as it always works. But, in my experience, downtime is far less tolerated in hosted solutions than it is in on-site infrastructure. And stories like this make executives nervous about this stuff.

It's spotty

[bobdehnhardt]

As I type this, I can get in to GMail just fine, but a friend in Texas can't (I'm in Nevada). Guess Google likes us better.

And kudos to the Google team for updating the status when they say they will. Looks like the script they use automatically puts current time + 1 hour in as the default next update time, and they're posting updates before that expire. Too many times, something simple like that gets overlooked.

Depend on something... pay for admin

[johnjones]

I think Gmail is a great service for personal accounts...

but for business sorry you need to pay a real live person or support company who will actually be able to deal with your data

how do you get the data out of gmail to switch providers ?

ever serviced a discovery litigation from google ?
(you know where they judge you guilty of you dont come up with the data)

sorry but there is a good reason to keep this stuff on site and working...

regards

John Jones

Re:Depend on something... pay for admin

[jcausey]

Feel like I'm feeding a troll, but johnjones's ID is so low that I feel this silliness may be taken seriously:

how do you get the data out of gmail to switch providers ?

Same way you would do any remote hosted email migration. POP and IMAP. Additional tools are provided for Google Apps (their for-pay version).

ever serviced a discovery litigation from google ? (you know where they judge you guilty of you dont come up with the data)

sorry but there is a good reason to keep this stuff on site and working...

Umm, an hour of downtime doesn't mean your data is gone. I'll also echo earlier comments -- locally hosted email generally has more problems, as no company but the largest enterprise has the same magnitude of IT equipment and experience as Google.

I've never really understood why so many Slashdotters have this attitude about hosted services. Perhaps they are local IT folks for smaller companies, and fear for their jobs?

Eating their own dog food

[Max Romantschuk]

It it's true that this outage is affecting Google too I have to say that is a good thing. Eating your own dog food, product-wise, is always a good idea.

eating your own dogfood

[Trepidity]

Upside: shows confidence in your products; makes it more likely that your engineers will spot problems if they use the software and services themselves; can increase how motivated people are to improve the products

Downside: tainted dogfood kills the engineers who would have investigated the issue

Re:Wow

[jellomizer]

Why would you fire the guy who caused it. He would probably be the most carful employee after that. People learn from mistakes firing people even for big mistakes isn't a solid business model and bad HR.

Re:Anti-Slashdot Effect

[sortius_nod]

Because your company and personal sandbox are valid representation of a mail system that serves millions of people. When either of your servers do that you can post bullshit like this.

Hell, even the company I work for has outages for both proactive and reactive maintenance, and that's only for 5000 people.

To say that because you've never had an outage you never will have an outage is absurd.

On top of this, saying that google should "have a backup" is silly. Do you even understand how redundancy works? Do you even understand how web based mail systems work? I really don't think so from this comment. If the error has nothing to do with servers falling over and is an issue with routing then you can have all the redundancy you want, but it won't make a difference.

At this stage it's any comments are merely conjecture, until google make a press release advising of what happened comments like "have a backup" are just troll posts.

Re:Anti-Slashdot Effect

[Mike Buddha]

Yeah, that's funny. You know what's also funny? The treadmill I bought 3 years ago and never used is in mint condtion. I've never had a problem with it sitting there under the pile of clothes in the corner. I read that 24 Hour Fitness has TONS of problems with their treadmills going down, but mine just keeps going without a single issue. I guess they just bought the wrong brand. Stupid idiots.

Re:Anti-Slashdot Effect

[Grishnakh]

Because your company and personal sandbox are valid representation of a mail system that serves millions of people. When either of your servers do that you can post bullshit like this.

The parent poster's simple little postfix system doesn't NEED to serve millions of people. That's a feature: by not needing the immense complexity that goes along with running a web-based email system serving millions of people, his system is smaller, simpler, and less prone to problems.

It's impressive that Google's Gmail runs as well as it does given its size, but smaller, simpler solutions are almost always preferable. For company email (especially in a small company, not some behemoth company with 100k employees needing lots of mail servers), it simply makes more sense to use a small, simple mail server like the parent's postfix system, rather than to rely on some external vendor's multimillion-user system. Especially since the software needed to run that system is all available for free.