GMail Experiences Serious Outage

JacobSteelsmith was one of many readers to note an ongoing problem with Gmail: "As I type this, GMail is experiencing a major outage. The application status page says there is a problem with GMail affecting a majority of its users. It states a resolution is expected within the next 1.2 hours (no, not a typo on my part). However, email can still be accessed via POP or IMAP, but not, it appears, through an Android device such as the G1." It's also affecting corporate users: Reader David Lechnyr writes "We run a hosted Google Apps system and have been receiving 502 Server Error responses for the past hour. The unusual thing about this is that our Google phone support rep (which paid accounts get) indicated that this outage is also affecting Google employees as well, making it difficult to coordinate."

Comments

Anti-Slashdot Effect

[ink]

Seems to be fine at the moment. Is this the first anti-slashdot-effect?

Re:Anti-Slashdot Effect

[sortius_nod]

Because your company and personal sandbox are valid representation of a mail system that serves millions of people. When either of your servers do that you can post bullshit like this.

Hell, even the company I work for has outages for both proactive and reactive maintenance, and that's only for 5000 people.

To say that because you've never had an outage you never will have an outage is absurd.

On top of this, saying that google should "have a backup" is silly. Do you even understand how redundancy works? Do you even understand how web based mail systems work? I really don't think so from this comment. If the error has nothing to do with servers falling over and is an issue with routing then you can have all the redundancy you want, but it won't make a difference.

At this stage it's any comments are merely conjecture, until google make a press release advising of what happened comments like "have a backup" are just troll posts.

Re:Anti-Slashdot Effect

[Mike Buddha]

Yeah, that's funny. You know what's also funny? The treadmill I bought 3 years ago and never used is in mint condtion. I've never had a problem with it sitting there under the pile of clothes in the corner. I read that 24 Hour Fitness has TONS of problems with their treadmills going down, but mine just keeps going without a single issue. I guess they just bought the wrong brand. Stupid idiots.

Re:Anti-Slashdot Effect

[jason.sweet]

ya'll'll

I'm from from Texas and that even makes my head hurt.
There's some poor schmuck in Bosnia right now googling how to pronounce it.

Re:Anti-Slashdot Effect

[drachenstern]

TL;DR - Southern boy using colloquialism, move along

Hahahaha, I'm from Houston, but grew up even further in the south (think mid Georgia/mid Alabama). Yeah, it's a little tricky, but it'll build character, no?

For the rest of the world, who don't quite grok our quaint pronunciation here in the good ole South, the word ya'll is pronounced like [yawl] similar to [yawn]. It should also be interesting to note that I have used one of the two forms of the conjunction, and I'm often told I use it wrong, but it's a little closer to how we pronounce it. The other spelling is y'all, but that would be pronounced like [ya-awl] and that's just a little to hick-ish even for me.

So if you can get to [yawl] then just tack an extra [ull] on the end and you'll have ya'll'll. You might notice that I tend to conjoin a lot of words, but that's just the spoken style where I've grown up, and as literary style derives from spoken style, well, yeah.

So, ya'll'll have to get a kick out of reading this, and just shake yer heads and mumble something about "that poor southern boy" and if you'd be so kind, drop a dollar in me alms cup as you pass by.

Ok, I've tried now to enlighten the world to some Southern-isms, and I tried to do at least part of it in properly written English, so we'll see. Also, I know it's WAY OT, so hit me with the mod, let's get this over with.

PS, it just means "you all will"

TL;DR - Southern boy using colloquialism, move along

Re:Anti-Slashdot Effect

[elashish14]

Maybe they should have thought twice about taking the beta label off!

Re:Anti-Slashdot Effect

[Grishnakh]

Because your company and personal sandbox are valid representation of a mail system that serves millions of people. When either of your servers do that you can post bullshit like this.

The parent poster's simple little postfix system doesn't NEED to serve millions of people. That's a feature: by not needing the immense complexity that goes along with running a web-based email system serving millions of people, his system is smaller, simpler, and less prone to problems.

It's impressive that Google's Gmail runs as well as it does given its size, but smaller, simpler solutions are almost always preferable. For company email (especially in a small company, not some behemoth company with 100k employees needing lots of mail servers), it simply makes more sense to use a small, simple mail server like the parent's postfix system, rather than to rely on some external vendor's multimillion-user system. Especially since the software needed to run that system is all available for free.

Re:Anti-Slashdot Effect

[Registered Coward v2]

So, ya'll'll have to get a kick out of reading this, and just shake yer heads and mumble something about "that poor southern boy" and if you'd be so kind, drop a dollar in me alms cup as you pass by.

Grammar Yankee Alert

The proper plural, when referring to a group, is all'ya'll'll

Indeed

[pablodiazgutierrez]

So much for handing your email over to Google because it's more reliable than hosting locally...

Re:Indeed

[DSW-128]

I dunno - I've been using G-Mail and Google Apps since each was introduced, and this is the first time one of their outages has impacted me, or anyone else that I talk to (true, that's not a lot of people, but...).

Re:Indeed

[tsotha]

Yeah, except that it has been extremely reliable. "Reliable" not being the same thing as "perfect".

Re:Indeed

[HangingChad]

So much for handing your email over to Google

We handed our mail over and it's the first time I've ever had a problem with them as a corporate mail provider. Almost two years. There may have been one other short outage, but I don't remember it being during business hours.

I doubt you could run a mail server more reliably. And, for the difference in cost, I'd stay with Gmail.

Re:Indeed

[ryanvm]

That was dumb. I have handed over our email because it's more reliable than hosting locally. This was the first time we've been affected in over a year and it was for a little more than an hour. That's an order of magnitude better uptime than we had before.

Can you beat Google's uptime? I doubt it. Sure, it's not impossible, but you won't be doing it for less than $50/user?

Re:Indeed

[ryanvm]

No problem.

Hi Boss -

I'm the guy that switched our email service to Google. See, it only costs us $50/year/user and this has been the first outage in over year. We used to pay a full time sysadmin to manage the mail server and would average about 12-20 hours of total downtime per year (maintenance, outages, etc.).

Obviously, the switch to Google has been much better for the corporate bottom line. Not to mention that we also get calendaring, wiki/sites, docs, and chat for the same price.

Ah, I'm glad you understand. You have a nice evening too.

Re:Indeed

[Facegarden]

I'm the guy that switched our email service to Google. See, it only costs us $50/year/user and this has been the first outage in over year...

Well, actually they had an outage in Feb of this year. And in May.

But they are awesome generally, I think their uptime over the past couple years has been beyond 99.99% or something crazy good.
-Taylor

Re:Indeed

[betterunixthanunix]

Except that instead of affecting one organization, this outage is affecting hundreds at a time.

Re:Indeed

[Anonymous Coward]

365 * 24 = 8760hrs/year "99.99% uptime" = 8759.124

So to meet 99.99% uptime, you have to have less than 52 minutes of downtime, total, planned and unplanned, in a year. That's really hard. Really. Think about it, few enterprise systems can rarely do that (Peoplesoft update in 50 minutes? HA!). But here, a 1.2hr outage puts them firmly out of the four nines club.

Not surprising

[Caustic Soda]

I don't know that this is actually news-worthy. I have never worked for a company which has not suffered email outages, no matter how their email is supported. Granted, GMail has a large list of client companies, but you are a fool of the highest order if you think the name will protect you from outages.

Re:Not surprising

[TheRealFixer]

I think the real story here is that it outlines the downside to moving everything to The Cloud, as a lot of people are trying to promote these days. As you said, email outages are pretty common even at large enterprises. The difference is, CIOs like to be able to go and yell at someone in their office for an outage, and know that it's being worked on in some measurable fashion. They don't like it when your answer is, "I don't know what's going on. Ask Google."

The Cloud is great, as long as it always works. But, in my experience, downtime is far less tolerated in hosted solutions than it is in on-site infrastructure. And stories like this make executives nervous about this stuff.

Good job, /.

[pablodiazgutierrez]

Great job slashdotting my email, dammit.

Re:Just a guess, but

[Kagura]

WTF how did you get access to my gmail account?

It's spotty

[bobdehnhardt]

As I type this, I can get in to GMail just fine, but a friend in Texas can't (I'm in Nevada). Guess Google likes us better.

And kudos to the Google team for updating the status when they say they will. Looks like the script they use automatically puts current time + 1 hour in as the default next update time, and they're posting updates before that expire. Too many times, something simple like that gets overlooked.

Depend on something... pay for admin

[johnjones]

I think Gmail is a great service for personal accounts...

but for business sorry you need to pay a real live person or support company who will actually be able to deal with your data

how do you get the data out of gmail to switch providers ?

ever serviced a discovery litigation from google ?
(you know where they judge you guilty of you dont come up with the data)

sorry but there is a good reason to keep this stuff on site and working...

regards

John Jones

Re:Depend on something... pay for admin

[Anonymous Coward]

.. Simply use a mail gateway..

http://www.google.com/support/a/bin/answer.py?hl=en&answer=60730

Was that so hard?

Re:Depend on something... pay for admin

[jcausey]

Feel like I'm feeding a troll, but johnjones's ID is so low that I feel this silliness may be taken seriously:

how do you get the data out of gmail to switch providers ?

Same way you would do any remote hosted email migration. POP and IMAP. Additional tools are provided for Google Apps (their for-pay version).

ever serviced a discovery litigation from google ? (you know where they judge you guilty of you dont come up with the data)

sorry but there is a good reason to keep this stuff on site and working...

Umm, an hour of downtime doesn't mean your data is gone. I'll also echo earlier comments -- locally hosted email generally has more problems, as no company but the largest enterprise has the same magnitude of IT equipment and experience as Google.

I've never really understood why so many Slashdotters have this attitude about hosted services. Perhaps they are local IT folks for smaller companies, and fear for their jobs?

Re:Depend on something... pay for admin

[ryanvm]

Bullshit.

I have been the full time sysadmin responsible for the mail server. I have had the job of keeping the mail service up. It's not cheap. You need redundant networking, redundant servers, redundant storage, redundant staff, and the glue to make sure it all works. For anyone spending less than a couple hundred thousand a year on IT, it's damn near impossible to beat Google's uptime for hosted mail.

As for your other concern about getting the data out of Gmail - you use the same protocols the rest of the Internet uses - IMAP/POP and SMTP. Not rocket science.

Re:Depend on something... pay for admin

[Atario]

I've never really understood why so many Slashdotters have this attitude about hosted services. Perhaps they are local IT folks for smaller companies, and fear for their jobs?

It's the same reason Slashdot has:

• such a large component of libertarians
• every tech/science story hit with a slew of +5ed comments questioning the basic underlying premise of the research and/or machinery
• every story about a study tagged with "correlationisnotcausation"
• etc.

...and that reason is that code-hackers, having succeeded in something most people find impossible, go on to generalize that they must simply be hypercompetent, and therefore anything done by others must be questionable by comparison. Thus, hosted services, being run by mere mortals, can't be as good as something set up by one's own brilliant self.

Re:Depend on something... pay for admin

[internic]

I think it's just the psychological impact of the lack of control. It's the same reason that people fear flying more than driving (one of the reasons, anyway) or that it's much scarier when you're the passenger during a dangerous maneuver than if you are driving the car and doing the same thing yourself.

Re:Depend on something... pay for admin

[Daniel Boisvert]

Umm, an hour of downtime doesn't mean your data is gone. I'll also echo earlier comments -- locally hosted email generally has more problems, as no company but the largest enterprise has the same magnitude of IT equipment and experience as Google.

I've never really understood why so many Slashdotters have this attitude about hosted services. Perhaps they are local IT folks for smaller companies, and fear for their jobs?

It's more than that. There are more moving and breakable parts between you and a hosted provider than between you and an internal service, which changes the math a bit.

Some of the single points of failure are shared between both approaches too, so they're a wash for a small implementation. If you're a small company and your non-redundant core switch fails, your email is down either way, because you can't get to your email server or to your hosted provider, no matter how redundant your provider is. There are various components for which this is true, which helps to mitigate the benefit of a hosted service where your mail server is replaced by a massively redundant cluster.

You also have additional dependencies. If you're a small business with a single T1 to the internet, let's say, and the telecom bunker outside your building catches fire and you lose internet access, you've got problems. With a local email service, internal mail works, but you can't send email to or receive email from external users (let's pretend you don't have an offsite secondary MX or an outbound mail spool where this stuff queues, mostly invisibly to users). For organizations that are hugely dependent on internal email, that's quite a bit better than having no access to your (hosted) email at all.

Additionally, you get concerns about "If we outsource this today and we have problems in 2 years, will we still have somebody here who can design/build/find a better solution, or will it cost us a fortune in consultants if we let the in-house expertise lapse?".

You also have support issues. Google specifically is well-known for only doing things that can be automated (and doing them well, mind you). Support isn't always one of those things, and small companies are well-acquainted with getting the shaft from vendors because your business isn't worth enough for them to care (check out the quality differences between the enterprise and SMB versions of various products for examples). Given the importance of email to most organizations today, folks are a bit reluctant to hand it over to an outsider with minimal financial incentive to devote resources to their specific problems.

If you're a 5-person business, outsourcing email is likely a good idea, but once you start getting into the teens and twenties or so, it's probably worth a look at your particular circumstances before continuing that assumption.

Full disclosure: I'm currently a local IT guy for a smaller company, with enough on my to-do list that if I thought outsourcing email would work well for my users and save us time & money, I'd be all over it.

Eating their own dog food

[Max Romantschuk]

It it's true that this outage is affecting Google too I have to say that is a good thing. Eating your own dog food, product-wise, is always a good idea.

eating your own dogfood

[Trepidity]

Upside: shows confidence in your products; makes it more likely that your engineers will spot problems if they use the software and services themselves; can increase how motivated people are to improve the products

Downside: tainted dogfood kills the engineers who would have investigated the issue

Re:eating your own dogfood

[Chyeld]

Solution: Hire food tasters, engineers should only be fed a set number of days after if the tasters aren't dead. Also resolves the issue of engineers complaining about being fed dog food.

Re:eating your own dogfood

[dkleinsc]

Solution: Make the beancounters eat the dogfood, not the engineers. Sure, you'll lose a few, but that's just solving another problem. Any BOFH would know that.

Re:Wow

[jellomizer]

Why would you fire the guy who caused it. He would probably be the most carful employee after that. People learn from mistakes firing people even for big mistakes isn't a solid business model and bad HR.

my domain via gmail

[The Yuckinator]

I drank the Google kool-aid about six months ago and moved my personal domain's mail over to the free gmail service. I've been extremely happy with it ever since.

I think it's interesting that I couldn't access my personal domain gmail during this outage, but my @gmail.com account worked without issue.

Re:Wow

[Gospodin]

That'd be a good name for a superhero: Apocryphal Guy. You always hear about his exploits but never actually see them.

Lemme call up Marvel. I mean, Disney.

Shades of AT&T, January 15, 1990

[Fencepost]

Most of the long distance in the country dropped that day, triggered by 4ESS switches hitting a bug, detecting, it, going offline (with load shifted to other switches). Increased load made the bug in question more likely to be hit, so those switches would in turn drop and shift load away (sometimes back to the originator). 9 hours of basically no long-distance service.

And just think, it was a year and a half before Berners-Lee announced the "World Wide Web" and Linus announced that he was working on this "Linux" thing.