Extended Gmail Outage Frustrates Admins

CWmike writes "A prolonged, ongoing Gmail outage has some Google Apps administrators pulling their hair out as their end users, including high-ranking executives, complain loudly while they wait for service to be restored. At about 5 p.m. US Eastern on Wednesday, Google announced that the company was aware of the problem preventing Gmail users from logging into their accounts and that it expected to fix it by 9 p.m. on Thursday. Google offered no explanation of the problem or why it would take it so long to solve the problem, a '502' error when trying to access Gmail. Google said the bug is affecting 'a small number of users,' but that is little comfort for Google Apps administrators. Admin Bill W. posted a desperate message on the forum Thursday morning, saying his company's CEO is steaming about being locked out of his e-mail account since around 4 p.m. on Wednesday. It's not the first Gmail outage. So, will this one prompt calls for a service-level agreement for paying customers? And a more immediate question: Why no Gears for offline Gmail access at very least, Google?"

Comments

The benefits of cloud computing

[mlwmohawk]

Someone else deals with all the problems, right?

Re:The benefits of cloud computing

[Thelasko]

Remember folks, it's still in beta!

Re:The benefits of cloud computing

[jellomizer]

Yes. In your organization how many times have your servers went down or had a problem... Compare that to Google Mail... You will probably find that there is a lot less downtime. The problem is just like flying on an airplane. You are statically safer flying an airplane then driving. However because you fate isn't in your control you feel more scared then if you could just drive there yourself. The same thing with SaaS models, you actually get better service however because you don't have the same amount of control you feel like it is riskier. But it isn't

Re:The benefits of cloud computing

[an.echte.trilingue]

As somebody who admins google apps in a business environment, I can say, that is not what they have to offer. What they have to offer is up-time that is better than what internal solutions could ever possibly offer at a price an internal solution could not pray to beat. Is it 100%? Is it free? Nope, but neither is the exchange server in the basement. Do I control my data? Nope, but realistically the alternative would be to contract my data storage out to somebody else anyway.

Bill W. is probably taking heat because he sold google apps to his superiors as having 100% uptime with no disadvantages, which of course it does not.

Re:The benefits of cloud computing

[electrictroy]

That's no excuse. When I need a word processor, I need it NOW, not tomorrow. I do not want my software to be dependent upon anything except my Cl drive. No net connection required.

Re:The benefits of cloud computing

[EraserMouseMan]

If Google doubts it's readiness for mission-critical usage it gets a "beta" slapped on it. Do real professionals actually think Google Apps is ready for prime time usage?

Re:The benefits of cloud computing

[mlwmohawk]

because you don't have the same amount of control you feel like it is riskier. But it isn't

I have a real problem with "cloud computing" and the lack of control is just once piece. With google, there is no assurance that *my* problem is being worked on. *My* problem will get handled in the order in which it was reported. (if at all) To me, MY problem is the most important problem.

The problem with "cloud" computing, and probably the biggest IMHO, is the importance of "you" and your interests to the company providing your service. Suppose that you build your own business on a company providing virtual machine services. All is going well, you are profitable, and poof!! they decide to drop the service because it isn't profitable for them. What if they see what you are doing and say "hey, that's a great business idea, how does he do that, lets look at the code." and so on.

I could go on, but there is a lot to be said about "building" your own business, and my rule of thumb is: "Committing to a single vendor lock-in, in the long run, will always be worse than doing it yourself."

Re:The benefits of cloud computing

[Sancho]

Statistics aren't magical. It's entirely possible that a safe, conscientious driver is safer driving than flying (I haven't seen any statistics which break it down that way before.) There are a whole lot of considerations that need to go into a statistic like that for it to have any real meaning.

Re:The benefits of cloud computing

[twistedsymphony]

Something tells me that an email system that only depends on your hard drive with no net connection wouldn't be very useful.

FWIW I don't seem to be having any problems getting gmail through my gmail-lite install.

Re:The benefits of cloud computing

[xant]

I started out as a personal gmail user. I was very happy with it, even routing my work email through it, but when the question for our mid-sized business came up, "should we outsource our email to gmail?" I said no. I said let's do due diligence, there are other outsourced solutions, this is something we really ought to get right.

Our CEOs (we have two, yeah..) both tried it and liked it, so we went with it.

So I'm in the unique position of having argued to management that we shouldn't risk anything on Gmail, and us doing it anyway because management wanted it. And you know what? I was wrong. Gmail has been a great productivity booster for our business, it's saving us money on salaries, and the downtime is less than we experienced when we were half-assedly running it ourselves.

Plus, when shit does hit, I just smile, and nobody tries to blame me. :-) On the ~two occasions that we had any noticable gmail outage, our CEOs weren't the ones complaining. They have realized that email may be important, but we can still get work done while gmail is futzing around with it.

Re:The benefits of cloud computing

[MightyYar]

Take a step back for a moment... Your email is down for a day. So what? For most people, this is really no big deal. Especially if it only happens once in a blue moon.

Have you never experienced a local failure of any kind? You never had a day without a computer before? Your local net connection never went down? Never had a power failure?

I'm sure some guy on here has some worst-case, niche scenario where losing email for a day means the end of the world - but for most of us, it just means people who REALLY need us use a phone. If your email is so important, you need some kind of redundancy that a single provider is just not going to ever give you. You'd better have a hell of a data center with redundant connections through multiple providers.

Re:The benefits of cloud computing

[A non-mouse Coward]

You are statically safer flying an airplane then driving

That's true, I always get shocked while riding in cars. Can't remember a time in a plane, though. Must be that the plane is off the "ground".

Re:The benefits of cloud computing

[elrous0]

My status as a Google shareholder is in beta too.

Re:The benefits of cloud computing

[cailith1970]

The problem I think isn't the fact that there is down time, the problem is that when you're performing internal maintenance, you can choose the best time to do it by coordinating with everyone else in the organisation. When downtime is imposed with little or no warning externally (or simply just goes down "for maintenance"), that's when the online model comes unstuck and people get frustrated.

Re:The benefits of cloud computing

[MightyYar]

With google, there is no assurance that *my* problem is being worked on.

While that is true, it is also true of your electricity and net connection. And any other utilities feeding your building that are critical to your business.

For most businesses, losing email for a little while is nothing compared to a snow day. It just means more telephone calls. And probably more productivity :)

Re:The benefits of cloud computing

[tolan-b]

Google Apps is a hosted service sold to businesses, Google are meant to provide that redundancy, and in theory they should be in a much better position to provide it than most small to medium size companies' IT departments.

In practice they seem to be sucking a bit.

Re:The benefits of cloud computing

[afidel]

Yes. In your organization how many times have your servers went down or had a problem... Compare that to Google Mail... You will probably find that there is a lot less downtime.

Sorry, but the total downtime I've ever caused ALL of my employers over my career has been a LOT less than 28 hours! Heck, even if you add up the downtime for all of the single systems I've admined their collective downtime is probably only close to that. I'm not bragging, I'm pointing out how bad of an outage this is. The only other outages I've personally heard of that were this bad are hosting providers who have critical systems physically damaged and a failed Exchange 2000 pilot at Cisco (They had a corrupted datastore that was so bad that MS and HP and EMC couldn't recover it so they had to fall back to a tape restore which took something similar to this gmail outage)

Re:The benefits of cloud computing

[Rakishi]

Yet amazingly enough planes crash. They crash quite often in fact. In fact from what I remember the chance to die per hour of travel is roughly the same between airplanes and cars. In other words the chance of dying from some random outside event is probably much higher in an airplane per hour of travel. So yes, a safe driver is much less likely to die in a car than in an airplane.

Re:The benefits of cloud computing

[Moebius Loop]

Honestly, I've maintained my own mail server for 5 years, and my company's corporate server for 2 years, and I can count on one hand the number of times either of them have failed in that period. When they did fail (because I was being irresponsible about configuration changes, or hardware failures, etc), there was pretty much no way I was going to be getting in bed before I got them back up.

Granted, I don't have millions of users and petabytes of email. But I also am not any kind of real system administrator, I don't have a massive redundant data storage facility, and neither do I have millions of dollars and endlessly brilliant engineers working at my beck and call.

Some GMail downtime is, of course, to be expected. But these kind of high-profile outages from Amazon and Google are truly shocking. I don't think it puts the nail in the coffin of SaaS by any means, but it does indicate a significant necessity of SLAs for paying customers.

I would desperately love to divest myself of the responsibility for these mail servers, but I want to know that I can trust GMail's response time during crisis as much as I can trust my own.

Re:The benefits of cloud computing

[Anonymous Coward]

What's a chlorine drive?

Re:The benefits of cloud computing

[wfeick]

If you paid a marginal fee. Then you are a paying customer, and your problem gets priority.

It's not actually that absolute. What is your priority relative to all the other customers who are competing with you for finite rexources? The reality is that more profitable customers may move through the customer support phone queue faster than you, and their issues may be addressed before yours.

Also, it's not uncommon, particularly after a company acquisition, for customers to be reassessed and prioritized according to profitability. Companies decide to cede whole markets to their competitors if they're not sufficiently profitable or they decide to go in a different market direction.

The customer is often not told they are no longer a priority, they just find their rates go up, the quality of their support goes down, their packets are routed through over subscribed network fabric, etc.

The company won't actually tell the customer to go away; they'll continue to accept money until the customer figures it out and goes away on their own.

Re:The benefits of cloud computing

[Kleen13]

I agree. All the downtime we have on our Exchange server is administrative.

Re:The benefits of cloud computing

[Firehed]

Yeah, that's usually what I was saying about email when the exchange server at my previous employer went down every third week or so. Or when some rodent chewed through the line that handled the VOIP. Or when some transformer down the street got plowed by some idiot driver causing sporadic access to the mains for half a day. Word processors in the cloud may not make a ton of sense right now, but email is fundamentally useless without every machine in the chain working properly.

You try running your software without a power connection and let me know how it goes for you. Laptop batteries don't last that long, and desktop UPSs even less so (assume that the generator, if present, can only keep the servers online indefinitely, not the whole building).

Gmail being down for a few hours is a minor inconvenience at worst. If your dirt cheap or free and completely awesome email being unavailable for two hours a year is causing you to lose business, then you seriously need to rethink your operations. You have a landline, a cell phone, and a fax (among others), and if those are all also out of commission then chances are you've got bigger problems. You know, that mushroom cloud hovering overhead.

Re:The benefits of cloud computing

[Anonymous Coward]

Isn't that a chlorIDE drive?

Re:The benefits of cloud computing

[reboot246]

I've not had any problems with my password, "invalidpassword".

Re:The benefits of cloud computing

[sribe]

It most certainly is riskier. If you own your computers & data, and your company goes out of business, you no longer need access to that data. If you SaaS provider goes out of business, you probably still need that data.

Re:The benefits of cloud computing

[Belial6]

People keep claiming that Google has better uptime than in house systems. I have to ask, what kind of monkeys are administering these systems? In the last decade, we have not had a single unscheduled email server outage. We haven't even come close to 28 hours of SCHEDULED downtime. Heck, my personal mail server hasn't come close to 28 hours of unscheduled downtime in the last decade. There just isn't that much that can go wrong on an even half assed administered email server. I get why it would take longer for Google to restore their backups, but that is just a weakness of centralized data storage. It's everybody trying to get a restore at the same time so they have to stand in line. That is just one reason that gmail is only good for personal email.

Re:The benefits of cloud computing

[Bryansix]

That gmail-lite project appears dead. That or they just forgot about the blog since 2005.

Re:The benefits of cloud computing

[Tony Hoyle]

Been using mine all day with no problems.. clearly the outage didn't affect everyone.

Re:The benefits of cloud computing

[Mista2]

Then think carefully about this before you rely on the cloud. Sometimes it's sunny and there is no cloud 8)

Re:The benefits of cloud computing

[Kalriath]

I don't think it puts the nail in the coffin of SaaS by any means, but it does indicate a significant necessity of SLAs for paying customers.

There is an SLA for paying customers, but with Google's track record they really need to up the penalties for non-compliance.

Re:The benefits of cloud computing

[kestasjk]

The worst thing is the update to iGoogle (which is extensive and undoubtedly the cause of the outage) is quite a step backwards. It's a pretty clunky hybrid of a window based system and a widget based system, with a lot more AJAX and a lot fewer clickable links.

Re:The benefits of cloud computing

[Anonymous Coward]

I do not want my software to be dependent upon anything except my C: drive. No net connection required.

Different strokes. If you were traveling to see a client and your laptop got stolen, you might see the upside of having your documents online. Viewable and editable whether on a free library terminal or iPod Touch.

I can't count the number of times myself and co-workers have mislaid a USB thumb drive. Can I just VPN in? No, that's why I have the damn thumb drive.

And I use Photoshop, so there's no promise that my client will have a spare terminal in their office with the latest version installed. I guess that's why Adobe is shooting for online apps too.

Re:The benefits of cloud computing

[brasscount]

Let's see, so 99.9% uptime...
Hours in a year: 8760
99.9% of hours in year: 8751.24

Hours in a year that Google has lost in this incident: >27 or 99.691780821917808219178082191781%. Lets round to 99.692% (I'm feeling charitable.)

Wait, we just said 99.9% uptime. Really we meant 99.9% uptime every ten years.
So, 99.9% of 87,600 hours allows us 87.6 hours out of service. No problem boys, we've got two weeks to get this thing working!

Re:The benefits of cloud computing

[Anonymous Coward]

email is fundamentally useless without every machine in the chain working properly.

Ummm, no. SMTP was designed a long time ago when computers & networks were much less reliable than they are now. You can have multiple email servers on multiple internet providers on multiple backbones for outstanding reliability and graceful failover.

Of course, you do have to configure the redundancy - it doesn't magically appear.

Re:The benefits of cloud computing

[pherthyl]

Uhh.. No. Drive from Vancouver to Halifax, you're in the car for a few days. Fly that distance and it's a few hours.

Risk/hour is completely irrelevant when comparing the safety of airplanes to cars. When was the last time you took a week long plane trip?

Re:The benefits of cloud computing

[Bandman]

You've got to love IT, where one-in-a-million events occur every other day...

Re:The benefits of cloud computing

[twistedsymphony]

yeah glite is on life-support in an official capacity, but there is work being done here. Though it's mostly just enough to keep it functional with the changes that google impliment.

It works well enough that I can run it from my own server and access my email in places where gmail might be blocked or otherwise difficult to access directly. which is all I need it for.

Re:The benefits of cloud computing

[Kugrian]

The worst thing is the update to iGoogle

Dear god yes! After 3 years of having a perfect homepage, I now have to change, as the new one no longer fits my needs. I'm all for change, but a 'Use original IG' option would be nice. Even if it wasn't updated and newer applications would no longer work. My perfect homepage is now destroyed :(.

Re:The benefits of cloud computing

[DocSavage64109]

Well, the Exchange server where I work reboots on a nearly weekly basis for the seemingly continual Microsoft patches. That has got to be bad for uptime stats.

Re:The benefits of cloud computing

[Bryansix]

What part of my post didn't explain how even with close to 0 downtime you STILL might have data loss?

Re:The benefits of cloud computing

[zooblethorpe]

Re-read the parent post. What businesses hate isn't customers, it's unprofitable customers. You know, the ones that cost more to provide services to than they bring in in terms of revenue.

Cheers,

Re:The benefits of cloud computing

[MightyYar]

They should pay for the lapse in service, and that is fair. I was just saying that - in a business sense - losing a day of email connectivity is not the end of the world for most businesses. I mean, just this year I can think of a Sprint outage and at least one "Exchange is down for emergency maintenance" message. Last year there was a power outage and the generators didn't kick in. Every year there are snow days. Business moves on...

Re:The benefits of cloud computing

[MadnessASAP]

Yes well inside computers, billions of event occur every second. Hell something as insignificant as a cache miss is a one in a million event.

Re:The benefits of cloud computing

[Anonymous Coward]

How are sites slashdotted when nobody reads TFAs?

They just want to look at the pictures.

Re:The benefits of cloud computing

[invalid_user]

Thief! You stole my password!

Re:The benefits of cloud computing

[TimSSG]

There is an SLA for paying customers, but with Google's track record they really need to up the penalties for non-compliance.

For some reason, I think Google are more likely to lower the penalties because of this than raise them. Tim S

Re:The benefits of cloud computing

[Bryansix]

No no no. We are talking about two completely different things here. My point was that with Google EVEN WITH downtime there is no loss of data. I then pointed out that RAID controllers and the hard drives attached to them sometimes fail in such a way that nothing is recoverable. Then I pointed out that you don't have real-time backups. Yes, the email gets delivered but nobody can get to their old mail until you do a restore in which case they probably lost some email between when the backup was done and when the server failed. Then you mentioned that you can throw extra servers at it at which point I mentioned that this really is cost prohibitive when you compare it to Google's solution.

If you want to argue features or speed or something like that then I will listen, but arguing against Google Apps because of reliability is lame and you don't have much going for that argument.

Outage Outrage

[Pantero Blanco]

It's a risk you take any time you let someone else handle something for you.

Re:Outage Outrage

[Xandar01]

This is what you get when you pick the cheaper option eh?

Re:Outage Outrage

[mshannon78660]

It's a risk you take any time you let someone else handle something for you.

Specifically, it's a risk you take anytime you use a free service for something critical. You can't have an enforceable service level agreement for a free service - in order to be binding, a contract has to involve consideration from both sides.

Re:Outage Outrage

[jeffmeden]

That's just it, google DOES charge for the hosted apps version of Gmail. See http://www.google.com/apps/intl/en/business/index.html for more info.

Re:Outage Outrage

[mshannon78660]

If that's what they're using, then they do have recourse:

Google Apps for Business

Claims right on that page that it provides 99.9% uptime.

Re:Outage Outrage

[Medievalist]

It's a risk you take any time you let someone else handle something for you.

Specifically, it's a risk you take anytime you use a free service for something critical. You can't have an enforceable service level agreement for a free service - in order to be binding, a contract has to involve consideration from both sides.

Having an "enforceable service level agreement" does not make things magically unbreakable.

One of the great business fallacies of our time...

A really smart provider will not sign a contract promising 100% uptime, but a stupid one will. Which one will deliver the better service? In practice, your real guarantee of reliability is quality work, and the best way to get quality work is hire the best and treat them well. Making them sign pieces of paper promising doom for failure does little or nothing useful.

I've been using Gmail all day, incidentally. Works fine for me - I've been corresponding with other gmail users no problem.

Stallman is laughing

[Rinisari]

Is that the sound of cloud computing advocates crying, or the sound of Richard Stallman laughing?

Re:Stallman is laughing

[Pantero Blanco]

Not to mention it is a "free" service, no one has a right to demand it to be up anyway. You are storing your e-mail on google's servers and you pay nothing for it. If they need access to their crucial time-sensitive data then they should keep it themselves or pay for a service which will guarantee availability.

Some of these companies are paying customers. The headline only mentions Gmail, but this is also about Google Apps as a whole.

Re:Stallman is laughing

[Etrias]

Have you really looked at it? I mean actually go to the Google Apps page at looked at it, or did you just hit reply without knowing. Oh, who am I kidding, this is /. and I'm lucky you read any of my message.

Seriously, here's a link to the Google Apps business page. Look around. This isn't free stuff. I'm not sure why you scoff at this and not other business webmail applications. There...do you see that...do you see how Gmail isn't always free?

My point remains...if they paid for it, Google owes them an explanation.

Re:Stallman is laughing

[That's Unpossible!]

No, it's the sound of one hand clapping.

What I mean to say is, what admin HASN'T had an outage like this?

Shit happens. I'd rather get email that works 99% of the time, and when there's a problem, google engineers are dealing with it, leaving me time to work on more important things.

If you need something done right do it yourself!

[EncryptedSoldier]

You can't count on Google to run your IT...sorry buddy. Using Google may be cost effective, but the obvious trade off is that someone else is really doing your job, and if that person drops the ball, then you really screwed the pooch, at least that's what your boss will think.

These guys need a brain transplant...

[Khyber]

"Admin Bill W. posted a desperate message on the forum Thursday morning, saying his company's CEO is steaming about being locked out of his e-mail account"

Run your own damned mail server if it's THAT IMPORTANT. Seriously, it's not hard to set one up, and you've obviously got the money to do it.

Once again, it's a case of rich people with more money than brains having the problems. Nothing important here, nothing of value lost.

Re:These guys need a brain transplant...

[Rary]

Run your own damned mail server if it's THAT IMPORTANT. Seriously, it's not hard to set one up, and you've obviously got the money to do it.

Right. Because some nerdy 20-something admin with a copy of "Sendmail for Dummies" can do a much better job than all the engineers at Google.

This is a paid service offered by one of the largest and most knowledgeable technology companies around. They should be able to do a much better job than any internal IT department. There are arguments in favour of doing it yourself, but there are definitely arguments in favour of outsourcing to a competent provider, which Google should be.

This is a PR disaster for Google.

Re:These guys need a brain transplant...

[iamhigh]

Who is running a single mail server that should be easily recoverable in a few hours? Who is running a service used my millions with complications I probably can't even fathom?

They might be "better" at running world class stuff than you, but you can be better at running simple stuff. Veritas, Acronis, etc... take your pick and you are back up and running. That probably isn't true for Google.

Re:These guys need a brain transplant...

[Builder]

How do you figure it's not that hard to run your own mailserver? Please tell me how to achieve the following, simply.

1. POP and IMAP access for all users
2. Simple ability to delegate administrative rights for the domain including creation of new user rights
3. Ability for users to manage aliases
4. Ability for users to manage mail lists
5. Spam prevention with similar success rates (including very low false positives) to Gmail or any other mass mail site
6. Webmail
7. Simple password changing for users
8. Authenticated SMTP
9. Simple SPF

I've spent a lot of time and effort, and have a good smtp / imap / pop / webmail solution, but it's still cobbled together from loads of stuff and it's still missing solutions to allow users to change passwords and delegate administration control of a domain.

Re:These guys need a brain transplant...

[slimjim8094]

Right. Because some nerdy 20-something admin with a copy of "Sendmail for Dummies" can do a much better job than all the engineers at Google.

Well, actually, there's a decent chance. These technologies are extremely mature, and pretty simple. For example, I ran a 25-user email server for well over two years with a 99.8% uptime... between age 15 and 17, and it handled at least 500 emails a day.

The difference is, I was dealing with tens of gigabytes of data and backing that up (external HD + gpg + rsync + my house's firesafe). Google is dealing with thousands of terabytes (including, frankly, a couple of mine - I use their free domain service)

I'm sure as hell not as smart, and I'm NOT criticizing Google here. But when you're dealing with MUCH less data and don't need any load-balancing or fancy DNS tricks, it's pretty easy to get right.

But they are doing load balancing, clustering, immediate failover with synchronization, and all those other fancy tricks, because they need more than one server, have a lot of data to back up, and need to keep it all in sync. That is Fucking Hard to get right. A server in my client's basement on a T1 realistically only needs to worry about fire and trees.

Re:These guys need a brain transplant...

[mortonda]

Run your own damned mail server if it's THAT IMPORTANT. Seriously, it's not hard to set one up,

Yes and no, unfortunately. Sure it seems easy to set up a mail server, but there are a lot of misconfigured mail servers out there that are open relays, or spew backscatter spam, or simply get hacked and turned into zombies for spammers.

That doesn't begin to touch the issues of administration of users, anti spam software, uptime and redundancy planning, etc.

I'd say, it's not that hard *for a competent system admin* to set up a mail server.

POP/IMAP

[superphreak]

Why no Gears for offline Gmail access at very least, Google?
I believe it's called POP/IMAP access, and it's been around a long time. Oh, downside - you might need a program called Outlook/Express or Thunderbird. Free download available.

Beta means never having to say you're sorry.

[Orgasmatron]

Who puts important mailboxes on a beta service? Sheesh.

Re:Beta means never having to say you're sorry.

[Zarel]

The same idiots that pay to use a BETA product.

Too many people make this mistake, calling Google Apps a beta product.

First off: Gmail is only called "beta" because it is constantly being improved. If you use the version of Gmail within Google Apps, which is stable and not beta, you will find that its features are usually a few months behind Gmail. The same goes for all of Google Apps - it is not a beta product (although it's possible to opt-in to beta features).

That's SaaS for you...

[scsirob]

This is the main reason not to turn to Software as a Service. Sure, it's nice to just rent some functionality, but you are not in control of your own destiny. What if Google decides that GMail no longer fits their business model? Poof...

Re:That's SaaS for you...

[tshak]

If you ran your mail server off a single $499 dell workstation you'd probably have availability problems as well. This is not a problem inherent with SaaS, it's a problem with using a consumer grade mail service for a corporate mail.

Re:That's SaaS for you...

[Todd Knarr]

Actually they did have an SLA. The users complaining were using Google Apps, for which they pay and which includes a Service Level Agreement. However, the users are learning 2 important life lessons:

1. Down is down, SLA nonwithstanding. All the SLA means is that you may get limited compensation when the service is down. It doesn't get your service up and running one second sooner.
2. In this case, the compensation is pretty poor. Google won't compensate for damage to their businesses due to e-mail being down. All Google has to do is provide 15 more days service at the end of the contract period. But then, what did you expect for $50/year?

Re:If you need something done right do it yourself

[whisper_jeff]

The thing is someone will always drop the ball. In this case, the CEO can't chew out the guy in IT who pooched the email server and is working frantically trying to get it back up and running because that guy works for a different company. Or do people honestly think that an internally-run email server never has problems?... Just because it's Google does not mean it's infallible.

it can go both ways

[Trepidity]

I've been at plenty of places that run their own mailservers where uptime is considerably worse than Gmail's, so it'd be an improvement to offload it. The biggest problem seems to be at medium-sized shops: big enough for there to be problems, but not so big that you have some sort of massively redundant setup with transparent failover and 24/7 staffing. The ideal of the cloud-computing style of outsourcing is that you'd outsource to someone who was big enough to have a massively redundant setup with transparent failover and 24/7 staffing. However Google seems not to have delivered on that ideal.

Re:If you need something done right do it yourself

[samkass]

So you think most companies have better IT departments than Google? I agree that using a free beta software to run mission-critical software is probably unwise, but there are other providers that offer way more uptime than probably most internal IT departments could manage. Pair Networks, etc. It will cost money, though.

Re:WTF? I see no outage.

[mapsjanhere]

Is that you, brenda@viagra.com?

Re:You get what you pay for!

[AKAImBatman]

Doesn't anyone RTFA?

Google Apps is a suite of hosted collaboration and communication software and services designed for workplace use. Its Premier edition costs US$50 per user per year and includes a 99.9% uptime guarantee for the Gmail service.

In August, Gmail had three significant outages that affected not only individual consumers of the free Webmail service but also paying Google Apps Premier customers. As a result, Google decided to extend a credit to all Apps Premier customers and vowed to improve its problem-notification methods.

$50/yr for each user is not "free". Nor is it in the domain of "you get what you pay for". $50 per user is actually a rather significant sum when we're talking about 100+ user companies.

There is an SLA for paying customers

[Que_Ball]

Quote from article: So, will this one prompt calls for a service-level agreement for paying customers?

Paying customers of the apps Premium account level DO have a service level agreement.

Free customers do not however which is probably what they were trying to say.

Revised quote: So, will this one prompt calls for a service-level agreement for free customers in addition to paying customers?

From the terms of service for Premier account edition:
http://www.google.com/apps/intl/en/terms/premier_terms.html

1.9. *Service Level Agreement*, or *SLA* means the Service Level Agreement located at the following URL: http://www.google.com/a/help/intl/en/admins/sla.html

Downtime period is a period of ten consecutive minutes of Downtime

Service Credit is
three days of service added to the end of your term at no charge for monthly uptime percentage between 99.0 and 99.9
seven days for between 99.0 and 95.0
fifteen days for worse than 95.0 uptime percentage.

You must request your service credit. It is not automatic.

Google SLA

[WPIDalamar]

http://www.google.com/apps/intl/en/terms/sla.html

There you go, the SLA for Google Apps. It's listed at 99.9%

But... the remedies for them failing that suck, only up to 15 days worth of service per month will be credited.

Also, it costs $50 per user per year

Re:Offline mail access?

[larry bagina]

Does it include a reverse bayesian filter to generate stock market and h3rb@l \/i4gr@ spam?

Re:If you need something done right do it yourself

[LWATCDR]

Because every company can afford redundant internet connections, back up generators, a fall over mail server, and a 24/7 IT staff and I don't mean some poor guy with a cell phone and no life.

Outsourced Monoculture

[Doc Ruby]

Outsourcing links in your essential service chains is risky enough. Outsourcing them to a single point of failure is too risky. So many independent places all outsourcing something so central to so many service chains is unacceptably risky.

I would never rely on GMail without a local cache of all the content GMail holds, or without a truly alternate server to serve my messages when GMail goes down, as it clearly does some percentage of the time.

Re:Better check your backups...

[moosesocks]

Um. I can't find where they made this excuse...

Re:WTF? I see no outage.

[An ominous Cow art]

I use (free) Gmail through Thunderbird and IMAP, and have been getting timeouts and spurious authentication failures for months.

Sometimes you do get what you paid for, but it was pretty smooth for a while.

Re:If you need something done right do it yourself

[snspdaarf]

The thing is, if I am going to take a screamin' reamin' from the boss, I prefer it be for something that is either my fault or that I can do something about. While a normal human can grasp these issues, some admin-types seem to think that if they throw a big enough shit hemorrhage that it will force the IT people to fix the problem. Tough to do when it is outside of their control.

Gmail's working fine for me, all day now.

[Medievalist]

It's a risk you take any time you let someone else handle something for you.
It's a risk you take, period. You're trying to tell me that you can guarantee no unplanned downtime if you were to handle it yourself?

I want 24 hours notice prior to any unscheduled downtime! And don't give me any of that technical mumbo jumbo, I have a SERVICE LEVEL AGREEMENT!

Re:If you need something done right do it yourself

[ajs]

You can't count on the USPS to deliver your mail...sorry buddy.

You can't count on Verizon to run your telecommunications...sorry buddy.

Every service you use was, at one point, decentralized and every large corporation ran it themselves. Then someone did a better job and companies slowly released the reins. Does Verizon's phone service go down? Yep. Does the USPS lose mail? Yep. Goes Google mail go down? Yep. But, in the end we've decided that we'd rather rely on these external services than continue to try to run increasingly large services with ever-diminishing returns for the individual business.

Re:If you need something done right do it yourself

[ajs]

Interestingly, phone service and physical mail have both gone through several iterations of increasing and decreasing scope and centralization within organizations, so my above examples are a bit simplistic, but overall I think they hold up. We're at the start of what will be a century-plus period of understanding the role of computer-based communications in the business world, and as that grows and changes, Google will continue to grow and change and others will compete with them in interesting and perhaps successful ways.

I'm not waving a Google flag, here, just reacting to the idea that a single outage makes for a useless corporate service (which, if true, would have every company in the world running their own fleet of planes and drilling for their own oil).

the "small IT shops are worse than SaaS" BS

[SuperBanana]

Yes. In your organization how many times have your servers went down or had a problem... Compare that to Google Mail... You will probably find that there is a lot less downtime.

Bullshit- this is an often-repeated myth that small or medium-sized IT shops can't offer competitive uptimes. It's simply not true- I'm a sole sysadmin, and my server (~200 users) has only had one time when we had an outage , and it took us all of about 15 minutes to fix. We have a number of people who choose to use GMail, and I'm constantly reminding them that they should not be relying on Gmail so much.

The problem is not downtime- it's lack of any way to mitigate the problems, and a complete and total lack of any customer service from Google. There is NOBODY you can call when there's a problem. PERIOD.

Compare and contrast. Google:

• If Google hoses someone's account, they're completely fucked. Google will shrug and say "meh, whaddya gonna do?", and point to their user agreement.
• If someone breaks into their account or changes the password, they're completely fucked. Google won't block access, can't prove who is who, getting logs will be a slow fight to the death, etc.
• If the user deletes a bunch of mail (or someone else does) or there's a bug with their email client (ie if they're using IMAP or POP access), they're completely fucked. Google won't do a restore. Their backups (if they even have any) are for "oh shit" system-wide fuckups (like, I'm guessing, the current one- I bet the accounts got deleted and they're restoring from backups.)

Me:

• If we hose someone's account, they need only wait about 15 minutes for the tapes in the jukebox to shuffle and we've got their entire account back from less than 24 hours ago. If I refuse or cannot, I'm pretty much out of a job.
• If someone breaks into their account or changes the password, I can lock the account in seconds, and I've got logs for forensics I can hand over immediately to the university police. Again, my paycheck is close to these people.
• If they delete a bunch of mail (or someone else does) or there's a bug with their email client (ie if they're using IMAP or POP access), again, they need only wait for the jukebox to shuffle tapes around. It's a few minutes of my time and perhaps a trip to the server room to feed the jukebox some tapes.
• If we have a crash, or a hacker breaks in, etc- we tell people what happened and we get the hairy eyeball from the administration. If Google hoses your account, you're told they had a 'service outage' or 'technical problem', and that's that.

The building I (and the server) are in in could burn to the ground, and I could have us back up in less time than this stupid outage at Google (I'm factoring the time to find/buy two commodity PCs, find/buy compatible tape drive/SCSI card, do an OS install, install the backup server, and fetch the off-site backups from across campus.)

If Google's datacenter burns to the ground, how long do you think you'll be without your GMail account?

Re:the "small IT shops are worse than SaaS" BS

[Terrasque]

Yeah, got a rude awakening there myself. My gmail account was closed a month ago, no prior warning. No way to get to support. No way to actually contact a human.

I still have absolutely no clue why it was closed. I also lost blog, gallery, docs, calendar, site stats, rss reader, notebook... You get the idea. Luckily I didn't have any serious data there, but still. It's a lot of things I used daily just suddenly gone.

And google's ToS says "We can close an account for any reason. Absolutely any reason. Like, we didn't like the color of the sky today is a good reason." Which isn't helping the issue much either.

Re:the "small IT shops are worse than SaaS" BS

[Anonymous Coward]

The building I (and the server) are in in could burn to the ground, and I could have us back up in less time than this stupid outage at Google

How? You'd be dead, or probably in a burn unit, or something...

Re:the "small IT shops are worse than SaaS" BS

[Bandman]

If Google's datacenter burns to the ground, how long do you think you'll be without your GMail account?

But that's one of the strengths of using gmail.

The loss of a complete datacenter for them would be a minor issue which would cause slow access. Accidental removal of one part of their infrastructure doesn't cause much of a problem. It's the administrative issues that get replicated across the network that cause major issues.

Re:the "small IT shops are worse than SaaS" BS

[jellomizer]

You work as the administrator. Your job is probably administering the system... Take You salary and add say 30% for your benefits. 35k - 90k depnding on your location vs. Say you use a professional SaaS shop and pay say $2000 (not just google free service) a month for the same services, that is 24k a year. With the $2000 a year service you are probably a good customer thus they make sure they are up. If not they will work just as hard as you to keep it running. This shop may have 10 - 20 people working on those servers if there is a problem they will go and fix it.

Dont confuse the Free Gmail as a true SaaS.

Lol Mailtrust Ad

[genner]

I wonder how much rackspace is racking in having their mailtrust ad slapped on top of this story.

Banner ads to the contrary

[HalInc]

In poignant irony, the banner ad I see above the story is a google ad that reads:

"So why not switch to Google Apps?
We maintain our hosted software 24/7 so you can sleep at night."

Strange Help Desk Phone Calls

[The Moof]

Employee: Hi, Help desk?
Help Desk: Yea, how can I help you?
Employee: I can't get my e-mail.
Help Desk: Hmm... I see. Yea, there's an issue. Hold on while I call the help desk.
Employee: Sure, no pro...wait, what?

Re:Can anyone tell me the difference...

[Todd Knarr]

The big difference is the affected company can do something about the problem. The CEO can come down and tell the admin he's not going home until this is fixed. They can call in any extra people they need. They can, if it's really that critical, have someone physically go and buy a new server and get enough software installed on it to get mail back up and running until the main system can be fixed. Expensive proposition there, but the company gets the option of deciding whether it's worth it or not.

Compare this to the situation Bill W.'s company is in. Their e-mail is down. All they can do is wait until it comes back up. No matter how crucial service is, no matter how much money they're losing because it's down, they've got absolutely no control over how fast the problem gets fixed. That'll be determined by how important to Google restoring service is. And the cost equation to Google is the cost of having staff working overtime all night to fix the problem vs. the cost of giving Bill W.'s company 15 days more service (about $2.06 at the $50/year rate for Google Apps).

SLAs aren't a cure-all

[Knara]

Yeah, okay, you get an SLA by paying for the business version, but anyone doing their homework and a few thought experiments will realize that SLAs are only potentially helpful after the fact.

$50/yr/user isn't going to get you 100% uptime, I don't care who is running it.

This strikes me kinda similar to the folks who try to run their businesses off Dreamhost shared web hosting servers (which don't even HAVE SLAs) and go ballistic when something breaks.

Re: Sunny!

[TaoPhoenix]

Icarus, is that you falling?

Re: usb drives, clouds...

[TaoPhoenix]

C. Other.

The online copies are backups. When your laptop went AWOL, you go to some new computer, download them, then do your thing.

Go redundant. When your laptop isn't available, these new phones can sometimes process your actual documents. We're one generation short of proper usability on this front. That will be fixed in about 2 years.

Phone not an option? Get a "disposable desktop". You know, some piece of junk for $100. There's a huge influx of machines due to hit this maturity stage within the same next two years when HeavyOS drives upgrades.

My USB drive is my watch. It's strapped to my arm. So unless I'm a twit and take it off, it's essentially unloseable. Oh look, I lost it. Here's one on my car key chain. Awww, I got mugged. Maybe in 10 years they'll be doing subcutaneous mods. (Gee. My beer belly holds 4 terabytes.)

Still more reliable than Exchange

[DustoneGT]

I'll gladly put up with the occasional outage...it's better than the almost weekly MS Exchange bugs on a crappy MS Windows Server system.

Too bad, so sad, serves them right!

[billcopc]

That's what they get for not having an SLA.

In today's bastardized world, if you don't have something in writing, you have absolutely nothing.

If a small fry were to screw up this bad, they would be afraid of their phone thanks to the many passive-aggressive office drones complaining repeatedly. Not that it helps in any way, but I'm pretty sure Google's techs aren't being harassed with phone calls every 5 seconds. I hate to enable the lusers, but when you've got paying clients breathing down your neck, you tend to take whatever measures necessary to fix the problem asap.

If anything, this should send the message that Google, contrary to popular belief, is not invincible. They mess up just like everyone else, which means maybe they're no better than anyone else.

Re:Beta?

[Tony Hoyle]

You're paying less than $1 a week per user. If email was important to you you'd be paying a *lot* more than that. Stop being a cheapskate.
 

Re:Google's stuff is breaking all over

[mollymoo]

I also get the impression nobody at Google can be bothered with the boring stuff. It's not a staffing problem, it's a management problem. Google is famous for its lack of management and it's starting to show in the quality of their products.