Google Apps Gets a 99.9% Guarantee

David Gerard passes along a posting on Google's official blog announcing that they have extended the three-nines SLA for the Premier Edition of Google Apps from Gmail alone to also cover the Calendar, Docs, Sites, and Google Talk services. 99.9% uptime translates to 45 minutes a month of downtime, and the blog post puts this in context with Gmail's historical reliability, which has been between three and four times as good over the last year (10-15 min./mo.). It also claims, based on research by an outside group, that Gmail's historical reliability beats that of in-house hosted solutions such as Groupwise and Exchange, on average. Reader Ian Lamont adds an article in The Standard that digs down into the details of the SLA, revealing for instance that outages of less than 10 minutes aren't counted against the monthly 45 minutes.

Comments

Umm...

[Sylos]

so if I have 60 1 minute downtimes, I'm keeping within the 99.9% uptime range? I call shenanigans.

Re:Umm...

[Threni]

Exactly. How many minutes per month of downtime inclusive of anything, for any reason?

Re:Umm...

[Creepy Crawler]

Most likely it's the time for node crash detection and load balancing to take effect.

If service is that bad or intermittent, nobody would buy service there.

Re:Umm...

[HappySmileMan]

Well you could have 90 downtime like this and still count? If it's down for 9 minutes, up for 1, down for 9 etc.

But of course measuring it googles way that would still be 100% uptime.

Re:Umm...

[ILongForDarkness]

Well if they cache the current session locally and it is just the connection to the back end that you lose temporarily I think it would be alright. Losing data sucks. That said who uses desktop suites without a crash? "Hopefully" (not sure if that is the right word to use when referring to an outage), they manage to have the downtime clumped together and planned in non-peak hours for the region (say upgrades done first Saturday of the month at midnight or something).

My big concern with this type of offering is it increases a companies dependence on their internet line. If your network is down not only can't retrieve files, email or browse, you now can't work on productivity software either. Essentially if your doing a job that requires a computer in this environment you can't work whenever the internet or network has a hickup. I like having something else to do in the rare instances where the network isn't working right.

Add to that the fact that wireless/laptops are becoming of larger importance in companies (and wireless is flaky at the best of times IMHO) you're really courting disaster not just in terms of outages but in terms of accidental data loss. Say your not so gifted technologically colleague decides to walk over to your desk with their laptop to show you the spreadsheet they've been working on. They get out of range of the router that they were using and presto session time out and the chance of data loss.

Re:Umm...

[cgenman]

Is that 99.9% uptime or 99.9% planned uptime? Many companies refer (rather facetiously) to *planned* uptime, which means that you can have unlimited downtime so long as it isn't unplanned.

Re:Umm...

[shadanan]

I believe they're referring to availability. In this case, planned downtime shouldn't affect availability since service can be proactively transferred to a backup system with zero impact to the user. Wikipedia has good article here.

Re:Umm...

[aaarrrgggh]

The concept of "unplanned downtime" seems to originate in the banking world, where something as benign as daylight savings time could force you to take down the mainframe for two hours. It has unfortunately spread to other industries (healthcare records management pops up). The real question is if Google's application architecture requires planned downtime for the service as a whole or individual users.

Based on their roots, I would expect them to be able to do any upgrades in the ten minute window they exclude from their SLA.

Re:Umm...

[mestar]

You could have WAY MORE downtime than this. You could have 9 minutes down, 1 second up, 9 minutes down, 1 second up.

This way it can work 2 minutes per day, and still be 99.9% up.

Re:Umm...

[ChrisA90278]

How would you notice a 60 second email outage unless you were running synthetic tests designe only to detect an outage. In normal use all you would see is a 60 second delay in email delivery.

One could even define email as a service that deliveres messages in a "short" time of about 5 minutes. If you accept this definition of email then a 60 second outage is not even an outage as long as emails do get delivered wthout a "short" time.

So I think Google is OK to not count such sort drop outs. email was never a contiously connected service. It is always done by periodically polling. Heck, most polling takes place on a longer interval than 60 seconds.

Re:Umm...

[SoopahMan]

Buggy offline access via Google Gears helps alleviate the internet connection issue - it might also alleviate what downtime actually occurs.

Re:Umm...

[afabbro]

The concept of "unplanned downtime" seems to originate in the banking world, where something as benign as daylight savings time could force you to take down the mainframe for two hours.

What are you talking about? I take it you've never seen or touched a mainframe.

Re:Wait..

[mikael_j]

It's called a cluster, "The cloud" is a really annoying buzzword for software as a service.

/Mikael

What about internet downtime?

[Dan East]

Yes, but what is the average company's internet downtime verses their LAN downtime for a single-campus outfit?

So instead of LAN / Exchange Server (or whatever is being used) you now have LAN / WAN / Google downtime. WAN gateway downtime is probably the weakest link in the chain, so wouldn't the total downtime be greater using something internet based?

Re:What about internet downtime?

[RiotingPacifist]

what use are webserver, email server, IM servers if your internet is down anyway?

Re:What about internet downtime?

[moderatorrater]

Internal email might be able to get around if your internet connection is down, but that's about it. If a company's seriously looking at outsourcing its email servers anyway, I doubt that keeping the internal email up during an internet outage is worth the headache of managing their own machines.

Re:What about internet downtime?

[vadim_t]

With an internal server, the mail you got it stays there so you can still read it, and compose replies. With an internal SMTP you can queue emails for delivery even if they don't get out (nice for laptops that may not stay around until the connection comes back). With an internal IM server you keep being able to talk to people inside the company, and can depending on the server, can queue messages until the connection comes back.

Now if you happen to use say, gmail, then you're out of luck. You can't read your mail, can't compose replies, can't IM people in the next room. All you can do is sit there and wait for somebody to fix the problem.

Re:What about internet downtime?

[afidel]

WAN gateway downtime, what? Our DS3 hasn't been down once since it was installed in September 2006 and the firewall cluster behind it has likewise never been down since it was installed in 2005. If you have significant internet outages you are doing something wrong. Forget LAN outages, it just doesn't happen. Of course that's why we paid the premium for Cisco chassis based switches with redundant supervisors for both the datacenter and the wiring closets.

Re:What about internet downtime?

[mysidia]

So instead of LAN / Exchange Server (or whatever is being used) you now have LAN / WAN / Google downtime. WAN gateway downtime is probably the weakest link in the chain, so wouldn't the total downtime be greater using something internet based?

E-mail is internet based and isn't going to work if your WAN is down, regardless (you can't e-mail anyone, or receive e-mail from other people).

One of the costs of using a service like Google Apps is the increased need to design a proper resilient network at your site that won't go down.

If you are multi-homed and have dual WAN links that take an independent path, with a standby router, and ensure your ISP provides redundancy, and your network is properly designed according to network industry standard and respected network equipment manufacturer's best practices: then a failure of your internet connection is unlikely.

Much less likely than the probability of failure of a single mail server.

The cost of internet link failure or congestion is significant for companies that rely on internet-based resources and online communications for productivity.

For companies that conduct eCommerce, it is unthinkable to have the website going down, or to not have planned enough capacity for the network connection to meet all anticipated needs in a failure scenario. Bad connectivity is already costly, even without relying on application service providers for business apps.

In a well-designed setup, the WAN itself should not much reduce that 99.999% figure. Although yes, there are some new failure modes introduced.

Loss of connectivity to Google, for example, even if the network is otherwise working. Some unexpected Tier1 depeering ala. Sprint/Cogent may cause issues on rare occasion.

Re:What about internet downtime?

[rcamans]

Obviously the answer to the question "what is the average company's LAN/WAN downtime" is 42

Re:What about internet downtime?

[afidel]

Our ISP is AT&T, I don't think they are going to get depeered =) Also the fact that a bunch of amateurs didn't know how to get STP under control has nothing to do with the reliability of Cisco hardware. It's like the condition of our network before the current staff got there. The company had a laser link between buildings that would go out with the slightest bit of precipitation, this would have been bad enough for the other building but whenever the network reconverged their was a spanning tree war because there was a switch in each building that was setup as a root bridge. Turning off the root bridge in the other building and setting spanning tree weights correctly eliminated the main network problems and digging a horizontal bore and running fiber eliminated the other buildings problems.

Re:What about internet downtime?

[TubeSteak]

Now if you happen to use say, gmail, then you're out of luck. You can't read your mail, can't compose replies, can't IM people in the next room. All you can do is sit there and wait for somebody to fix the problem.

Isn't that problem the idea behind Google Appliances?
You plug it into your network and :BAM: locally hosted Google products.

Re:What about internet downtime?

[SuperQ]

And if you are a big enough customer you'll likely have fiber to one of the various pops out there, and you can just buy a cross-connect directly to google's peering network. I don't know if anyone that size has yet to sign up for hosted gmail. I know a couple of schools have rolled it out.

Re:What about internet downtime?

[Xugumad]

You apparently have an operational budget that probably has an extra zero over ours. We don't even have UPS for the network, which takes the whole lot out for a while every 3-4 months. Having had both the power substation for the cluster of buildings we're in, and the power substation for the network a few hops upstream, catch fire, hopefully they've fixed a lot of the old hardware that was causing those power outages, but...

I suppose if you've got the network, Google stuff makes sense. For us, we'd love a lot of these things in appliance version, but depending on them day to day wouldn't make sense.

Re:What about internet downtime?

[SanityInAnarchy]

If you're using Gmail, you could avoid this problem with an IMAP client. Of course, that may defeat the point of Gmail...

The other cool point about web apps is: If your Internet goes down, you can go wandering. Maybe there's some nearby wifi you can leech. Maybe you go to Starbucks. Maybe you go home.

With an internal server, if you lose power, you're SOL. If your server is down, you're also SOL. If a worm runs rampant, bogging down network traffic and even making your desktop unusable, guess what? You're shit-outta-luck.

Re:What about internet downtime?

[Splab]

It's called backhoes and they will eat up your uptime in one scoop.

Re:What about internet downtime?

[afidel]

Nah, our SONET loop goes out to two different streets and to two different POP's which are routed out separate directions. I know that isn't typical or universally available but we did our homework =)

Re:What about internet downtime?

[aaarrrgggh]

Google appliance unfortunately is just for search. Here's to hoping they add app support as well in the future.

Re:What about internet downtime?

[aaarrrgggh]

My company's IT budget is likely a good order of magnitude smaller-- 20 person organization. Not having your network on UPS is just stupid!

We are in a major metropolitan area, but we have a UPS for workstations even if they are being used by a part-time student just above minimum wage. We get about four hits a year, and that alone is enough for it to make sense.

We also have a T1 and ADSL from different providers. While automated failover isn't in place, it is on our list as time allows.

Re:What about internet downtime?

[Splab]

Well that's good thinking, I remember hearing about someone who bought connectivity from two different companies to have a redundant circuit, unfortunately both companies bought traffic through the same cable upstream, so every spring their connection got eaten by a backhoe.

Re:What about internet downtime?

[Xugumad]

For systems I'm directly responsible for, we've got three cases. There's desktop systems, which are actually laptops, so they have their own UPS essentially. Servers doing useful work have a UPS. Servers that can die and no-one notices for three weeks (real story) don't.

Unfortunately, I don't make the policy beyond my own research group. I hope they're going to start UPSing the network as they have funds, but it's going to be an ongoing task.

Re:What about internet downtime?

[pddo]

gmail does give you the ability to connect via POP or IMAP so storing your emails off line (even drafts) is no issue. Companies that use google apps simply needed to educate their users how to use this type of connectivity. Google calendar also lets you sync easily to outlook although they have not (as far as I am aware) provided syncing apps for other calendering apps. The biggest issue will be for google docs and other google apps for which there isn't an offline equivalent.

Re:What about internet downtime?

[sevrugaflocked]

Well - the client I have most difficulties with at the moment in an Exchange / Leased Line situation is at 20 x 5 downtime minutes for the last year, which given the usual level of reliability I expect is pretty horrendous - and despite the credit crunch we're making efforts / investment to make it more reliable. That's still 8.3 minutes a month, which would make it amply within Google's SLA's. Most of the leased lines are 99.9% SLA as well for example, but certainly as far as I'm concerned if any of my Leased Line operators actually delivered 99.9% uptime I'd desert them in a heartbeat for an inadequate service. And remember the above example isn't a 'mission-critical' application, which we would pour more resources on in any case. I suppose the three nines 'in the cloud' (shoot me if you must) will end up kind of be like the badge we all expect, but something which doesn't actually have any correlation to the actual reliability of the product - you know, like those regulatory badges on PSU's we never bother to find out what they're exactly about.

Re:What about internet downtime?

[lucifuge31337]

Now if you happen to use say, gmail, then you're out of luck. You can't read your mail, can't compose replies, can't IM people in the next room. All you can do is sit there and wait for somebody to fix the problem.

I can't im somebody, but I sure can queue outbound mail and read already delivered mail....I use IMAP with gmail.

Re:What about internet downtime?

[mikkelm]

Heh, you don't think Cisco equipment with redundancy up the ass goes down every now and then as well? Try managing 1500 boxes.

Re:Wait..

[Anonymous Coward]

Google is a company. Saying "Google doesn't have 100% uptime" makes as much sense as saying "Microsoft takes 40 minutes to install". What specifically are you trying to say?

3 9's is meaningless without customer support

[syousef]

The 99.9% guarantee is great, if there's someone to talk to who'll actually look at the problem when those three 9s aren't met. Otherwise it's marketing propaganda.

Re:3 9's is meaningless without customer support

[hackstraw]

The 99.9% guarantee is great, if there's someone to talk to who'll actually look at the problem when those three 9s aren't met. Otherwise it's marketing propaganda.

Keep in mind that most software comes with no warrantee whatsoever that it will be worth what the marketing propaganda says it will be.

Also, I don't think that Google would put out a product like this without adequate support. And if your only problem you have with the software is only getting 99.9% availability, then a simple status webpage would be suffice that says something like, "We are experiencing troubles, we anticipate having the system back up in 10 minutes".

In my eye, I see this as setting the bar a little higher in terms of system architecture and software engineering than anything Joe sixpack, Joe Executive, or Joe the plumber or myself have seen to date.

Re:3 9's is meaningless without customer support

[lysergic.acid]

so it's all propaganda unless you have someone to complain to on the phone? you know that customer support reps usually aren't the ones that maintain servers/networks or fix them when they go down, right?

if it makes you feel any better, you can pick up the phone and call your ISP and bitch at them until the problem is fixed. i mean, it's all the same. it's not like complaining to customer server/tech support ever gets a service outage fixed.

this is a service agreement. it states their company policy, and if they break the agreement you can file a lawsuit. what you're asking for is that Google waste resources on useless tech support line operators (which are usually outsourced to overseas call centers anyway) just so you can delude yourself into thinking that complaining to a customer service rep over the phone is going to fix any server problems.

if you need to feel busy when their service is down, why not do something that's actually productive?

Re:3 9's is meaningless without customer support

[cerberusss]

Google Domains its Premier membership is something like $45 per year and definitely has a support number which you can call.

Re:3 9's is meaningless without customer support

[Daengbo]

Well, you know what the BOFH would do ...

I was going to fix the service problem as quickly as possible, but now that you've called and complained, I guess I'll go to lunch instead.

Re:3 9's is meaningless without customer support

[syousef]

And what is it you want to hear from the person on the phone? That they know there is a problem? That they are working on it?

That they'll pass it on to someone who will momentarily know about it and work on it.

Re:3 9's is meaningless without customer support

[syousef]

so it's all propaganda unless you have someone to complain to on the phone? you know that customer support reps usually aren't the ones that maintain servers/networks or fix them when they go down, right?

If a company is willing to allocate staff to liasing with a customer, they're more likely to make the technicians aware of the problem in a timely manner and it is more likely to be fixed sooner. They're also more likely to have good technical staff if they have decent customer facing staff.

it states their company policy, and if they break the agreement you can file a lawsuit.

So you're solution is to pick an ISP with horrible service, not complain that the service is horrible, and sue when things go wrong. This is exactly why Americans have a terrible reputation for litigating. Never mind that usually it's just the lawyers that win as the costs skyrocket. Yeah that's a much better plan than doing your research and choosing someone with a decent level of service in the first place that has customer facing staff who can pass on problems to technical staff. Brilliant plan.

if you need to feel busy when their service is down, why not do something that's actually productive?

Tell that to someone who trades in futures. I'm told 10 minutes can be the difference between profit and bankruptcy

Server uptime is not the issue.

[B5_geek]

The issue is your internet connection AND your ISPs connection to the world. Your connection to the world is more likely to go down before a Google cluster would. Think of how often Telco's, ISP, and major hubs go down. This is the point behind having LOCAL copies of apps/servers/services, the odds that the hub/switch dies (with nothing else inhouse to patch around) is very slim compared to the odds of internet connectivity going south.

Re:Server uptime is not the issue.

[Predius]

As a commercial user of Google Apps, I have observed this not being the case. GMail does go down, and the cause is not our connectivity. What's worse is when there is a problem, all the 'phone support' does is tell you to post on their forums... not impressed.

Re:Server uptime is not the issue.

[Predius]

Gee... you don't think I haven't brought it up, multiple times, with data? I pointed out the pitfalls before we jumped in, and we got bit. If I had control we'd be off GMail, but it's not my final decision.

That doesn't make my observation any less salient.

Re:Server uptime is not the issue.

[glwtta]

Think of how often Telco's, ISP, and major hubs go down.

Very rarely? I worked for three companies over the last 7 years, I can remember losing internet connectivity exactly once: we were down for 3-4 hours after a construction crew damaged our T1 line.

Hell, I lose my home connection only once or twice a year.

great, now all we need

[inzy]

is an internet service that's 99.9% reliable, or this is all moot

Re:great, now all we need

[afidel]

Even my very worst site out of 13 that I monitor has 99.8% availability, if you are getting much worse than that then I strongly suggest you change ISP's.

Your Net

[hhawk]

It seems correct that Google's end of the network works very well.

The other side of the network, yours, is the other consideration; how good your connection? LAN? desktops? Etc., Etc...

Then beyond that, i've used Gmail since 2004 from Korea to Paris and NYC to Cali... I've had it run fast and slow.. is that the Google Server? the network? my computer(s)? I would think it's mostly network congestion but that's a hard one for an average user to determine (where and why and how to fix).

Nothing has 100% uptime

[EsJay]

If your organization will fail without 100% email uptime - bon chance in the real world, mon friend, bon chance.

Make sure your users have a phone directory available on their local PCs (or paper copies on their cubicle walls). Have a phone tree notification system scheme in place in case the network is REALLY down.

And prepare for the troublesome PRODUCTIVITY SURGE when your users cannot reach the Internet!

Re:Nothing has 100% uptime

[Edzilla2000]

It's "bonne chance"...

Re:Nothing has 100% uptime

[tomhudson]

It's "bonne chance"...

... his Google Apps spellchecker only has a 99.9% SLA, you ignorant clod!

Re:Nothing has 100% uptime

[turbidostato]

"... his Google Apps spellchecker only has a 99.9% SLA, you ignorant clod!"

Yeah, but he missed 2 characters out of 12, that makes just 83.3%, you spellchecker SLA infringing clod!

Still beta?

[Anonymous Coward]

If their service is so solid, then why not remove the 'BETA' tag from Gmail?

Re:Still beta?

[David Gerard]

The service they sell isn't beta. The service they give away is what they inflict new features on.

Re:Wait..

[Drakonik]

A Beowulf cluster?

What I actually posted

[David Gerard]

was their claim that this is 4x less outages than on-site-maintained Exchange or GroupWise.

(Notes, of course, gets 45 minutes of uptime a year.)

wouldn't like to be customer support on this one.

[timmarhy]

fuck that, imagine how many disputes they will have with retarded IT departments who can't manage their own network properly causing goolge apps to be unavailable. they are going to get the blame for every isp fuck up that happens as well.

Wow, that's pretty terrible

[yttrstein]

I achieved four nines (%99.99) 8 years ago with Netscape's broken mail server "Suite Spot" running on a (at the time) three year old Sun E450 with 4 gigs of RAM. As I recall, it served about 120,000 clients on a large cable network in Chicago.

This whole "new web" thing is very pretty, but it seems like about three steps back to me.

Re:Wow, that's pretty terrible

[hax0r_this]

That may be true, but what you were able to achieve and what you guarantee clients you will achieve are two very different things.

Re:Wow, that's pretty terrible

[TheLink]

The penalty for failing to achieve it also influences the guarantee.

Re:Wow, that's pretty terrible

[Plutonite]

I hate to tell you this, but your 'Homepage' website is down. What happened to the 4 nines now, batman? :)

Re:Wow, that's pretty terrible

[gillbates]

Yeah, but you need to remember that ten years ago, vendors like HP, Sun, DEC, and SGI (IIRC) were putting uptime guarantees in their marketing. IBM wouldn't guarantee uptimes, but they would state the average mainframe had less than 5 minutes of downtime a year. Strangely, Microsoft was pushing Windows as "Enterprise Class", but would make no uptime guarantees. Guaranteeing your clients 5 nines of uptime is nothing new in the enterprise market. It wasn't until Microsoft started pushing Windows as a server that people started to think of 5 nines as some kind of remarkable feat - it had been done in the industry by IBM mainframes and UNIX machines for decades prior.

Re:Wow, that's pretty terrible

[aaarrrgggh]

I was actually working with a client on an SLA with a co-lo facility recently. The co-lo would only commit to three nines, despite the fact that their infrastructure should theoretically support a solid four nines. They keep that extra order of magnitude in their back pocket-- be it to improve profit margins later or "value-added" services soon to come.

You have to keep in mind that statistical reliability and real-world availability are different animals. Some of it comes down to luck, and some is about looking at the full life-cycle.

Re:Wait..

[game kid]

It's a King Arthur cloud, maaan. Get with the times!

Re:Wait..

[TooMuchToDo]

On a related subject, next person who says "in the cloud" is going to get cockpunched. As parent said, there are no clouds, just highly available clusters.

Re:Wait..

[Worthless_Comments]

I am a cloud, you insensitive clod!

Re:Wait..

[moosesocks]

There'd be no need for a Beowulf-type cluster in this case.

Have a bunch of machines running identical instances of Apache, and randomly fire requests at them individually. This balances the load, and ensures that the servers themselves aren't a single point of failure.

It's quite a bit more complicated than this in reality, although you should get the basic idea.

Beowulf is typically used for clusters that seek to emulate a supercomputer (usually for scientific number-crunching), rather than a server. For this reason, something like Google's setup would more typically be referred to as a "server farm"

Push

[pbrammer]

Gmail might have a better uptime than Exchange, but at least Exchange has push-email.

Re:Push

[networkzombie]

Push email is actually very important when there are donuts in the break room. When you alert everyone they all get the email at the same time and no one gets left out of the Monday morning cofee and donuts feeding frenzy (gotta be fast to get the eclairs, though).

Re:Push

[cgenman]

Gmail might have a better uptime than Exchange, but at least Exchange has push-email.

Why is push better in practical terms than, say, a 1-minute pull?

Re:Push

[TrancePhreak]

Power useage on a phone with push can be much lower. A 1 minute poll means your data connection is constantly active. Push means the message goes through the phone network and connects to you only when necessary. It's kind of like SMS, but instead it goes to your email.

Re:Push

[pbhj]

Why is push better in practical terms than, say, a 1-minute pull?

How many "donuts" [sic] can you eat in a minute ...?

Re:Wait..

[Anonymous Coward]

Yeah, punch those bastards. Punch 'em so hard they'll go flying up high in the sky. In the cloud, even.

Re:Wait..

[osu-neko]

Google doesn't have 100% uptime? They have never gone down when I've noticed, guess its that sweet cloud setup they have there.

Seriously? I see it happen at least once every few weeks or so. It's usually very temporary, like as in less than a minute, but I'm quite familiar with the look of Google's error/service unavailable page...

Re:Wait..

[glwtta]

I thought it was a really annoying buzzword for compute capacity as a service?

Re:Wait..

[Midnight Thunder]

It's called a cluster, "The cloud" is a really annoying buzzword for software as a service.

An from my experience clouds are full of unpredictable vapour and they tend to have this annoying tendency to turn to rain - not really something I would want for my data ;)

What is more imprtant is...

[rcamans]

How much data loss occurs? big corp email loss is bigtime (I worked at one of the biggest, "most knowledgable", and exchange server crashes were frequent. You just had to have email set to move to a local folder. Of course, the local machine was also running M$windoze, so you lost it there.
Network connectivity issues were rare (think "invented the internet").
Lost, corrupted, unavailable, and stolen data must be the primary determinants of usability. So unavailable comes in third. Think backup tapes, and you will see that although not easily available, buttsave is possible has long been good enough.

So the questions are, in order:
How much mission critical data is lost? one major contract or project is too much.
Corrupted data is probably useless, nearly lost.
Can google data be restored? I have not heard.
Can google apps be hacked?

Re:What is more imprtant is...

[rcamans]

I actually have never worked for a blowhard full of hot air and lies.
The guys who invented ethernet invented the internet, in my opinion.
I would have to "think" of Al Gore using "anti-think." Where did you learn "anti-think"? Oh, yeh. I forgot. That is what they teach in most colleges. I actually am taking a critical thinking course right now, and even in that course, I am able to use critical thinking to blow apart some of the class's basic propositions.

Funny boy.
Heh heh.
You made me laugh.
I always appreciate a good laugh.

What's The Recourse?

[bschorr]

So if somebody is running their business on the free version of Google Apps and they have more downtime than the SLA allowed exactly what do they get? A refund? And by the way, I'd like a list of those Exchange customers who are suffering 2.5 HOURS of downtime per month. Sounds like they should be about ready to change service providers...

Re:What's The Recourse?

[Daengbo]

So if somebody is running their business on the free version of Google Apps ...

He/she deserves to lose his/her business. If your business depends on something, you make sure it's reliable. That generally requires money somewhere.

Re:Wait..

[Anonymous Coward]

I am a clod, you insensitive cloud!

99.9 is not so good

[mschuyler]

I dunno, if my servers had 45 minutes of unplanned downtime per month, I think the condition would be called 'chronic repeated failures' and be subject to some 'employee counseling.' I can understand planned, scheduled downtime after hours, but I don't think that's what they are saying here. Our users get nasty when the net is not available for 20 seconds. 45 minutes a month isn't acceptable around here. And saying, "Hey, this isn't a hospital. It's not as if anyone was at risk!" is not something you'd want to say either. Nope, I think they need to add some more nine's there. Not something to earn you bragging rights or put on your resume.

Re:99.9 is not so good

[mschuyler]

Google isn't a 9 to 5 shop, I don't think.

Re:You don't get what you pay for

[palegray.net]

Your POTS isn't delivering five nines worth of uptime. You just aren't on the phone 24x7, so you don't notice.

I've got a couple of decades of experience with AT&T systems. Oh, and I'm a programmer.

Microsoft has 5 nines ...

[tomhudson]

0.00099999.

Hey, it's five nines ... and with all the "exceptions" and bogus metrics in google's SLA, they're not offering 3 nines.

Re:well well well

[ozphx]

Wading through Coward's latest dense prose, I felt disappointed. What could have been a truely epic troll was falling flat. I suspect that by padding out the homosexual references somewhat, and by pacing himself, Coward could have really delived with this one. There was some unique content here but terrible delivery.

A poor performance. One and a half stars.

99.9% is merely good advertising copy.

[Dzimas]

Imagine an airline that offered 99.9% reliability. Or a new car that runs only 99.9% of the time. 45 minutes per month of downtime sounds reasonable, until that 45 minutes happens to take our entire organization's email system offline for a few adrenaline-filled minutes on a busy Monday morning. Cloud computing is still in its infancy, and will be until the "cloud" offers near-perfect redundancy on both a software and network level.

Re:99.9% is merely good advertising copy.

[Shados]

Cars don't get anywhere close to 99.9%, since just a flat tire can take you out for a while, nevermind a broken transmition. If you need to let the car sit overnight, thats pretty much done for. And well over 50% of flights are delayed, even just 99% reliability is a pipe dream (i know you basically meant plane crashes, but still).

Re:99.9% is merely good advertising copy.

[pbhj]

Cars don't get anywhere close to 99.9%, since just a flat tire can take you out for a while, nevermind a broken transmition.

Example, I ordered a new shock absorber and was sent the wrong part ... silly me removed the old one (broken) before checking the new one. Downtime 1 week = 2% (send back parts, order new ones from a company that knew what part they should have sent me!).

I'd guess adding up time for refuelling would give you less than 99.9% uptime.

Re:Wait..

[mortonda]

Seriously? You must not use it much.

Sysadmin = Punching Bag

[lucm]

When Google is down, all you get is access to lousy forums with little or no support, while your end users keep asking for an ETA or at least for an explanation. You end up being a punching bag for the failure of a solution you probably never agreed with and that was forced down your throat by the management.

I guess this is an ok deal for small biz with no technical employees, but as soon as your users headcount goes over 20, Novell Groupwise or Microsoft Small Business Server becomes more interesting. And when hosted locally, it will at least work as internal groupware and allow users to access shared documents while the internet connection is down.

Re:Sysadmin = Punching Bag

[krenaud]

Google offers telephone support for "System critical issues" if you are a paying customer. I haven't had the neeed to try that support so I can't comment on the quality of it.

Re:Sysadmin = Punching Bag

[lucm]

One of our sysadmins called and basically was told that some accounts were temporarily unavailable (duh!), that the problem would be solved as soon as possible, and to check the forums for public announcements. (And yes, we are a premium customer).

That's how you spot crappy support: when you get a list of symptoms (that you already know) and no specific cause or resolution process other than "the engineers are working on it". No valuable info to pass to the end users to calm them.

Re:Sysadmin = Punching Bag

[loyukfai]

Guess when Google's guaranteed 99.9% services is downed, she is probably also trying to get it back up ASAP, without any mails, calls and/or forum postings asking she to do so.

I haven't looked into the contract terms, but if the remuneration paid back by Google for not achieving the 99.9% uptime cannot mitigate one's agony (in case it happens)... There are plenty of other options out there.

Re:Sysadmin = Punching Bag

[lucm]

Customer care basics: communication! Keep your customer aware of what is going when there is a problem, tell him what you do to get it fixed and try to provide an ETA for the resolution. This way the user don't feel like he is kept in the dark and stress level will go down a notch or two. Look at how Tylenol handled the poisoned bottles crisis in the early 80's; this is textbook crisis management.

Google is going the other way: being vague, hiding behind its interpretation of the SLA, etc. They might be very good in the search engine business, but as a service provider, and especially in the customer management area, they suck big time. Their downtime is not the biggest problem; its the nonchalance with which the service disruptions are handled that is unpleasant. Email and groupware services are the lifeblood of many companies, and if Google cannot offer a decent customer service for the money they ask, they should either raise the fees and behave like a responsible provider or get out of the business.

Penalties?

[Jeff Hornby]

Google guarantees 99.9% uptime, right? So what do you get if they don't deliver? A lollipop? A cookie? A profound apology personally signed by Larry and Sergey?

Actually you get extra time.

If the system is down for betwwen 45 minutes and 7.2 hours, you get an extra three days. &.2 hours is pretty much a full business day if it starts at the wrong time.

If the system is down for 7.2 hours to 36 hours you get 7 free days.

And if the system is down for more than 36 hours you get 15 free days.

I don't know about the rest of you, but most of my clients would be losing at least tens and perhaps hundreds of thousands of dollars an hour if all of their key systems went bust. Email is down? No communications because not only is that a communication channel, that's also where you keep most of your contact information. Productivity suites are down? There goes work for the entire office for the duration. Not only are they unable to create new documents, they're unable to access existing information.

You can say what you want about Microsoft Office (or even move to something else like OpenOffice or StarOffice) but at least when something happens to Office, it only stops one user. If Google goes down, your entire enterprise grinds to a halt for the duration.

Re:Penalties?

[blincoln]

That was my question as well. I think it's especially important to have answered given that Google has already failed to meet a 99.9% uptime at least once this year.
When they fail in that way, how long is it before they are able to make a particular uptime claim again? Apparently the answer is "less than a few months" at best. I suspect it's actually "immediately" instead of the more appropriate "when they prevent their service from going down long enough to meet the stated uptime percentage including statistics for as long as the service has been offerered."

Re:Penalties?

[Daengbo]

It amazes me that so many sys admins are worried about the availability of Google, which is a company that bases its whole existence on clustering and high-availability.

I want to know how many of these guys have over three nines in-house. I smell a conflict of interest in their responses.

Re:Wait..

[theeddie55]

so i'll be an insensitive 0.1% cloud to complete the cycle.

Planned vs. Unplanned

[valen]

Upgrades are planned to have no impact on availability. Check the article; they have no 'unplanned' outages. I don't think Google have ever suffered a user-visible data loss; when they talk about downtime, it means user connections can't get to the backends for some reason. If your network connection dies, Google Docs retries till you get back in coverage. Hopefully that'll happen sometime.

If you plan to take out a service with an upgrade, you are doing it wrong. Seriously.

Re:Wait..

[valen]

  Um...not as such. There are many parts of Google Apps that are better off running on a cluster than a 'server farm'.

  Especially searching. If you need to search through your 100GB of indexed documents, you want to be able to transparently break up that search query over multiple machines.

  Obviously, a cluster is just a fabric running on top of a server farm. But having a fabric that spots slow/loaded machines and repairs or unloads them, and gives a rich API to things like GFS/Bigtable/Chubby/Mapreduce etc. means a big difference to Apps developers.

cloud data exposed to zillions

[datadefender]

it might be true that Google apps have better uptime than any internal application. My big fear is that anything in the cloud is physically exposed to the "the world" including some 1000 bad guys. Any Software has bugs so its just a matter of time that these bad guys will steal/erase/modify my data in the cloud. It has happened so often and will never stop. Granted, Exchange also has bugs (of course) - but at least these are only exposed to the people in my company - a few 1000 rather than zillions. So no matter ho trusted and reliable a cloud provider might be - I wont trust my data to the cloud.

Re:Wait..

[dkf]

If you need to search through your 100GB of indexed documents, you want to be able to transparently break up that search query over multiple machines.

Actually, it's building the index of the documents that is especially computationally intensive. Particularly chunky is the algorithm to assign a significance score to each document. Once you've done that, actual searching can then be done by merging streams of information suitably, which it is pretty easy to do fast.

Re:Wait..

[pbhj]

Google is a company. Saying "Google doesn't have 100% uptime" makes as much sense as saying "Microsoft takes 40 minutes to install". What specifically are you trying to say?

If you're going to be pedantic ... Google is not a company. Google is the name of a company.

Yes I'm sure someone can one up my pedantry, something about visual representations of sound tokens representing names, or somesuch ...

99.99% 2 days outage I had recently. No way!

[Anonymous Coward]

i had a 2 day outage which just said 'sorry an error has occurred' there is no way i could be without professional mail or documents for 2 days. I would only consider this for social non-critical uses.

Re:Wait..

[AlecC]

And if you enter a cloud, you usually get lost. Flying in clouds is regarded as very bad thing if you can void it.

Counter-pedantry

[ais523]

If you're going to be pedantic ... Google is not a company. Google is the name of a company.

Yes I'm sure someone can one up my pedantry, something about visual representations of sound tokens representing names, or somesuch ...

Nope. Google is a company. "Google" is the name of a company. The use-mention distinction strikes again!

Re:Counter-pedantry

[pbhj]

Err, yes, -ish.

Use-mention distinctions are not necessarily indicated by quotation marks.

In Warll's statement [GGP] "Google doesn't have 100% uptime" it was clear that Google is a placeholder for a real-referant and not a mention. But that real-referant is not *the company bearing the name "Google"* [GP] but a non-specific service of that same company.

But it seemed more pedantic to claim contrary to the Anon. Cow. [GP] that it was a mention, by which [of course] I do not mean it but Google.

Clear?

Gmail is broke!

[Pope Raymond Lama]

For me it happens that my 0.1% share of service outtage maps directly to the functionality of sending e-mails?

I get an "invalid from address" whenever I try to send a message through gmail's interface. The "support" for this is almost impossible to find out, and there has been no feedback from my complaint there. And yes, I've tried several settings that could affect the "from" address already (and it worked up to 2 weeks ago)

Re:Wait..

[quarterbuck]

You control clusters, but clouds are really not under your control -- Google controls them and you only worry about your data. The seperation of ownership of data and computers is the first difference.
Another is the fact that you don't need a cluster to form a cloud - I could set my computer to provide some sort of document processing and offer it as a service to you. I only have a computer, while your computer and mine don't form any meaningful cluster. You do receive services from me, which constitutes a "cloud".
Similarly, in clusters each node is aware of other nodes or central servers (more or less) , but in a cloud it is the client which makes a lot of decisions - for example the Google Docs can all be on a cluster while Google mail is an entirely on a different cluster - If I provide you a service which couples both of them, then it is the client machine which decides which cluster is going to be used .
Cloud encompasses a cluster, but also has the idea of specific services that run on it.
They are different, but if you don't like the word cloud, thats different. I believe IBM came up with the term and for lack of a better term, it stayed.

Re:Wait..

[xombo]

I just completed a report on this subject for a class that I've posted online. I discuss the specifics of Google's clustering technology.
http://googlepleasehireme.com/

99.9% uptime of a pain in the butt?

[Roadkills-R-Us]

Google apps is NOT enterprise ready. It's taken us a month, an outside consultant, and a week's worth or intermittent, screwed up email to even get close to what we had before, email-wise. We haven't had any time to work on calendars, etc. It was extremely difficult getting google's attention at all, much less a path to anyone who could actually help. This has been the most painful rollout I've worked on in years.

"It all depends on what your definition of 'evil' is."

YMMV. I would only recommend google apps to a competitor I wanted to hurt. 8^)

Re:Wait..

[afabbro]

I just completed a report on this subject for a class that I've posted online. I discuss the specifics of Google's clustering technology. http://googlepleasehireme.com/

You don't seriously expect Google to hire you based on that "paper" do you? It's 5 pages, of which 3 to 3.5 are consumed with big oversized (and sometimes uncaptioned) pictures and whitespace. The rest reads like a marketing brochure written for people with no technical background.