Slashdot Mirror
Outages Leave Google Apps Admins In the Hotseat
snydeq writes "This week's Google outages left several Google Apps admins in the lurch — and many of them are second-guessing their advocacy for making the switch to hosted apps, InfoWorld reports. The outages, which affected both Gmail and Apps, 'could serve as a deterrent to some IT and business managers who might not be ready to ditch conventional software packages that are installed on their servers,' according to the article. 'If we began to experience a similar outage more than about two or three business hours per quarter, we'd probably make Google Apps and Gmail a backup solution to a locally hosted mail system, if we used it at all,' said one Apps admin. 'And it would likely be years before we'd try a cloud-based collaborative system again from any vendor.' Coupled with recent Apple and Amazon cloud issues, these Google outages are being viewed by some as big wins for Microsoft."
When my boss tells me he wants 0 downtime (or even five-9 downtime), I show him a quote for the 7-figure cost of creating such a system.
Apparently Google is expected to hit that level of uptime all while charging either nothing for their standard edition or $50 per user per year for the premier.
I wonder how much downtime the companies that are using Google Apps would experience if they had to pay for their own redundancy?
I'm a big tall mofo.
It is not a big win for Microsoft, it is a big win for corps hosting their own app servers. I would think that eventually Google will release google apps on a server that corps could install in their own data centers.
Google Apps Premier is not free - it's 50$ per year per account.
I'm using it for my private mail. I like it. But i don't expect 100% uptime - especially for just 50Ã per year per account.
we only have one or two unexpected downtimes per year
What about your planned downtime? If you're running Windows, you're rebooting to install patches on a regular basis or you're running unpatched systems. What about software installs?
In the context of the article, do you think the users of Google Apps (or any users) would be happy with, "Oh, no you don't understand. This is PLANNED downtime. This doesn't affect you or our downtime numbers."
you can have 0 unexpected downtime with a single server, if you are lucky.
You can win the lottery too, if you are lucky. How many people win the lottery though?
I'm a big tall mofo.
Google has a Service Level Agreement. If they have excessive downtime, you can get up to 15 days of free service. No refunds.
Tell that to your boss. It's not your problem. That's what the company signed up for. Welcome to "cloud computing".
Do you honestly believe that you or your employees are going to build a system with higher availability than Google? In the magical fantasy world we all wish we lived in, you may have the budget, skill, manpower, and infrastructure resources to do this. In the real world it is not even remotely possible. I know how much it sucks when your system is down and there's nothing you can do but wait on some status dashboard to from Red to Green. That said, we should recognize that while being frustrated at this lack of control is normal, that doesn't mean you actually could do it better. It's easy to say "this would have never happened if we were self-hosted" while never thinking about the bullets you dodged by running hosted applications.
That means you, as a single customer, are insignificant. And that shows daily when dealing with any large service provider.
The only thing that my service provider should care about is the availability of the platform. I am completely insignificant, but the only reason my hosted app would be down is if the platform is down, and that sure as hell is significant to them. The advantage of hosted applications and cloud computing is that no one needs to ever look at or touch my app, the platform is all that matters.
Because it's a refutation of Google's business model (cloud based, for want of a better way of phrasing it) compared to Microsoft's (locally based tech).
I remain sceptical, as it it would seem that Google would have to be less reliable than local kit in order to make it worth switching back, even before you take into account extra costs for doing it locally. (How much more do you want to spend to get an extra hour per quarter in reliability?)
Nevertheless, IANASA so I don't know the data behind this decision.
All intents and purposes. Not intensive purposes.
I scan Slashdot nearly every day and didn't remember seeing anything about outages at Google this past week. A search through the story history confirmed that fact. So I thought I'd visit google.com and see what Google itself had to say. Nothing on the blog; nothing in the press section.
So why is this the first time these outages have been discussed here? From reading the article it appears we're talking about multiple outages over the past couple of weeks. Doing a Google search for "google outages" brings up one blog posting about these recent events. The blog posting includes this unsourced quotation, "Google spokesman Andrew Kovacs said via e-mail that 'a small number' of Gmail users and 'some' Apps users were impacted by the problem, which is still outstanding and being worked on as of 5:30 p.m. U.S. Eastern Time on Friday."
So all these events seem rather shrouded in mystery. How big was the outage? What explanations did Google give for the outage? I've certainly had servers go down, lost network connectivity, etc., etc., but I don't maintain huge server farms with enormous redundancy and multiple high-bandwith connections to the Internet. I don't recall search on Google ever going down; what's up with gmail and Apps?
The suspicious among us might start to think that outside parties might be responsible. After all, if companies start migrating to the "cloud," disrupting those services could have a substantial, economy-wide impact.
Those IT manager using the free service and expecting mission critical uptime should really go out more often and get a grip on reality.
Let's see, to set up my own five/nine email servers I would need at least two hosting location on different backbone, each of them should have at least two redundant servers. And of course I should have one spare that I can ship express whenever one fail.
Fixed Cost (Investment)
Monthly Recurring Cost
Implementation time
Of course I pulled the numbers out of my hat but it should be enough to show that there is no way a SOHO will ever have the mean to do it and that it is unrealistic to expect that kind of service for free or cheap.
We have all seen it. Ebay a couple of years ago going down due to Oracle corruption. Royal Bank of Canada failure due to an improper software upgrade. Now, Google with Gmail and other Google Apps failing. All of these organizations were geared towards having the highest uptimes available and failed spectacularly.
Whether you host your own or use someone else its the illusion of control that somehow clouds our judgment into believing that it would somehow be different if I did it. Example: Is it better to drive or fly? Pure numbers state that its safer to fly on a commercial carrier by an order of magnitude but somehow we feel safer when we drive. Whether we choose to acknowledge it or not the world is full of 6 sigma events. As long as you are doing everything you can and within your budget when your hosting your own apps or auditing your provider to ensure they have, backup systems, redundancy, offsite bunker, etc. then you have done everything you can to prepare for this inevitability.
In a lot of ways designing systems is like playing poker. You can play your hand perfectly, design all the systems redundancy and recovery you like, but sometimes even after all that your opponent (risk) draws a lucky card on the river to beat you. Just because you got beat doesn't mean you shouldn't continue to play the same way, it just means you hit one of those events that you cannot plan.
There are only two IT solutions out there in the minds of too many people: Microsoft, and non-Microsoft.
To go with Microsoft is the easy, sure road. It is the standard. It is what is expected, what is known to be safe, what will always work. Any problems you encounter here are met with "well, computers always have problems don't they?"
To go with non-Microsoft is hard and uncertain. It is not expected, nor "the standard", and suspected to be extremely unsafe. The smallest problem will be countered with "you and your stupid ideas. Now go and call LocalRetailerInc for a certified Microsoft solution, and be glad I don't fire your ass over this fiasco!"
Google is not Microsoft, so according to the business logic described above, if it doesn't work the only possible alternative is to use Microsoft.
Two weeks ago a transformer blew out in the building I work in. First there was no power for 3 hours, then temporary power as a large generator was hooked up, but it was not big enough to run the AC, so we did no turn on the servers. It took another day to get a large enough generator (about the size of a tractor trailer). In total, our business was shut down completely for a day and a half due.
I don't think you can even get a SLA from the power company.
Google Apps went down for 3 hours.
Shit happens.
We ran into one of these "gotcha" features in hosted Gmail that's been giving me fits and it all started with a simple mistake. I misspelled a user name. You can change the spelling in the admin module, but it doesn't change the spelling in the contacts and the misspelling still showed up when she logged in. So I tried deleting the user name and recreating the account.
Big mistake.
When you delete a user name you can't recycle it for five days, which pushed us past our roll out date. Their crip work-around is creating a mailing list with that user name. But that has its own set of problems, especially when trying to migrate a large number of users. There's no support unless you get the premium edition. So now we're stuck in the position of paying for support on a service we're not certain will work for us. I'm not inclined to throw money at something to see if it will work when what we're already paying for is working.
Unfortunately, it was one of our key sales people who already had that account name on her business cards. Rolling without her is a non-starter.
It's frustrating because I'm the one who recommended Google and I feel really let down. It's a stupid problem that shouldn't exist in the first place. Even if there's a good reason for it, there should be a giant warning banner with a flashing red neon border warning you that deleting a user results in a five day lock out. Actually, it's been more than five days and I still can't recreate the account.
This one niggling little incident is making me rethink hosted applications. So, yeah, it does sort of benefit MS. Not in our case, we're using hosted SendMail instead of Exchange, but if this type of "feature" deters other companies already using MS solutions, then yeah. Who wants to take a chance on looking bad? There will still be outages with any solution but no one gets fired for recommending MSFT. There's a certain period of time that users are looking for an excuse not to like a new service, just because it's different. If you can get past that time frame, then a small outage can be overlooked. But those first few months have to be smooth. Maybe not flawless, but close to it.
It would almost be better if the free version was a trial and corporate users could get support from day one. This is just maddening. Shape up, Google.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
In my company Google Apps is the most reliable thing we use. Microsoft products are my biggest headache. We have clients that need their work done and I don't have any more time to waste on these crappy machines. We will be switching to Apple for all mission-critical machines in the next three weeks.
If my MS computers could have only 3 hours of downtime a quarter I would be really happy. I used to work for an IT company and they primarily used MS servers for their clients. Big mistake. MS products are a nightmare. Their clients would have been happy with 3 hours of downtime instead of days and days down dealing with MS server issues. I would only avoid cloud computing if there were serious concerns with privacy or hacking.
Non-Microsoft can be :
I think the basic problem is impatience. I can understand that people want for business purposes something that is quickly implemented, but my experience is that when a Microsoft implementation is chosen, you have two long-term issues : you will time and time again have to solve the same problems over and over, and you can be sure that Microsoft will try to pressure you into upgrades, willing or not.
My experience with Linux and associated programs is this. Over time, everything gets better and better. Sometimes, you might need some time to investigate a problem and solve it, but once solved, it will not recur again (be sure that you have a good system to record such findings, but that would be same when using Microsoft).
I have already three people (not much, yes, but important for me) using Linux : my father, my brother (who shares with my father's PC) and my sister. Unless there is a hardware problem, I can be sure that I do not have to solve software issues on a regular basis, only help them with functional questions : what software to use and how to use it.
They use on a regular basis :
I am pretty sure that for most parts of a business, this would be enough.
Now, I think that the usage of Exchange is more of a perception thing, than a real technical obstacle. At my work, Lotus Notes was swapped for Exchange, but I do not consider this a progress, as it reminds me too much of PCTools 4.0 or 5.0 (about 1990) : I really do not see anything innovative in this area (and while some people here seem to loathe Lotus Notes (mostly without any reasons given), it was much faster than Exchange, I find speed very important for computer programs).
Anyone here which as implemented or is using alternatives to Exchange ?
The sign-up page for Google Apps Premier says you get 99.9% uptime. That's about 1/3 of a day downtime per year, or a couple of hours per quarter.
Google seems to be managing to hit that 99.9% uptime, just not exceed it. VERY few in-house e-mail systems actually manage 99.9% uptime, especially when you consider scheduled maintenance and downtime (remember, Google's 99.9% is for all downtime)
In fact, I have seen very few Exchange systems that manage much more than 99% uptime. However, for those organizations, there are other compelling advantages to Exchange.
ERROR: Null
I love being the asshole, but let's be honest here: how many in-house systems actually deliver better uptime than Google ?
Not that many. If they did, all us sysadmins would be out of a job. Apps are not perfect. The fact that you can pay Google a few pennies to manage your email, even with some downtime, makes it several orders of magnitude cheaper than an in-house solution for most people.
Give them a break, people can survive without email for a few hours.
-Billco, Fnarg.com
Our mail platform has beaten google in uptime and security "bugs" for the past 40 months. Why? I attribute it to using proven technologies and not everyone wanting an account being able to get one: we charge every system user. You would be surprised how much this cuts down on spammers/excessive usage.
Google has had their mail in beta for years. The last time I checked SMTP was ratified as an RFC over a decade ago.
Website Hosting
Precisely. And while I use Gmail most of the time, and the rest of my office use an Exchange server hosted in same said office, guess who has the better uptime...?
That would be me. They frequently (3-4 time a month) loose half a day, as the under resourced, high maintanence, auto-destructing, sorry updated, blackhat honey pot splutters in the corner. I've lost two half days in the however many years I've used Gmail.
Because you can - or because you should?
Linux has been rock-solid from version 1. Version 3 isn't being planned yet.
The main complaint against Linux is that it requires someone who "knows what he is doing". If the same is required of Microsoft solutions, then why not just use Linux?