Slashdot Mirror
Outages Leave Google Apps Admins In the Hotseat
snydeq writes "This week's Google outages left several Google Apps admins in the lurch — and many of them are second-guessing their advocacy for making the switch to hosted apps, InfoWorld reports. The outages, which affected both Gmail and Apps, 'could serve as a deterrent to some IT and business managers who might not be ready to ditch conventional software packages that are installed on their servers,' according to the article. 'If we began to experience a similar outage more than about two or three business hours per quarter, we'd probably make Google Apps and Gmail a backup solution to a locally hosted mail system, if we used it at all,' said one Apps admin. 'And it would likely be years before we'd try a cloud-based collaborative system again from any vendor.' Coupled with recent Apple and Amazon cloud issues, these Google outages are being viewed by some as big wins for Microsoft."
isnt there any other vendor out there providing business solutions ? its not like everyone is going to jump into exchange wagon because they couldnt do with google apps. geez.
Read radical news here
When my boss tells me he wants 0 downtime (or even five-9 downtime), I show him a quote for the 7-figure cost of creating such a system.
Apparently Google is expected to hit that level of uptime all while charging either nothing for their standard edition or $50 per user per year for the premier.
I wonder how much downtime the companies that are using Google Apps would experience if they had to pay for their own redundancy?
I'm a big tall mofo.
It is not a big win for Microsoft, it is a big win for corps hosting their own app servers. I would think that eventually Google will release google apps on a server that corps could install in their own data centers.
we only have one or two unexpected downtimes per year
What about your planned downtime? If you're running Windows, you're rebooting to install patches on a regular basis or you're running unpatched systems. What about software installs?
In the context of the article, do you think the users of Google Apps (or any users) would be happy with, "Oh, no you don't understand. This is PLANNED downtime. This doesn't affect you or our downtime numbers."
you can have 0 unexpected downtime with a single server, if you are lucky.
You can win the lottery too, if you are lucky. How many people win the lottery though?
I'm a big tall mofo.
Google has a Service Level Agreement. If they have excessive downtime, you can get up to 15 days of free service. No refunds.
Tell that to your boss. It's not your problem. That's what the company signed up for. Welcome to "cloud computing".
Do you honestly believe that you or your employees are going to build a system with higher availability than Google? In the magical fantasy world we all wish we lived in, you may have the budget, skill, manpower, and infrastructure resources to do this. In the real world it is not even remotely possible. I know how much it sucks when your system is down and there's nothing you can do but wait on some status dashboard to from Red to Green. That said, we should recognize that while being frustrated at this lack of control is normal, that doesn't mean you actually could do it better. It's easy to say "this would have never happened if we were self-hosted" while never thinking about the bullets you dodged by running hosted applications.
That means you, as a single customer, are insignificant. And that shows daily when dealing with any large service provider.
The only thing that my service provider should care about is the availability of the platform. I am completely insignificant, but the only reason my hosted app would be down is if the platform is down, and that sure as hell is significant to them. The advantage of hosted applications and cloud computing is that no one needs to ever look at or touch my app, the platform is all that matters.
I scan Slashdot nearly every day and didn't remember seeing anything about outages at Google this past week. A search through the story history confirmed that fact. So I thought I'd visit google.com and see what Google itself had to say. Nothing on the blog; nothing in the press section.
So why is this the first time these outages have been discussed here? From reading the article it appears we're talking about multiple outages over the past couple of weeks. Doing a Google search for "google outages" brings up one blog posting about these recent events. The blog posting includes this unsourced quotation, "Google spokesman Andrew Kovacs said via e-mail that 'a small number' of Gmail users and 'some' Apps users were impacted by the problem, which is still outstanding and being worked on as of 5:30 p.m. U.S. Eastern Time on Friday."
So all these events seem rather shrouded in mystery. How big was the outage? What explanations did Google give for the outage? I've certainly had servers go down, lost network connectivity, etc., etc., but I don't maintain huge server farms with enormous redundancy and multiple high-bandwith connections to the Internet. I don't recall search on Google ever going down; what's up with gmail and Apps?
The suspicious among us might start to think that outside parties might be responsible. After all, if companies start migrating to the "cloud," disrupting those services could have a substantial, economy-wide impact.
Those IT manager using the free service and expecting mission critical uptime should really go out more often and get a grip on reality.
Let's see, to set up my own five/nine email servers I would need at least two hosting location on different backbone, each of them should have at least two redundant servers. And of course I should have one spare that I can ship express whenever one fail.
Fixed Cost (Investment)
Monthly Recurring Cost
Implementation time
Of course I pulled the numbers out of my hat but it should be enough to show that there is no way a SOHO will ever have the mean to do it and that it is unrealistic to expect that kind of service for free or cheap.
We have all seen it. Ebay a couple of years ago going down due to Oracle corruption. Royal Bank of Canada failure due to an improper software upgrade. Now, Google with Gmail and other Google Apps failing. All of these organizations were geared towards having the highest uptimes available and failed spectacularly.
Whether you host your own or use someone else its the illusion of control that somehow clouds our judgment into believing that it would somehow be different if I did it. Example: Is it better to drive or fly? Pure numbers state that its safer to fly on a commercial carrier by an order of magnitude but somehow we feel safer when we drive. Whether we choose to acknowledge it or not the world is full of 6 sigma events. As long as you are doing everything you can and within your budget when your hosting your own apps or auditing your provider to ensure they have, backup systems, redundancy, offsite bunker, etc. then you have done everything you can to prepare for this inevitability.
In a lot of ways designing systems is like playing poker. You can play your hand perfectly, design all the systems redundancy and recovery you like, but sometimes even after all that your opponent (risk) draws a lucky card on the river to beat you. Just because you got beat doesn't mean you shouldn't continue to play the same way, it just means you hit one of those events that you cannot plan.
Another issue is web/network attacks. They are going up big time and are even state-sponsored. Look at what Russia is, and has been doing to Georgia.
I don't understand how anyone in this day and age can justify going with remotely-hosted applications. The ability to reach remote servers can be taken away even by morons and botnets who might not like your company.
In my opinion, remote web hosting of applications that are presumably important for a company to be able to run is just asking for trouble. I wonder how many fingers will get pointed when some critical deadline looms and nobody can run their applications to be able to meet it.
It's reckless and risky for business to expose themselves like that. As others have pointed out, OpenOffice is free and it is good. Why waste money on training people on both the Google (or other) remotely-hosted application and OpenOffice (if that is your emergency backup). Just train people on OpenOffice and now you don't need a backup plan in case the network goes down and you can't run the remote stuff.
Remote applications may have been a solution before the Internet got nasty but these days, running business-critical stuff over it when you don't need to does not make sense to me.
Maybe I'm missing the huge economic advantages that justify the unknown and growing risk, but I see network (Internet) applications as being at huge risk for outages, a security risk, a data privacy risk, etc.
Two weeks ago a transformer blew out in the building I work in. First there was no power for 3 hours, then temporary power as a large generator was hooked up, but it was not big enough to run the AC, so we did no turn on the servers. It took another day to get a large enough generator (about the size of a tractor trailer). In total, our business was shut down completely for a day and a half due.
I don't think you can even get a SLA from the power company.
Google Apps went down for 3 hours.
Shit happens.
We ran into one of these "gotcha" features in hosted Gmail that's been giving me fits and it all started with a simple mistake. I misspelled a user name. You can change the spelling in the admin module, but it doesn't change the spelling in the contacts and the misspelling still showed up when she logged in. So I tried deleting the user name and recreating the account.
Big mistake.
When you delete a user name you can't recycle it for five days, which pushed us past our roll out date. Their crip work-around is creating a mailing list with that user name. But that has its own set of problems, especially when trying to migrate a large number of users. There's no support unless you get the premium edition. So now we're stuck in the position of paying for support on a service we're not certain will work for us. I'm not inclined to throw money at something to see if it will work when what we're already paying for is working.
Unfortunately, it was one of our key sales people who already had that account name on her business cards. Rolling without her is a non-starter.
It's frustrating because I'm the one who recommended Google and I feel really let down. It's a stupid problem that shouldn't exist in the first place. Even if there's a good reason for it, there should be a giant warning banner with a flashing red neon border warning you that deleting a user results in a five day lock out. Actually, it's been more than five days and I still can't recreate the account.
This one niggling little incident is making me rethink hosted applications. So, yeah, it does sort of benefit MS. Not in our case, we're using hosted SendMail instead of Exchange, but if this type of "feature" deters other companies already using MS solutions, then yeah. Who wants to take a chance on looking bad? There will still be outages with any solution but no one gets fired for recommending MSFT. There's a certain period of time that users are looking for an excuse not to like a new service, just because it's different. If you can get past that time frame, then a small outage can be overlooked. But those first few months have to be smooth. Maybe not flawless, but close to it.
It would almost be better if the free version was a trial and corporate users could get support from day one. This is just maddening. Shape up, Google.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
Expecting five-9 or 0 downtime for a system used by only ONE company might be a very high expectation with a high cost vs. usage obtained from it afterwards.
But how many companies rely on Google's systems? When you offer your application or suite to the whole nation or WORLD, and campaign for its use - then YES, you do need to keep a very near-0 downtime to be really successful.
Cloud apps have the same problem. When google apps or EC2 go does, it's news.
In my company Google Apps is the most reliable thing we use. Microsoft products are my biggest headache. We have clients that need their work done and I don't have any more time to waste on these crappy machines. We will be switching to Apple for all mission-critical machines in the next three weeks.
If my MS computers could have only 3 hours of downtime a quarter I would be really happy. I used to work for an IT company and they primarily used MS servers for their clients. Big mistake. MS products are a nightmare. Their clients would have been happy with 3 hours of downtime instead of days and days down dealing with MS server issues. I would only avoid cloud computing if there were serious concerns with privacy or hacking.
The sign-up page for Google Apps Premier says you get 99.9% uptime. That's about 1/3 of a day downtime per year, or a couple of hours per quarter.
Google seems to be managing to hit that 99.9% uptime, just not exceed it. VERY few in-house e-mail systems actually manage 99.9% uptime, especially when you consider scheduled maintenance and downtime (remember, Google's 99.9% is for all downtime)
In fact, I have seen very few Exchange systems that manage much more than 99% uptime. However, for those organizations, there are other compelling advantages to Exchange.
ERROR: Null
I love being the asshole, but let's be honest here: how many in-house systems actually deliver better uptime than Google ?
Not that many. If they did, all us sysadmins would be out of a job. Apps are not perfect. The fact that you can pay Google a few pennies to manage your email, even with some downtime, makes it several orders of magnitude cheaper than an in-house solution for most people.
Give them a break, people can survive without email for a few hours.
-Billco, Fnarg.com
I migrated my company of 80 users to Google Apps hosted email about a year ago, and yeah, sometimes there has been interimitent issues. People want to use it like Exchange via IMAP, but there are quirky issues, like Thunderbird sending the wrong delete command, Thunderbird somehow corrupting the user's password (the only way to correct is to login to the user's account on the hosted Gmail site), etc. So there definitely are some quirks sometimes.
That said, it's free. Somebody a few posts back posted the cost of an RHEL install with server costs etc. Using Exchange, the price increases even moreso (license costs, CALs, etc.). Ultimately, you're getting a hosted, web-based email solution with the capability for shared calendars and document collaboration, all for absolutely $0.00.
Free vs. $20k+ solution? In my oh-so-humble-opinion, users can deal with (and quite frankly, should continue to periodically expect) some downtime.
Our mail platform has beaten google in uptime and security "bugs" for the past 40 months. Why? I attribute it to using proven technologies and not everyone wanting an account being able to get one: we charge every system user. You would be surprised how much this cuts down on spammers/excessive usage.
Google has had their mail in beta for years. The last time I checked SMTP was ratified as an RFC over a decade ago.
Website Hosting
The part that is being misunderstood is simply this. Instead of just complaining about Google Apps... compare it to the alternatives.
How many companies rely on Microsoft Outlook with Microsoft Exchange Server? When you offer an application or suite to the whole nation or WORLD, and campaign for its use - then YES, you do need to keep a very near-0 downtime to be really successful.
Except, Microsoft Exchange (while often reliable) does have its moments. Sometimes, just from getting clogged by tons of spam, it can come to a crawl. The server can become unavailable to do network issues. Microsoft Outlook has a tendency to run slowly on some machines, or crash regularly. Expecting ANYTHING that uses computers to work 100% perfectly all of the time, although desirable, is completely unrealistic.
I don't think the people here are saying "expect downtime and just deal with it." What is really being said is, "when MS Exchange goes down... or there are internal network hiccups... or when Outlook locks up on your machine... complain loudly on the Internet instead of to your local admin... that way, the world can get a real comparison between Google Apps and the alternative."
The only reason Google Apps seems like the "bad one" here is because people go posting on blogs and news sites about it. Why? Because it's news... it's rare... it's not what people expect of Google. When Exchange server craps out, Outlook locks up, your computer gets a blue-screen-of-death, a hard drive goes bad, a router needs restarting, power goes out to the building, a UPS battery goes bad, etc, etc, etc... nobody bothers posting this on blogs or news sites because, well, it's an every-day occurrence... it's not exactly news.
Then, when you compare systems that are "always up and available 24/7, can be easily accessed from outside of the company without a complicated VPN, have admins that don't gripe if they are taking up dozens of gigs of storage, with the capability of searching through millions of emails in a fraction of a second" to Google Apps... you'll likely notice that these other systems (with you take into account the cost of the servers, routers, admin hours, electricity, software, etc) cost much much more than $50/year per user.
What's happening here is people are comparing Apples to Orangutans and are creating unrealistic expectations. If these companies really have that much cash to just waste on something they have been brainwashed into thinking is perfect, then they're next likely step in these economic times is to lay off some of their admins because, after all, why do you need admins if the systems are perfect?
Linux has been rock-solid from version 1. Version 3 isn't being planned yet.
The main complaint against Linux is that it requires someone who "knows what he is doing". If the same is required of Microsoft solutions, then why not just use Linux?