Slashdot Mirror
Outages Leave Google Apps Admins In the Hotseat
snydeq writes "This week's Google outages left several Google Apps admins in the lurch — and many of them are second-guessing their advocacy for making the switch to hosted apps, InfoWorld reports. The outages, which affected both Gmail and Apps, 'could serve as a deterrent to some IT and business managers who might not be ready to ditch conventional software packages that are installed on their servers,' according to the article. 'If we began to experience a similar outage more than about two or three business hours per quarter, we'd probably make Google Apps and Gmail a backup solution to a locally hosted mail system, if we used it at all,' said one Apps admin. 'And it would likely be years before we'd try a cloud-based collaborative system again from any vendor.' Coupled with recent Apple and Amazon cloud issues, these Google outages are being viewed by some as big wins for Microsoft."
It is not a big win for Microsoft, it is a big win for corps hosting their own app servers. I would think that eventually Google will release google apps on a server that corps could install in their own data centers.
What about your planned downtime? If you're running Windows, you're rebooting to install patches on a regular basis or you're running unpatched systems. What about software installs?
A non-issue for these customers, as they do not run 24/7 businesses. Larger stuff can be done over the weekends, smaller stuff after everyone left.
For a 24/7 enterprise with distributed locations etc. things might be different.
But then again: This is "just" e-mail. While becoming more and more critical even in the smallest of businesses, there are worse things that could crash. For instance, your production control application that controls machines (if you are a producing company), or your ERP software if you're mainly selling oriented.
There also might be some companies that depend on E-Mail as their main business application, but these should then invest into providing highly reliable internal E-Mail. That costs more than 50$ / year / account, though.
You can win the lottery too, if you are lucky. How many people win the lottery though?
The chance of having a single server run through a year is much, much higher than winning the lottery.
I scan Slashdot nearly every day and didn't remember seeing anything about outages at Google this past week. A search through the story history confirmed that fact. So I thought I'd visit google.com and see what Google itself had to say. Nothing on the blog; nothing in the press section.
So why is this the first time these outages have been discussed here? From reading the article it appears we're talking about multiple outages over the past couple of weeks. Doing a Google search for "google outages" brings up one blog posting about these recent events. The blog posting includes this unsourced quotation, "Google spokesman Andrew Kovacs said via e-mail that 'a small number' of Gmail users and 'some' Apps users were impacted by the problem, which is still outstanding and being worked on as of 5:30 p.m. U.S. Eastern Time on Friday."
So all these events seem rather shrouded in mystery. How big was the outage? What explanations did Google give for the outage? I've certainly had servers go down, lost network connectivity, etc., etc., but I don't maintain huge server farms with enormous redundancy and multiple high-bandwith connections to the Internet. I don't recall search on Google ever going down; what's up with gmail and Apps?
The suspicious among us might start to think that outside parties might be responsible. After all, if companies start migrating to the "cloud," disrupting those services could have a substantial, economy-wide impact.
We have all seen it. Ebay a couple of years ago going down due to Oracle corruption. Royal Bank of Canada failure due to an improper software upgrade. Now, Google with Gmail and other Google Apps failing. All of these organizations were geared towards having the highest uptimes available and failed spectacularly.
Whether you host your own or use someone else its the illusion of control that somehow clouds our judgment into believing that it would somehow be different if I did it. Example: Is it better to drive or fly? Pure numbers state that its safer to fly on a commercial carrier by an order of magnitude but somehow we feel safer when we drive. Whether we choose to acknowledge it or not the world is full of 6 sigma events. As long as you are doing everything you can and within your budget when your hosting your own apps or auditing your provider to ensure they have, backup systems, redundancy, offsite bunker, etc. then you have done everything you can to prepare for this inevitability.
In a lot of ways designing systems is like playing poker. You can play your hand perfectly, design all the systems redundancy and recovery you like, but sometimes even after all that your opponent (risk) draws a lucky card on the river to beat you. Just because you got beat doesn't mean you shouldn't continue to play the same way, it just means you hit one of those events that you cannot plan.
Expecting five-9 or 0 downtime for a system used by only ONE company might be a very high expectation with a high cost vs. usage obtained from it afterwards.
But how many companies rely on Google's systems? When you offer your application or suite to the whole nation or WORLD, and campaign for its use - then YES, you do need to keep a very near-0 downtime to be really successful.
Do you honestly believe that you or your employees are going to build a system with higher availability than Google?
Why, yes, I do. I've worked as an enterprise technical architect for 2 of the baby bells and a 3 hour outage is outrageous.
I've designed systems that will fail over within 2 minutes with with under 30 seconds of data loss. The users just need to re-login and the load balancers (also redundant) will redirect them to a different data center 600 miles away from their primary location.
This solution is possible regardless of the crap code provided. And when you build the entire network, you know where the weaknesses are - and you aren't at the whim of some ISP for connectivity. Redundancy, management, monitoring and good overall system designs are your friend. Cost and cocky software developers are your enemy. Having fully tested DR solution for the price of a Prod/Test set of systems is win/win, if you ask me. Most of the time, executive management agreed with me and we built systems with less than 30 minutes of downtime for disasters even on the cheapest projects. We test fail over every other week by swapping the primary location as a matter of course.
Google's weakness is in believing that having 1,000 of CPUs is all you need to deal with redundancy.
You have to plan for outages.
You have to practice your fail over plan - at least every other week or on game day, when it counts, YOU WILL FALTER.
"Hope is not a plan." -JG
Do you honestly believe that you or your employees are going to build a system with higher availability than Google? In the magical fantasy world we all wish we lived in, you may have the budget, skill, manpower, and infrastructure resources to do this. In the real world it is not even remotely possible. [...]
Google has to run a massively sized setup catering to a vast diversity of customer types. Sure, they have more manpower and know-how than my employer's IT dept. But they have to distribute this manpower across a very wide field, working on dozens of products and issues in parallel. They have to deal with and prepare for basically any issue imaginable to the IT savvy part of mankind.
My employer's IT infrastructure, communications system and document/information management system are tailored to our needs. We have everything we need, but nothing we don't. We follow a safety and recovery protocol that reflects our business structure and priorities.
Short of the annihilation of Western Europe there aren't too many scenarios that would compromise our infrastructure in a way that would impact business. Why should we trade this situation for a little more convenience?
(Notwithstanding the fact that for legal reasons we won't actually even consider outsourcing anything more delicate than delivering lunch to our IT folk. Mere uptime is not our greatest concern.)
Rudolf Hess edited Mein Kampf. He was the very first grammar nazi.
I feel for you.
I'm trying to set up a collaborative network for a small non-profit organization. Right now, 'free (at least) as in beer' is not an option for any such tool.
However great Gmail's interface, Google Apps is not really ready for serious corporate/pro team use IMHO.
You may recall that up until a few days or weeks ago, members contacts were not automatically populated for new accounts--everyone had to manually fill them in for keeps. Although that was fixed, there's no way to create and propagate contact groups (as opposed to mailing lists) automatically--and mailing lists won't just cut it as users can't see who actually is on the list. Nor can they make new groups and propagate them to their sub-team.
Also, the inability to "force" chosen calendars into members calendar areas is almost a showstopper for us. I wonder how many larger corporations can cope with that. The way I'm working around this is adding calendar widgets to relevant sections and asking users to click the button and add the agenda to their calendars. We're still testing, but I'm not seeing satisfactory compliance...
Finally, support is egregiously non-existent. They have an apparently smart policy on the discussion groups: if you post a lot (presumably in response to people's queries), you get benefits. Well, I have yet to receive a single answer as to why "naked domain" addressing (http://mydomain.com/) used to work on my host but doesn't after CNAME modifications. And I'm getting 404's, not DNS errors! I may be getting what I'm paying for, but then again, my EUR 0 are worth a lot more on most other freeware communities around the Web.
For one, I've gotten better support and answers from the guys at Zoho.com, while on my free plan. Their offerings are also way more complete and functional, though slightly worse performance-wise, than Google's. Too bad they don't have 2nd-level domain customization (AFAIK) and, more importantly, they have a price for multiple projects we can't afford as of yet.
And don't even get me started on the lame Google Pages -> Sites app... Either you have a full-time coder dishing out Google Gear applications for you, or you're much better off clicking away at Wordpress or Joomla GUI admin pages... Layout templates? They look all the same. CSS customization? Severely limited. Actually I think you can customize your Blogger.com site more than you can Google Sites.
All in all, organizations of all sizes considering/using Google Apps have a lot more to worry about than a few sparse hours of email downtime...
Ok, enough with the tedious rant, but just a quick reminder to those using the free Apps edition: be advised that one of the main selling points of Google Apps, unified logins, is closed unless you sign up for the paid plan. You'll have to pay to let users login from your actually-usable-and-professional-looking blog/CMS into their Gmails, Docs and Calendars using the same account.
Non-Microsoft can be :
I think the basic problem is impatience. I can understand that people want for business purposes something that is quickly implemented, but my experience is that when a Microsoft implementation is chosen, you have two long-term issues : you will time and time again have to solve the same problems over and over, and you can be sure that Microsoft will try to pressure you into upgrades, willing or not.
My experience with Linux and associated programs is this. Over time, everything gets better and better. Sometimes, you might need some time to investigate a problem and solve it, but once solved, it will not recur again (be sure that you have a good system to record such findings, but that would be same when using Microsoft).
I have already three people (not much, yes, but important for me) using Linux : my father, my brother (who shares with my father's PC) and my sister. Unless there is a hardware problem, I can be sure that I do not have to solve software issues on a regular basis, only help them with functional questions : what software to use and how to use it.
They use on a regular basis :
I am pretty sure that for most parts of a business, this would be enough.
Now, I think that the usage of Exchange is more of a perception thing, than a real technical obstacle. At my work, Lotus Notes was swapped for Exchange, but I do not consider this a progress, as it reminds me too much of PCTools 4.0 or 5.0 (about 1990) : I really do not see anything innovative in this area (and while some people here seem to loathe Lotus Notes (mostly without any reasons given), it was much faster than Exchange, I find speed very important for computer programs).
Anyone here which as implemented or is using alternatives to Exchange ?
I am currently in the process of starting a new company. At the moment we only have four persons as staff, two if which (including me) work mostly from home.
We could never afford a dedicated server at this time.
Signing up for Google Apps was a no brainer for us and we are very happy with it so far.
We use Google Apps for our email hosting. We use email all day long, especially because many of our automated processes during the day kick out emails with various status reports.
What happened last week is the first time since we began using it (1 year ago) that I have not been able to get into my google mail. I'd say that's not too bad.
And further, I've been using gmail since it was released 4 (?) years ago, and I can't recall a single other time I haven't been able to access it.
That's pretty darn good uptime.
I migrated my company of 80 users to Google Apps hosted email about a year ago, and yeah, sometimes there has been interimitent issues. People want to use it like Exchange via IMAP, but there are quirky issues, like Thunderbird sending the wrong delete command, Thunderbird somehow corrupting the user's password (the only way to correct is to login to the user's account on the hosted Gmail site), etc. So there definitely are some quirks sometimes.
That said, it's free. Somebody a few posts back posted the cost of an RHEL install with server costs etc. Using Exchange, the price increases even moreso (license costs, CALs, etc.). Ultimately, you're getting a hosted, web-based email solution with the capability for shared calendars and document collaboration, all for absolutely $0.00.
Free vs. $20k+ solution? In my oh-so-humble-opinion, users can deal with (and quite frankly, should continue to periodically expect) some downtime.
Our mail platform has beaten google in uptime and security "bugs" for the past 40 months. Why? I attribute it to using proven technologies and not everyone wanting an account being able to get one: we charge every system user. You would be surprised how much this cuts down on spammers/excessive usage.
Google has had their mail in beta for years. The last time I checked SMTP was ratified as an RFC over a decade ago.
Website Hosting
Do you honestly believe that you or your employees are going to build a system with higher availability than Google? In the magical fantasy world we all wish we lived in, you may have the budget, skill, manpower, and infrastructure resources to do this. In the real world it is not even remotely possible.
Do I believe it? You betcha! While my company doesn't have 100% uptime for every employee all the time, we haven't suffered an across-the-board outage of a critical system (i.e. email, ERP, core business applications, etc.) in the 11 years that I've been here. Sure, we'll lose an email server once in a while, but we have many such servers, so the loss of a single system only impacts a few thousand employees tops. That's far better than impacting ALL our employees if Google has an outage. And don't get me started about the idea of not being able to do word processing just because a WAN link is down. How on earth could you run a business that way???
And it's possible to provide uptime even in the event of widespread events, such as flooding, tornadoes, etc. We have multiple datacenters, geographically dispersed. Each center has multiple Internet connections through multiple providers carefully chosen such that the lines go to different cities (i.e. one link to Chicago, one to Denver). Similarly, our power is connected to multiple grids, with the feeds coming in on opposite ends of the buildings. Critical centers have on-site generators spec'd to handle 100% load of the datacenter and requisite support stations, plus enough battery backup to allow for all systems to continue running between loss of grid feed and when the generators are spun up, not to mention on-site diesel sufficient for several days of operations and contracts to get more as needed.
Was this cheap? Not in the least. Was it worth it? Definitely. We kept our main datacenter running without interruption during a week that saw multiple weather events (i.e. tornadoes, flooding, lightning-related power loss, etc.) when every building around ours for multiple miles was without power.
With those you know WHEN it's going to happen. You can schedule it for out of hours.
The position you're touting is completely foreign to me. I don't want to discount it, I just think it must be because you work for a small company and don't have any experience administering widely used web sites.
Even for medium sized companies, I have to imagine that "out of hours" are few and far between.
I'm a big tall mofo.
is in-site and outsource. failed in house tech tends to put alot of pressure on the in house support staff hired to maintain the tech, but im thinking most management is wondering if they could withstand the black eye of losing something like this if they hosted it at google
this would be much less of a concern if they open sourced the entire group of apps, and offered hosting as an option. IT Managers could evaluate it on a more level cost benefit ground.
i guess another question, is this really something that should be web based?
Good people go to bed earlier.
Precisely. And while I use Gmail most of the time, and the rest of my office use an Exchange server hosted in same said office, guess who has the better uptime...?
That would be me. They frequently (3-4 time a month) loose half a day, as the under resourced, high maintanence, auto-destructing, sorry updated, blackhat honey pot splutters in the corner. I've lost two half days in the however many years I've used Gmail.
Because you can - or because you should?
What you're saying makes sense for something like google apps, but it's exactly backwards for something like gmail. A small organization is much more vulnerable to a DOS attack than google is.
This is more complicated than it might seem at first glance. Google, for example, has a published privacy policy. It may or may not be acceptable to a particular organization, but I'd expect Google to do a pretty good job of following it competently. On the other hand, many organizations that manage their own IT services do a really lousy job of managing security and privacy. For instance, Ameritrade had a problem for years where people would sign up for accounts, and immediately start getting pump and dump spam. Ameritrade tried to blame it on dictionary attacks, viruses, etc., but users thoroughly and publicly documented the fact that it was happening to single-purpose email accounts that were not vulnerable to dictionary attacks and were not on Windows boxes. Years later, Ameritrade finally admitted that there was a problem, and said it looked like it must have been an inside job -- some employee selling the addresses to spammers. You also get issues with employees bringing home laptops with sensitive data. Realistically, most of the security issues that IT departments deal with on a day to day basis are issues with users getting their machines infected with malware. I would expect that kind of thing to be less of a problem if your apps are remotely hosted. Your machine is probably less likely to get infected from clicking on a malware attachment, and if worst comes to worst, you can always do a clean install on the infected machine, meanwhile using a different machine to access the web app. No downtime, no lost data.
Find free books.
Some good points but they can also be turned around. Something like GMail will definitely be more robust than trying to get the mail down to a local server. But web applications are subject to those same kinds of service interruptions. Even if Google has the bandwidth and distributed systems to be that robust, the choke point is the link in/out of the company and a DOS attack there can still close off access to the apps people need to run.
And I think we're all starting to get a feeling for what other company's privacy policies are worth. While a local server at your company might be more likely to be penetrated depending on the skill of the admins and how well it is hardened compared to Google, other companies and the links between you and them are still subject to being compromised. You can also have issues with malicious insiders and I'm sure that Google is no different in that respect. Looking just at that, there could be a bigger risk at Google just based on the numbers of people that have access to your data. And your point with Ameritrade is just more proof of that risk.
But malware will be a risk regardless of where the data is hosted. Even if you are working on some document remotely, you are still seeing it locally and if that system is exfiltrating data, you are still compromised.
I am sure that Google has good security policies, good backups, good admins. But I stand by myself not wanting to risk the added exposure and possibility of being shut down by events outside of my control. I tend to think that with remote applications, I still have all of the local risk that I had before and add extra risk by using remote applications. About the only good I see is that a place like Google really should have excellent data backup practices that probably exceed what most companies call adequate.
I think a lot of the "good arguments" that have been voiced are just recycled arguments that could also be used to suggest that using computers in general is a bad idea. We should go back to pen and paper and horse drawn carriages. Simpler times back then.
To be honest, the only valid argument I could make (that I haven't seen mentioned before) is that a hosted app has multiple points of failure in terms of network availability. If Google's servers go down, the service is unavailable. If any of the three or four Internet backbones between you and Google's servers has an issue, the service is down. If your own Internet access goes down, the service is down.
So, there are more things to possibly go wrong in that respect. However, all of these same arguments could also be said if you were trying to reach your company email from home.
The thing that keeps getting overlooked are the reasons WHY people are switching to Google Apps. They aren't switching because they just want to use some other system. They aren't switching because the price is so low (or free.) They aren't switching just because you can store a lot of data and search lightning fast. They are switching because there are collaboration and revision services. They aren't switching because it is accessible from all over the world with the same ease as accessing it from at work. They aren't switching because of the excellent spam filters. They're switching because of ALL of these.
It's a cost/benefit thing. Telling a Ferrari owner that a Hyundai is much more reliable is missing the point that the person probably doesn't own the Ferrari only for its reliability.
When Google Apps (as a service) is running well (which is the majority of the time,) it isn't just an alternative to software solutions. It is leaps and bounds beyond it. Gone are the days where people are emailing attachments to a group, trying to collaborate through a spreadsheet or specification in a Word Doc. Gone are the days where one must connect through a VPN first, then remote desktop to a machine, just to access certain files remotely. Gone are the days when admins have to stand over people's cubicle walls and say, "Do you know that you're using 10 gigs in your email? Can you please start clearing some stuff out, or we'll have to clear it for you."
When things aren't running so smoothly, and Google Apps is inaccessible, then you end up with a pretty good (but not amazing) set of software (thanks to Gears.) Thankfully, this is very rare.
But again, if you're ok with the software you're currently using and the price you are currently paying, there is absolutely no reason to switch. This shouldn't be a case about Hyundai owners trying to get Ferrari owners to switch to Hyundais and Ferrari owners trying to get Hyundai owners to switch to Ferraris. Everyone should use the software and services they are comfortable with.
I just don't understand that instead of someone saying "it's just not for us at this time" they instead talk about the impending doom that is just around the corner. In reality, there is simply a technology shift taking place... but it's still happening. For a time, it's still ok for VHS owners to keep hanging onto their VHS collection while DVDs start flooding the market. For a time, it is still ok for those with black and white televisions to hang onto them a while longer even though color televisions have been out for a while. They're nothing wrong with diversity, taste, and opinion.
The time has not come yet where those who are still using local software are out-of-touch. We're still a long way from that. But there is a certain personality type known as the innovators. The early adopters who are willing to take the risks needed to gain the bigger rewards. Sure there are some learning curves to deal with and the growing pains. But in the end, the innovators consider these as worthwhile costs to justify the end result. Eventually the time will come where the late adopters will be paying money to the early adopters to help them make the switch. To make their VCR stop flashing "12:00" so-to-speak.
Do you honestly believe that you or your employees are going to build a system with higher availability than Google?
Our qmail/vpopmail based mail server is much more reliable than google. So why have we moved many accounts to Google if we have better availability? Because Google rocks when it comes to spam filtering. I wouldn't recommend Google for high availabilty and performance, but if you've got the too much spam blues, Google can take that headache off your hands. The perpetual battle against spam isn't something we want to devote extensive resources to beyond a few sensible, easy to implement measures.
Loose lips lose spit.