Slashdot Mirror
Microsoft's Azure Cloud Suffers Major Downtime
New submitter dcraid writes with a quote from El Reg: "Microsoft's cloudy platform, Windows Azure, is experiencing a major outage: at the time of writing, its service management system had been down for about seven hours worldwide. A customer described the problem to The Register as an 'admin nightmare' and said they couldn't understand how such an important system could go down. 'This should never happen,' said our source. 'The system should be redundant and outages should be confined to some data centres only.'"
The Azure service dashboard has regular updates on the situation. According to their update feed the situation should have been resolved a few hours ago but has instead gotten worse: "We continue to work through the issues that are blocking the restoration of service management for some customers in North Central US, South Central US and North Europe sub-regions. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers." To be fair, other cloud providers have had similar issues before.
Until it isn't.
Shoes for Industry. Shoes for the Dead.
Basket?
Or how about "Never outsource your core functionality?
I am Slashdot. Are you Slashdot as well?
The companies I deal with tend to say things like, we want to go with a company like this so we can can get "Support". Which usually means, so we can blame them if something goes wrong.
Scott Carr
Leap year strikes again?
Yay, cloud!
Cluster at the application level and have nodes at different providers. If your volume is too high for that, you are big enough to host your own stuff.
Go green: turn off your refrigerator.
This is not helping, guys!
The hilarious part of this link is that the article detailing how screwed people are for depending on Microsoft's cloud services is stuffed with rollover ads for...Microsoft's cloud services!
Last time a Microsoft cloud product went down, users sustained real data loss. Of course, Microsoft claimed it couldn't happen with Azure.
"First they came for the slanderers and i said nothing."
I agree, I have nothing against the idea of cloud services, but they do need to work and reputations are based on events like this. After an outage this long, it takes a LOOONG time to earn your way back to five nines (which works out to 5.5 minutes of downtime per year).
Since the image that "Azure" and "Cloud" conjurs up is more "sky" than "cloud" it would be my suggestion that Microsoft simply register chickenlit.tl and set up an Azure service status monitor/report page there.
They could have an adorable cartoon chicken that, when the system is working normally, runs around scratching and pecking(speed dependent on load). When downtime occurs, it would begin squawking about how the sky is falling. What could make failure more endearing?
Just to add that Microsoft touch, they could do the entire thing as a Microsoft Agent ActiveX control!
Well...maybe not right now...
After an outage this long, it takes a LOOONG time to earn your way back to five nines (which works out to 5.5 minutes of downtime per year).
Only 84 years per the article, and growing at a rate of a year every 5 minutes.
Thats probably about how long it would take me to trust MS in an enterprise environment.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
It's funny how those of us who bring up issues of data security and service resiliency are dismissed as just trying to protect our jobs.
Like so many other things, the actual technical underpinnings of "the cloud" are great, and have been standard fare for years. Virtual machines + flexible networking are a godsend for systems guys tasked with getting capacity for a new project up and going yesterday. I love being able to build and rip down entire test environments just to try something out...that used to mean a rack of physical servers, switchgear, etc. tied up while it was being used. That's why everyone's slowly coming around to the "private/hybrid cloud" model, which is really just code for "VMs + network capacity + something to tie it all together + maybe some external hosting".
The problem is that "the cloud" is very badly misunderstood. As sson as a CIO sees "virtual, on-demand capacity without those pesky physical on-site machines and IT staff, for a fixed cost per compute-hour" everything else takes a back seat. Then, it's "why do we need IT staff on-site, everything's being taken care of in the cloud." Public clouds like Amazon or Azure are great for startups who can't really afford their own data centers, or even bigger businesses to offload some of the nonessential stuff. When you start looking at hosting everything though, the marketing hype of the cloud sometimes distracts people from realities that they have to contend with.
Also, I'm not saying that businesses who go the private cloud or traditional hosting/outsourcing route won't have downtime -- they will. However, having onsite staff and infrastructure means you can work those staff until they fix the problem, and you have control over them. Most sane outsourcing contracts have SLAs in them stating that the vendor will expend X amount of effort to fix your problems. Cloud provider agreements, unless specifically mentioned otherwise, are "as is, where is, best effort restoration with no warranty." OK, maybe some providers will give you an SLA, but all that does is buy you free service at a later date if they violate it...it doesn't bring your application back online. You still have no choice but to sit and wait around for the provider to fix whatever's wrong...just ask Amazon EC2 customers about what happened during their last outage...
Companies need to draw sane boundaries around hosted systems, and decide what is critical and what can be offloaded. Do I care about a set of development/test machines that get used once a month? Probably a lot less than the critical database/application servers that run my core business. Comfort level, cost per minute of downtime vs. cost of dedicated resources and other factors need to be carefully considered before jumping into the cloud with both feet.
One of the selling points of using cloud services was that it would be more reliable than managing your own hardware/software. But to date, every single big player has suffered major downtime. If I would be hesitant to believe the sales pitch.
But still, for most companies that are good candidates for cloud offerings, even 8 hours of downtime once a year is probably better than they can guarantee using their own infrastructure. Companies in this range tend to not have redundant servers, offsite backups, disaster recovery sites, etc. Larger companies that can build redundant infrastructure (and staff it properly) are probably better off staying away from the cloud since they can guarantee any level of uptime and redunancy they want to pay for.
Of course, when a small company Admin spills a cup of coffee in the Exchange server and they are down for 5 days while building a replacement server, it doesn't make the news so you never hear about it...while when a large cloud provider has a 2 hour outage, it's all over the news.
Use the MCSE mantra:
1. Perform virus scan.
2. If that doesn't work, find a different program that will display a reassuring green graphic.
3. If that doesn't work, reboot.
4. If that doesn't work, reformat, reinstall.
5. If that doesn't work, GOTO 1.
Microsoft wouldn't know anything about data center running if it were chase aftering them at full speedo.
Google this: "Microsoft Sidekick / Danger"
http://techcrunch.com/2009/10/10/t-mobile-sidekick-disaster-microsofts-servers-crashed-and-they-dont-have-a-backup/
https://www.pcworld.com/article/173470/microsoft_redfaced_after_massive_sidekick_data_loss.html
http://www.appleinsider.com/articles/09/10/11/microsofts_danger_sidekick_data_loss_casts_dark_on_cloud_computing.html
I'll see your senator, and I'll raise you two judges.
Put your servers in the Azure cloud to have an uptime of 9.999999999%
I wrote a comment on slashdot a while back which questioned the sensibleness of running services in the cloud. I used to be a sceptic.
Since then I've used Rackspace Cloud and found that it's actually a very good idea, for certain things.
The benefits of using a cloud system are scalability and no commitment- it's not about reliability or higher availability - but you do get a little win in those areas.
To give some examples, I was recently able to play around with mysql clustering. I followed a mysql clustering howto and played around with it, setup a mysql cluster with load balancers. Once I was finished geeking about, I saved the VMs to the file storage and deleted the cloud instances. Total cost a £/$2-3 maximum. I hadn't previously been able to do this, I would have had to rent a dedicated server which would serve websites, email etc. I couldn't really use the dedicated server to play with new technology in case it had a negative impact on the live systems. I did have development box for a while, but it essentially doubled my costs without making any more money, just offering some protecting.
Now I have staging/development instances in the cloud - and no commitments to them - I don't have to worry about a £250 monthly bill or sign a 12 month contract to get my own box. I can fire up some resources, use them, and throw it away when I'm done.
The upshot is that I can play around with other peoples cool open source software without risk or buggering something up on my live box, and the costs are insignificant since I'm only renting it per hour. I can try something new, if it works great - it might go/stay in production. If not, delete it and move onto the next cool thing.
If I need high availability, I would use Rackspace, Amazon, Azure, and I'd ensure that I have a plan to deal with a major outage with any of the providers. Each have APIs, so in theory I could create new instances automagically and failover between different cloud providers with a quick DNS change, while keep costs low.
To recap, the cloud isn't all about high availability - no matter what the marketing says. It's about scaling systems and running resources for small amounts of time, and is perfectly suited to services which have peak demand (ticket sales for example).
it's a leap year, they can be down a full day and still claim they were up for 365 days this year!
You mean besides Amazon, SalesForce, VMWare, Google Gmail, Yahoo Mail, Apple iCloud. Seriously who hasn't had downtime?
Well, there's spam egg sausage and spam, that's not got much spam in it.
I've always had to laugh at the name "Office 365" -- the fact this happened on Leap Day amuses me to no end.
I went to eat some animal crackers and the box said, "Do not eat if seal is broken." I opened the box and sure enough..
I've always had to laugh at the name "Office 365" -- the fact this happened on Leap Day amuses me to no end.
In light of Excel's horribly buggy code of handling Leap Day, I have to wonder if Microsoft's problems here might not be because it's Leap Day? Whaddaya bet Azure comes back up all fine and dandy once the date rolls over to 1 March instead of 29 February? I'm actually serious about this conjecture, this is not just an attempt at humor.
On a different angle, does anyone else find it amusingly ironic that this service is named Azure, and now it's blue-screened? They've only gone up one letter -- now it's the ASOD.
Cheers,
"What in the name of Fats Waller is that?"
"A four-foot prune."