Slashdot Mirror
Certificate Expiry Leads to Total Outage For Microsoft Azure Secured Storage
rtfa-troll writes "There has been a worldwide (all locations) total outage of storage in Microsoft's Azure cloud. Apparently, 'Microsoft unwittingly let an online security certificate expire Friday, triggering a worldwide outage in an online service that stores data for a wide range of business customers,' according to the San Francisco Chronicle (also Yahoo and the Register). Perhaps too much time has been spent sucking up to storage vendors and not enough looking after the customers? This comes directly after a week-long outage of one of Microsoft's SQL server components in Azure. This is not the first time that we have discussed major outages on Azure and probably won't be the last. It's certainly also not the first time we have discussed Microsoft cloud systems making users' data unavailable."
It is almost a year ago to the day Azure was down for a day because no one accounted for leap year for validating certificates, lol. AWS seems to have issues too, but they don't seem to revolve around blatant stupidity and result in an entire day of downtime.
I wonder how long it will be before there's a major failure loop in the cloud, something like the certificate for cloud X is stored in service Y, which actually uses cloud X as its backend. So when certificate for X stops, the whole thing grinds to a halt with no way to restart it (unless backdoors)...
Non-Linux Penguins ?
You'd think that, but there's contract stuff. The thing is, you basically need a department in charge of renewing shit like this when you have enterprise level services. We've got a site with millions of hits daily and still manage to let it expire every couple of years. You try the credit card thing, but credit cards expire. You try recurring billing and then you get into a contractual nightmare with the registrar. The registrar isn't going to do you any favors, you might get millions of hits daily, but they still only get $5/year even from google.com so fuck you, figure out the billing yourself.
The only real way to do it effectively is build yourself a database of all the crap you need to renew regularly, then hire someone to renew that stuff. But who are you going to hire? It usually ends up being some assistant that doesn't know a damned thing about tech... and it's still going to cost you $60k a year in pay and bennifits to retain them. That's an expensive way of keeping track of such things... ah, the website admins can remember right?
Actually, there's a bit more to being "cloudy" than just virtual servers over the internet (indeed, they not even need be over the internet - you can have your own local cloud and many companies have internal clouds). Virtual servers over the internet is merely client/server. For a service to be "cloudy", generally it'll have attributes like HTTP (in other words, RESTful interfaces and each request being treated no different to the first request, in other words, the service doesn't hold state from request to request, just like with HTTP) and distributable. The main benefit of "cloudiness" is because of this you can easy scale up services when demand is high, and scale them back when demand is low. It makes it easier to make a resilient service than the traditional client/server type service where the server side has to keep state. Infrastructures like Amazon's EC2 allow you to scale things up and down easily and economically because you can turn on the "virtual server over the internet" part of it on and off very rapidly, and you only pay for the instances you've instatiated. But just using Amazon's EC2 doesn't automatically make your service "cloudy" if it does not have all the other necessary attributes.
Oolite: Elite-like game. For Mac, Linux and Windows
On the other hand, I've worked at places where the worst thing you could do is leave things that the company can't live without *in* the control of the company. Sometimes certain areas of expertise require specializations that the company just doesn't have and isn't interested in acquiring. Of course handing the responsibility of those things off to *Microsoft* is not necessarily any better.
So wrong in so many ways. Any reason you wouldn't purchase a 100 year certificate and just roll with it? Too bad about 1/3 of all Azure disk space is used for endpoint backup. This reminds me of the leap-year calculating bug - Feb 29 2012, you couldn't generate a site because the default is to generate a certificate for 1 year, and well, Feb 29 2013 just doesn't exist. http://blogs.msdn.com/b/windowsazure/archive/2012/03/09/summary-of-windows-azure-service-disruption-on-feb-29th-2012.aspx