Slashdot Mirror
Certificate Expiry Leads to Total Outage For Microsoft Azure Secured Storage
rtfa-troll writes "There has been a worldwide (all locations) total outage of storage in Microsoft's Azure cloud. Apparently, 'Microsoft unwittingly let an online security certificate expire Friday, triggering a worldwide outage in an online service that stores data for a wide range of business customers,' according to the San Francisco Chronicle (also Yahoo and the Register). Perhaps too much time has been spent sucking up to storage vendors and not enough looking after the customers? This comes directly after a week-long outage of one of Microsoft's SQL server components in Azure. This is not the first time that we have discussed major outages on Azure and probably won't be the last. It's certainly also not the first time we have discussed Microsoft cloud systems making users' data unavailable."
What's an expirty?
Not the first time they've made such blunders:
http://slashdot.org/story/03/11/06/1540257/microsoft-forgets-to-renew-hotmailcouk
If only Redmond had some sort of calendar system to help them remember this stuff.
Maybe rtfa-troll and Timothy's spell checkers were hosted on Azure.
If you can't trust Microsoft for such kind of small but essential things, should you trust them with bigger ones?
The really amazing thing is that if you look at their service dashboard, it took them 12 hours to update the certificates on their site:
http://www.windowsazure.com/en-us/support/service-dashboard/
They spent several hours doing "test deployments" ... while it's great to make sure you aren't going to make something worse, updating an SSL cert isn't exactly rocket science. I'd had to see how long it took to recover from a more serious service issue triggered by a software bug.
I wonder how long it will be before there's a major failure loop in the cloud, something like the certificate for cloud X is stored in service Y, which actually uses cloud X as its backend. So when certificate for X stops, the whole thing grinds to a halt with no way to restart it (unless backdoors)...
Non-Linux Penguins ?
So you are saying that MS is the lowest common denominator. I guess you can say that again.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
Eh, don't put anything too important that you can't live without on systems outside of your control.
this is my sig
Finally the Microsoft Blue Screen of Death has made into the new mobile cloud age.
I mean the Azure Screen of Death, excuse me Mr. Ballmer.
Mod me down, my New Earth Global Warmingist friends!
The Blue Sky of Cloud Death
I am a free slashdotter. I will not be modded, blogged, DRM'd, patented, podcasted or RFID'd. My life is my own.
... this is what you get. Sure, it's possible the same thing can happen for any company. But at least then you can fire your incompetent staff.
.. 'Cloud' computing is just remote virtual servers over the Internet. It's really not something new and original. People act like it's some amazing new 'thing'. Well .. it's not. It's just another way of letting companies with limited or no tech skills put up a web site or store data. It's expensive, proprietary, and I doubt very cost effective in the long run.
Once you deploy to a vendor, you are stuck. From what I've seen, you can't easily move data and code from one vendor to another. One of our clients is in the UK Azure cloud and we have to BCP about 6M rows from their server to our system every week. Takes over 90 minutes, and constantly fails because of losing the connection. We've looked at deploying systems to various clouds, and the costs were not worth it.
I will NEVER put any critical business system in someone else's cloud. At worst, I might put it in someone's data center on *MY* servers. The cloud seems to be fine for small business startups and non-important data for personal use. Businesses who no one would even notice if their site was down for a day.
BTW
I rarely read replies, it's my opinion and if you thought about your opinion a little more, I'm OK with that.
Somehow I feel those worker visas are the issue here.
Anything else you'd like to blame on foreigners?
Declining population of ducks in the local pond?
Chips no-longer served in old newspaper?
Lack of respect for elders?
Banning of blackboards in schools?
Rampant rape and violence all foreigners bring to your little Daily Mail reading village?
I find it hard to believe anyone who maintains such a large fleet of services wouldn't have setup some sort of trivial monitoring (I know they own a product or two) that would include SSL Certificate expiration warning. 30+ days out, a ticket (or some sort of actionable tracking mechanism) should have been generated, alerting those responsible to start taking action. Said ticket should have become progressively higher severity as the expiration date loomed (meaning nothing had been updated), which in any sane company, would have implied higher and higher visibility.
That way, if an extensive test plan for such a simple operation was required, they had plenty of time to execute upon it and still not miss the boat.
Working with MS in other ways, and combined with both the lack of foresight and inability to act quickly, just shows that this sort of customer-forward thinking just doesn't exist inside the MS mind.
$ man woman *
-bash:
Outperforms in reliability, huh? bullshit
Of course it doesn't work, but look how fast it is!
I guessMS somewhere in their licensing of this stuff have a clause that states they are not liable. Basically, 'bollocks to the Customers' when we fuck up [again].
So I cannot understand why people use them at all (once bitten, twice shy, twice bitten.. etc.).
Right and I think this is an important aspect to the problem here.
There is simply no substitute for having all your I's dotted and T's cross with large integrated systems like this. This is a culture problem not a individual screwed up problem. If you just fire the guy, there will be lots of awareness but the take away most of your remaining people will get is "don't forget to check the certificate expiry dates, that'll get you canned" many of them traumatized by the experience will dutifully check certificate dates for the rest of their careers but this will do nothing to prevent your next major outage; because that will almost certainly be the result of something else.
Everyone is pushing this vitalization + "dev ops" + management/monitoring is going to let us have one admin do what was once the work of ten. The fact is it just does not work like that. Management/monitoring like Microsoft Mom for example requires you to have all the failure modes identified and the scripts written to check conditions like expiry dates and trigger the alerts. Unless everyone is really good about all the routine maintenance tasks in there is won't help with something like this. That takes time you ONE admin has not got and discipline that breaks down when someone is overworked.
The "dev ops" and vitalization stuff is all great in terms of how much can be automated. Someone has to develop that automation though. Your ONE guy does not have time to build and test his generic deployment scrip when you promised your customers you'd have their infrastructure stood up last week.
It comes down to the business recognizing its important to have good people, enough people, and willingness to invest in making sure the job is done correctly and completely every time, and that documentation is maintained and in a way everyone knows how to use it. Check lists need to be kept and followed etc. IT got away from plant engineering style discipline when hardware got cheap. You know longer had to worry about that one computer you had failing. As we move back to more consolidated and integrated solutions; management is going to have to get used to the idea again that there is some people time investment that must be made. Its great you can save on power, cooling equipment, and headcount but you can't cut headcount to far because the more consolidated you get the less you can afford for anything to go wrong so it all must be check, doubled checked, and checked again just to be sure. This is if you do it yourself or if you pay your cloud provider to do it. Either way cloud services so far have been mostly a race to the bottom and that is going to cause some to have to learn some very painful lessons if the industry remains on its current trajectory.
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
On the other hand, I've worked at places where the worst thing you could do is leave things that the company can't live without *in* the control of the company. Sometimes certain areas of expertise require specializations that the company just doesn't have and isn't interested in acquiring. Of course handing the responsibility of those things off to *Microsoft* is not necessarily any better.
Azure is webscale? I never knew!
So wrong in so many ways. Any reason you wouldn't purchase a 100 year certificate and just roll with it? Too bad about 1/3 of all Azure disk space is used for endpoint backup. This reminds me of the leap-year calculating bug - Feb 29 2012, you couldn't generate a site because the default is to generate a certificate for 1 year, and well, Feb 29 2013 just doesn't exist. http://blogs.msdn.com/b/windowsazure/archive/2012/03/09/summary-of-windows-azure-service-disruption-on-feb-29th-2012.aspx
From a business perspective, it makes perfect sense: If Azure were reliable, secure and fast, customers could start to wonder why the other products by MS are not. This could heighten customer expectations, and that would be bad as MS really does not have the engineering capabilities to build, say, a good OS or a good office productivity suite and then customers may leave for the alternatives. So I applaud them for their foresight in making Azure just as bad as their other things are. This may actually be quite beneficial for their bottom-line.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Yes, the single point of failure works!
But I thought "the cloud" wasn't supposed to have a single point of failure, otherwise it would be just a "remote server" rather than "the cloud"?
Also FatPhil on SoylentNews, id 863
Azure - bright blue in color, like a cloudless sky
"From the depths of my skeptical and rationalist soul, I ask the Lord to protect me from California touchie-feeliedom."
If you want to defend H1B1 workers and dirt-cheap Indian code monkeys, perhaps you should make a logical argument.
I don't think the guy you're responding to had the most well thought out argument but your response did nothing to refute it. You accuse him of xenophobia when it's obvious that he wasn't talking about foreigners in general, he was talking about specific foreigner workers that are hired by American firms that are looking to cut costs. That doesn't mean that all foreigners are incompetent -- the assumption is that the most competent foreigners don't have to accept lower than deserved wages to undercut American workers. There's a reason the foreigners who undercut American jobs are willing to accept less money -- they're not worth as much.
Shame on the four mods who upvoted your post.
"From the depths of my skeptical and rationalist soul, I ask the Lord to protect me from California touchie-feeliedom."
Imagine if someone's signature on your PGP identity expired. It might be a bit of a blow, but people would still have other trust pathways toward you. Then you get a new signature from 'em, or someone else.
Certs can fail in so many ways, both false positives (compromised CAs) or false negatives (such as this expiration), and a myriad of subjective failures since different people have different reasons to trust (or not trust) different CAs. The risks aren't even theoretical. Failure really happens, to the extent that it's almost routine and we see a story about it here on Slashdot every month.
And Phil Zimmerman totally solved the problem(!) in, what, 1988? Why are we still using obsolete-the-day-it-came-out single signer systems? So brittle. So unrealistic.
The only reason I can think of, is that it would work too well. MitM attacks would become nearly impossible for even the most powerful governments. Certs would become so competitive and cheap that the CA business would collapse.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Calling someone a cunt because they missed a typo is not constructive criticism.
Waterfox - a Firefox fork with legacy extension support, security updates and better privacy by default.
Calling someone a cunt for any reason wouldn't make constructive criticism. When I use say it, it definitely isn't an attempt at anything constructive. I still love the word though.
Pull my finger for my public key.
* expire date: 2013-11-15 18:15:53 GMT
Call this from a cronjob script which should then take suitable action if the date is too close.
IE10 has a spell checker now. They're only 5 years late, but they got there.
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".