Certificate Expiry Leads to Total Outage For Microsoft Azure Secured Storage

rtfa-troll writes "There has been a worldwide (all locations) total outage of storage in Microsoft's Azure cloud. Apparently, 'Microsoft unwittingly let an online security certificate expire Friday, triggering a worldwide outage in an online service that stores data for a wide range of business customers,' according to the San Francisco Chronicle (also Yahoo and the Register). Perhaps too much time has been spent sucking up to storage vendors and not enough looking after the customers? This comes directly after a week-long outage of one of Microsoft's SQL server components in Azure. This is not the first time that we have discussed major outages on Azure and probably won't be the last. It's certainly also not the first time we have discussed Microsoft cloud systems making users' data unavailable."

Comments

Lolwut?

[Anonymous Coward]

What's an expirty?

Re:Lolwut?

[Nidi62]

I think you get them from storage vendros

Re:Lolwut?

[drinkypoo]

Vendro is Destro's cousin, who works on the supply side.

Re:Lolwut?

[flyingfsck]

Look, if Destro is selling expirtys in my neighbourhood, then I want a slice!

Re:Lolwut?

[Anonymous Coward]

Which part of "Microsoft product" did you not understand?

Expirty?

[Anonymous Coward]

Timothy!! It's your fucking JOB!

Somebody

[Anonymous Coward]

Had better get fired. I normally don't condone firing over mistakes, but this is pretty huge.

Although, it's also a point of proof of the cloud's inability to be reliable if not set up right.

Re:Somebody

[jhoegl]

uh, this was user error.
Caused by either corporate inadequacies in procedure, documentation, or checks of their systems. Or a corporate system designed to say "this isnt my problem".
Somehow I feel those worker visas are the issue here. Inexperience causes user errors, users administrate these large cloud systems, cloud configurations are complex and take years to master, yet cloud systems are a service provided by the lowest common denominator.
Corporate brilliance at its best.

Re:Somebody

[flyingfsck]

It seems to be a point of poof...

Re:Somebody

[flyingfsck]

So you are saying that MS is the lowest common denominator. I guess you can say that again.

Re:Somebody

[Glendale2x]

Eh, don't put anything too important that you can't live without on systems outside of your control.

Re:Somebody

[Anonymous Coward]

Because he's an expirt.

Re:Somebody

[Anonymous Coward]

Somehow I feel those worker visas are the issue here.

Anything else you'd like to blame on foreigners?

Declining population of ducks in the local pond?
Chips no-longer served in old newspaper?
Lack of respect for elders?
Banning of blackboards in schools?
Rampant rape and violence all foreigners bring to your little Daily Mail reading village?

Re:Somebody

[DarkOx]

Right and I think this is an important aspect to the problem here.

There is simply no substitute for having all your I's dotted and T's cross with large integrated systems like this. This is a culture problem not a individual screwed up problem. If you just fire the guy, there will be lots of awareness but the take away most of your remaining people will get is "don't forget to check the certificate expiry dates, that'll get you canned" many of them traumatized by the experience will dutifully check certificate dates for the rest of their careers but this will do nothing to prevent your next major outage; because that will almost certainly be the result of something else.

Everyone is pushing this vitalization + "dev ops" + management/monitoring is going to let us have one admin do what was once the work of ten. The fact is it just does not work like that. Management/monitoring like Microsoft Mom for example requires you to have all the failure modes identified and the scripts written to check conditions like expiry dates and trigger the alerts. Unless everyone is really good about all the routine maintenance tasks in there is won't help with something like this. That takes time you ONE admin has not got and discipline that breaks down when someone is overworked.

The "dev ops" and vitalization stuff is all great in terms of how much can be automated. Someone has to develop that automation though. Your ONE guy does not have time to build and test his generic deployment scrip when you promised your customers you'd have their infrastructure stood up last week.

It comes down to the business recognizing its important to have good people, enough people, and willingness to invest in making sure the job is done correctly and completely every time, and that documentation is maintained and in a way everyone knows how to use it. Check lists need to be kept and followed etc. IT got away from plant engineering style discipline when hardware got cheap. You know longer had to worry about that one computer you had failing. As we move back to more consolidated and integrated solutions; management is going to have to get used to the idea again that there is some people time investment that must be made. Its great you can save on power, cooling equipment, and headcount but you can't cut headcount to far because the more consolidated you get the less you can afford for anything to go wrong so it all must be check, doubled checked, and checked again just to be sure. This is if you do it yourself or if you pay your cloud provider to do it. Either way cloud services so far have been mostly a race to the bottom and that is going to cause some to have to learn some very painful lessons if the industry remains on its current trajectory.

Re:Somebody

[Nerdfest]

On the other hand, I've worked at places where the worst thing you could do is leave things that the company can't live without *in* the control of the company. Sometimes certain areas of expertise require specializations that the company just doesn't have and isn't interested in acquiring. Of course handing the responsibility of those things off to *Microsoft* is not necessarily any better.

Re:Somebody

[Anonymous Coward]

Why is it every time there's a simple mistake or somebody overlooks something and the only result is a bit of inconvenience, there's people clamoring for someone's career to be ruined? Seriously, it's childish.

Anybody who trusts their data to "the cloud" needs to understand that this can happen.

Re:Somebody

[RazorSharp]

If you want to defend H1B1 workers and dirt-cheap Indian code monkeys, perhaps you should make a logical argument.

I don't think the guy you're responding to had the most well thought out argument but your response did nothing to refute it. You accuse him of xenophobia when it's obvious that he wasn't talking about foreigners in general, he was talking about specific foreigner workers that are hired by American firms that are looking to cut costs. That doesn't mean that all foreigners are incompetent -- the assumption is that the most competent foreigners don't have to accept lower than deserved wages to undercut American workers. There's a reason the foreigners who undercut American jobs are willing to accept less money -- they're not worth as much.

Shame on the four mods who upvoted your post.

Re:Somebody

[sjames]

It's not their foreign-ness that is the problem, it's their dirt-cheap-ness. Those just happen to go hand-in-hand due to the way H1-B works.

It all falls under "If you pay peanuts, you get monkeys". It just happens that the current fashionable way to get monkeys is to import them.

Re:Somebody

[Kalriath]

Yeah, but who is? AWS has more outages than I care to remember, Rackspace has had it's share of outages, Google goes down like once a month, even Apple can't keep a service up - and that's pretty much all the big players counted out.

Re:Somebody

[PRMan]

Having worked in hiring of H1B workers, I can assure you it's illegal to offer them significantly less than an American. Maybe they get a couple thousand less because they don't have the experience, but it's the same as if you hired a US worker with the same experience. They don't work for half and they are not indentured servants (they can find another H1B sponsor and stay in the country).

Re:Somebody

[Anonymous Coward]

Right and I think this is an important aspect to the problem here.

There is simply no substitute for having all your I's dotted and T's cross with large integrated systems like this. This is a culture problem not a individual screwed up problem.

Posting as AC to avoid losing mod points.

It is a human problem. People wanted the cloud for all the wrong reasons. Companies like Microsoft wanted the cloud to wedge in their Software as a Service, Corporate suits saw it as a way to get rid of people, and some IT people just gave up, I think.

And the big issue always seemed to be the encryption. Pah. Not really all that much of an issue. The real issue is that simple little things like a certificate can bring an entire system down. This is what "we" wanted, and this is what we will get. All that is left is the No True Scotsman defenses of the true believers.

The cloud can never fail, only we can fail the cloud

Re: Somebody

[GigaplexNZ]

I can't even have a phone conversation with one most of the time and you're lucky if they ever follow through with what they say they'll do.

Funny, I can say the same thing about most Americans I've had to deal with.

Re:Somebody

[RazorSharp]

I won't try to argue against your anecdote, though I'd like to point out that your experience is a single instance and I've heard other anecdotes that are contradictory. Different situations are different.

My major qualm was the AC's accusation of xenophobia where xenophobia wasn't present. It sickens me when people try to use political correctness as a trump card in a debate when it's really just a red herring.

Re:Somebody

[shutdown -p now]

There is still a person who wrote the manual for that procedure. And the HR or manager who approved such by-the-rote practice in the first place. Ultimately, there is a responsible party - a person, or several people.

Also, "it was not in my work manual" is not a valid excuse for the failure to apply common sense. Are all the people who deal with those certs failed to notice the expiry date? Did it never cross their mind to consider what would happen when the cert expires? I'm not talking about showing "fire" here, but just dropping a simple email to a suitable contact, along the lines of "Hey Joe - I just noticed that our cert is 1 month away from expiry date - do you know if someone is taking care of that?" would most likely prevent this whole mess.

Re:Somebody

[cheekyboy]

heres a tip, dont expire by N years or months.

Check the date it expires on, never expire on sunday.

Or some holiday.

Oh and we need better notifications, than just an email, or sms, how about a computer TTS voice to call you and speak it to you, "wahh dude, your gona expire in 1 day"

Typical.

[berchca]

Not the first time they've made such blunders:
http://slashdot.org/story/03/11/06/1540257/microsoft-forgets-to-renew-hotmailcouk

If only Redmond had some sort of calendar system to help them remember this stuff.

Re:Typical.

[Stormthirst]

Does MS not have a credit card its vendor can keep on file?

Re:Typical.

[hsmith]

It is almost a year ago to the day Azure was down for a day because no one accounted for leap year for validating certificates, lol. AWS seems to have issues too, but they don't seem to revolve around blatant stupidity and result in an entire day of downtime.

Re:Typical.

[rtb61]

M$ has a history of lack of customer focus hence it will fail ay any industry that demand the highest levels of customer focus. For cloud services to be down for a down is inexcusable and seriously any IT management staff that fails to acknowledge these failures and uses or recommends Azure should be fired. Any down time should be measured in minutes not days, this should be considered catastrophic failure. M$ is far to used to it's EULA's a warranty without a warranty and has become woefully complacent about actually guaranteeing a supply of service, meh, it mostly works it their motto and we'll fix it net time round, for sure this time.

Re:Typical.

[Charliemopps]

You'd think that, but there's contract stuff. The thing is, you basically need a department in charge of renewing shit like this when you have enterprise level services. We've got a site with millions of hits daily and still manage to let it expire every couple of years. You try the credit card thing, but credit cards expire. You try recurring billing and then you get into a contractual nightmare with the registrar. The registrar isn't going to do you any favors, you might get millions of hits daily, but they still only get $5/year even from google.com so fuck you, figure out the billing yourself.

The only real way to do it effectively is build yourself a database of all the crap you need to renew regularly, then hire someone to renew that stuff. But who are you going to hire? It usually ends up being some assistant that doesn't know a damned thing about tech... and it's still going to cost you $60k a year in pay and bennifits to retain them. That's an expensive way of keeping track of such things... ah, the website admins can remember right?

Re:Typical.

[chrism238]

"...and it's still going to cost you $60k a year in pay and bennifits to retain them."

Sure, but this incident, alone, has probably generated more than $60K of negative publicity for Azure. Gotta get the basics right first.

Re:Typical.

[Anonymous Coward]

Just do what we do. When you register anything that can expire you make a meeting request reminder and send it out to every IT guy in the company. While it's still usually one person's job to do the renewal you have a reminder for everyone in case it slips via personnel changes.

Re:Typical.

[symbolset]

I think your estimated minimum of the damage is several digits short.

Re:Typical.

[Anne Thwacks]

send it out to every IT guy in the company

When the entire IT department is incapacitated by incoming chair damage, it might not help. Every janitor is the required email target.

Re:Typical.

[drinkypoo]

It seems like a competent registrar would send a bill[ing statement] to the billing contact.

Re:Typical.

[Kalriath]

Except that companies like Microsoft and Google register domains through "Enterprise" registrars like MarkMonitor, who charge upwards of a few hundred (possibly even thousand) dollars per year for their service - which supposedly includes "not letting the fucking things expire" and "making sure other people don't register our damn marks".

Microsoft actually has even less excuse in this instance, believe it or not - Microsoft's certificate vendor is itself. All MS certificates are chained up to a Microsoft subordinate CA (with a GTE root). So it wasn't a failure to pay for a certificate since they don't actually pay for them - it was just someone not bothering to open the certificates MMC and click "Renew Certificate". Hell, it's probably even AD integrated so there's no need to fill in any fields.

Re:Spellcheck...

[mystikkman]

Maybe rtfa-troll and Timothy's spell checkers were hosted on Azure.

Tip of the iceberg

[gmuslera]

If you can't trust Microsoft for such kind of small but essential things, should you trust them with bigger ones?

Re:Tip of the iceberg

[pr0nbot]

For me the confusing thing is that there was a single point of failure. I thought that much of what the cloud was about was resilience; I would expect that someone designing cloud infrastructure would have done an analysis of failure points, and implemented failover mechanisms (or at least monitoring and recovery procedures). Ok, maybe not a cloud-startup-du-jour, but certainly a big enterprise-style entity like Microsoft.

Re:Tip of the iceberg

[Junta]

The reality is, if you outsource your hosting to a single company, there will always be single points of failure.

There will be architectural ones, like root of trust expiring resulting in security framework taking everything down.

There will be bugs that can bite all of their instances in the same way at the same time.

There will be business realities like failing to pay electric bills, or collapsing, or simply closing down their hosting business for the sake of other business interests.

Ideally:
-You must keep ownership of all data required to set up anywhere at all time. Even if you host nothing publicly yourself, you must assure all your data exists on storage that you own.
-You either do not outsource your hosting (in which case your single point of failure business wise would take you out anyway) or else you outsource to financially independent companies. "Everything to EC2" is a huge mistake, just as much as "everything to azure" is a huge mistake.
-Never trust a providers security promises beyond what they explicitly accept liability for. If you consider the potential risk to be "priceless", then you cannot host it. If you do know what your exposure is (e.g. you could be sued for 20 million, then only host it if the provider will assume liability to the tune of 20 million)

Re:Tip of the iceberg

[dbIII]

I think there's multiple single points of failure, such as the leap year problem that caused an entire day of downtime last year.

Re:Does Timothy Have Brain Damage?

[Anonymous Coward]

How does Timothy fuck up so many words?

Occam's Razor applies here. The simplest explanation is: because he's an incompetent, stupid cunt who can't do basic things correctly.

12 hours to update the certs?

[crt]

The really amazing thing is that if you look at their service dashboard, it took them 12 hours to update the certificates on their site:
http://www.windowsazure.com/en-us/support/service-dashboard/

They spent several hours doing "test deployments" ... while it's great to make sure you aren't going to make something worse, updating an SSL cert isn't exactly rocket science. I'd had to see how long it took to recover from a more serious service issue triggered by a software bug.

Re:12 hours to update the certs?

[Glendale2x]

Maybe they tried rolling back to an older version of the cert first.

(Yes, that was sarcasm.)

Re:12 hours to update the certs?

[sribe]

Maybe they tried rolling back to an older version of the cert first.

No, first they would have tried reinstalling the current cert. Three times. Only then would they have moved on to rolling back to the prior version.

Re:12 hours to update the certs?

[gweihir]

Maybe they tried rolling back to an older version of the cert first.

(Yes, that was sarcasm.)

You know, from their track record, I would consider this entirely possible....

Re:12 hours to update the certs?

[rjr162]

Pretty sure they tried rebooting first to solve the solution, which cause windows system repair to start on boot up. System repair ran for the whole time (since theres a grayed out cancel button you cant click) after which it reported system repair was unable to repair the system

Re:12 hours to update the certs?

[dbIII]

It's not that amazing when you consider the service level of their hosted email. A week to correct an internal DNS entry, and meanwhile a customer with sixteen thousand email users just had to wait in queue to get it fixed. The large print pretends to give, but the fine print says you just have to wait for as long as it takes and SLA's be damned.

Re:12 hours to update the certs?

[ewertz]

> Maybe they tried rolling back to an older version of the cert first.
Probably after trying to reset the date to 24 hours earlier. Setting the time backwards can do wonders for a system.

Clueless Ballmer

[Anonymous Coward]

There's an awful lot of BS'er in Microsoft these days. They'll have had a process manual written long ago. Someone will have been taught that following that manual is the definition of quality, and a load of BS middle managers will have been looking for any departure from the manual so they can pass blame over to someone else.

I could point a finger, but that's for the MS Board to do, and if they fail it's for the shareholders to intervene.

Entwined failure loop...

[dargaud]

I wonder how long it will be before there's a major failure loop in the cloud, something like the certificate for cloud X is stored in service Y, which actually uses cloud X as its backend. So when certificate for X stops, the whole thing grinds to a halt with no way to restart it (unless backdoors)...

Re:Entwined failure loop...

[gweihir]

Hehehehehe, nice!

I expect we _will_ see things like this though.

Re:Entwined failure loop...

[Njovich]

And I wonder when Slashdot commenters will get how certificate infrastructures work these days... I guess neither of us will get lucky.

Where do I pay?

[Trailer Trash]

Anyone have the link?

Then what the hell was this Slashdot article?

[Anonymous Coward]

http://slashdot.org/story/13/02/21/2216221/microsoft-azure-overtakes-amazons-cloud-in-performance-test?sdsrc=prev

"Microsoft Azure's cloud outperformed Amazon Web Services in a series of rigorous tests conducted by Nasuni, a storage vendor that annually benchmarks cloud service providers (CSPs). Nasuni uses public cloud resources in its enterprise storage offering, so each year the company conducts a series of rigorous tests on the top CSPs' clouds in an effort to see which companies offer the best performing, most reliable infrastructure. Last year, Amazon Web Services' cloud came out on top, but this year Microsoft Azure outperformed AWS in performance and reliability measures. AWS is still better at handling extra-large storage volumes, while Nasuni found that the two OpenStack powered clouds it tested — from HP and Rackspace — were lacking, particularly at larger scales."

Outperforms in reliability, huh? bullshit

Re:Then what the hell was this Slashdot article?

[Anonymous Coward]

That's why I didn't even bother with that article. Astroturf.

When Azure failed practically completely on Feb 29 not long ago you should already know the level of reliability Microsoft would provide.

I know people using Azure and they say it's a piece of shit and still a piece of shit. The CDN and blob stuff is ok, but any idiot can make that reliable. The rest of Azure is halfbaked crap.

Only use Azure if Microsoft pays you megabucks to do so.

Re:Then what the hell was this Slashdot article?

[multi io]

Outperforms in reliability, huh? bullshit

Of course it doesn't work, but look how fast it is!

Re:Then what the hell was this Slashdot article?

[ThreeKelvin]

Azure is webscale? I never knew!

Re:Then what the hell was this Slashdot article?

[Kalriath]

Apple. iCloud uses Azure Storage for (some) document storage.

Ahhh.. the cloud

[wbr1]

An out of reach place where you give other people your stuff and hope they will hand it to you when you ask.
I don't want my head in the clouds.

Re:Blew their support contracts..

[binarylarry]

Finally the Microsoft Blue Screen of Death has made into the new mobile cloud age.

I mean the Azure Screen of Death, excuse me Mr. Ballmer.

Re:Blew their support contracts..

[click2005]

The Blue Sky of Cloud Death

Microsoft's Azure cloud

[Mister Liberty]

Microsoft's Azure could!

When you depend on other people ...

[johnlcallaway]

... this is what you get. Sure, it's possible the same thing can happen for any company. But at least then you can fire your incompetent staff.

Once you deploy to a vendor, you are stuck. From what I've seen, you can't easily move data and code from one vendor to another. One of our clients is in the UK Azure cloud and we have to BCP about 6M rows from their server to our system every week. Takes over 90 minutes, and constantly fails because of losing the connection. We've looked at deploying systems to various clouds, and the costs were not worth it.

I will NEVER put any critical business system in someone else's cloud. At worst, I might put it in someone's data center on *MY* servers. The cloud seems to be fine for small business startups and non-important data for personal use. Businesses who no one would even notice if their site was down for a day.

BTW .. 'Cloud' computing is just remote virtual servers over the Internet. It's really not something new and original. People act like it's some amazing new 'thing'. Well .. it's not. It's just another way of letting companies with limited or no tech skills put up a web site or store data. It's expensive, proprietary, and I doubt very cost effective in the long run.

Re:When you depend on other people ...

[Alioth]

Actually, there's a bit more to being "cloudy" than just virtual servers over the internet (indeed, they not even need be over the internet - you can have your own local cloud and many companies have internal clouds). Virtual servers over the internet is merely client/server. For a service to be "cloudy", generally it'll have attributes like HTTP (in other words, RESTful interfaces and each request being treated no different to the first request, in other words, the service doesn't hold state from request to request, just like with HTTP) and distributable. The main benefit of "cloudiness" is because of this you can easy scale up services when demand is high, and scale them back when demand is low. It makes it easier to make a resilient service than the traditional client/server type service where the server side has to keep state. Infrastructures like Amazon's EC2 allow you to scale things up and down easily and economically because you can turn on the "virtual server over the internet" part of it on and off very rapidly, and you only pay for the instances you've instatiated. But just using Amazon's EC2 doesn't automatically make your service "cloudy" if it does not have all the other necessary attributes.

Re:When you depend on other people ...

[Viol8]

"The main benefit of "cloudiness" is because of this you can easy scale up services when demand is high, and scale them back when demand is low."

Do you genuinely think this wasn't done until some marketdroid thought up the term "cloud"?

This is supposed to be a tech website FFS, at least pretend to have some sort of tech nous. Scaling available services up and down has been done since the days of fscking mainframes!

Re:When you depend on other people ...

[DeathFromSomewhere]

Yes and it was done by buying a shit ton of hardware and all the complexities and expenses that come with it. The problem is that 90% of the time that hardware was sitting around idle. Or that you would have to purchase a bunch of hardware for a one time project and then hope and pray that someone would buy that hardware from you when you were done. It doesn't take a tech website genius to realize how incredibly inefficient that is.

Re:When you depend on other people ...

[Anonymous Coward]

Once you deploy to a vendor, you are stuck. From what I've seen, you can't easily move data and code from one vendor to another.

RHEL is CentOS is RHEL is Amazon Linux wherever you are. A basic of the cloud is that, as you migrate to it you migrate almost everything to Linux.

One of our clients is in the UK Azure cloud and we have to BCP about 6M rows from their server to our system every week. Takes over 90 minutes, and constantly fails because of losing the connection. We've looked at deploying systems to various clouds, and the costs were not worth it.

There have been outages in Amazon; almost nothing has ever crossed from one Availability zone to another. Multiple countries have never happened. At the same time there have been many total outages in Azure. Whilst Microsoft regularly loses data; every time a Google system fails totally, it turns out they have a tape backup. These are not "minor issues between very simlar services"; these are the fundamental differences which matter. If you attempt to use a Yugo as a form of armoured transport and then get shot, you shouldn't start saying "all armoured vehicles are terrible". Instead you should go and look for a system which fits your needs.

I will NEVER put any critical business system in someone else's cloud. At worst, I might put it in someone's data center on *MY* servers. The cloud seems to be fine for small business startups and non-important data for personal use. Businesses who no one would even notice if their site was down for a day.

You are asking the wrong question I think. It's not "Jakes cloud vs. mine". Instead the question is; what is the distribution; what is the split. Systems all in my own data centre are unlikely to survive if the data centre is flooded. Systems all in the cloud are going to be impossible to reach if the internet fails. A proper engineering decision probably uses some of your own servers backed up with multiple independent Linux based clouds each with a different technology base. E.g. Amazon +Rack space (OpenStack) +a private eucalyptus + a few dedicated servers.

BTW .. 'Cloud' computing is just remote virtual servers over the Internet. It's really not something new and original. People act like it's some amazing new 'thing'. Well .. it's not. It's just another way of letting companies with limited or no tech skills put up a web site or store data. It's expensive, proprietary, and I doubt very cost effective in the long run.

This is a total misunderstanding. It is "remote virtual servers over the Internet" which can be created, destroyed or modified in a small number of minutes or seconds using a clearly defined API. That makes it possible to do a whole bunch of interesting things (duplicate your entire Amazon system to Rack Space in five minutes if the whole of Amazon collapses for some reason) which are not possible with older fixed servers. It also makes a whole load of different potential problems (someone who has your master key can delete all your servers in one command).

In the end cloud computing is just another a tool. Just like a huge circular saw, nothing it does is impossible with a hand saw. You could end up getting cut in half. However, if you need to regularly saw up huge number of trees, you may find you need one.

Re:When you depend on other people ...

[Todd Knarr]

And you think the cloud works differently? It's just that someone else is buying all that hardware to have sitting around idle until you need it. You hope. But, being a business, I'll bet one of their policies is to not buy more hardware than their projected needs, to avoid having any more sitting around idle than they absolutely have to to cover their own short-term needs. Anything else increases their costs without providing any revenue, so as a business they're going to avoid it just like you are.

What makes it work is that they have so many customers that when one needs more capacity they can take a bit away from everybody else and each customer's share will be so small they won't notice. With a large number of customers, hopefully not too many will need a lot more capacity at the same time. What could possibly go wrong?

Re:When you depend on other people ...

[DeathFromSomewhere]

It's just that someone else is buying all that hardware to have sitting around idle until you need it.

That's no longer my problem. It's now an operating expense for me instead of a massive up front capital expense.

What makes it work is that they have so many customers that when one needs more capacity they can take a bit away from everybody else and each customer's share will be so small they won't notice.

Nooo... when you reserve a VM that VM is yours whether you use it or not. You are paying for it after all. I have a very tough time buying that any of the major cloud platforms are oversubscribed. You will have to back up that claim.

It doesn't matter anyways. If you have grown to such a monstrous scale that you start to outgrow the capabilities of these cloud platforms, the capital cost of rolling out your own data center is likely no longer an issue. Operations on that scale are few and far between.

Re:When you depend on other people ...

[Todd Knarr]

That's no longer my problem. It's now an operating expense for me instead of a massive up front capital expense.

Exactly. Now, answer me this: you've decided that you can't afford that large up-front capital expense and having that capacity sitting around unused to deal with the occasional large spike in demand. So why is your cloud provider not following exactly the same business logic that you find sound? Why are they not trying to avoid exactly the same large capital expenditure that you're trying to avoid? ISPs, cel phone carriers, airlines, they all oversubscribe for good business reasons. So why do those reasons not apply to your cloud provider the same as they apply everywhere else?

I've lost count of how many times I've heard someone go "But this is different!". The number of times it really has been different? I can count those on my fingers, using only one hand, and still not run out of fingers.

Re:When you depend on other people ...

[DeathFromSomewhere]

You seem to have only read the first 2 sentences of my post. I'm going to go ahead and let you read that again because it's relevant to your post.

Re:When you depend on other people ...

[johnlcallaway]

The only people who bought a bunch of hardware and had it sitting around idle were people that didn't know how to manage data centers. You still have to project loads for the cloud, and you still have to pay for the ability to scale up. In fact, in our cost estimating, the cost of moving data into and out of someone else's cloud, and the cost of having those large data sets on their servers, was the reason it was more pricey than having our own servers locally even if we had to buy extra servers.

And of course, you missed the entire point. Once you pay someone else, you are at their mercy. Now, if you are a large corporation that can bring pressure to bear, that may not be a bad thing. But if you are a $10M company, they don't care if you pull everything and go elsewhere. In fact, they are betting you can't afford to.

Both Microsoft and Amazon have had major outages in the last year. At our data center, we have not had ANY in the five years I've worked here.

I'm still waiting for someone to come up with a good reason for letting someone else host my data. Lower price and higher availability don't seem to be valid arguments from my experience. It seems the only reason is because people would rather trust Amazon's and Microsoft's incompetent staff over their own incompetent staff.

error protection

[GPierce]

Back in the bad old days, IBM had a solution for down time in mission critical systems - such as for United Airlines. It was called redundancy - a complete dual system. Or as we described it: when one of the two parallel systems detected an error, it automatically sent a signal to the second system so that it could go down too.

Re:error protection

[gweihir]

I think this design was also used in the first Ariane 5 flight! You know the one where 800 Million Euros in solar-research satellites went up in smoke, because some manager was too stupid to understand that you cannot just plug-in an Ariane 4 guidance module and expect it to work.

Re:error protection

[Anne Thwacks]

IBM never provided anything inexpensively.

And they still nearly went bust.

The system works!

[Virtucon]

The system works! Certificates work! Yeah!

Now fire the idiot who forgot to update the certs and we can get on with life.

Re:The system works!

[Anonymous Coward]

I hope the next time you make a mistake at your job, you are immediately fired. And if you leave the toilet seat up even once, your wife leaves you, taking the kids and half your shit. And if you go 1Mph over the speed limit, you are dragged from your car and summarily executed. What a wonderful world it would be, no?

Re:The system works!

[fatphil]

Yes, the single point of failure works!

But I thought "the cloud" wasn't supposed to have a single point of failure, otherwise it would be just a "remote server" rather than "the cloud"?

Re:The system works!

[kqs]

There are always single points of failure. Always. In this case, it was that x509 is poorly designed, but there are others.

The point of "the cloud" was never to have no single points of failure. It is to avoid any single points of failure it can, and hire smart people to avoid and fix the SPoFs it cannot, all at a far lower price than you could afford. And it works well (unless you choose to use an incompetent cloud provider). Most companies screw up certificate expirations at some point, then spend days rather than hours debugging and fixing the problem.

So you trade off an occasional screwup which you make yourself (and can try to avoid via effort, money, training) or n even more occasional screwup by your cloud provider (which you have little control over). It's a valid choice, and for those companies which will not pay for expert IT employees, a no brainer.

Re:The system works!

[Anne Thwacks]

So all is revealed: renewing the certificates was your job!

Re:The system works!

[Virtucon]

Well the cloud works on open web standards and while certificate servers can have redundancy built in, the underlying certificate would still essentially be a single point of failure in the design. Any TLS that relies on certs will have to take this into account. The good news is that while somebody goofed at MSFT, the underlying principles of Certs prevailed and people were denied access to resources because their clients wouldn't trust the MSFT resources protected by those certs. Now, I would be more concerned if clients just ignored them and were allowed to use the services, that would be the more intriguing data point here. You can always "accept" an expired cert in your browser, but if an underlying application just did it on your behalf, then I'd be worried.

Monitoring Fail

[HTMLSpinnr]

I find it hard to believe anyone who maintains such a large fleet of services wouldn't have setup some sort of trivial monitoring (I know they own a product or two) that would include SSL Certificate expiration warning. 30+ days out, a ticket (or some sort of actionable tracking mechanism) should have been generated, alerting those responsible to start taking action. Said ticket should have become progressively higher severity as the expiration date loomed (meaning nothing had been updated), which in any sane company, would have implied higher and higher visibility.

That way, if an extensive test plan for such a simple operation was required, they had plenty of time to execute upon it and still not miss the boat.

Working with MS in other ways, and combined with both the lack of foresight and inability to act quickly, just shows that this sort of customer-forward thinking just doesn't exist inside the MS mind.

Re:Monitoring Fail

[ageoffri]

Believe it. When I worked at IBM, there was a certain automation team who let the critical SSL certificate for an ID provisioning tool expire not just once, but two years in a row causing a major outage to a large client.

Re:Monitoring Fail

[rabbitfood]

Simple operation? You've clearly never worked for a large company.

Even if a warning wasn't trickled down a month ago, and we've no reason to assume it wasn't, the person whose job it is to act on it, provided they weren't on vacation, won't have simply thrown five dollars at a registrar. They'll have had to put in a request to the finance department, probably via a cost-management chain of command, with a full description of what needed to be paid to whom and why, with payee reference, cost-center code, expense code and departmental authorization, and hoped it would arrive in time to be allocated to the next monthly rubber-stamp meeting. Assuming the application contained no errors, was suitably endorsed and was made against an allocated budget that hadn't been over-spent and wasn't under review, then, perhaps, in the fullness of time, it might have received approval and have been sent back down the chain for subsequent escalation to the bought-ledger department, who'd have looked at the due date, added ninety days and put it on the bottom of the pile. After those ninety days, when the finance folk began to take a view to assessing its urgency, unless they found a proper purchase order from the supplier, and a full set of signed terms and conditions of purchase, non-disclosure agreements, sustainability declarations and ethical supply-chain statements, as now required by any self-respecting outfit, it'll have been put aside and, eventually, sent back round to be done properly. Or, if it all checked out first time, it'll have been put on the system for calendering into the next round of payment processing.

I'm sure it might be possible to streamline aspects of such mechanisms, but to suggest there's anything trivial about them is a touch hasty. But you never know. Perhaps they're already thinking of planning a meeting to discuss it, and are working on a framework for identifying the stakeholders as I write.

Re:Monitoring Fail

[jader3rd]

I would be shocked if some sort of monitoring didn't fire. The problem would be that it would have gotten lost in the noise of all of the other monitors firing for other issues.

Re:Monitoring Fail

[TheLink]

After the infamous Feb 29th incident MS should have set up an Azure cluster identical to production stuff but with all the clocks set to 1 week or more ahead. Have it continuously running regression tests. Certs even getting close to 3 days before expiring is stupid.

Microsoft has billions of dollars, so if this 12 hour downtime is the best MS can do when they're "All In" (Ballmer's words not mine), it's not a good sign.

Re:Monitoring Fail

[HTMLSpinnr]

You'd be mistaken. The large company I work for has indeed learned from these mistakes.

Re:Monitoring Fail

[Common Joe]

All this... and if the dude quits, there is usually no backup for that person. Large company mentality at its finest.

Liability

[Skiron]

I guessMS somewhere in their licensing of this stuff have a clause that states they are not liable. Basically, 'bollocks to the Customers' when we fuck up [again].

So I cannot understand why people use them at all (once bitten, twice shy, twice bitten.. etc.).

Re:Liability

[FreelanceWizard]

Actually, Microsoft has a wide variety of SLAs with financial penalties covering the Azure cloud. I expect customers will be able to claim at least a 10% service credit on this, as it's definitely an issue within Microsoft's control and definitely would cause a miss of the monthly availability number.

Review http://www.windowsazure.com/en-us/support/legal/sla/ if you're interested in the Azure SLAs. Interestingly, Amazon has a much less tough SLA, as it's calculated on a yearly basis and doesn't have as brutal penalties (Amazon at most credits 10%; Microsoft credits up to 25%).

Re:Liability

[Skiron]

99.9% is stated there a lot of times. Is that over a 1000 years?

If not, that is about 1 day a year outage (when Customers go tits-up).

They are keeping their promise, it seems.

Re:Liability

[Todd Knarr]

My problem with those SLAs is that they're for a credit for a fraction of the cost of the service for that month. Which is fine if your business doesn't depend on the service and you suffer no disruption when the service is down. But if you're hosting a Web site on the service, or using it for anything business-critical? The cost of the service is going to be the smallest part of the cost to you of the disruption (that's why you went with the service after all, because it was so much cheaper than doing it in-house). The SLA doesn't cover you for lost sales, lost business, lost customers, the cost of employees sitting around idle because the systems they need to do their jobs aren't working...

Re:Liability

[fatphil]

99.9% is between 7 and 8 hours down-time a month (which is the unit they measure in). If it took them 12 hours to get new certificates up, then they are not keeping their promise, they are failing.

Of course, if that downtime coincides with your working hours, that's an entire working day down. It's a shitty level of service. Nobody hosting their own services, and having skilled staff managing their systems, would find that acceptable. I will admit that 99.999% uptime/connectivity is hard (we've had it one year in the last decade), I'm never satisfied with anything less than 99.99% uptime. The majority of the downtime is because of something beyond my control (electricity to the whole suburb goes, ISP drops off the net, etc.)

Re:Liability

[symbolset]

The Azure SQL Database reporting facility just completed a 5-day outage this month so they may be a couple years over their downtime quota this month. Or as somebody else put it recently: "Five nines: 9.9999".

Re:Liability

[BitZtream]

If your website is that important you should have it all in the hands of one UNREGULATED company.

Its acceptable to trust one telephone company or one power company as these are WELL regulated industries that the government requires them to provide certain TESTED redundancies.

Depending on a single (anything in the tech industry) is ... well to put it bluntly, is stupid.

Re:Liability

[dbIII]

If that's the case then they truly broke it last year with the 24 hour leap day outage.

Re:Liability

[petermgreen]

umm 99.9% is about 0.73 hours per month, not 7-8 hours per month.

Re:Liability

[fatphil]

very true. ooops

Laugh

[koan]

Where there are clouds there is rain.

Anyone remember Hotmail?

[BetaDays]

Remember when they forgot to renew it's domain name. http://slashdot.org/story/03/11/06/1540257/microsoft-forgets-to-renew-hotmailcouk

Microsoft has it's own internal CA

[ejoe_mac]

So wrong in so many ways. Any reason you wouldn't purchase a 100 year certificate and just roll with it? Too bad about 1/3 of all Azure disk space is used for endpoint backup. This reminds me of the leap-year calculating bug - Feb 29 2012, you couldn't generate a site because the default is to generate a certificate for 1 year, and well, Feb 29 2013 just doesn't exist. http://blogs.msdn.com/b/windowsazure/archive/2012/03/09/summary-of-windows-azure-service-disruption-on-feb-29th-2012.aspx

Re:Microsoft has it's own internal CA

[antdude]

FYI, it's = it is. You're welcome. ;)

Re:Microsoft has it's own internal CA

[Kalriath]

Because they are forbidden to issue a certificate with a greater validity than 39 months in accordance with the CA/Browser Forum Baseline Requirements for Publicly Trusted Certificates (warning: PDF). If they were to violate this, they'd have GTE's root certificate de-listed by Apple, Google, Mozilla, KDE, Opera, Blackberry and ... um ... Microsoft. Which would invalidate their subordinate certificate.

Makes good business sense!

[gweihir]

From a business perspective, it makes perfect sense: If Azure were reliable, secure and fast, customers could start to wonder why the other products by MS are not. This could heighten customer expectations, and that would be bad as MS really does not have the engineering capabilities to build, say, a good OS or a good office productivity suite and then customers may leave for the alternatives. So I applaud them for their foresight in making Azure just as bad as their other things are. This may actually be quite beneficial for their bottom-line.

Re:Blew their support contracts..

[RazorSharp]

Azure - bright blue in color, like a cloudless sky

Sick Cert Solutions Suck

[Sloppy]

Imagine if someone's signature on your PGP identity expired. It might be a bit of a blow, but people would still have other trust pathways toward you. Then you get a new signature from 'em, or someone else.

Certs can fail in so many ways, both false positives (compromised CAs) or false negatives (such as this expiration), and a myriad of subjective failures since different people have different reasons to trust (or not trust) different CAs. The risks aren't even theoretical. Failure really happens, to the extent that it's almost routine and we see a story about it here on Slashdot every month.

And Phil Zimmerman totally solved the problem(!) in, what, 1988? Why are we still using obsolete-the-day-it-came-out single signer systems? So brittle. So unrealistic.

The only reason I can think of, is that it would work too well. MitM attacks would become nearly impossible for even the most powerful governments. Certs would become so competitive and cheap that the CA business would collapse.

Re:Sick Cert Solutions Suck

[BitZtream]

You're just fanboying for PGP.

Everything that applies to certs applies to PGP and vice versus. They are essentially the exact same thing, only very minimally different implementations.

You're acting like its an SSL issue that MS decided to consider expired certs invalid in their systems rather than accepting them.

If it was PGP, the same thing would have happened because they wouldn't trust expired PGP certs either.

A policy on how to behave in a situation isn't effected just because you change the name of the software implementation and a couple of low level algorithms that have no effect at a high level.

SSL can also be done in a 'web of trust' kind of shitty way, its an active choice not to. If you knew anything about implementing things in the real world you'd understand why PGP is a relatively rarely used system of encryption compared to other systems like it. Nothing prevents you from adding your own signature to an SSL certificate.

You have absolutely no idea how either PGP or SSL work.

You fail to understand that the reason people use SSL over PGP is because PGP DIDN'T SOLVE SEVERAL MAJOR PROBLEMS. You some how interpret the rarity of PGP use with it being superior. It isn't. The only people who think PGP is great are freetards who think paying for a verification service is unacceptable and everything should be free.

Rather than giving you a single trust source like SSL (which you imply, but is entirely not the case by design), PGP gives you a certificate with no way to verify it with a known source. Everyone picks their own source of verification! Awesome idea!

You're probably one of those morons who think self-signed certs shouldn't trigger warnings too aren't you?

Re:Sick Cert Solutions Suck

[Sloppy]

You're acting like its an SSL issue that MS decided to consider expired certs invalid in their systems rather than accepting them.

No. I'm saying it's an SSL issue that when The One and Only cert that can possibly exist, expires, there is no backup trust path. When the expiration happens, the number of valid certifications falls from 1 to 0. With a real world trust model, when an expiration happens, the number of valid certifications could fall from, say, 4 to 3.

If you lose your drivers license, your passport should still work. Or it should somewhat still work, successfully persuading some people, maybe even a majority of people, that you're you. And a passport plus a CostCo card plus a note from your mother, ought to work a little better than just a passport alone.

PGP gives you a certificate with no way to verify it with a known source. Everyone picks their own source of verification!

Didn't your irony alarm go off, when you wrote something that dumb, and then said I don't understand how PGP and SSL work?

You're probably one of those morons who think self-signed certs shouldn't trigger warnings too aren't you?

You're probably one of those morons who think a complete lack of any cert or encryption -- plaintext which can be passively snooped, or actively altered, without even bothering to MitM -- shouldn't trigger warnings, aren't you? ;-)

The debate about how UIs should present the risks associated with unauthenticated connections, has always been about relative degrees. I don't really have a solid position about whether a self-signed cert should trigger a warning or not; the severity of the risk depends on the situation. I do hold, though, that SSL with a self-signed cert is safER than eschewing SSL altogether. MitM-vulnerable crypto is better than lack of crypto. If a web browser shows a modal warning for https and self-signed certs, and doesn't also show that warning (or something more severe) for http, then it was either written by fools or is micromanaged by clueless PHBs.

Re:Does Timothy Have Brain Damage?

[MrL0G1C]

Calling someone a cunt because they missed a typo is not constructive criticism.

Re:Does Timothy Have Brain Damage?

[6ULDV8]

Calling someone a cunt for any reason wouldn't make constructive criticism. When I use say it, it definitely isn't an attempt at anything constructive. I still love the word though.

CEO Backgrounds

[BoRegardless]

My perception of Ballmer and Dell is that they virtually started with their companies and neither person has a wide ranging training in business management & psychology of managing. Ballmer is famous for his chair throwing and viscous firing with a loud voice, sometimes for trivial reasons & banning Apple products in most places inside the company. Dell has been reported to become physically withdrawn when competitor Apple is mentioned.

Neither of those responses to common activities speak good of a stable CEO who delegates well & thus the company's results suffer.

Re:Blew their support contracts..

[grumpy_old_grandpa]

Behind the clouds, the sky is always blue.

Re:Blew their support contracts..

[wisty]

That's the great thing about cloud computing. It's always there, like a cloud. Except when it decides to go away. Like a cloud.

Microsoft cloud rains on world's parade.

[sjames]

n/t

SSL rocket science

[ei4anb]

$ curl -vIs https://www.windowsazure.com/ 2>&1 >/dev/null | grep "expire date"
* expire date: 2013-11-15 18:15:53 GMT

Call this from a cronjob script which should then take suitable action if the date is too close.

Re:Spellcheck...

[Kalriath]

IE10 has a spell checker now. They're only 5 years late, but they got there.

Re: Does Timothy Have Brain Damage?

[grcumb]

Shyeah. Even a total idiot knows it's 'expirtation'.

Re:MS cloud services have been good to me

[Eunuchswear]

I'm not a shill, I like all OSes from Windows to VxWorks

You're not a shill, you're Dr Pangloss.

What about Amazon's S3 recent MAJOR outage?

[elabs]

Why wasn't this mentioned in the story? Amazon has had several very high profile glitches in S3 and EC2 in the last few months. One of them recently brought down Netflix.