Microsoft Azure Failure: SSL Certificates Were Updated... Sort Of

judgecorp writes "Microsoft has published an explanation of the failure of Windows Azure earlier this month. Users of the Azure storage saw that an SSL certificate had expired. Microsoft's explanation says that the certificate had in fact been renewed, but an update with the new certificate details was not prioritized, and hadn't actually been implemented till after the old certificate expired. There are more interesting details, but Microsoft says better alerts and more automation will stop this particular fault happening again."

Comments

Re:When will they accept Windows 8 as a failure?

[theRunicBard]

Look, I know nobody cares, but Microsoft Azure has nothing to do with Windows 8. I'm also not sure it's a failure. Microsoft tried something new after getting great positive reviews for Windows 7, which is the BEST time to try something risky. Worst case, people skip one generation of Windows, and stick with... Windows. Best case, you redefine the PC interface. It is innovative, no matter how poorly implemented. Besides, Microsoft has a history of creating a shitty first version and then fixing kinks as time goes by. Was anyone expecting a good first version of Metro? The slow adoption numbers can easily be credited to how good Windows 7 is. Why would you switch? It costs $0 for me to stay on 7, and > $0 to upgrade. We won't be seeing many Windows 8 devices for a while. The timely upgrades brought about by Windows Blue might even spur more adoption (too early to tell, I think). Windows Phones I won't attempt to defend since I know nothing of them.

It won't happen again

[Nerdfest]

Unless I'm horribly mistaken, they've let certificates expire before. Why would I think they won't let it happen again?

Re:It won't happen again

[phantomfive]

Yeah, and they also had the Sidekick outage with actual data loss. A lovely quote from that article:

"I asked Microsoft for comment Saturday when I was writing this, in particular as to how the rest of its cloud might differ from the Danger set up. Microsoft said Sunday that its the fabric controller that manages the Azure service is built with redundancy in mind. "

It may be built with redundancy in mind, but apparently it still has at least one single point of failure.

Re:It won't happen again

[norpy]

Pretty sure the last one was a bug that was something to do with the cert expiring on a leap-day though.

This is a much deeper problem that shows that there is not a whole lot of good process going on behind the scenes.

Re:It won't happen again

[93 Escort Wagon]

Some of us remember when they forgot to renew hotmail.com. I'd say that might be worse...

Re:It won't happen again

[Anonymous Coward]

I always back up my cloud data to a local harddrive, just to be safe.

Re:It won't happen again

[girlintraining]

It may be built with redundancy in mind, but apparently it still has at least one single point of failure.

Yeah. It's the same single point of failure present in every IT project: It's called The Manager, and it goes something like this:

Engineer: "I sent you the e-mail!"
Manager: "Oh? I never got it."
Users: "Oh f---."

Re:It won't happen again

[phantomfive]

I would suggest two reasons:

1) They don't actually have a lot of customers, and for those customers they do have:
2) There aren't a lot of options. Building your own site with 99.999% uptime is really hard, Amazon's cloud has outage problems as well, where exactly are you going to host your site in a way that doesn't go down? If there aren't options, you just go with the best you can, and Azure at least seems to be easy if your team is only semi-competent (like a lot of programmers these days).

Re:It won't happen again

[phantomfive]

Maybe. It seems to me that if the engineers have let the manager become powerful enough to be a single point of failure, they've designed the system wrong.

Re:It won't happen again

[kamaaina]

Didn't they let one of their domains, passport.com, expire as well

http://news.cnet.com/Good-Samaritan-squashes-Hotmail-lapse/2100-1023_3-234907.html

Re:It won't happen again

[TheLink]

Given Microsoft's resources and Ballmer proclaiming they were "All In" on Azure, what they could have done after the leap year bug was to set up test systems that are replicas of production shards/clusters but with the time set to one week ahead or so. Then have the test systems run the usual regression tests 24/7.

Then if stuff fails because of leap years, expiration or other time related stuff, it's more likely to fail in a test system first and they'll have a week to fix the problem before their users notice.

This is a pretty frigging obvious thing to do right? If they really are "All In" they'd have the money, resources and people to do this and it would already have been done by now.

So given that this failed publicly and the failure wasn't in some fringe beta feature (which would have been somewhat excusable) it just reinforces my long held opinion that Azure is not ready for serious use.

There are plenty of very smart people working for Microsoft but they didn't or weren't able to help prevent this problem despite Microsoft supposedly being "All In" on Azure so I think Microsoft needs to fix itself too, or they're not really "All In" on Azure.

Re:It won't happen again

[wvmarle]

I still wonder why those certificates need an expiry date. And why they just don't put it like 100 years in the future.

But first of all, why have expiry dates at all?

I have seen often enough certificates being revoked for being compromised or whatever; and I have seen quite some trouble due to expired certificates leaving web sites inaccessible, for example.

It doesn't seem to add much if any security (if it's compromised, you'll want to revoke it now, and not wait until it expires months or years later). When setting up a Google Play developer's account they advised to have my certificate expire 10 years in the future - making me again feel like it is just an inconvenience, and I'm sure 8 or 9 years from now (when it expires - I already forgot when) I will have forgotten about it and it's going to cause all kinds of issues with my apps not working any more or whatever.

Re:It won't happen again

[girlintraining]

Maybe. It seems to me that if the engineers have let the manager become powerful enough to be a single point of failure, they've designed the system wrong.

You're fired. Anyone else have a problem with the manager?

Re:It won't happen again

[purpledinoz]

To be fair, it is really hard to remember so many dates and appointments when you're so busy. If only there was some sort of software, which can remind you of appointments and dates....

Re:It won't happen again

[phantomfive]

Of course I don't tell THEM that, I just build around them and let them think they are useful. Sometimes we have meetings whose sole purpose is to affirm the usefulness of the manager.

Re:It won't happen again

[Anonymous Coward]

Oh, I see you know my manager!

Though you forgot the part about "You! To the bottom of the stack rank, NOW!"

At Microsoft, that typically comes at least a few months before actually being fired.

(Posting this as AC for hopefully obvious reasons).

Re:It won't happen again

[girlinatrainingbra]

Nice! I like the fact that it was a linux user who paid the renewal fee and got passport.com back up again, allowing further logins into hotmail. Linky to credit card receipt of individual user : The lapse, which was first reported on the Internet news service Slashdot.org, was apparently caused when Microsoft's registration for the Passport.com domain name expired sometime Dec. 24, Chaney said. The Passport.com site verifies user identification and passwords for access to Hotmail and about 25 other services, according to Chaney. Chaney said he paid the bill Dec. 25 at about 2 p.m. EST and was given invoice #11395965 documenting the transaction. An electronic copy of the receipt can be viewed at his Web site at "www.doublewide.net."

Re:It won't happen again

[Alioth]

Certificates need an expiry for the same reason that passwords ought to have them. The probability that a certificate has fallen into unauthorized hands increases with the passage of time, so having certificates expire means you can limit the usefulness of a stolen certificate.

Re:It won't happen again

[wvmarle]

Interesting you mention expiry dates on passwords as plenty of security people will argue that having expiry dates on passwords tends to decrease the security of passwords, as people select easier ones.

Having a multi-year expiry date pretty much beats the purpose: after falling in the wrong hands the certificate is useful only until it's detected that it's in the wrong hands. And that's usually not very long after it's being used.

And a short expiry date (weeks, months) where it may actually have an effect on stopping unauthorised people to use it, is so inconvenient that no-one uses such short periods. In case of passwords that have to change monthly, expect many users to have a password like "March03" for this month. Or "March!03". Dictionary word? Well "hcraM!03" should circumvent that one, too. So much for security. And yes, first-hand experience on that part.

Re:It won't happen again

[DarkOx]

The security guys that argue passwords should not expire are crappy security guys. Passwords should be long enough and not expire at to great a frequency, I would say probably not less than 90 days. Many password attacks are inside jobs. Did the guy who does the backups take home a copy of the sam database? If you don't rotate your passwords he can probably brute force them if they are weak pretty quickly. They might hold up several months if they are strong. Once he gets a password to a privileged account or a valuable account like C?O he is now free to do things and it will be hard to account for them or prove who did it. That is one of the biggest reasons to rotate passwords.

Certificate expiration is important as well. You talk about revocation. That only works if the system can get a revocation list or reach a oscp sever. There are plenty of cases where a system may need to validate a certificate where it does not have access to the specified revocation authority. That is probably not the case with the Azure cloud but certainly is true of a device like a smart phone which moves between networks. Often the most you can do in this situation is assume if everything else is okay the certificate is valid or fail. If you go with the former at least having an expiration date forces the presenter of the certificate to "re-authenticate" with the authority at some point to get a new cert. Otherwise that 'revoked' certificate would seem to be authentic to any device that can't do a revocation check either forever or until the CA certificate expires.

Re:It won't happen again

[RaceProUK]

Building your own site with 99.999% uptime is really hard, Amazon's cloud has outage problems as well, where exactly are you going to host your site in a way that doesn't go down?

There are Cloud providers other than Amazon & Microsoft.

None of which can claim to be better than 99.999% uptime, since it's practically impossible to achieve.

Re:It won't happen again

[FireFury03]

The security guys that argue passwords should not expire are crappy security guys.

It depends what the password is protecting. If someone gets your password, the chances are they are going to use it immediately. A timed expiry of passwords can prevent repeat-uses of it, but if the attacker already had chance to install malware when the account was originally compromised, they probably don't even need the password the second time around. Additionally, if a repeat attack isn't going to get the attacker anything extra over the original attempt, its probably not worth worrying about.

Conversely, regularly expiring peoples passwords almost always leads to them changing "mypassword1" to "mypassword2" and then to "mypassword3" which arguably doesn't actually do a lot to improve security. Additionally, if people are having to change password frequently and they don't do the above, they are probably using fairly weak passwords in order to remember them and/or keeping a post-it note on their monitor with the password written on it.

There is a lot to be said for encouraging people to have a very secure password that is never shared with other services and rarely changes (so they can memorise it, even if its strong).

At the end of the day, different policies lend themselves to different situations. Blindly forcing everyone to change their password every month is probably very counterproductive - the chances of someone compromising a strong password that isn't shared between services is very low, whereas the temptation to use a weak, shared password with a deterministic monthly change is extremely high.

Certificate expiration is important as well.

Certificate expiration can't be considered anything like the situation with password expiration. Certificates are machine-readable, they can be made as secure as you like and you don't have to remember them. Other than being a mild pain in the arse, renewing expired certificates shouldn't be a big problem and there is no incentive to make them weak as a result.

Re:It won't happen again

[Trailer Trash]

And I would have renewed their cert had I been able.

Look, the bottom line is that they haven't learned anything in the past 13 years (wow, I feel old). The sloppiness that allowed a domain registration to lapse is the sloppiness that allows a cert to expire. This is a cultural issue that will likely never be overcome.

To step into another industry, let's look at phone service. The "Phone Company" (AT&T back in the day, then the baby bells) had a culture of "this service has to work, period". I'm 45 today and there have been now 3 times in my life that I've picked the phone up and there wasn't a dial tone. In our parlance, they have good "up time".

The cable company, on the other hand, has never had that culture. Their product isn't necessary in the way that a phone is and outages are fairly common. If they have to work on something your service may be unexpectedly down for a few minutes. Or hours. Whatever. My internet has more outages every month than my phone service has had over my lifetime.

The point is that Microsoft's culture is more like the cable company. They are a software company and having to keep servers "up" hasn't been their deal until kind of recently. Companies like Amazon or Google, on the other hand, have had to have a phone-company-like culture from the beginning. They write software, yes, but their main product is a web site that has to be up come hell or high water. And, yes, I know that there have been a couple of high-profile outages, but those outages weren't caused by the kind of sloppiness that results in someone forgetting to renew a domain registration.

So I use Amazon for my stuff but would switch to Google if there were problems at Amazon.

But Microsoft? Are you kidding? I feel like I diapered them on Christmas Day of 1999, so I probably have less respect due to that.

Re:It won't happen again

[93 Escort Wagon]

"The lapse, which was first reported on the Internet news service Slashdot.org, ..."

... and was again reported on Slashdot.org two days later...

Re:It won't happen again

[frinkster]

None of which can claim to be better than 99.999% uptime, since it's practically impossible to achieve.

Having worked for half a decade on mobile communications infrastructure that regularly exceed 99.999% uptime, I feel qualified to say that it is neither impossible nor super difficult. If it is a goal and you are willing to spend a lot of money than you can accomplish it.

But nobody is going to pay $X for 99.99999% uptime when 98% uptime is available for $X / 100 unless they are forced to. Look at all of the various highly-funded internet services that go down completely when a single Amazon data center has an outage. They aren't even willing to pay a little bit extra and do the extra work to make their services run on multiple data centers at a time. Clearly, it is not a requirement of the venture capital that they are getting.

Re:It won't happen again

[Chris Mattern]

Building your own site with 99.999% uptime is really hard

Building your own site with better uptime than Microsoft's Cloud, on the other hand, doesn't look that hard at all.

Re:It won't happen again

[tehcyder]

I always back up my cloud data to a local harddrive, just to be safe.

Isn't that cheating or something?

Re:It won't happen again

[Lisandro]

God. I'm torn between modding +1 "Funny" or "Insightful"...

Re:It won't happen again

[RaceProUK]

The network as a whole may achieve better than 5 9s, but I doubt any of the users are experiencing less than 30s downtime a year.

Re:It won't happen again

[DarkOx]

Again its about not enabling someone to get your password in the first place. Rotation absolutely helps with that in the even a master password database is stolen. There are any number of reasons you might not be aware of that as well, not the least of which is an admin who has rights to copy the file decides to do so. He might find it very useful to be able to brute force the CEOs passwords and take a look around at the companies financial statements or his mailbox with out appearing in any logs for example and without having to tamper with logs, which he might also get caught doing.

As far as passwords like mypassword1, followed by mypassword2 there are simple technical methods to prevent users from doing that as well and they should be employed. Passwords written down are big problem, which is why I suggested in the vast majority of situations make the passwords complex enough to stand up for 90 days and not requiring users to change them more often than that. One thing we suggest to people is if you absolutely have to write down a password keep it in your wallet/purse. You will usually know right away if its gone missing and change your password immediately or phone the helpdesk to have your account locked if this happens.

Re:It won't happen again

[FireFury03]

Again its about not enabling someone to get your password in the first place. Rotation absolutely helps with that in the even a master password database is stolen.

Of course, but as with all security, this is about balancing the odds - what are the chances of the master password DB being stolen, cracked and the password(s) used, vs. the chances of one of your users being pushed into having a weak password through having to change it regularly?

You will usually know right away if its gone missing and change your password immediately or phone the helpdesk to have your account locked if this happens.

More likely they will be more concerned about their money and credit cards having been stolen and will completely forget that they had a password in there...

Re:It won't happen again

[DarkOx]

Google is not a good example just yesterday, their 8.8.8.8 dns server was returning anything for www.youtube.com ( no not NXDOMIN ) just nothing would time out. Yet every other query I could think to send it worked fine. It was really odd actually. Gmail was down just a couple months ago. Slate even had an article about how debilitating it was for everyone.

Re:It won't happen again

[im_thatoneguy]

It may be built with redundancy in mind, but apparently it still has at least one single point of failure.

It's extremely rare that an entire services goes down. Generally what happens is one region goes down. The fact that people don't *pay* for the full redundancy and fail-over protection doesn't mean it's not technically built into the system.

Re:It won't happen again

[TheRealSlimShady]

Actually if you bother to read the article it looks like they had a reasonably good process going on behind the scenes i.e. cert owners got alerted & pushed the new cert in an update. The only problem was that they forgot to mark it as containing critical information (well, and their monitoring tools didn't alert them say a week out to say that the certs hadn't been renewed). So there is definitely room to improve the process, but saying that there is not a whole lot of good process is drawing a long bow.

Re:It won't happen again

[Trailer Trash]

Right, there will always be outages. The point is that most of them aren't caused by general numbskullery.

Re:It won't happen again

[Trailer Trash]

Yep, still here.

It WON'T happen again because of automation

[Anonymous Coward]

It definitely won't happen again, instead the team responsible for keeping the automation software running will fail. Or an automatic upgrade to Windows will break it, or the libraries needed to run it will have been deprecated.

So yeh, it won't happen again, the next time it will be something else to blame.

Never of course a management that chops up roles into such small increments, dis-empowering it's workforce so much that the simple job of updating a certificate becomes a major obstacle each and every time it happens. No, never a load of BS managers, no sir!

Typical scenario of ...

[Skapare]

... managers saying "we need to get this up and running sooner ... automating it reliably is hard to do ... just get it working and update things manually for now and we will automate it later". When later comes, everyone is working on something else.

inexcuseable re: ...cert expiring on a leap-day...

[girlinatrainingbra]

re Pretty sure the last one was a bug that was something to do with the cert expiring on a leap-day though. [emphasis mine]
.
$gt begin{sarcasm} Well, if it was a leap-day event, well that's totally excuseable because there's no predictable way to know that a particular year might be a leap-year with a leap-day in it, and even if there were, my goodness, you'd need some sort of computational device to carry out the algorithm (that Al Gore, he invents everything!) that would let you figure it out, and who could afford a computational device??? end{sarcasm}
;>p
Come on, you can't let Microsoft off the hook for screwing up things like that. It's supposed to be a software company. Y2k was known about well before it occured; leap-year days are well known about and recur on an amazingly well-understood and defined schedule. This is not a much deeper problem. It's just another basic problem that shows that there are not any good processes going on behind the scenes at Microsoft. And Apple screwed up their alarm clock functionality that kept messing up on iOS at the beginning of the New Year, too. That was also just as inexcuseable.

automation and alerts

[holophrastic]

uhuh. I think people, especially technology companies, forget that the easiest task to automate is one that a human can simply do.

"Executive assistant in charge of renewing certificates". Make it someone's job. It'll get done. You don't need a robot. You just need it to be in someone contract. That's it.

Re:automation and alerts

[Lennie]

Yeah, we can see how well that worked this time.

The biggest problem with all this is something else of course, within big companies people get assigned to different tasks all the time.

Sometimes that means that simple task but very important task gets handed over to some one else who doesn't fully understand the implications when it doesn't get done.

Hm.

[stoploss]

I always back up my cloud data to a local harddrive, just to be safe.

That sounds like vaporware.

Even more inexcuseable

[dbIII]

You'd think after people made fun of the MS Zune for being out of action on a leap day that MS would take a bit more care before the next one.

Re:When will they accept Windows 8 as a failure?

[hairyfish]

I haven't tried Windows Azure, which is the subject matter here...

questlove and windows-phone-suckage

[girlinatrainingbra]

Yeah, all of the window phones silliness is so worth laughing at. I remember the crazy ad that came out for the windows phone last year that had QuestLove in the commercial. I believe that /. had a story about MS cancelling that phone the SAME DAY that the commercial had just aired.
.
And what the fVCk is it with the stomping and jumping and slapping around of hardware in the ms tablet ads? Is that all that the MS tablets are good for? Throwing them around and clunking them onto tables and benches? What's with the ugly mean-faced girl-scouty attired girls in that first MS tablet surface ad? I think MS just saw the Apple ipod and iPhone ads that had a single song playing in the background with cool activities and decided to copy the style without any substance. Hey, that kind of explains most of the things that they do!

Re:When will they accept Windows 8 as a failure?

[kbg]

When you charge an arm and a leg for an OS and your company basically has unlimited money, then there is no excuse for not delivering perfect software with no bugs. So yes I was expecting a perfect version of Metro.

Re:When will they accept Windows 8 as a failure?

[phntm]

the adoption rates for students who get windows 8 for free is non existant at least by the anecdotal evidence in my faculty (computer science).
even during exam season (when you suddenly get the urge to clean the room, re-check the fridge or format your laptop).
you can piss on my face but don't tell me it's raining.

Car analogy...

[girlinatrainingbra]

Read the MSDN blog for how screwed up this really was. Here's the car analogy: We have a "Secret Store" that tells "the team that owns the tires" that the tires are just about worn out and that they will be useless on a certain specific date. The "team that owns the tires" buys new tires and tells the "Secret Store" that new tires have been bought. But the team does not install the new tires, but places the task of installing the tires in an "unprioritized queue"!!!! Somehow, more important tasks like replacing the windshield washer fluid and replacing that pine-tree air freshener hanging off the mirror get prioritized on the queue and performed. Lo and hehold, the tires get too old, expire, and are taken off of the car. No one bothers putting new tires on the car. The car is nonfunctional. MS FTW, yet again!

It's incredible how they keep shuffling blame around, or hot-potato-ing it:

In this case, the Secret Store service notified the Windows Azure Storage service team that the SSL certificates mentioned above would expire on the given dates. On January 7th, 2013 the storage team updated the three certificates in the Secret Store and included them in a future release of the service. However, the team failed to flag the storage service release as a release that included certificate updates. Subsequently, the release of the storage service containing the time critical certificate updates was delayed behind updates flagged as higher priority, and was not deployed in time to meet the certificate expiration deadline. Additionally, because the certificate had already been updated in the Secret Store, no additional alerts were presented to the team, which was a gap in our alerting system. [source link] [bold emphasis mine]

Laughable, if it were not so stupid.

Re:Car analogy...

[girlinatrainingbra]

It's incredible that you're so dense that you don't know about car analogies or soviet russia jokes! What, are you a neutron star?

Re:Car analogy...

[drinkypoo]

It's laughable mostly because they told us just how incompetent they are. They came right out and told us that their service, which they want you to make mission-critical, is managed with a process that would make the three stooges weep with uncontrollable laughter.

Bad infrastructure management

[DaveV1.0]

This is what happens when you have bean counters and MBA running the IT department.

Re:When will they accept (Linux|Mac) as a failure?

[DaveV1.0]

Very few like it and use it. (Linux|Mac) desktop? less than 5% of the market share. Now that I have shown the fallacy of your statements, how about you just shut the fuck up.

Re:When will they accept Windows 8 as a failure?

[DaveV1.0]

Guess what. Almost nobody cares that it comes with a secure bootloader. The only people who do care are a small number of geeks.

Re:When will they accept Windows 8 as a failure?

[RaceProUK]

When you charge an arm and a leg for an OS and your company basically has unlimited money, then there is no excuse for not delivering perfect software with no bugs. So yes I was expecting a perfect version of Metro.

The cost of certifying a modern OS totally bug-free would exceed the GDP of the entire world, hundreds of times over.

Re:Maybe not putting incompetent cretins in charge

[gweihir]

A true coward: Nothing of worth to say and that without any grace...

Re:When will they accept Windows 8 as a failure?

[LordLimecat]

Troll = Fed.

In the future, mod him down and move on.

Ouch

[thoth]

Good lord, last year it was a 12 hour outage on leap day, this year it was a 12 hour (as far as I can tell) outage due to expired certificates. They won't be able to claim six 9's uptime for ~274 years!

At the rate of a half day of failure every year, so far, I'm not even sure I'd trust Azure for storage no matter what the discount they offer.

How many days does it take to push an update?

[brtech]

They pushed the update out on Jan 7. By Feb 22, it hadn't been completed. Something is not right with this explanation. Doesn't matter how low a priority it was, it should have been pushed out within in what? Two weeks?, a month?

Re:When will they accept Windows 8 as a failure?

[X0563511]

Almost nobody cares about a lot of things that matter a great deal.

Re:When will they accept Windows 8 as a failure?

[drinkypoo]

As usual, we see the failure of using the closed source model for an operating system. They have to get the users to fund development somehow, so they sell them a shitty version every other time to pay for the real versions, and get the new ideas into the hands of the customers where they can tell them which ones are good and which ones are bad. It can work fine for applications where they can bring out a new version when they're ready, with incremental updates for features or fixes which must and can be hacked on to remain relevant and keep customers from jumping ship, but it's just not a good model for operating systems while the state of the art continues to change so rapidly.

Re:When will they accept Windows 8 as a failure?

[tlhIngan]

I really like the OSX system, I feel it's a stronger OS overall, but Microsoft will win because they have a strong platform in one crucial area: backwards compatibility. If you write something for OSX, there is no reason to believe that it will still work in five years. On Microsoft's platform, it has a good chance of working as compiled. This is even more important to businesses than to home users.

That's because most programmers suck.

OS X is actually quite compatible. Provided you stick to the public APIs. Do anything funny and yes, things will break. The presence of private headers though makes it way too easy to use a private API to do so something (as part of "get-it-working") that leads to OS version dependency.

Oh yeah, it happens on Windows too. Next time you run Process Explorer, look at the window title for explorer.exe. You'll see it's called "Program Manager". Because despite it being 2013, there are applications that STILL hard code it. Or apparently some apps hardcode resource IDs or DLL export IDs (you normally export by name, but you can export by ID) in their apps, forcing what was one auto-assigned IDs to be hardcoded IDs. (And let's not forget the apps that instead of calling an API to get the user's home directory, program files, or windows directories, they hardcode "C:\Document and Settings", "C:\Program Files" and "C:\Windows", breaking installs on computers that didn't install Windows on C drive, use localized (non-English) Windows, or... use Windows Vista and above. It's why Vista+ have symlinks to C:\Users and various directories within. Or why UAC broke everything in Vista. Or why 64-bit Windows is complex because it has to rename/virtualize C:\Program Files to C:\Program Files (x86) at runtime.

Yes, Windows is great because it's backwards compatible. However, it also makes it extremely crufty with a lot of hacks having to stay in purely because some app needed it. (And I think WinSxS was a partial solution to that) Vista is proof - Microsoft tried a "new start" with it, and broke so many apps that it was Vista being blamed because they got rid of a lot of compatibility cruft.

Re:When will they accept Windows 8 as a failure?

[phantomfive]

You are wrong. There is nothing compiled for OSX before 2005 that still works on their most recent OS. The shift to 64 bit is further causing Apple to remove public APIs. Apple has demonstrated again and again they have no commitment to backwards compatibility, and there is nothing you can do as a programmer to avoid it.

Re:When will they accept Windows 8 as a failure?

[tehcyder]

That's because most programmers suck.

So what we need to do is make it illegal for the majority of people to become programmers. It should remain a tiny elite class, a bit like being a Catholic Cardinal, but with less sex.

MS Azure Failure ...

[fahrbot-bot]

...otherwise know as the Microsoft Blue Cloud of Death?

[ "Azure" is a shade of blue, for those that don't know,
and why MS would go with this kind of name, given their history with things "blue" is beyond me. ]

Re:When will they accept Windows 8 as a failure?

[kbg]

I don't think so. NASA makes almost bug free code with very stringent testing at a cost of $1000 per line of code I believe, so for example Windows 7 which has about 50 million lines of code, would only cost 50 billions, and given the profits of Microsoft that would only take two or three years of their profit.

Re:When will they accept Windows 8 as a failure?

[RaceProUK]

The main difference being NASA software runs on fixed hardware, but Windows has to support a near infinite variety. So you'd have to certify on an unbelievable number of systems to be sure. Also, even NASA software occasionally has bugs ;-)

Re:What a massive joke

[Anonymous Coward]

I don't think think the numbers are that skewed, actually, probably about a 70/30 split, maybe even 60/40. Unfortunately the morons typically hold substantially more power due to their title or caste. Speak the right dialect of Hindi? You can get away with anything, even if you're an idiot. In fact, you'll probably get promoted into management. Speak English as your native language? Your best work will be attributed to the guy who speaks the correct dialect, and everything else will be declared "average" if you're lucky.

Point it out and suddenly you're a 5 in the stack rank (and that's bad news).

So, the great engineers over time quit trying to fix things, and dumb down their work so they don't get stabbed in the back by their politicking H-1B coworkers, because at the end of the day we're here because we need a paycheck.

CA emails me when my cert is about to expire

[Beeftopia]

My certificate authority sends me nagging emails like 6 weeks before my certificate's about to expire. Microsoft's certificate authority group needs to create a database and automated emails when certificates get near expiration. Start emailing a bunch of folks. It's very simple. Probably most CA's have such a setup.