The Story Behind a Windows Security Patch Recall

bheer writes "Raymond Chen's blog has always been popular with Win32 developers and those interested in the odd bits of history that contribute to Windows' quirks. In a recent post, he talks about how an error he committed led to the recall of a Windows security patch."

Comments

An error he committed?

[drinkypoo]

he talks about how an error he committed led to the recall of a Windows security patch.

Okay, he made an error. Why the HELL wasn't it caught in QA? Microsoft wants us to believe that the reason that we have to wait for patches is that they are getting some kind of exhaustive QA. This patch and executable were specifically created to avoid problems with invalid shell extensions. Don't you think that given that fact the thing to do would be to test it with some invalid shell extensions?

This is the reason that Windows admins have to be so much more paranoid about patches than the rest of us. A Windows patch is highly likely to be a big pile of crap that causes your system to not work properly. I think we can all remember certain service packs that broke various versions of Windows NT pretty much completely...

If you can't have confidence that security patches will fix more than they break, how can you have sufficient confidence to even install that vendor's products, let alone count on them for mission-critical applications?

Re:An error he committed?

[bmajik]

I'm a software tester at Microsoft, although I'm not involved with the Windows team or the security process.

Just so we're clear:

Microsoft is not selling you products that have gone through exhaustive QA, nor are we issuing patches that have gone through exhaustive QA.

The key word here is "exhaustive".

You can imagine that as much as it costs a business when they get a hotfix from us that breaks them, it costs us _at least_ that much in real employee hours (dollars), not to mention the direct and indirect, monetary and non-monetary costs of having to admit that we screwed up a patch.

Software testing cannot tell you how good your product is, only in what ways it doesn't appear to be bad. Every release decision is a _decision_, and its based on necessarily incomplete data put together by imperfect humans with non-infinite time.

A release decision is a culmination of many nested risk/reward tradeoffs. Sometimes, that decision gets made incorrectly, or at least gets made in a way with known or even unknown downsides.

You'll notice that the patch was an interaction problem with an antique 3rd party product. From my time doing admin work on Solaris, IRIX, and Linux machines, I can tell you the big difference between this situation and "those" situations. I never _ran_ 3rd party software on Solaris, IRIX, or Linux (well, I ran 3rd party software on linux all the time, but i just expected it to break anytime i patched anything.. it was a mandatory recompile of any dependant libraries and applications).

I also think your glasses are a little rosy. There were some IRIX patches back in the day that you couldn't back out. Or that wrecked your XFS volumes. I think in every operating system there has been at least one instance of a patch / upgrade / new version that some user opted to back out, because it hurt them and their scenarios more than it helped.

I run very little non-Microsoft software on my windows machines and thus I rarely worry about patches from MS. If you're doing something weird, you need to be more risk averse. IIRC, Microsoft's official recommendation for businesses with critical systems is to install patches in a pre-production environment to ensure compatability with the specific intricacies of your business. You can choose to play fast and loose, but you should be aware that you're making a risk/reward tradeoff decision, based on incomplete data.

Just like we have to do.

Honesty

[florescent_beige]

This illustrates the kind of employee I like to have. One who can talk about his mistakes the same way he talks about anything else work-related.

Some years ago I myself made a rather expensive mistake which involved the design of an aircraft structure. The fellow I was working for at the time had one of those razor-blade intellects and I got called into his office for a chat. When he asked me what happened I had two choices, weasel or turkey. In engineering it's always possible to talk the complicated talk and hope to obfusticate your way out of a situation, but fortunately I said "I make a mistake." And you know what? That was exactly the answer he was looking for.

You see, the most important thing is not to be perfect, it's to be honest. That's what a boss, of which I am one now, wants.

If you have a boss that doesn't want that, better watch out for yourself.