The Story Behind a Windows Security Patch Recall

bheer writes "Raymond Chen's blog has always been popular with Win32 developers and those interested in the odd bits of history that contribute to Windows' quirks. In a recent post, he talks about how an error he committed led to the recall of a Windows security patch."

Comments

If this happened at Apple...

[Jadware]

Raymond Chen would be iFired, or at least told to iRTFM.

The Money Quote

[SixFactor]

You're about to be Slashdotted.
Seriously, it's good to get a glimpse of the interactions in the dev side of MS. It's astonishing that MS even allows this to happen at all. The March 07 Wired had a feature on Channel 9 that humanized the MS organization quite a bit, IMO. It's not just about chair-throwing, marketing hyperbole, and world domination after all... oh wait.

Fascinating

[wbean]

This is fascinating. The system for exiting a process is so complicated that a lot of implementations fail. In fact, it's so complicated that even Microsoft can't get it right. Sounds like an unbounded loop to me.

Re:Fascinating

[Timesprout]

Raymond has touched on the complexity of their software before and noted that oftentimes the complexity was not acually a product of the fuctionality but due to fixes, patches and additions to the code over time. To his credit he has in the past admitted that issues similar to this one were introduced because the core problem ie loading faulty shell extensions was not addressed directly for reasons of time/money/too scared to touch it/whatever and the hacks and workarounds only served to pointlessly bloat the complexity of the whole system. It's also worth noting that this complexity creep was not entirely due to MS. They had 10s of millions users with god knows how many applications which the MS dev teams struggled to support with backwards compatability etc. Raymond has admitted in the past that specific checks were put in the OS for certain applications to keep them functioning. Nice if you are a third party developer but just asking for trouble for your OS.

An error he committed?

[drinkypoo]

he talks about how an error he committed led to the recall of a Windows security patch.

Okay, he made an error. Why the HELL wasn't it caught in QA? Microsoft wants us to believe that the reason that we have to wait for patches is that they are getting some kind of exhaustive QA. This patch and executable were specifically created to avoid problems with invalid shell extensions. Don't you think that given that fact the thing to do would be to test it with some invalid shell extensions?

This is the reason that Windows admins have to be so much more paranoid about patches than the rest of us. A Windows patch is highly likely to be a big pile of crap that causes your system to not work properly. I think we can all remember certain service packs that broke various versions of Windows NT pretty much completely...

If you can't have confidence that security patches will fix more than they break, how can you have sufficient confidence to even install that vendor's products, let alone count on them for mission-critical applications?

Re:An error he committed?

[NickFitz]

As he points out in his response to the second comment on his blog post, internal testing can't possible cover every single third party shell extension on the planet. (Nor does he try to use that as an excuse for his screw-up.)

Re:An error he committed?

[bmajik]

I'm a software tester at Microsoft, although I'm not involved with the Windows team or the security process.

Just so we're clear:

Microsoft is not selling you products that have gone through exhaustive QA, nor are we issuing patches that have gone through exhaustive QA.

The key word here is "exhaustive".

You can imagine that as much as it costs a business when they get a hotfix from us that breaks them, it costs us _at least_ that much in real employee hours (dollars), not to mention the direct and indirect, monetary and non-monetary costs of having to admit that we screwed up a patch.

Software testing cannot tell you how good your product is, only in what ways it doesn't appear to be bad. Every release decision is a _decision_, and its based on necessarily incomplete data put together by imperfect humans with non-infinite time.

A release decision is a culmination of many nested risk/reward tradeoffs. Sometimes, that decision gets made incorrectly, or at least gets made in a way with known or even unknown downsides.

You'll notice that the patch was an interaction problem with an antique 3rd party product. From my time doing admin work on Solaris, IRIX, and Linux machines, I can tell you the big difference between this situation and "those" situations. I never _ran_ 3rd party software on Solaris, IRIX, or Linux (well, I ran 3rd party software on linux all the time, but i just expected it to break anytime i patched anything.. it was a mandatory recompile of any dependant libraries and applications).

I also think your glasses are a little rosy. There were some IRIX patches back in the day that you couldn't back out. Or that wrecked your XFS volumes. I think in every operating system there has been at least one instance of a patch / upgrade / new version that some user opted to back out, because it hurt them and their scenarios more than it helped.

I run very little non-Microsoft software on my windows machines and thus I rarely worry about patches from MS. If you're doing something weird, you need to be more risk averse. IIRC, Microsoft's official recommendation for businesses with critical systems is to install patches in a pre-production environment to ensure compatability with the specific intricacies of your business. You can choose to play fast and loose, but you should be aware that you're making a risk/reward tradeoff decision, based on incomplete data.

Just like we have to do.

Re:An error he committed?

[Blakey Rat]

And the manpower to run it all costs... how much?

Seriously, though, just putting all that equipment in one building would create a zeppelin-hangar-sized building. Finding any specific router or PCI modem would be near impossible. The logistical difficulties of your plan I think would be insurmountable, not even considering the manpower question.

The real point Raymond mentions is that if MS does tons of testing on all the hardware they have available, they get bad press for being slow to release patches. If not, they get bad press for having to recall buggy patches. It's a lose/lose situation for them.

Lesson

[Jeffrey Baker]

I think the lesson here is not that this guy should have been more careful about programming, it's that no amount of careful programming can overcome a stupid design. It's stupid that there are magical filenames in the form of UUIDs that cause Explorer to load and run arbitrary DLLs. You can't get around this stupidity with some kind of speculative watchdog thread that works with what sound to me like some seriously questionable heuristics.

They should have simply got rid of the magic naming system in favor of something explicit, such as a Shell Extension Interface that a shell extension must fully implement.

Honesty

[florescent_beige]

This illustrates the kind of employee I like to have. One who can talk about his mistakes the same way he talks about anything else work-related.

Some years ago I myself made a rather expensive mistake which involved the design of an aircraft structure. The fellow I was working for at the time had one of those razor-blade intellects and I got called into his office for a chat. When he asked me what happened I had two choices, weasel or turkey. In engineering it's always possible to talk the complicated talk and hope to obfusticate your way out of a situation, but fortunately I said "I make a mistake." And you know what? That was exactly the answer he was looking for.

You see, the most important thing is not to be perfect, it's to be honest. That's what a boss, of which I am one now, wants.

If you have a boss that doesn't want that, better watch out for yourself.

This one bit a client of mine...

[ktakki]

On the day after Patch Tuesday, January 2006, I got a somewhat frantic call from a client. She's a lawyer, had a filing deadline, but could not save a document in MS Word. That's not all that this patch broke: you couldn't open My Computer or My Documents on the desktop (though you could navigate to them by typing the path in the Start -> Run box), and IE wouldn't let you type just "www.[website].com" in IE's address bar. You had to prepend the "http://".

I verified that "Save" and "Save As..." were not working in Word. Word would just hang and only Task Mangler could shut it down. I carry the Sysinternals utilities on CD and USB key, so I rebooted and ran FILEMON, REGMON, and PROCEXP to see what was happening when I tried to save a doc in Word. Sure enough, Word would spawn verclsid.exe as a child process and then hang.

I googled "verclsid" and "Explorer", got nothing on the web and about a dozen Usenet posts from people having the same problem. I played a hunch and renamed verclsid.exe to verclsid.exX. I do that when I'm manually hunting malware that leaves .exe and .dll files that are named just like Windows system files. Keeps my foot bullet-free.

Problem solved. When the patch for the patch came out, a working verclsid.exe was dropped in %system% and I deleted the .exX.

Oh, and the buggy third party shell extension came with a very common HP DeskJet printer. As for Google, the next day I googled "verclsid": there were hundreds of web results and Usenet hits. The day after, tens of thousands. This one bit a lot of people in the ass.

k.

A bit more background info

[Marton]

This pretty much rendered Windows useless (explorer, file open / save dialogs and the IE7 addressbar were not working) if you had software installed for HP cameras, HP scanners, or any HP DeskJet printer that included a card reader.

Courtesy of JSI FAQ:

You experience one or more of the following strange behaviors:

- You are unable to open special folders, like My Documents or My Pictures.

- Some 3rd party applications hang when accessing My Documents.

- Office files won't open in Microsoft Office if they are stored in My Documents.

- Entering an address into Internet Explorer's address bar does nothing.

- The Send TO context menu has no effect.

- The plus (+) sign on a folder in Windows Explorer does nothing.

- Opening a file via an applications File / Open menu causes the application to hang.

This behavior is caused by a new VERCLSID.EXE binary, which validates shell extensions before Explorer.exe, the Windows Shell, can use them. VERCLSID.EXE is installed by the MS06-015 (908531) security update.

The following 3rd party applications cause VERCLSID.EXE to hang:

Hewlett-Packard's Share-to-Web Namespace Daemon ("%ProgramFiles%\hewlett-packard\hp share-to-web\Hpgs2wnd.exe), auto-started from the Registry Run key and the Startup menu, which ships with:

                HP PhotoSmart software
                Any HP DeskJet printer that includes a card reader
                HP Scanners
                Some HP CD-DVD RWs
                HP Cameras

Sunbelt Kerio Personal Firewall which has a feature that prompts when Explorer launches VERCLSID.EXE, but you can configure it not to prompt.

To workaround this behavior, add the HP shell extension to the VERCLSID.EXE white list:

1. Open a CMD.EXE window.

2. Type the following command and press Enter:

REG ADD "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Sh ell Extensions\Cached" /V "{A4DF5659-0801-4A60-9607-1C48695EFDA9} {000214E6-0000-0000-C000-000000000046} 0x401" /T REG_DWORD /F /D 1

3. Shutdown and restart your computer.

NOTE: If you find other COM controls or shell extensions that cause this behavior, you can add them to the white list.

Re:A bit more background info

[jZnat]

You see, it's this sort of advice that is preventing the year of Windows on the desktop. Once you say, "open a command line", you've failed as an operating system. ;p