Time to End Microsoft's Patch Tuesday?

buzzardsbay writes "Techtarget's resident security curmudgeon, Dennis Fisher, is calling for an end to Microsoft's monthly security patching cycle. Fisher points out that 'a hacker only needs one unpatched system, one little crack in the fence in order to launch a major attack on a given network. The sheer volume of the patches Microsoft releases each month makes it quite difficult for even the most conscientious IT department to get every patch out to all of the affected systems in a reasonable amount of time.'"

Comments

I have always wondered...

[AxemRed]

Why don't they just release patches as the make them? Is there a specific reason that they hold them all until "patch Tuesday?"

Re:I have always wondered...

[kcurtis]

It allows IT departments to specifically set aside 1 (or more) days a month on a regular schedule to test the updates before rolling them out to the client computers.

If the updates come out on a random schedule, as done before, you cannot plan ahead for the testing required to ensure the updates don't break functionality.

Re:I have always wondered...

[Matt Perry]

It allows IT departments to specifically set aside 1 (or more) days a month on a regular schedule to test the updates before rolling them out to the client computers.

If the updates come out on a random schedule, as done before, you cannot plan ahead for the testing required to ensure the updates don't break functionality.

Nonsense. Companies are free to test and upgrade on a given day no matter when updates come out. I test patches and update my Linux servers once a month even though patches for said machines may come out at any point in time between my patch days. I make exceptions to this only for patches that we deem critical enough to apply outside of our schedule.

Re:I have always wondered...

[LurkerXXX]

You always wondered? You must be fairly new to IT. MS switched to that format well within the past 10 years. I think it was around 5 years ago. Before that they released them as each was finished.

As for why they do them that way now, their large corporate customers asked them to. In large corporate settings there are often lots and lots of in-house-developed applications the company runs. Each time a new patch comes out, the IT dept must go through a lengthy (sometimes several weeks) process of testing the new patch, on test beds of the various models/configurations of computers the company uses, to make sure it doesn't break any of those apps, or any other purchased applications. They often run into many bugs/conflicts that MS doesn't in their testing.

If MS comes out with a patch, the company starts testing it out, then 3 days later MS comes out with another patch, the big corp now has multiple cycles of testing trying to go on at the same time, using up tons of IT resources, backing things up in the pipeline. If their testing cycle is 2 weeks, and MS releases 6 patches during those two weeks, the pipeline is now filled up with 12 weeks worth of throughput. Not fun.

If, on the other hand, MS releases on a regularly scheduled day each month, the company can easily run their test suite just a single time, freeing up IT resources, and also letting them plan for the patches/testing, rather than being surprised and having to pull folks off of other projects to work on testing if MS suddenly goes on a streak of releasing several patches in a row.

Re:I have always wondered...

[edwdig]

Does Windows gracefully handle the situation where a DLL which is currently in use is replaced, or will I wind up with applications calling two different versions of the DLL depending on when they started?

The reason Windows updates require reboots is because open files cannot be replaced. So if a DLL is in use at the time of update, it won't actually be installed until you reboot.

Unix systems, otoh, have decided that the name of a file (the thing the user has control over) is not what actually ids a file, but instead the location on disk is the id. Hence why Unix updates don't require reboots and instead result in the problems you've mentioned.

I've always wondered how someone could consider the Unix design a good idea. Two different programs can open what they think is the same file, yet get completely different results. And yet some people don't seem to get why this is a really bad thing for shared libraries (or even files in general).

Re:I have always wondered...

[Tanktalus]

I still love the ability to replace in-use libraries. The only problems that ever crop up are when you dynamically load another library, and that library disappears (Windows doesn't help here, either), or its API changes (although usually that results in a new library name, so you still get the old one). If you still have a library loaded when it gets deleted, you maintain a filehandle to it so its disk space is not reclaimed or reused. Shut down all applications still loading the old library, and then the disk space gets reclaimed.

I've updated X.org at least a couple times since the last time I restarted my X server. So I have a bunch of old libraries still sitting on my disk with no way to refer to them (well, there are ways to get them back involving funky lsof/proc tricks, but let's not go there). Nothing will overwrite them. But, when I feel I have the time, I can shut down all my X apps, restart my X server, and free up all that space. But I don't need to take down mysql, apache, or anything not X-based to do so.

I don't get how anyone could consider this a bad idea. The only times it falls over is when people don't follow convention (change your library number when changing APIs!), or in cases that Windows will fall over, too (dynamically loading libraries that don't exist anymore - although that usually doesn't crash as hopefully most people catch the error return and handle it). Otherwise, it maximises the uptime of your server, so that you only need to restart programs that actually use your library when you want to.

(PS - thanks for this thread - it answers a question my wife posed - why her windows machine rebooted overnight when she was in the middle of sorting digital photos to send to be printed, and there was no power outage.)

Re:I have always wondered...

[Kijori]

When Microsoft releases a patch for an exploit, it's immediately known that computers are wide open to this attack. Malicious hackers - virus writers and the like included - can reverse engineer the patch to find out what vulnerability is being patched exactly, and know that, since your organization doesn't patch until such-and-such day, you're wide open to attacks. "Exploit Wednesday", the day after patch Tuesday, is a testament to the importance of Microsoft's patches in the development of exploits. Companies can't afford to gear up for patches every day, but can't afford to risk the ramifications of not applying a patch immediately either. Patch Tuesday gets them out of this catch-22.

Re:I have always wondered...

[pe1chl]

Unlike the Unix mechanism, where the library is replaced and you would need to voluntary restart your application to make it use the new library, there is no easy way to update a DLL in Windows after it has decided a reboot is required.

Windows update will try to replace each file, and when it succeeds everything is fine. When not, it will put the file on disk under a different name, add a "rename" operation to a list, and continues with the next file. At the end, when the list is not empty, it requests a reboot. At reboot, the list is processed (the new files renamed over the old ones), and the list emptied.
But merely stopping an application and closing the file that was in use will not make it rename that file and remove it from the list. You will need to reboot.

The Real Reason for Patch Day

[Gary W. Longsine]

Dennis Fisher fails to grok. Patch Day was created because Microsoft was getting hammered by the poor press which resulted from releasing many patches in one month. Patch Day, as much as it sucks, is probably here to stay.

I call Bullshit on the Red Bull

[The Media Mechanic]

"Known in some circles as Black Tuesday, the second Tuesday of each month in the last few years has become a kind of national day of mourning in the IT industry, as admins call all hands on deck and load up on pizza and Red Bull for the long night ahead."

I call bullshit on this anecdotal bit of trivia. Is the author of the article actually suggesting that some companies rush to test the new Winblows patches all through the night on Tuesday so that the patches are ready to deploy on Wednesday ? This sounds like a fresh steaming load of bullshit... what places actually force their employees to work ridiculous hours like this just due to an arbitrary vendor schedule! I would not work at such a place, regardless of the amount of free pizza or Redbull available.

My point is that this bit of exaggeration in the article has no basis in fact and should be supported by quotes from someone who actually enforces this policy at their IT department.

Re:I call Bullshit on the Red Bull

[Zontar_Thing_From_Ve]

Is the author of the article actually suggesting that some companies rush to test the new Winblows patches all through the night on Tuesday so that the patches are ready to deploy on Wednesday ? This sounds like a fresh steaming load of bullshit...

You may be right. My previous job was with a company that did a lot of VAR stuff, including various email systems. It didn't matter to us what you wanted - Notes, Exchange, Unix, anti-virus, anti-spam - we could sell you whatever combinations you wanted. I didn't work with Exchange, but the Exchange guys told me that in the past they used to rush out and patch systems with every "critical" Microsoft patch release and then they applied some patch that totally broke Exchange. The patch had nothing to do with Exchange, but it broke it. It took hours to fix the broken servers. After that fiasco, we regarded all Microsoft patches as suspect and we had a group in another state that one of their jobs was to test new patches on Exchange servers and see if Exchange still worked. It didn't matter to us how "critical" Microsoft considered a patch. We didn't patch any of Exchange servers until our test group gave the OK, which was usually a month later.

Re:Volume of patches won't get better

[gad_zuki!]

Patch day was started because administrators didnt want random patches being pushed out at random times. Its supposed to help the process by giving people a schedule, especially for people who arent using SUS.

The real question is when are they going to patch the patch system. The 100% CPU svchost bug is killing me and KB916089 (and its predecessor) doesnt do squat.

My Thoughts

[KenshoDude]

I am the Sys Admin for ensuring that our roughly 1800 desktops and notebooks get updated with the latest updates. Microsoft's strategy is the very least of my concerns. The patches show up on WSUS the Wednesday morning after they are released. I read up on them, noting any "caveats" in the KB articles and inform our help desk if I find anything signficant. Then, I set my approvals and decline any superseded updates. The clients check in and install the updates over night. I am not sure where all this talk about long nights with Red Bull and whatever come into play. If we have mission critical systems, we withold approval for that group for a week or so until we are confident that there are no undisclosed "caveats." Super simple.

I like having a regular schedule for updates. But I wouldn't mind a little more frequency. Why not the first and third tuesday of every month? Sounds reasonable to me.

Now if were only that easy for all the other software vendors out there like Adobe (Acrobat / Flash), Sun (Java), and so on. Where are their enterprise patch management solutions? Why can't I configure my Java clients to check into to one of my servers to automatically apply security updates? Instead I have to spend more money on a 3rd party patch management solution. And I haven't found one yet that is as reliable and simple as WSUS.

Re:End Patch Tuesday

[businessnerd]

Except for the fact that Linux also requires patching. Every other day I have a little star on my desktop notifying me of updates to various libraries, applications, and yes the kernel itself. Mac's have patches too. This is not necessarily a Windows vs. , this is about what the best way of releasing patches is. It's an Incremental vs. Bulk release debate. MS chose the bulk method. Is that a good decision? Maybe, maybe not. Regardless of the OS, patching is always required. No piece of software is bulletproof.

That's the Problem

[bill_mcgonigle]

It allows IT departments to specifically set aside 1 (or more) days a month on a regular schedule to test the updates before rolling them out to the client computers.

Your comment is accurate, and gets to the heart of the problem. The current system minimizes cost, at the expense of security.

The pundit would rather companies get more staff, do rolling testing, etc., whatever it takes - to maximize security.

Now, as a non-user of Microsoft products and a victim of attacks by unpatched machines, some of them corporate, it's clear that the current strategy just shifts the costs off of the companies and onto me. If it just crashed their networks I couldn't care less. But it's more than that.

So I need to side with the proposal - the users need to improve their security. They can do this by having rolling patches from Microsoft or picking a more secure product to use. I don't care how they do it, but they need to stop expecting me to pay for their poor performance.

Unfortunately, liability is poorly defined in this realm, otherwise I could theoretically sue for damages, and their insurance company would make sure they were in good shape or charge them through the roof for being in bad shape.