Patching Paranoia - How Fast Do You Patch?

selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."

Comments

Better safe than sorry?

[Soulfader]

After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree.

To many people, however, that means that you wait to install a patch until it has been tested. It is going to depend on your environment and needs; there is no one correct answer on this one.

Re:Better safe than sorry?

[Chazmyrr]

It has nothing to do with network environment. It has to do with the fact that the operating system and software are rarely kept in a default install configuration. Some of these patches can go south in bizarre ways when the configuration isn't what the patch expected. You simply have no idea what is going to happen when you install the patch. That may be fine in a mom and pop operation. It is clearly not acceptable for an enterprise.

In my company, a bad patch can mean 10,000 reps sitting around with nothing to do and impact hundreds of thousands of customers. Do you want to be the person who pushes the patch without testing it first? Actually, let me rephrase that. Do you want to be the person looking for new employment after the bad patch costs the company millions of dollars?

Pretty much immediately.

[Godeke]

I first patch my local systems and try them for a few hours as I run similar configurations to my clients for development. If I survive the patch, I patch the development systems at my client sites. If those remain stable for a period of time, I patch production clients, and then finally production servers.

If at any point a glitch appears, I stop at that point, minimizing damage. Usually that means I have a glitch locally and my clients would never know that there was a glitchy patch unless I tell them. Pretty much a similar approach that a big company would take (patch the test LAN) except I am the test LAN.

Re:Paraniod?

[sphealey]

I run a SUS server for my organization, and it checks for patches nightly. The next day, my servers and workstations are patched.

Serious question: what do you do when (a) the patch breaks {may or may not cause Windows to become unusable} (b) the patch breaks critical applications?

How do you know? What do you do when a Critical Update does in fact break something (as a recent Critical Update broke Citrix)?

sPh

Re:Quick fix at the firewall

[easyfrag]

For a lot of these advisories, you can plug the hole at the firewall, or maybe the mail server.

There's one big gotcha here: notebooks. Your users are firewalled at work but once they get home they're probably wide open. Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.

Depends on the patch

[Medievalist]

We don't EVER install a patch on a production machine without testing it first on some less crucial machine.

Any machine that accepts connections from outside the firewall (SMTP, IMAPS, HTTPS, & SSH are all we take, and only to specific machines) gets any remotely exploitable bug patched ASAP. Typically I will run the patch on a non-production machine for 24 hours to make sure it's reasonably stable, then patch.

Once the patch has proved itself in production on the remotely accessible machines, say for a week or so, we load it everywhere else.

Stuff that's not remotely exploitable is dealt with on a more relaxed schedule, generally at least two weeks after the patch has begun testing on a non-production machine. Sometimes longer.

We also always test our backup strategies before loading MS or HP patches, since sometimes they completely trash the system.

HP-UX patches come out months or years after the exploit, Microsoft patches come out weeks or months late, DEC patches used to come out within days (Oh, how we miss ye DEC) and BSD and linux come out within hours, usually.

patch?

[dakkon1024]

This is one of those grand broad questions with no answer. If you have an entire redundant system to test with, you can patch that instant, test it, and roll it out. But then again the new patch might fail in some way you never expected. If you are talking a 100+ servers, then you might need to test a group, before you patch your core group. Then there are the questions you need to ask. Is someone likely to break in? Did it work for someone else? Is it a MS product? What do your clients want? When will have the least effect on service? Did the patch come to you via email? You know, the important questions. To answer the question though, we patch, after we know the patch works.

Re:Paraniod?

[Anonymous Coward]

I also run an SUS server at my organization.

SUS allows you to choose which patches you want installed on your client. We have the patch server check patches nightly, and install those on a testbed IT machine we have set apart (it's actually the machine I use).

If I notice any problems I try to figure out which update did it and I just don't approve that update, if I can't figure it out in time I don't aprove any updates until I find out what happened.

Turn around time ends up being 48-72 hours on non critical patches which is the time it takes me to evaluate for problems, but I never have to touch any of the client machines.

Why Windows has a Higher Cost of Ownership

[darkonc]

Not only do you have cheaper acquisition costs, but things like this don't get in your way. You can patch 'live' and rarely need to reboot.

I've got roommates who've moved to the Linux desktop. I usually do the upgrades from my desktop. The only reason why I tell them that I'm doing upgrades is that it's annoying if they shut down the system in the middle of an RPM Install. (one dual boots to Windows so he's more likely to reboot, the other runs solely on Linux he really only powers off if he's heading out. I think I've installed one or two kernel upgrades in the last year (which require reboots to enable), but since my roommates reboot so often, I can just wait for their next reboot.

There's also much less need to do testing with Linux patches... You generally know EXACTLY what subsystems are being affected by a patch, so if it's not a critical component, you can often install blindly. Even if it is a critical component, the patches are often well defined and if you have any questions you can read the source code.

The problems with Windows is that it's the large-scale version of spaghetti code. The relationship between various pieces are ill-defined and numerous. Patches spider into various areas and it seems like nobody (even at Microsoft) knows precisely what a patch fixes (or what it breaks).

This doesn't just apply to desktops. I'm in the middle of putting together scripts to enable controlled push of patches to a large number of varied servers. In truth, the hardes part is going to be figuring out which patches go to which boxes -- not figuring out if the patch is going to break things.

Yep. I'm spoiled. Linux makes life both easy and cheap.

Re:MS

[Kobold Curry Chef]

Most of the patches to Apple OS X also require a reboot. Even for patches to things like iCal, iTunes, and what not. It's one of the more disappointing things about OS X. You'd think that a BSD-based OS wouldn't require so many reboots. Maybe they just wanted to carry on the old Mac tradition of "you have just touched your computer; do you wish to reboot?"

That being said, there are FAR fewer patches to install on OS X than on Windows.

Re:I wait until...

[dthable]

There's really no logic behind why their patches do some of the things they do.

There has to be...the computer is just a simple machine following the instructions. Things would be safe, secure and stable if we all went back to being happy with command lines and single running tasks.

Let Your Customer Decide

[reallocate]

Once upon a time, I worked at a large content organization with the usual large IT infrastructure, supported by a single large firm. Per the requirements of the support contract, these guys were compelled to down the system and install patches as soon as they got their hands on the code. No-notice outages eere the rule. Managers, customers and employees pitched fits until someone finally woke up and explained that the support vendor would be in violation of contract if he didn't move that fast.

So, we changed the contract. Unscheduled downtime projected to last more than 30 minutes required getting permission from several designated management types. Any one of those managers could postpone the maintenance.

This worked because the support contractor always made sure that those designated managers understood the implications of delaying the maintenance.

Re:Paraniod?

[Anonymous Coward]

That idea works, except for one major oversight. Your IT guys are rarely doing the same tasks or using the same applications as your core business users are. Joe Bob doing data entry doesn't care if the patch breaks VPN or SSH, but if it hoses the one ActiveX control he uses to input data, he's going to scream. When data entry screams, that's the sound of money running out the door.

Your desktop test cases should include more than IT if you're going to be doing rapid deploys of patches like this. If you don't want to take the time to do that, please let me know what company you work for so that I can make sure I don't use you guys to manage my health care.

Scientific method, man...it's not just for researchers anymore.
jb

Re:I wait until...

[jon3k]

Its not moot to the thousands of people who couldn't get back online. We agree and disagree at the same time. For you to say "they'll just call their tech friend" is a little absurd, or spending an hour on the phone with microsoft to fix THEIR probleme. Are they going to reimburse me for that time? I do agree that people need more training before using a computer. I believe using a public system, like the internet, should require licensing. Just look at these poor families who's children were using Kazaa, and will now be sued into a homeless shelter, its sad. Lets face it, Microsoft made billions by putting idiots on the internet.

Re:I wait until...

[crawling_chaos]

SCSI RAID arrays don't plug into ATA controllers. Microsoft's installer thought my IBM controller was a generic Adaptec and installed the wrong drivers. As soon as 2k got rid of the BIOS and moved to its internal routines it went blotto. Then the recovery console refused to load the old drivers or the latest one downloaded from IBM. I ended up doing a bare metal recovery of the system disk. I think there might have been another way, but I'm real confident in my backups, so it was the path of least resistance. If we were just a little bigger, we could afford a second staging server to test patches on before rolling them out, but we're not big enough to afford two identical servers.

Needless to say, we stayed on SP2 until SP4 was out for a few months. I now basically dread any major upgrades of that server.