Slashdot Mirror
Patching Paranoia - How Fast Do You Patch?
selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."
... I am to post to a new Slashdot article
I wait until I get feedbacks from sites such as The Register to make sure that the patch doesn't break anything.
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
Or common sense?
I run a SUS server for my organization, and it checks for patches nightly. The next day, my servers and workstations are patched.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Constant re-booting seems to be an exclusive MS-phenomenon. Installing patches on Linux only requires a restart of the affected services unless a kernel upgrade is involed - and even this can be worked around in some cases.
You will reboot less when patching a Linux machine. Guaranteed.
Have you guys looked at MS SUS 1.0 to automatically deliver critical updates? It's kinda lame--not the greatest management capabilities--but it does work. I have a company similar to Thrive & use it to deliver patches to end-user desktops at several clients.
Depends on the patch....security patches get applied, ASAP. If it's a patch fixing something that is not used much or that we don't have an issue with, it gets applied when the next Maintenence Level (IBM speak for Service Pack) comes out. Luckily, AIX does not have very many security issues. That covers the OS. Our application we are way behind in patches and we only can pacth after hours. Since we're in the middle of conversions, there are processes constantly running on the server and we also cannot patch when we have reps from the vendor in working on the conversion because the expect thigns to be the same while they are there and patches can really mess them up. So, needless to say, we are WAY behind on app patches but we are reasonably caught up with OS level patches.
Gorkman
Middle of the day reboots are normal, so we patch whenever we want.
sPh
There's one big gotcha here: notebooks. Your users are firewalled at work but once they get home they're probably wide open. Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.
I tend to follow at least the following criteria when deploying patches:
1- If the patch is a Microsoft patch, I deploy it immediately, regardless of severity, because Microsoft has repeatedly lied about the severity of security flaws that were actually quite critical.
2- If the patch is for a very theoretical problem, such as many of the recent OpenSSL patches, I tend to let it wait for the next big update. Good examples are those problems where key-breaking time is reduced to only 50 years or so on a $10,000,000,000 budget.
3- Patches that fix vulnerabilites that are only a problem in stupid configurations (Such as recent OpenSSH problems.) get ignored until the updates have been tested.
4- Patches from Sun go out immediately, because they seem to take so long that the exploits for bugs have been integrated into script-kiddie toolkits.
We don't EVER install a patch on a production machine without testing it first on some less crucial machine.
Any machine that accepts connections from outside the firewall (SMTP, IMAPS, HTTPS, & SSH are all we take, and only to specific machines) gets any remotely exploitable bug patched ASAP. Typically I will run the patch on a non-production machine for 24 hours to make sure it's reasonably stable, then patch.
Once the patch has proved itself in production on the remotely accessible machines, say for a week or so, we load it everywhere else.
Stuff that's not remotely exploitable is dealt with on a more relaxed schedule, generally at least two weeks after the patch has begun testing on a non-production machine. Sometimes longer.
We also always test our backup strategies before loading MS or HP patches, since sometimes they completely trash the system.
HP-UX patches come out months or years after the exploit, Microsoft patches come out weeks or months late, DEC patches used to come out within days (Oh, how we miss ye DEC) and BSD and linux come out within hours, usually.
If it's windows patch early, and patch often. If anyone asks why you rebooted a box, lie about it and say "It crashed." That's one everyone will believe.
Less Talk, More Beer.
..broke all KINDS of things. On my home machine, I now get 5 USB power errors that I didn't get with 10.2.6, as well as unexplained freezes & crashes.
I reverted to 10.2.6 and all was well once again.
And this was 10.2.8 redeux - remember the first time that it came out, machines were breaking all over the place. (ethernet issues, IDE oddities..)
I just wish we had 1/3 of the balls of that company and that fucking up with the company computer was seen as destructive and damaging as it actually is.
The countless whining we get over passwords ("My boss says I dont hafta have one.."), applying updates to desktops(!), removing shit like comet cursor, and the people that toss laptops around and then bitch that they don't have the right laptop after they've broken it.
I'd love to see 2 or 3 people in particular have to sit down in front of the CFO and be told:
1) The computer you broke won't be replaced until you pay for the old one.
2) If you can write a check today, we won't dock your paycheck, but if we do, we'll spread the payment over at least 4 paychecks.
3) Any work you don't get done due to no computer will be considered against you in your next performance review and may be considered grounds for dismissal.
There's lots of reasons not to do it that way, but geeze, if there were real consequences (financially especially) for being a fuckup with computers, I think the users would toe a much tighter line.