Slashdot Mirror
Patching Paranoia - How Fast Do You Patch?
selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."
I wait until I get feedbacks from sites such as The Register to make sure that the patch doesn't break anything.
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
Constant re-booting seems to be an exclusive MS-phenomenon. Installing patches on Linux only requires a restart of the affected services unless a kernel upgrade is involed - and even this can be worked around in some cases.
You will reboot less when patching a Linux machine. Guaranteed.
Depends on the patch....security patches get applied, ASAP. If it's a patch fixing something that is not used much or that we don't have an issue with, it gets applied when the next Maintenence Level (IBM speak for Service Pack) comes out. Luckily, AIX does not have very many security issues. That covers the OS. Our application we are way behind in patches and we only can pacth after hours. Since we're in the middle of conversions, there are processes constantly running on the server and we also cannot patch when we have reps from the vendor in working on the conversion because the expect thigns to be the same while they are there and patches can really mess them up. So, needless to say, we are WAY behind on app patches but we are reasonably caught up with OS level patches.
Gorkman
How do you know? What do you do when a Critical Update does in fact break something (as a recent Critical Update broke Citrix)?
sPh
sPh
There's one big gotcha here: notebooks. Your users are firewalled at work but once they get home they're probably wide open. Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.
We don't EVER install a patch on a production machine without testing it first on some less crucial machine.
Any machine that accepts connections from outside the firewall (SMTP, IMAPS, HTTPS, & SSH are all we take, and only to specific machines) gets any remotely exploitable bug patched ASAP. Typically I will run the patch on a non-production machine for 24 hours to make sure it's reasonably stable, then patch.
Once the patch has proved itself in production on the remotely accessible machines, say for a week or so, we load it everywhere else.
Stuff that's not remotely exploitable is dealt with on a more relaxed schedule, generally at least two weeks after the patch has begun testing on a non-production machine. Sometimes longer.
We also always test our backup strategies before loading MS or HP patches, since sometimes they completely trash the system.
HP-UX patches come out months or years after the exploit, Microsoft patches come out weeks or months late, DEC patches used to come out within days (Oh, how we miss ye DEC) and BSD and linux come out within hours, usually.
If it's windows patch early, and patch often. If anyone asks why you rebooted a box, lie about it and say "It crashed." That's one everyone will believe.
Less Talk, More Beer.
I'm the network admin/windows/active directory guy for a healthcare company. We run multiple SUS servers, several for desktops, and one for servers. Our procedure is, when a patch is released, that day I.T. downloads and installs it on our desktops and test servers. If it's successful, it gets approved on our desktop SUS servers. If those work OK, the next day it gets approved for our severs. So far we've had no problems with that process.
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.
I just wish we had 1/3 of the balls of that company and that fucking up with the company computer was seen as destructive and damaging as it actually is.
The countless whining we get over passwords ("My boss says I dont hafta have one.."), applying updates to desktops(!), removing shit like comet cursor, and the people that toss laptops around and then bitch that they don't have the right laptop after they've broken it.
I'd love to see 2 or 3 people in particular have to sit down in front of the CFO and be told:
1) The computer you broke won't be replaced until you pay for the old one.
2) If you can write a check today, we won't dock your paycheck, but if we do, we'll spread the payment over at least 4 paychecks.
3) Any work you don't get done due to no computer will be considered against you in your next performance review and may be considered grounds for dismissal.
There's lots of reasons not to do it that way, but geeze, if there were real consequences (financially especially) for being a fuckup with computers, I think the users would toe a much tighter line.