Slashdot Mirror
Patching Paranoia - How Fast Do You Patch?
selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."
Or common sense?
I run a SUS server for my organization, and it checks for patches nightly. The next day, my servers and workstations are patched.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Have you guys looked at MS SUS 1.0 to automatically deliver critical updates? It's kinda lame--not the greatest management capabilities--but it does work. I have a company similar to Thrive & use it to deliver patches to end-user desktops at several clients.
My memory is hazy, but last I remember a patch breaking something was about 4 years ago for me. I think it was with the MDAC patch.
Who else has had problems on patches lately?
The Department of Defense has no specific timeline for patches to be put in place, mainly because each team out here is free to do this as it wishes, when it wishes. This leads to disorganization and chaos of a level hitherto unimagined except on networks run by the most rank amateurs imaginable -- which may well be our status.
:(
In any case, my office has a bi-weekly reboot period set aside wherein they apply critical patches. Since this is scheduled downtime, our customers have already agreed by way of an SLA (service level agreement) that services won't be up during a brief window every other Friday. At least, that's when our guys are supposed to add patches -- it's mostly at the discretion of the admin on duty and how late he or she is in getting out for their weekend.
Past that there's no allowed downtime except when servers crash or when the Quarterly Outage rolls around. As such, patching is infrequent and often incomplete. It is distressing in the extreme.
I've pushed for (and received) tools to automatically download patches from Microsoft, and have other tools on hand to push these out to servers, but thanks to the Windows architecture I can't simply stop and restart services to make sure the patches take effect. Reboots are called for, and because that necessitates Downtime (capitalized most intentionally), it is verboten.
Things changed a bit when Nimda and Welchia hit, mainly because all of our suborganizations were busy scrambling like hell to uninfect themselves. My group, a rather high component in DoD, did not get hit by the worms -- our firewalls were properly configured and didn't allow random incoming RPC. (Though having seen how many orgs *did* get hit.. well let's just say there's a bright, bright future for college graduates with no real world experience, hmm? All you have to do is qualify for a security clearance!)
Anyway, we wound up patching in very short order in that specific case, but only because of immediate impact. If the writer of Nimda had half a brain and had used his exploit to write a very quietly installed trojan horse instead of a stupid reboot script he would have had control of hundreds of systems at the Pentagon. Lucky for us he was busy making a statement.
Patching does not happen nearly fast enough to suit me.
[Posted anonymously. I don't mind losing my job -- our contract's over in forty five days -- but I do mind federal prison.]
My USB Ports mysteriously stopped working immediately following a patch I applied to Windows XP about a month ago. Then I installed Steam, which basically destroyed my Windows installation (choppy sound, extended periods of no system response, etc..) so I reformatted the partition and reinstalled XP with all the patches..this time my USB ports kept working with the patches.
Just goes to show how touch-n-go a windows patch can be..sometimes it borks your system, sometimes it doesn't. There's really no logic behind why their patches do some of the things they do.
Maybe you should get your clients to run servers that don't require a reboot for most application patches.
I tend to follow at least the following criteria when deploying patches:
1- If the patch is a Microsoft patch, I deploy it immediately, regardless of severity, because Microsoft has repeatedly lied about the severity of security flaws that were actually quite critical.
2- If the patch is for a very theoretical problem, such as many of the recent OpenSSL patches, I tend to let it wait for the next big update. Good examples are those problems where key-breaking time is reduced to only 50 years or so on a $10,000,000,000 budget.
3- Patches that fix vulnerabilites that are only a problem in stupid configurations (Such as recent OpenSSH problems.) get ignored until the updates have been tested.
4- Patches from Sun go out immediately, because they seem to take so long that the exploits for bugs have been integrated into script-kiddie toolkits.
We have constant problems with patches where I work because Hpaq/Sun seem to think that the versions of certain software they ship with Solaris/Tru64 are sacrosanct.
Every time we patch our primary DNS server (on an E-250) Sun's patch stomps on our custom build of BIND. Similarly, HPaqs patch kits won't install properly if they involve any patches for sendmail because we got tired of waiting for patches for 8.9.3 (even under 5.1A they stay with 8.9.3!) while we prefer to run our own build of 8.12.10. HPaq is also bad about making security patches depend on their version of the software unnecessarily. As a f'rinstance, I recently installed Aggregate Patch Kit 5 for Tru64 5.1A. It included about a half-dozen patches to fix weaknesses in the init scripts. The patches for the init scripts REFUSED to install until I downgraded sendmail to 8.9.3 configured as it was during the system installation! After the patches were installed, I had to re-upgrade sendmail to our preferred version. To the best of my ability to determine there was absolutely NO reason for those patches to depend on sendmail being at v 8.9.3.
utter rubbish
It's a side-effect of the DOS legacy that still hangs over Win2000/XP. Unix separates files and inodes, so you can delete a file and replace it with a new one whilst the existing services are still using it, then restart the services to pick up the update. Windows has no such split, which means if a file is 'in use' you can't delete/overwrite it - this is what requires a reboot.
They could have fixed this in NTFS but chose not to, presumably to keep compatibility with DOS. TBH it's about time they sorted it out.
Of course taking the side of rebooting no matter what, due to poor Windoze OS memory management rebooting workstations and servers is usually a good idea anyway. Starting with Windoze 2000 they started improving memory management but I know from Windoze 95 through 98 and ME and Windoze NT 4.0 are all notoriously poor memory managers. So the reboots will probably do some short term good in terms of system utilization...
There's one big gotcha here: notebooks. Your users are firewalled at work but once they get home they're probably wide open. Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.
A company I did some work for (re: I was a contract monkey..yes, I admit it) had a policy that plugging in a company laptop to your home network constituted grounds for firing.
Yup. They were that strict. It wasn't a technology company, (so the "brass" were a bit... over the top) and they'd been bitten hard by folks bringing infected-at-home company notebooks back into the environment, so I can understand some paranoia, but sheesh...
My company writes enterprise software, albeit badly. The QA process I feel could be much better, but at least it gives a support rep like me a job.
Twice a month, we release patches which fix any number of bugs we may have found since the original release of the software. About 1/3 of the patches we release introduce NEW bugs that weren't there before the patch! These new bugs can easily and often cripple important parts of the software.
I knew a 4 month stretch where this happened on every release for those 4 months, 8 patches in a row!
Most of our customers update every few months, and they keep an eye on our website, and the public customer email lists constantly throw out emails which the bleeding edge leaders complain of problems introduced on new builds (which they have every right to complain about).
Now I can't speak for any other company, including Microsoft, but sometimes upgrading right away when you aren't really currently experiencing an active problem is worse than not upgrading at all.
"All great wisdom is contained in .signature files"
I just wish we had 1/3 of the balls of that company and that fucking up with the company computer was seen as destructive and damaging as it actually is.
The countless whining we get over passwords ("My boss says I dont hafta have one.."), applying updates to desktops(!), removing shit like comet cursor, and the people that toss laptops around and then bitch that they don't have the right laptop after they've broken it.
I'd love to see 2 or 3 people in particular have to sit down in front of the CFO and be told:
1) The computer you broke won't be replaced until you pay for the old one.
2) If you can write a check today, we won't dock your paycheck, but if we do, we'll spread the payment over at least 4 paychecks.
3) Any work you don't get done due to no computer will be considered against you in your next performance review and may be considered grounds for dismissal.
There's lots of reasons not to do it that way, but geeze, if there were real consequences (financially especially) for being a fuckup with computers, I think the users would toe a much tighter line.
I run windows xp pro and I usually check windows update at least once a week. I keep my virus defs updated too. Ironically, this is "proactive" security measures.
The windows patches I download are usually the critical updates and some of the "recommended updates." I am doubtful of the driver updates because the current NVIDIA driver wasn't too stable. I don't enable automatic updates, but I do that for my parents' and sister's computer because like most people they don't understand what patch security is.
I haven't had any real problems with patches screwing up my computer, except for that NVIDIA driver. But I did take comfort in Window's driver rollback that allowed me to the older driver that was stable.
I think that this system up update patches at one source makes things a lot easier than finding patches for windows 95 like back in the day. But obviously if they base system was more stable and secure, I wouldn't have to update as frequently.
My company recently became a Windows-only shop, and replaced the Solaris network. Last week we had to reboot our systems three times for patches. This week we've already done it once (it's only Tuesday). The master install image for a whole product line was infected with a virus.
Oh, but we're so much more productive now with Windows than with Solaris, that I guess it's okay. I can crank out ten flimsy hyperbolic presentations with PowerPoint in the time it used to take me to write up one detailed spec in FrameMaker. That's progress!
Don't blame me, I didn't vote for either of them!
Let's just say that I approached Service Pack 4 with a great deal of apprehension. I've had good luck with workstation upgrades, but my server experience is decidely mixed.
You can only drink 30 or 40 glasses of beer a day, no matter how rich you are.
-- Colonel Adolphus Busch
If your ATA/IDE controller is no longer properly recognized by Windows and you can no longer boot because of that, you can usually put the drive right onto the motherboard's built-in IDE adapter and boot successfully, so you can install the drivers for your controller and reboot with them switched back again. Of course this will only work if you're just using an ATA controller or have a RAID1 setup. There's other ways (using the recovery console) to install/load drivers, but I agree, it's pretty tricky.
I didn't have any problems with SP3/SP4 and SQL server.
Not All Who Wander Are Lost
Goodness - perhaps you don't realize.
He's got an IBM server - probably a big production machine. It's almost certainly a SCSI Raid setup.
It's not possible to plug the array into the regular controller.
In any case, doesn't matter if this would fix it or not. It shouldn't happen EVER.
I'm not sure which is worse, I take the box down to patch, and get heart palpatations when it goes down catestrophically, or someone roots my box.
Either case, I'd be pissed.
Cheers,
Greg
I have l33t z3r0 day patches! I patch before the bugs are even discovered. :)
/. crowd.
Seriously. Yeah. Let's have a bunch of people describe for us exactly where they work and what their window of vulnerability is. That would rock. I've got paper and pencil handy.
I bet the boss of the guy who submitted this is thrilled to see this information broadcast to the whole
This space intentionally left blank
Am I the only person that uses LVS for redundant servers? Whenever they'res a kernel patch needed, or something to that sort, I just bring one down, let it come up, test it, then repeat. This way, I have 100% uptime. We use LVS and OSPF throughout our network for that 100%, and are damn proud of it. :P