Slashdot Mirror
Patching Paranoia - How Fast Do You Patch?
selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."
... I am to post to a new Slashdot article
I wait until I get feedbacks from sites such as The Register to make sure that the patch doesn't break anything.
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
Or common sense?
I run a SUS server for my organization, and it checks for patches nightly. The next day, my servers and workstations are patched.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Constant re-booting seems to be an exclusive MS-phenomenon. Installing patches on Linux only requires a restart of the affected services unless a kernel upgrade is involed - and even this can be worked around in some cases.
You will reboot less when patching a Linux machine. Guaranteed.
Have you guys looked at MS SUS 1.0 to automatically deliver critical updates? It's kinda lame--not the greatest management capabilities--but it does work. I have a company similar to Thrive & use it to deliver patches to end-user desktops at several clients.
Depends on the patch....security patches get applied, ASAP. If it's a patch fixing something that is not used much or that we don't have an issue with, it gets applied when the next Maintenence Level (IBM speak for Service Pack) comes out. Luckily, AIX does not have very many security issues. That covers the OS. Our application we are way behind in patches and we only can pacth after hours. Since we're in the middle of conversions, there are processes constantly running on the server and we also cannot patch when we have reps from the vendor in working on the conversion because the expect thigns to be the same while they are there and patches can really mess them up. So, needless to say, we are WAY behind on app patches but we are reasonably caught up with OS level patches.
Gorkman
I have 6 machines at home to administrate, all are connected to the same LAN, 4 are RedHat Linux, and the rest are Windows 2K/XP, I have no problem for the RedHat boxes, as up2date automatically detects new updates and notifies me, so I download and patch, and as you know, no need for reboots, one of the reasons I love Linux.
As for the 2 Windows machines, I try to apply critical updates as soon as possible, I download them off MS Download Center so I reinstall them in case of a format.
The IT section color scheme sucks.
Middle of the day reboots are normal, so we patch whenever we want.
I first patch my local systems and try them for a few hours as I run similar configurations to my clients for development. If I survive the patch, I patch the development systems at my client sites. If those remain stable for a period of time, I patch production clients, and then finally production servers.
If at any point a glitch appears, I stop at that point, minimizing damage. Usually that means I have a glitch locally and my clients would never know that there was a glitchy patch unless I tell them. Pretty much a similar approach that a big company would take (patch the test LAN) except I am the test LAN.
Sig under construction since 1998.
sPh
There's one big gotcha here: notebooks. Your users are firewalled at work but once they get home they're probably wide open. Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.
Maybe you should get your clients to run servers that don't require a reboot for most application patches.
I tend to follow at least the following criteria when deploying patches:
1- If the patch is a Microsoft patch, I deploy it immediately, regardless of severity, because Microsoft has repeatedly lied about the severity of security flaws that were actually quite critical.
2- If the patch is for a very theoretical problem, such as many of the recent OpenSSL patches, I tend to let it wait for the next big update. Good examples are those problems where key-breaking time is reduced to only 50 years or so on a $10,000,000,000 budget.
3- Patches that fix vulnerabilites that are only a problem in stupid configurations (Such as recent OpenSSH problems.) get ignored until the updates have been tested.
4- Patches from Sun go out immediately, because they seem to take so long that the exploits for bugs have been integrated into script-kiddie toolkits.
We don't EVER install a patch on a production machine without testing it first on some less crucial machine.
Any machine that accepts connections from outside the firewall (SMTP, IMAPS, HTTPS, & SSH are all we take, and only to specific machines) gets any remotely exploitable bug patched ASAP. Typically I will run the patch on a non-production machine for 24 hours to make sure it's reasonably stable, then patch.
Once the patch has proved itself in production on the remotely accessible machines, say for a week or so, we load it everywhere else.
Stuff that's not remotely exploitable is dealt with on a more relaxed schedule, generally at least two weeks after the patch has begun testing on a non-production machine. Sometimes longer.
We also always test our backup strategies before loading MS or HP patches, since sometimes they completely trash the system.
HP-UX patches come out months or years after the exploit, Microsoft patches come out weeks or months late, DEC patches used to come out within days (Oh, how we miss ye DEC) and BSD and linux come out within hours, usually.
If it's windows patch early, and patch often. If anyone asks why you rebooted a box, lie about it and say "It crashed." That's one everyone will believe.
Less Talk, More Beer.
This is one of those grand broad questions with no answer. If you have an entire redundant system to test with, you can patch that instant, test it, and roll it out. But then again the new patch might fail in some way you never expected. If you are talking a 100+ servers, then you might need to test a group, before you patch your core group. Then there are the questions you need to ask. Is someone likely to break in? Did it work for someone else? Is it a MS product? What do your clients want? When will have the least effect on service? Did the patch come to you via email? You know, the important questions. To answer the question though, we patch, after we know the patch works.
I work for a rather large webhosting company and on the M$ side of things, we normally update all of the shared M$ boxes within 12 hours of learning of the patches (be it windows patches or software specific patches), but only if it's a security-sensitive update. Major version updates can take up to a month (ie: PHP 4.2.x - PHP 4.3.x). On the other client machines, it can be anywhere from 24 hours to a week. It all depends on how severe the patch is. The Blaster patch was applied within 36 hours on all M$ machines when it originally came out. The unix side is a bit slower... they have dev boxes that they test and retest the new patches on. Once it's deemed suitable for our enviroment, they will go ahead and apply it. That can take up to 5 hours to auctually apply the patch ;D
Harm
I've got roommates who've moved to the Linux desktop. I usually do the upgrades from my desktop. The only reason why I tell them that I'm doing upgrades is that it's annoying if they shut down the system in the middle of an RPM Install. (one dual boots to Windows so he's more likely to reboot, the other runs solely on Linux he really only powers off if he's heading out. I think I've installed one or two kernel upgrades in the last year (which require reboots to enable), but since my roommates reboot so often, I can just wait for their next reboot.
There's also much less need to do testing with Linux patches... You generally know EXACTLY what subsystems are being affected by a patch, so if it's not a critical component, you can often install blindly. Even if it is a critical component, the patches are often well defined and if you have any questions you can read the source code.
The problems with Windows is that it's the large-scale version of spaghetti code. The relationship between various pieces are ill-defined and numerous. Patches spider into various areas and it seems like nobody (even at Microsoft) knows precisely what a patch fixes (or what it breaks).
This doesn't just apply to desktops. I'm in the middle of putting together scripts to enable controlled push of patches to a large number of varied servers. In truth, the hardes part is going to be figuring out which patches go to which boxes -- not figuring out if the patch is going to break things.
Yep. I'm spoiled. Linux makes life both easy and cheap.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I'd run openBSD if they would release a version of Gator for it.
slashdot: where everyone yells sarcastic metaphors to themselves to understand the issue
There's one big gotcha here: notebooks. Your users are firewalled at work but once they get home they're probably wide open. Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.
A company I did some work for (re: I was a contract monkey..yes, I admit it) had a policy that plugging in a company laptop to your home network constituted grounds for firing.
Yup. They were that strict. It wasn't a technology company, (so the "brass" were a bit... over the top) and they'd been bitten hard by folks bringing infected-at-home company notebooks back into the environment, so I can understand some paranoia, but sheesh...
..broke all KINDS of things. On my home machine, I now get 5 USB power errors that I didn't get with 10.2.6, as well as unexplained freezes & crashes.
I reverted to 10.2.6 and all was well once again.
And this was 10.2.8 redeux - remember the first time that it came out, machines were breaking all over the place. (ethernet issues, IDE oddities..)
Once upon a time, I worked at a large content organization with the usual large IT infrastructure, supported by a single large firm. Per the requirements of the support contract, these guys were compelled to down the system and install patches as soon as they got their hands on the code. No-notice outages eere the rule. Managers, customers and employees pitched fits until someone finally woke up and explained that the support vendor would be in violation of contract if he didn't move that fast.
So, we changed the contract. Unscheduled downtime projected to last more than 30 minutes required getting permission from several designated management types. Any one of those managers could postpone the maintenance.
This worked because the support contractor always made sure that those designated managers understood the implications of delaying the maintenance.
-- Slashdot: When Public Access TV Says "No"
My company writes enterprise software, albeit badly. The QA process I feel could be much better, but at least it gives a support rep like me a job.
Twice a month, we release patches which fix any number of bugs we may have found since the original release of the software. About 1/3 of the patches we release introduce NEW bugs that weren't there before the patch! These new bugs can easily and often cripple important parts of the software.
I knew a 4 month stretch where this happened on every release for those 4 months, 8 patches in a row!
Most of our customers update every few months, and they keep an eye on our website, and the public customer email lists constantly throw out emails which the bleeding edge leaders complain of problems introduced on new builds (which they have every right to complain about).
Now I can't speak for any other company, including Microsoft, but sometimes upgrading right away when you aren't really currently experiencing an active problem is worse than not upgrading at all.
"All great wisdom is contained in .signature files"
I just wish we had 1/3 of the balls of that company and that fucking up with the company computer was seen as destructive and damaging as it actually is.
The countless whining we get over passwords ("My boss says I dont hafta have one.."), applying updates to desktops(!), removing shit like comet cursor, and the people that toss laptops around and then bitch that they don't have the right laptop after they've broken it.
I'd love to see 2 or 3 people in particular have to sit down in front of the CFO and be told:
1) The computer you broke won't be replaced until you pay for the old one.
2) If you can write a check today, we won't dock your paycheck, but if we do, we'll spread the payment over at least 4 paychecks.
3) Any work you don't get done due to no computer will be considered against you in your next performance review and may be considered grounds for dismissal.
There's lots of reasons not to do it that way, but geeze, if there were real consequences (financially especially) for being a fuckup with computers, I think the users would toe a much tighter line.
I certainly didn't like patching OpenSSH on a machine I can only reach via SSH.
Programming can be fun again. Film at 11.
Hm, rebooting. Rebooting. Oh yeah, I remember now. I had to do that to my GNU/Linux system once when I upgraded my motherboard.
--Just the place for a snark!
I have emerge rsync && emerge -U world in cron.daily you insensitive clod.
Unix uses a system called reference counter. Each file which exists on disk has on reference counter.
Normal files, which have only one filename, have a reference count of 1.
File wich have multiple names, e.g. hard links have an increased reference count.
For example, if /bin/sh is hard link to /bin/bash . Both filenames point to the same file on disk, which haves a reference count of 2.
Another example: supose you run a program called /usr/sbin/named and you update the program with another version, you will have the following scenario:
Note: You cannot overwrite a running process program. But you can delete the filename from the directory.
DOS and NT do not allow this. (And sometimes even with files with the same name, but in an other directory!)
I run windows xp pro and I usually check windows update at least once a week. I keep my virus defs updated too. Ironically, this is "proactive" security measures.
The windows patches I download are usually the critical updates and some of the "recommended updates." I am doubtful of the driver updates because the current NVIDIA driver wasn't too stable. I don't enable automatic updates, but I do that for my parents' and sister's computer because like most people they don't understand what patch security is.
I haven't had any real problems with patches screwing up my computer, except for that NVIDIA driver. But I did take comfort in Window's driver rollback that allowed me to the older driver that was stable.
I think that this system up update patches at one source makes things a lot easier than finding patches for windows 95 like back in the day. But obviously if they base system was more stable and secure, I wouldn't have to update as frequently.
http://www.securityfocus.com/archive/1/272695/2002 -05-13/2002-05-19/0
My company recently became a Windows-only shop, and replaced the Solaris network. Last week we had to reboot our systems three times for patches. This week we've already done it once (it's only Tuesday). The master install image for a whole product line was infected with a virus.
Oh, but we're so much more productive now with Windows than with Solaris, that I guess it's okay. I can crank out ten flimsy hyperbolic presentations with PowerPoint in the time it used to take me to write up one detailed spec in FrameMaker. That's progress!
Don't blame me, I didn't vote for either of them!
Whare are these "patches" of which you speak?
:)
Just run a VAX/VMS system as your firewall... it's so old and obscure that no hacker will have any hope of remembering how to hack it.
I keep one browser open to windowsupdate all the time, constantly refreshing, so I never miss an update. Why, sometimes, I even get truncated downloads because the upload on their end hasn't finished to the server yet!