Slashdot Mirror
Patching Paranoia - How Fast Do You Patch?
selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."
Constant re-booting seems to be an exclusive MS-phenomenon. Installing patches on Linux only requires a restart of the affected services unless a kernel upgrade is involed - and even this can be worked around in some cases.
You will reboot less when patching a Linux machine. Guaranteed.
Depends on the patch....security patches get applied, ASAP. If it's a patch fixing something that is not used much or that we don't have an issue with, it gets applied when the next Maintenence Level (IBM speak for Service Pack) comes out. Luckily, AIX does not have very many security issues. That covers the OS. Our application we are way behind in patches and we only can pacth after hours. Since we're in the middle of conversions, there are processes constantly running on the server and we also cannot patch when we have reps from the vendor in working on the conversion because the expect thigns to be the same while they are there and patches can really mess them up. So, needless to say, we are WAY behind on app patches but we are reasonably caught up with OS level patches.
Gorkman
I have 6 machines at home to administrate, all are connected to the same LAN, 4 are RedHat Linux, and the rest are Windows 2K/XP, I have no problem for the RedHat boxes, as up2date automatically detects new updates and notifies me, so I download and patch, and as you know, no need for reboots, one of the reasons I love Linux.
As for the 2 Windows machines, I try to apply critical updates as soon as possible, I download them off MS Download Center so I reinstall them in case of a format.
The IT section color scheme sucks.
RAS was broken with a recent NT 4.0 server update, took a few weeks for MS to fix it.
sPh
SUS allows you to approve a patch before distributing it. In practice, this means applying it to your test lab (or test cupboard in my case) before approving it for everyone else.
Predictive text is shiv!
I guess you didn't hear about the patch for XP that disabled Internet access for hundreds of thousands of users. And while I had good luck with service packs, many others did not.
BTW, you may want to change your sig because at first, I thought that it was part of the message. Most mods won't know the differents and will mod you flamebait.
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
Service Pack 3 broke a workstation we have that runs EDI and is uh well, pretty critical.
I work for a rather large webhosting company and on the M$ side of things, we normally update all of the shared M$ boxes within 12 hours of learning of the patches (be it windows patches or software specific patches), but only if it's a security-sensitive update. Major version updates can take up to a month (ie: PHP 4.2.x - PHP 4.3.x). On the other client machines, it can be anywhere from 24 hours to a week. It all depends on how severe the patch is. The Blaster patch was applied within 36 hours on all M$ machines when it originally came out. The unix side is a bit slower... they have dev boxes that they test and retest the new patches on. Once it's deemed suitable for our enviroment, they will go ahead and apply it. That can take up to 5 hours to auctually apply the patch ;D
Harm
I'm the network admin/windows/active directory guy for a healthcare company. We run multiple SUS servers, several for desktops, and one for servers. Our procedure is, when a patch is released, that day I.T. downloads and installs it on our desktops and test servers. If it's successful, it gets approved on our desktop SUS servers. If those work OK, the next day it gets approved for our severs. So far we've had no problems with that process.
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.
But they did. That story even made it to CNN.com. I did not apply that patch until MS released a fixed patch.
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
Uninstall the sendmail and BIND Solaris pacakges and the patches will stop trying to patch them. You could also install your custom software in /usr/local like everyone else.
For DigitalUNIX/Tru64, the only thing I can say is "yeah, it's a pain, but hey, HP will stop supporting it soon enough!"
that only works if it's ok to reboot those machines at night.
:)
(Then again, neither does windows, but I'm only one man in a 1400 man company)
Makes me happy when people correct themselves.
Some of our machines are processing 24/7, and we have to wait until they can afford to reboot. And if some patches are waiting in queue, then the new ones don't get deployed until after the existing (waiting) patches are applied. SUS isn't meant for machines that need to do real processing, or at least it doesn't work well for them.
I would recommend you setup some sort of patching schedule ( and SUS+group policies works well for this ), maybe use a rotating schedule so there are at least a few systems online at any given time, but make this a "Company Policy". If it's expected, PHBs are usually cool with it.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
What you SHOULD do is put your versions of the binaries, and startup scripts OUTSIDE the vendor tree. That way they will NOT be modified by vendor patches.
Saves a LOT of time.
..broke all KINDS of things. On my home machine, I now get 5 USB power errors that I didn't get with 10.2.6, as well as unexplained freezes & crashes.
I reverted to 10.2.6 and all was well once again.
And this was 10.2.8 redeux - remember the first time that it came out, machines were breaking all over the place. (ethernet issues, IDE oddities..)
Not to play favorites, my Windows 2003 server recently crashed and burned after a patching incident, requiring a full re-install. Luckily it only took a couple of hours with the ASR disk and DLT tape. Try doing that with Linux. BTW, the 2003 box was a replacement for that RH8 server....
The recent problems with Apple's Mac OS X 10.2.8 update are a good example of a patch breaking things (ie. killing network connections). Now the problem I see with how updates are administered is that in many cases you can't select between a security update and a feature update. 10.2.8 addressed the OpenSSH, OpenSSL bugs that were recently reported on in addition to sendmail and a couple of others. At the same time, it installed new USB 2.0 drivers and NIC drivers for G4 desktops.
One solution I believe is to make every patch and update available separately. In addition provide an update tool with presets that choose only the latest security fixes or feature updates or all updates, and allow administer's to customize their own presets. You are then faced with the issue of dependencies however these can be easily addressed by warnings letting you know what additional software is required and will be installed.
I dont even recommend this to any clients. I just tell them I will be doing it afterhours. Sure, its less convient for me, but they arent working for me. IMO, it just goes with the territory.
How fast do you (or your IT group) install patches for major exploits like this?
You kind of have to do a risk analysis on it. If it is a critical exploit, it moves higher. If it is exposed to raw internet, a critical should be done immediately. If it is a web server, likewise. If it is just a server on an internal LAN, it can probably wait a while. And while the parent was scored as funny, in reality it never hurts to wait for feedback on something if you can. You dont get a higher score for being the first person to install a patch.
What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet?
You can always reasure your clients that things are low risk, but low doesnt mean no. Better safe than sorry is a good attitude, and one which will make your client feel more secure (in an emotional sense). Seems you made the right choice.
As an aside, now that MS is planning on holding their security patches to one a month, what does everyone else think? Should they release them asap, or wait once a month? Personally, we just scheduled once or twice a month to do the patching on our servers, but I think putting it out asap is better.
Manipulate the moderator system! Mod someone as "overrated" today.
Unix uses a system called reference counter. Each file which exists on disk has on reference counter.
Normal files, which have only one filename, have a reference count of 1.
File wich have multiple names, e.g. hard links have an increased reference count.
For example, if /bin/sh is hard link to /bin/bash . Both filenames point to the same file on disk, which haves a reference count of 2.
Another example: supose you run a program called /usr/sbin/named and you update the program with another version, you will have the following scenario:
Note: You cannot overwrite a running process program. But you can delete the filename from the directory.
DOS and NT do not allow this. (And sometimes even with files with the same name, but in an other directory!)
http://www.securityfocus.com/archive/1/272695/2002 -05-13/2002-05-19/0
Every patch for MS Office broke groupwise. It was actually the address book that it broke.
It is not about speed...it is about risk management. Patching is not a race. Some patches must be deployed quickly, i.e. the vulnerability poses a substantial risk to the enterprise. Other patches can be deployed more slowly (e.g. a flaw which can only be exploited from the console when the server lives in a machine room with good physical security and trusted IT staffers). Other vulnerabilities can be mitigated until the patch can be applied (e.g. new firewall rules).
Also, patches often do not fix the problem they are trying to address or produce unwanted side effects. In such cases, you must weigh risk vs. rewards. (Something that gets overlooked while your staff blindly pushes out the update to 10,000 machines in 48 hours).
Speed patching is really the wrong mindset. Remember, effective information security is all about effective risk management.
As pointed out by others the guy is refering to a IBM server, most like with SCSI. Outside of that point, a service pack should *NEVER* overwrite 3rd party drivers without at least warning you. 3rd party drivers were installed for a reason...*hint* b/c the drivers that are in windows were either not available or not working.
Let's just say that I approached Service Pack 4 with a great deal of apprehension.
SP4 broke, among others, the Terminal Services (for win2k TS servers) -- the logins now take over 30 seconds (from 5 sec earlier). During TS sessions the TS freezes few times an hour for around 20-30 seconds at a time, making it unusable for some tasks and wasteful (of time and nerves) for the remaining ones.
Other patches and "upgrades", especially those for IE, have been degrading win98se performance and stability (such as annoying 1 minute freeze ups [WaitForMultipleObjects() that never occur] after deleting or copying 'large' number of files). I suppose that's one way to "help" customers decide on upgrading to XP -- just "accidentally" select the most incompetent programmers and QC to provide patches for earlier OS versions. It reminds me of the common tactic by insurance companies to staff the dumbest and the rudest on the refund/payment side of the business.