Slashdot Mirror
Patching Paranoia - How Fast Do You Patch?
selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."
As soon as possible
... I am to post to a new Slashdot article
I wait until I get feedbacks from sites such as The Register to make sure that the patch doesn't break anything.
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
But I'm busy applying some patches. Damn this Windows.
Or common sense?
I run a SUS server for my organization, and it checks for patches nightly. The next day, my servers and workstations are patched.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Constant re-booting seems to be an exclusive MS-phenomenon. Installing patches on Linux only requires a restart of the affected services unless a kernel upgrade is involed - and even this can be worked around in some cases.
You will reboot less when patching a Linux machine. Guaranteed.
Have you guys looked at MS SUS 1.0 to automatically deliver critical updates? It's kinda lame--not the greatest management capabilities--but it does work. I have a company similar to Thrive & use it to deliver patches to end-user desktops at several clients.
ASAP with some caution. Best to do at least _some_ QA on a patch that supposedly received QA from your vendor. You never know what they might have bungled.
That said, you should assume that attackers already have the capability to exploit the problem. Implementing mitigating strategies to keep yourself at least somewhat safe while you work out the QA is a good idea too.
Depends on the patch....security patches get applied, ASAP. If it's a patch fixing something that is not used much or that we don't have an issue with, it gets applied when the next Maintenence Level (IBM speak for Service Pack) comes out. Luckily, AIX does not have very many security issues. That covers the OS. Our application we are way behind in patches and we only can pacth after hours. Since we're in the middle of conversions, there are processes constantly running on the server and we also cannot patch when we have reps from the vendor in working on the conversion because the expect thigns to be the same while they are there and patches can really mess them up. So, needless to say, we are WAY behind on app patches but we are reasonably caught up with OS level patches.
Gorkman
On my desktop computer, I do it as soon as I see a warning issued. The network? Whenever I can convince 20-30 people that I need to interrupt whatever it is they are in the middle of, to let me use their system for 15-20 minutes. Usually takes 2 weeks to accomplish that task.
I have 6 machines at home to administrate, all are connected to the same LAN, 4 are RedHat Linux, and the rest are Windows 2K/XP, I have no problem for the RedHat boxes, as up2date automatically detects new updates and notifies me, so I download and patch, and as you know, no need for reboots, one of the reasons I love Linux.
As for the 2 Windows machines, I try to apply critical updates as soon as possible, I download them off MS Download Center so I reinstall them in case of a format.
The IT section color scheme sucks.
secure the perimeter! He just forgot to mention to secure it with Linux.
Most of our customers have Linux gateways installed. Patching can be done calmly after work hours.
[...] This is extremely difficult because no customer wants to be interrupted by a reboot during business hours.
So? Why would "what they want" be of any importance? They hired to you do a job, right? Or did they hire a yes-man?
This is one of the things MS needs to improve on. Servers shouldn't need to be rebooted for every minor patch. Over the last few years, they seem to have improved on the number of reboots required, however, there are still far too many reboots on this platform. This would make it much easier to perform patching - no significant interruption of networking functions.
Middle of the day reboots are normal, so we patch whenever we want.
For a lot of these advisories, you can plug the hole at the firewall, or maybe the mail server. Do you really need to allow MS messenger service to be running outside your LAN? Sure, it is a good idea to quickly patch the systems, but it may take days to get them all patched. Fixing the problem outside the Windows boxes can be done within minutes of reading the advisory, depending on where the problem is.
I first patch my local systems and try them for a few hours as I run similar configurations to my clients for development. If I survive the patch, I patch the development systems at my client sites. If those remain stable for a period of time, I patch production clients, and then finally production servers.
If at any point a glitch appears, I stop at that point, minimizing damage. Usually that means I have a glitch locally and my clients would never know that there was a glitchy patch unless I tell them. Pretty much a similar approach that a big company would take (patch the test LAN) except I am the test LAN.
Sig under construction since 1998.
Just think how mad they'd be when they get rooted because you waited til after business hours to patch their servers. It's a lose-lose situation for you!
I run Debian Woody servers, so I apt-get update && apt-get upgrade every morning. Since I never have to reboot, that is not an issue.
Sincerely,
Cliff
Thrive Networks
...but since we don't use the only operating system where applying a patch requires rebooting, our clients don't complain.
I suppose this would be better suited for an "ask slashdot" question, but *how* do you roll out the patches? There are several solutions out there, involving central local server dedicated to the job and using Norton Ghost among others.... How do you do it?
I have the following in a cron job:
apt-get update && apt-get -y upgrade
Thus, my systems are patched every night. Works for me. I'd be interested to see if anyone working in IT and taking care of lots of systems (instead of just a handful for me) do the same.
I just use gentoo linux. The portage package system lets me type in two simple commands and update my entire system. I can upgrade to stable builds or even develpment builds if I see fit. Thats why the 3 servers I administer run gentoo. Keeping them secure is a fairly simple and painless job.
-=You might be a geek if your computer is worth more than your car=-
But it has been a while, so I may be wrong.
Slashdot: Where people pretend to be twice as smart as they really are by behaving like children.
Patches?
Patches!?!
We don't need no STINKIN' PATCHES!!!
Best to patch immediately, especially if you are running a server.
For clients, it really depends on the severity of the exploit. If the virus is something that comes through email and attacks other machines then you better patch right away or else risk a network-wide infection.
Otherwise I usually leave client machines to update themselves at night. If you have a decent firewall then there is no reason to go crazy over something that only comes in through an open port. It is important to patch ASAP but I won't lose any sleep over it in this situation.
I use GNU/Linux.
Thanks in advance,
Kilgore Trout
Impeach G. W. Bush
of course, for some platforms might have this
"built-in" -- i don't remember a >1min break as result of applying security patches from red hat or debian.
Is it just me or does this sound like a clever little advertisment for the aforementioned company? Shouldn't this be up at the top of the page along with the other advertisements?
Reboot? Who the hell needs to reboot? Oh yeah, Windows. And seriously, even if you need to reboot, if your computers are fast, it takes what, 30 seconds? less? If that amount of time is going to interrupt you that much you have a problem. And if your computers take longer to reboot than that, you have another problem. Using network installers that will patch and reboot all the systems from a central location it should take you this long to patch
1) download patch depends on connection, 1 minutes for me for average large wad of Windows Update stuffs. probably be like 10 minutes for people with less than college quality connections.
2) time it takes to send patch to all systems and patch them. It takes me about 1 minutes to patch my single xp machine, so well generously give it 5 minutes if it's a whole bunch of machines.
3) Reboot all patched machines. As I already said, this should take maybe 30 seconds on an average pc. Unless you set it up bad. So we'll give this a minute.
So, I generously estimate 16 minutes it should take to patch a network of windows boxen with the latest fix. If you don't have a means to patch all the machines from one location, consider getting one. Patching should be something you can do over lunch break. And of course, use non-windows and you wont have to reboot. Doing that should make patching transparent to the employees.
The GeekNights podcast is going strong. Listen!
If you ran openBSD servers then
1) you would save your clients money
2) you would not likely have to reboot
3) you would probably not have the exploit in the first place
Windows is a big make work project.
IF a workaround isn't available, and IF the patch doesn't break something else (Microsoft comes to mind immediately), and somebody else with a similar setup to mine has verified this...
THEN I patch. Security paranoia is good and all, but look what happened to all the chumps who went nuts over the recent SSH exploit. Some of them A) Ran to an even more insecure LSH, or B) Upgraded immediately, opening themselves to another attack based on a bug in the ported patch!
Although the 'Window of Exploitation' becomes null and void on a security announcement, recent history has pretty much cemented the fact that large scale internet attacks are reactive to exploits, and not proactive.
Which explains why until last semester we were still running Solaris 2.7. Finally got it updated to 2.9. I guess there are advantages to running a relatively obscure OS.
My enterprise just rolled out something called BigFix that is supposed to allow them to provide patches immediately upon their release. It seems to be basically a cusomizable Windows Automatic Updates system. My understanding is that the user still has to allow the patches to be installed, which is a potential problem.
Does anyone have any experience or additional info about BigFix?
"I'm not, like, that smart. I, like, forget stuff all the time." -- Paris Hilton
I have to manage a server farm and lord knows how many workstations (about 130+)... I make life easy on myself and use remote patching on all our systems. Right before lunch time (a solid half hour) I use the intercom to let everyone know that they will have to reboot their machines. Only real pain in the butt for us was Blaster and Nachi... A different tech from another division said he already patched the system.... the problem was he was only referring to his one freakin machine!
Business \Busi"ness\, n.;
A scam in which all people involved perceive as beneficial...
Back when I dealt with a large number of windows machines I held to the notion of running patches our test network immediately, if that went for a day without noticly blowing up I would start installing on a select number of low priority boxes in IT and a few brave soles that worked closely with us. Then after another two days of observation (and a lot of reading to see if anyone else has had significant problems) then we would start a blitz on the whole of the network.
they hack the code and fix it in binary....
how long until
How much of that 48 hours was spent testing the patches with your client's configurations?
If you were running a *nix based OS, you wouldn't have to restart every time you patched something. Windows is a flawed design, and the reboots are one of the chief reasons I would NEVER use windows as a server unless I am forced to.
The only reason a *nix box would have to be restarted, is if the patch goes against the kernel.
Solution:Drop Windows. Grow some real technical knowhow, and install a *nix to do your serving.
This is just sig!
i always read the stories then i....
Windows has detected the time has changed
please press ok to restart
Within 24 hours on Windows systems, 72 hours on Macs... but within 4 hours --with NOTHING higher priority-- if patches are announced same-day for both Windows and Mac.
As part of my belt-and-suspenders paranoia, we run two separate parallel-function network backup machines, one OSX based, the other Windows (mumblemumblemumble). I live in terror of the hypothetical multi-platform/multi-mode Warhol worm taking out my systems.
//Information does not want to be free; it wants to breed.
... should be how fast others apply patches to my servers if I get it right. It seems that the clients didn't do anything in order to get their servers patched. So why do i press no whenever MS "scrambles to get my my software patched" to a new version?
We test first. In general, we respond to a vulnerability by first checking to see if it effects us (for example, ssh has had some recent problems that did not effect us because we did not use the features that were compromised). If it is some thing that we need to worry about, we make do some testing to make sure it doesn't break anything. Then we determine the best way to patch the effected systems on a case by case (or class by class) basis; in general, we try for minimal disruption (only patch what needs to be patched).
For everything else, we do rolling full-upgrades, on a as-much-time-as-it-takes-to-get-it-right schedule, again testing as we go.
The bad guys(tm) can hurt you, but they're nothing compared to what you can do to yourself if you make a habit of pulling the trigger first and aiming later.
-- MarkusQ
IAALS.
At least where I work, the general turnaround time is about six months. And even then, they only do the servers, and they only patch the most critical of vulnerabilities.
We're looking at 2,500 servers here, but still - six months? Absurd.
(No, I'm not directly involved in the process, though I *did* write many of the docs about it that they use. They take forever to do the simplest of tasks.)
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
...whenever we can get around to it. Usually quickly, unless we cannot devide our processes (assigned tasks) up, then we can do a few, work on something else, patch a few more, work on something else... ya know, just like your CPU does.
-- Liberalism is a mental disorder.
Critical patches for important services get applied ASAP. If I can't turn it off or firewall it off, then we notify clients of impending emergency downtime and go at it.
That said, if I can firewall it off or turn it off, the patch can wait until its well tested.
If I have been able to see further than others, it is because I bought a pair of binoculars.
Seems Slashdot has a new form of advertising. Disguising ads as stories.
I hardly ever install patches. I run Windows ME and it makes more sense for me to just let the thing sit and wait till it gets so screwed over that I need to completely reinstall. Then I do so and repatch and repeat the process. I just keep a firewall andnorton around to make sure I don't get any already know viruses, etc but beyond that patching isn;t going to really help taht much, especially since WinME crashes even quicker with patches than it does if I just leave it alone.
There's a growing sense that even if The Future comes,
most of us won't be able to afford it.
-- Lemmy
On our servers we tend to do nothing more than make sure that we are firewalled so tight that even the ugliest security hole will go unnoticed by black-hats, because they won't be able to get that far. That being said, we STILL patch the servers once we see that the patch has been in use for a few days without problems by other users.
There's nothing better than installing a patch-for-a-patch. The recent SSH exploits come to mind, had to do everything twice!
Nobodies Prefect
Tidbits for Techs Technology Blog
I use a very simple cron.daily script, which essentially does:
apt-get update
apt-get -f -u --yes upgrade
So as not to break my system, I actually add another cmd line param that uses a separate config file and sources.list file, which ensure that I only download the latest security packages.
how many hours of time i wasted patching machines. in a company with 20,000 computers easily in each building.... and only about clusters of 4 are the same, it's a pain. we can't sus the patch to them, or anything fun like that...
we're still trying to make sure all 1 million of our computers are patch world wide. It's a big pain.
We have to test the patches on our test servers and such before we issue them, because it might be worse havoc than the virus would cause...
i never realized how much i hated virus' until that week of 2 patches, back to back... gah!
What are these patches you speak of? Oh crap, I don't have alot of time to finish this, theres an error with something called RPC I only have a few seconds l^H^H^H^H^H^H^H^H^H^H^H^H^H^H
CARRIER LOST
The Department of Defense has no specific timeline for patches to be put in place, mainly because each team out here is free to do this as it wishes, when it wishes. This leads to disorganization and chaos of a level hitherto unimagined except on networks run by the most rank amateurs imaginable -- which may well be our status.
:(
In any case, my office has a bi-weekly reboot period set aside wherein they apply critical patches. Since this is scheduled downtime, our customers have already agreed by way of an SLA (service level agreement) that services won't be up during a brief window every other Friday. At least, that's when our guys are supposed to add patches -- it's mostly at the discretion of the admin on duty and how late he or she is in getting out for their weekend.
Past that there's no allowed downtime except when servers crash or when the Quarterly Outage rolls around. As such, patching is infrequent and often incomplete. It is distressing in the extreme.
I've pushed for (and received) tools to automatically download patches from Microsoft, and have other tools on hand to push these out to servers, but thanks to the Windows architecture I can't simply stop and restart services to make sure the patches take effect. Reboots are called for, and because that necessitates Downtime (capitalized most intentionally), it is verboten.
Things changed a bit when Nimda and Welchia hit, mainly because all of our suborganizations were busy scrambling like hell to uninfect themselves. My group, a rather high component in DoD, did not get hit by the worms -- our firewalls were properly configured and didn't allow random incoming RPC. (Though having seen how many orgs *did* get hit.. well let's just say there's a bright, bright future for college graduates with no real world experience, hmm? All you have to do is qualify for a security clearance!)
Anyway, we wound up patching in very short order in that specific case, but only because of immediate impact. If the writer of Nimda had half a brain and had used his exploit to write a very quietly installed trojan horse instead of a stupid reboot script he would have had control of hundreds of systems at the Pentagon. Lucky for us he was busy making a statement.
Patching does not happen nearly fast enough to suit me.
[Posted anonymously. I don't mind losing my job -- our contract's over in forty five days -- but I do mind federal prison.]
We use Linux! We use the Red Hat network to patch our computers, and they are usally patched in 24 hours.
I get you're using Windows. Using Linux, or any unix, patching is usually transparent to the user, unless you parch the kernel. Patching the RPC server in such a context could be invisible (RPC server was the latest patch in windows I know of enough to talk about it)
...
rpm -Uvh
service portmap restart (if required)
done.
Scriptable, possible remotely, no reboot. Even ssh can be upgraded via itself (the old executable runs for your connection untill you disconnect, the new one handle the new connections, and the blocks on the disk are freed as soon as the last instance of the old exe (running connections) finishes. Super elegant, super state of the art since the 80's or something. And no monkey involved in clocking buttons.
Get a real OS. AIX, HP-UX, Solarix, Linux, *BSDs all have those feature.
I don't manage any servers, but on my personal Windows XP pc, I patch asap. There's nothing critical on there that would be impacted by a lousy patch, and I know enough that I can always unpatch it if need be.
:-)
The only patches I hold off on are motherboard bios patches, cause those are such a bitch to debug if something goes wrong
I wish there weren't so many WIndows XP patches, but you have the admit they've got an amazing patch delivery service. It must help for them to have so much practice delivering patches.
$8.95/mo web hosting
My patching shedule? What's a patch?
Between
- how much disruption and lost time will be caused by a potential exploit of the vulnerability and
- how much disruption and lost time will be caused by patches that break your mix of applications.
The only thing you can do is to start testing patches in a micro-environment as soon as they're released.That, and check your firewall rules to insure potential exploits can't enter via the easy routes.
"Provided by the management for your protection."
Linux, on the other hand doesn't require reboots after installing the patches. I think this is due to the fact that the required modules can be installed on demand without bringing the system down (feel free to correct me if I'm wrong here).
What prevents Windows from doing the same?
An Indian-American Hindu committed to non-violent thought/speech/action alarmed by the global explosion of radical Islam
We usually let our people go about 3PM on Fridays if there isn't a major project do, so I usually am gone by 4:30.
Now on the FreeBSD machines, we haven't patched them at all because they run on a closed network and handle our file and print servers. The OpenBSD boxes serve as our company intranet, mail server, and firewall/router. These are patched when needed, but that is usually with updated every 6 months.
"The problem with socialism is eventually you run out of other people's money" - Thatcher.
I try to follow an idea from public heath services' response to virus outbreaks; I create a perimeter around the outbreak consisting of vaccinated systems. In this case, the virus carrier is the internet, so all of the systems the control passage of information to and from the internet are vaccinated (patched) first. Of course, if you have no proxy server, and way to block harmful web content at the edge, that means patching everybody...
Your customers don't "want to be interrupted by a reboot during business hours?" Remind them that they used to reboot every 4 hours before windows 2000. Either patch or go back to pre-windows 2000!
Maybe you should get your clients to run servers that don't require a reboot for most application patches.
It would seem that patching is becoming the biggest source of downtime for MS-based systems. How can any hosting place claim a bazzilion-9s uptime when they need to patch'n'reboot for the security flaw of the week? I suppose all OS types have this issue. Anyone have comparitive data on patches-per-year for different OS species and the associated downtime to install and reboot for each patch?
On the other hand, I suppose a hosting company could maintain seemlingly high uptime by never patching -- a great strategy until they get hit by a big exploit.
Two wrongs don't make a right, but three lefts do.
Speak for yourself, troll.
I never patch the live servers immediately but follow this regime
1. My local development machine. ( This is done as soon as I get the reports that the patch is out.)
2. Our development servers
3. Our staging server
4. Live web servers
5. Database server
The time from top to bottom is usually about 2 - 3 days, unless a problem was found ( which there was with KB824141)
Patriotism is the opium of the masses
We use a product called Patchlink to patch all of our windows systems. A small agent runs on the client machine (be it server or workstation) and we have one sql/web server (the patchlink server) that communicates with both our clients and the Patchlink corp's systems. They do all the silent install prep and testing of the patch in their labs, and then they put it on their servers. Our Patchlink server picks up the new patch, checks it's database to see what clients it can apply the patch too, and then it sends the patch out to them at night according to our schedule. It silently installs the patch and reboots it at night.
It helps us out A LOT - makes patching easy and fairly painless for us. We can also choose not to roll out certain patches to ease the pain of breakage.
My university patches things only when something goes wrong. When the whole campus went down because of blaster, they then decided it would be a good idea to patch all of the lab computers with windows update. They're still running lots of old Apache incarnations as well... from 2.0.40 to 1.3.6. I'm consistantly amazed by their seeming retardation.
Because I need to install windows first and there are no plans for an instalation in the short/medium /long term
So I'll keep with apt thanks
I tend to follow at least the following criteria when deploying patches:
1- If the patch is a Microsoft patch, I deploy it immediately, regardless of severity, because Microsoft has repeatedly lied about the severity of security flaws that were actually quite critical.
2- If the patch is for a very theoretical problem, such as many of the recent OpenSSL patches, I tend to let it wait for the next big update. Good examples are those problems where key-breaking time is reduced to only 50 years or so on a $10,000,000,000 budget.
3- Patches that fix vulnerabilites that are only a problem in stupid configurations (Such as recent OpenSSH problems.) get ignored until the updates have been tested.
4- Patches from Sun go out immediately, because they seem to take so long that the exploits for bugs have been integrated into script-kiddie toolkits.
What do you consider to be an acceptable turn around time for a vulnerability patch
Faster than it takes someone to attack you.
Our desktops and internal servers get service packs only and then only after they have been thoroughly tested by our certification group. This means we get SP1 installed at about the same time that SP2 is released.
I have to assume that our web facing servers get better treatment but I have no direct experience with them.
Miraculously, the network seems to stay up most of the time.
We don't EVER install a patch on a production machine without testing it first on some less crucial machine.
Any machine that accepts connections from outside the firewall (SMTP, IMAPS, HTTPS, & SSH are all we take, and only to specific machines) gets any remotely exploitable bug patched ASAP. Typically I will run the patch on a non-production machine for 24 hours to make sure it's reasonably stable, then patch.
Once the patch has proved itself in production on the remotely accessible machines, say for a week or so, we load it everywhere else.
Stuff that's not remotely exploitable is dealt with on a more relaxed schedule, generally at least two weeks after the patch has begun testing on a non-production machine. Sometimes longer.
We also always test our backup strategies before loading MS or HP patches, since sometimes they completely trash the system.
HP-UX patches come out months or years after the exploit, Microsoft patches come out weeks or months late, DEC patches used to come out within days (Oh, how we miss ye DEC) and BSD and linux come out within hours, usually.
If it's windows patch early, and patch often. If anyone asks why you rebooted a box, lie about it and say "It crashed." That's one everyone will believe.
Less Talk, More Beer.
Gotta do this one anonymous, cause the DMCA would probably allow Microsoft to supoena /. for my personal information because there is evidence of copyright infringement in my post.
At the office we patch as soon as they are available, but I don't patch at home at all. I'm a "software lover", so my copy of xp wouldn't work if I patched. Anywho, without patching at all(ever), and with a DSL connection that is always on, and with Norton 2002 running, I have never gotten a virus through XP vulnerabilities. I have however acidentally clicked on an attachment in a newsgroup and gotten a virus. Damn finger twitch!
What we do is a simple processes. All people on staff, tech related or not, have been told of the importance of patching and why it occurs. When a problem occurs and a patch is released we quickly package the patch in a zip file with installation instructions and send out an immediate email. All people within our organization are responsible for patching their own systems (or asking for help if they can not) and replying within 24 hours of the email. We ten audit the systems every month or so to make sure that the patches have been applied, if someone is caught not following this procedure (which has never happened) the person would be seriously reprimanded if not fired due to the sensitive information we house.
This is one of those grand broad questions with no answer. If you have an entire redundant system to test with, you can patch that instant, test it, and roll it out. But then again the new patch might fail in some way you never expected. If you are talking a 100+ servers, then you might need to test a group, before you patch your core group. Then there are the questions you need to ask. Is someone likely to break in? Did it work for someone else? Is it a MS product? What do your clients want? When will have the least effect on service? Did the patch come to you via email? You know, the important questions. To answer the question though, we patch, after we know the patch works.
Then tell them how long it will take to reboot now.
Then tell them how much your company will charge for said system restore.
See how many complaints you get.
Extended Warranty? How can I lose!
I like to get security issues resolved within a few hours of the problem being announced.
Usually this can be accomplished in minutes without disruption.
-josh
You mean you have to reboot after you patch!? Must be an obsolete OS!
Since it seems like there's a new MS exploit every thursday, it's easier to just declare every thursday as a day of downtime so you can get all your windows machines patched. Smart companies will give their employees the day off on Thursday and make them come in on Sunday instead.
Microsoft: We're changing the world. Ask us how!
This guy posts the URL to slashdot then informs everyone how great he is because his company patch. I guess he need either a new webserver or wants to see how long it takes someone to break in.
We run a secure shop. After the umpteenth virus/worm infection/buffer overflow, we decided that Windows was no longer affordable. We ditched our Windows machines, and replaced them with Linux and Solaris boxen, and since then, we haven't had a single security incident! Really, how difficult is it to understand?
.... If your firewall, anti-virus, office security etc are all taken care of properly. Protect the network then think about individual servers/pc's. A friend of mine works for a large law firm in London that got screwed by Blaster, his network security was spot on until he looked at all those bogged down home user vpn connections with no Personal firewall. Leave no stone unturned on the network and you should be ok.
There is no right or wrong, just the consequences of your actions.
Sorry, I couldn't resist.
If you can read this sig - the bitch fell off.
I like to know that I haven't screwed up the machine's ability to boot properly.
And since I'm not the only person with superuser privs, I like to make sure my cohorts haven't screwed up the machine's boot process, either.
You don't know unless you test. Patching's a good excuse to do a test boot - you're logged on anyway, and you can justify any interuption of services by pointing to the need for the patch.
In my area I see other computer support businesses advertising their awesome abilities at fixing systems affected by virus/worm attacks. The use of Windows is reducing the computer profession to the level of a "Pests-R-Us" business. Rather than really helping customers by pointing them toward a product that does not suffer from the poor quality of Windows, support companies are actually basing their livelihood on the inherent problems and fostering the idea that its all part of the computing XPerience. These companies should be putting up signs that say "Migrate-R-Us", let us help you save money by moving away from such a bug ridden platform as Windows!
Pretty sad.
Why not run the patch, then have someone reboot the systems after hours?
/. article.. :)
Just have a spare engineer, or two, on standby in the event a system doesn't come back.
I forgot, that's too easy and wouldn't have resulted in a
I just recalled that a friend told me a couple years ago that BSD systems do not need to reboot when patching the kernel, is that true or was he just full of it?
Linux requires reboots all the time. I wonder what you're smoking. Always more FUD.
All Microsoft patches/hotfixes and service packs have command-line switches that can set "no reboot" (-z or /z). Install the patches (as your SLA seems to dictate), inform the customer, and let the customer reboot when ready. Suggest to the customer the need for a timely reboot.
I subscribe to ntbugtraq.com and read what others are saying about the patches. Inevitibly, there are some that patch immediately, and a few of them are kind enough to report their findings.
Saying Android is a family of phones is akin to saying Linux is a family of PCs.
What do you mean, reboot? You didn't need to patch the kernel. Unless you work with machines that need a reboot every time the mouse moves...
... can you do patches for me to apply ???
And no, I'm not new here!
"Ask not what your country can do for you." --John F. Kennedy
I just run up2date over the weekend. Fixes any security issues that exist. All this time and we havent had any downtime due to crap like MSBlast.
Take that MS!
you need a patch in your fp software, before the next major story
litigious bastards
suck it sco!
So there.
It uses, um... dark energy, or something.
--- Ban humanity.
This is totally OT, but wft.
...
:) /.= 7263 334
So, there are these 2 homosexual computer programmers.
The first says to the second - "Do you want the key to my heart?"
The second says - "No, I just want your ASCII".
Get it? ASCII? lol
OK. Its pretty bad. But, wtf, its not the worst joke you've ever heard.
I'm posting this again, because I want to. Gotta luv AC.
-----
AC: The choice for OT posts on
http://slashdot.org/comments.pl?sid=82912&cid
I install every patch which Microsoft emails me as soon as I get it. I can hardly keep up nowadays.
:-)
(Yes, I know MS never emails patches
I've got roommates who've moved to the Linux desktop. I usually do the upgrades from my desktop. The only reason why I tell them that I'm doing upgrades is that it's annoying if they shut down the system in the middle of an RPM Install. (one dual boots to Windows so he's more likely to reboot, the other runs solely on Linux he really only powers off if he's heading out. I think I've installed one or two kernel upgrades in the last year (which require reboots to enable), but since my roommates reboot so often, I can just wait for their next reboot.
There's also much less need to do testing with Linux patches... You generally know EXACTLY what subsystems are being affected by a patch, so if it's not a critical component, you can often install blindly. Even if it is a critical component, the patches are often well defined and if you have any questions you can read the source code.
The problems with Windows is that it's the large-scale version of spaghetti code. The relationship between various pieces are ill-defined and numerous. Patches spider into various areas and it seems like nobody (even at Microsoft) knows precisely what a patch fixes (or what it breaks).
This doesn't just apply to desktops. I'm in the middle of putting together scripts to enable controlled push of patches to a large number of varied servers. In truth, the hardes part is going to be figuring out which patches go to which boxes -- not figuring out if the patch is going to break things.
Yep. I'm spoiled. Linux makes life both easy and cheap.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
I have a regular patch cycle, based on availability requirement (i.e short for DNS's, longer for application servers). If an emergency erratum shows up I evaluate our exposure and balance it against the risk of breakage. For me, that risk is usually quite low, since all my machines are unix or linux. Very few errata require me to reboot or pose a serious threat to my service level. I also run minimal services and local firewalls to reduce my exposure. Hence I can usually sit back and wait to see how a patch performs in the wild before applying it. If that process fails, it is usually quite simple to back out.
It helps that relatively few unix exploits are packaged and widely distributed in kits. In the past eight years I have encountered such a compromise twice. Once at work (sorta, it was at a University affiliated with my employer) and once at a consulting gig. Both were on vanilla, wide open RedHat 6.2/wu-ftpd installs (blech). I have seen other compromises, but they were manual entry into archaic crap SunOS 4 and Irix 5 (double blech!) machines.
Windows is another story. Luckily I don't manage any of our 80 or so Windows servers. I have no idea how I would handle patching them. It seems like their exposure is quite high. More importantly, the patch process appears fatally flawed. Our admins are afraid to apply patches because of the frequency with which they break machines. And almost all of them require rebooting. The recent round of RPC exploits hit us hard and we had no choice but to patch all our Windows servers. Unfortunately, the patches broke our backup system. That left me working 14 hours a day for six days (I admin the backup server) with backline engineers from two vendors. We got the overall system back up, although it meant rebooting every Windows server (some more than once), but a couple of Windows backups still fail every night. Not much we can do about it, as we have no access to the source code. Besides, backing out of the patches would not only be suicidal but would require another massive disruption of service.
It is cowardly, and a betrayal of whatever it means to be a Jew, to act as a white man
-James Baldwin
Host the sites we manage. We farm out the hosting to third party hosts. But we're on their backs to make sure they get patched ASAP. Business hours or not, patch and reboot as soon as a patch is available.
Dance monkey, Dance!!
-Goran
Carpe Scrotum - The only way to deal with your competition.
...either another Monster (along the lines of blaster or welchia) doesn't rear its head comes along, which if everyone keeps patching within 48 - 96 hours wont happen for a while. And we according human nature become complacent and let 48 - 96 become 96 - 192 becomes 192 - 384 becomes 384 - 768
OR
We get another patch that breaks things
Either way the cycle will begin again, 'tis the circle of life.
And I wonder if someone at Thrive has moderator access today because all the other messages calling this out as an advertisement are getting modded down.
...we have a push-tool on every XP machine that forces users to patch. about a month ago we were patching several times a day for a week! (incremental patches).
usually it's at least one a month if not more.
it's pretty funny, someone will be projecting a presentation and suddenly the patch manager pops up and takes control. a real hassle, but the IT team has really put security ahead of convenience.
Look at me! I'm getting free advertising under the guise of an "insightful" question for Slashdot!
If its something that I have a hole punched through the firewall for I'll patch it right away, like ssh, apache, bind, etc. Something that affects 135 or 137 or a linuxconf exploit which is behind the firewall I'll take a bit more time to do.
slashdot, news for crazed liberal socialist zealots
Your risk analysis will determine a few things. Among them:
;).
1) Criticality of systems (how important is this system to your business?)
2) Sensitivity of the data on those systems (are you gonna get sued or lose your customers?)
3) Impact of availability (does anybody care if your news server is down for half an hour per night?)
4) Availability of exploits/platform exposure (Windoze bugs are exploited in less than a week these days)
5) Stuff I probably forgot (home taking care of the kid, and only half a pot of coffee...)
For us, our "critical" systems are patched immediately. Typically, we reboot that night (except Unix/Linux boxes, which typically don't need reboot). As part of our risk analysis, we went even further to minimize risk by doing things like not allowing sensitive data to be stored (use the mainframe for that- let someone else worry about it
There exists no way of exchanging information without making judgments. --Bene Gesserit Axiom
Isn't that simply a security hole? An exploit would be code taking advantage of said hole...
+1 Insightful
to keep the cravings at bay. Sometimes I wear two.
Yep I'm the "miracle man" as the boss slaps me on the back and introduces me to his golfing buddies. "You know this guy has saved the company?"
The whole intranet and servers are Mac OS X, one minor patch which was done via memo. Oh, a few temps needed help from a coworker with the update.
The strategy of IT making work for themselves and justifying bloated staff by endorsing faulty Microsoft software is backfiring.
There are a lot of hungry college educated Indians who want your job.
provided you dont plug in the power cord.
The One Rule Of Chess You'll Ever Need: Don't play someone who carries a kit in their bookbag.
What be these "patches" you speak of?
Patch your stuff right before they show up. This is key. Another measure you can use is to evaluate afterwards: if it broke stuff, you patched too fast, if you were infected, you patched too slow.
...and when I have understanding of computers, I will be the Supreme Being.
Okay, so it's a valid question, but Slashdot is read by how many people and most of the home page write-up deals with how quickly his/her company services their clients, complete with link?
This smells like "Astroturfing" to me - a hell of a win for their PR agency.
The only acceptable defense of scientific results is to say that they were the product of the Scientific Method.
...how often do you patch?
Every damn day. I've been trying to quit smoking forever. LOl!!!11111
>:^)
Spread the RC luvin'
There is a small independent Linux company out there which guarentees they will apply security patches to your system within 48 hours.
This isn't an automated service, their techs actually login remotely, backup config files, install the patches, and test everything.
It looks like it is priced for businesses.
My company uses these guys because we've had problems with up2date and similar automatic patching software in the past. We've lost configs, had apache and other daemons not restart, and have had lost log files as a result. But since this company has been patching our systems, our patches are always applied on time and our services have not been interrupted for more than a few minutes at the most.
sounds like what you need is a the 'cron'ed' use of 'apt-get' at about 2:00 am in the morning; updates for us linux users can be boring sometimes. of course i have the system clock updated then to...
or, if your boss 'enjoys' microsoft; then maybe the ones who want microsoft should live with their descions.
or, you might consider investing in moller's sky car, you might as well lay back and enjoy the ride from patch site to patch site to patch site to patch site to patch site to patch site to patch site to patch site to patch site to patch site....
What you SHOULD do is put your versions of the binaries, and startup scripts OUTSIDE the vendor tree. That way they will NOT be modified by vendor patches.
Saves a LOT of time.
Man, you need to get a life, stop using CAPS, and grow-up.
Everyone is allowed to have a different point of view.
If you say linux, I guess you've mastered the two-kernel monte with dissimilar kernel symbol tables? To get past those kernel patches that required a reboot?
That makes you a better linux admin than Alan Cox. I'm impressed!!
Please share with us your technique for replacing the kernel without rebooting. We'd all like to know.
After 7 years of working in IT (started with NetWare, then NT, now Linux), life has never been easier with Debian GNU/Linux. Most of the time, a reboot is not necessary, which means servers can be updated from remote with a high degree of certainty that a visit will not be required. I now live about an hour's drive from my nearest client. They're running two servers, one in a DMZ with an Internet-acessible app, the other behind the firewall with SAMBA, backup and intranet web server. Both run Debian GNU/Linux stable for a small network of about 30 Windows workstations.
Working with Microsoft products is emotionally not worth it. Too much change in the way administration needs to be done. Too many problems with viruses, worms, bad patches, politics, hardware requrements, and application interaction. I have other colleagues who work more than me with longer hours and make a lot more money because they're constantly fixing Windows, but I'm happily married with two children and focussing my efforts on Python and Perl scripting so I can automate even more adminsitration.
Ruby on Rails Screencast
You haven't learned yet? Throw Microsoft products in the trash. *n[ui]x vulnerabilities never require a reboot except in the rare event of a kernel level exploit. Your clients will thank you for switching.
It's probably down the memory hole by now, but to the best of my recollection one of Microsoft's claims prior to the release of Windows NT 4.0 was that it would almost completely eliminate the need for reboots during installation.
It always amazes me how Microsoft is consistently able to deflect criticism by saying that the problem will be addressed in the next major release... and when it gets there and it is clear that the promise hasn't been kept, nobody--certainly not the people to drive IT purchasing decisions--seems to mind.
"How to Do Nothing," kids activities, back in print!
/.
Red Hat's kernel patches for the 6.x, 7.x, 8.0, and 9.x series have ALL REQUIRED REBOOT.
That's right, all these assholes are either:
A) Unpatched and too stupid to know it
or
B) lying liars telling lies
--Charlie
this was in no way a blatant plug for said company
as an antidote to all the stupid "linux can patch anything without rebooting" lies.
I update as soon as RedHat has the patches up on the RHN and up2date lets me. This is usualy in a day or 2 from the announcment.
Switching to Linux can be an adventure!
..broke all KINDS of things. On my home machine, I now get 5 USB power errors that I didn't get with 10.2.6, as well as unexplained freezes & crashes.
I reverted to 10.2.6 and all was well once again.
And this was 10.2.8 redeux - remember the first time that it came out, machines were breaking all over the place. (ethernet issues, IDE oddities..)
Not saying that patches shouldn't be applied, but anyone that follows BugTraq knows that there are exploits for fully patched systems that have yet to be addressed by Microsoft. This includes a couple of problems with the RPC service.
This isn't a good solution for home users, but perhaps something like Cisco Secure ACS Remote Agent. This is a kernel level shim that intercepts function calls and uses a ruleset and heuristics to decide whether to permit the activity. I've seen this software protect a Windows 2000 server directly connected to the 'net with no service packs or patches.
Once upon a time, I worked at a large content organization with the usual large IT infrastructure, supported by a single large firm. Per the requirements of the support contract, these guys were compelled to down the system and install patches as soon as they got their hands on the code. No-notice outages eere the rule. Managers, customers and employees pitched fits until someone finally woke up and explained that the support vendor would be in violation of contract if he didn't move that fast.
So, we changed the contract. Unscheduled downtime projected to last more than 30 minutes required getting permission from several designated management types. Any one of those managers could postpone the maintenance.
This worked because the support contractor always made sure that those designated managers understood the implications of delaying the maintenance.
-- Slashdot: When Public Access TV Says "No"
My company writes enterprise software, albeit badly. The QA process I feel could be much better, but at least it gives a support rep like me a job.
Twice a month, we release patches which fix any number of bugs we may have found since the original release of the software. About 1/3 of the patches we release introduce NEW bugs that weren't there before the patch! These new bugs can easily and often cripple important parts of the software.
I knew a 4 month stretch where this happened on every release for those 4 months, 8 patches in a row!
Most of our customers update every few months, and they keep an eye on our website, and the public customer email lists constantly throw out emails which the bleeding edge leaders complain of problems introduced on new builds (which they have every right to complain about).
Now I can't speak for any other company, including Microsoft, but sometimes upgrading right away when you aren't really currently experiencing an active problem is worse than not upgrading at all.
"All great wisdom is contained in .signature files"
I patch as soon as I see the GLSA email! :P
:D
Then I run "emerge -u " right away.
There, obligatory Gentoo plug AND it's on topic.
Just be smart, 2 years ago I planned to make a cluster inside a web compagny, And we planned to used Linux and a lot of open packages, that can be updated within the hours, if we encounter some DDOS... We also make this web cluster redundant, so if a web server is potentially weak under any kind of attack, we drop-it (and changing version of apache or whatever) and nobody heard about a down time ! Still interested to heard a MicroSoft Tech sing the same song, I don't think so !
I don't patch or upgrade anything unless I have
to. It costs time and money and usually introduces
new bugs into my working systems.
Weigh the unknown benefits against the known cost.
Unless you're using the patch to shift the
blame to others it's pretty expensive.
I wasn't bothered by any of the recent spate of
virii since my firewall blocked rpc exploits.
The bayesian spam filtering on my email took care
of the email. A well thought out firewall
policy would have prevented a lot of the RPC
and SQL server exploits. I can't think of any
good reasons why those services should have
been accessable to attackers.
-- Programming with boost is like building a house with lego. It's a cool but I wouldn't want to live in it
We try to respond as quickly as we can but with
1500+ PCs in our division/region and
only 6 to 8 techs, it can be a daunting task.
It's very difficult to put a price on prevention,
however, with recent exploits it's safe
to "assume" that it could be relatively high depending
on impact and nature of the group/process.
I have a question: Do any of you get paid for patching time since most of it is after hours?
I'll cast the first vote: No
No, I'm New Here
Even disregarding the fact that, by running Windows XP, you have to patch a lot in the first place, the problem is your OS: unless you need to patch the kernel, there should be no need to reboot. In fact, if the vulnerability is in a loadable module, you should still not have to reboot. Some OSs even let you replace the kernel on the fly.
If you are running an OS that was designed for desktop usage (and realistically, that's what Windows was designed for), you can't expect good uptimes or server-appropriate behavior.
Most of our systems are set to auto-update weekly from windows update.
We're behind a firewall/nat and running Symantec Antivirus Corporate edition. Our server checks for new virus definitions daily. It does a good job of catching viruses on the systems of those who insist on using Eudora with IE rendering. Virus problems are very rare, and most of the alerts I send out are to remind people to patch their home pc's.
We've thought about using Microsoft's Software Update Services to reduce the number of downloads involved with auto-updates, but so far it hasn't been too big of a bandwidth hog and our server doesn't meet the advertised requirements for SUS.
It really depends on the severity of the exploit however. A patch to update say... nedit, probably isn't as critical as a patch for OpenSSH. In the case of OpenSSH patch it right away. Then again. Then (oh crap, another one?) again...
Really though, patches come out all the time to fix bugs in other patches. If you've had to deal with the Solaris patchwork of patches, you'll see newer patches negating older patches which themselves turned out to break things. So you get the task of hunting down the newest patch that negates the one you need, and sometimes it ends up being a patch from a completely different category than where you originated!
Please don't fling me in that update patch.
we have several legacy apps which are known to screw up with new hotfixes. we can't even run the latest mdacs because our legacy apps crash unless using the specific version they were coded for!
we rarely have a major issue with hotfixes and our new apps, but there have been instances where a patch does break them and we have to figure out why. once QA gives us the nod, we go patch happy and do about 500 servers ASAP.
And taking down a service isn't disruptive? It's not too bad for usually-stateless and quick services like http, but restarting a database can take a while and really hose performance since it flushes the db's internal caches.
I didnt know slashdot's new add server now put ads in the first line of submissions.
Hm, rebooting. Rebooting. Oh yeah, I remember now. I had to do that to my GNU/Linux system once when I upgraded my motherboard.
--Just the place for a snark!
Yes, but patches that fix exploitable bugs are very time sensitive. One can ususally wait until 3:00 am or the weekend for a power cycle. If you need machines working 24-7, round robin the restarts so you have continous functionality (as suggested by another in this thread).
If you are running a critical 24-7 service on one machine then that's bad implementation. Also, plan on outages in advance so you have a time to fix something, (also suggested earilier in this thread). A lot of servers plan an hour of down time every week. This will screw up your 5 nines uptime, but if it saves you from having unscheduled downtime then it is worth it. Plus it makes you comfortable with the process of fixing stuff and rebooting. If there is an unexpected outage (and there will be) then you will get it back up faster becasue you reboot machines every week. s/week/month/ if need be. YMMV.
Administrators Commandment #1 - plan for failure.
Why, o why must the sky fall when I've learned to fly?
For all these people that don't test before patching, what are you in a 1-2 server enviornment with 10 workstations?
Where I work we have well over 250 servers, with god only knows how many different applications running on them. Patching without testing isn't an option.
I think the posters question is valid. Do you patch immediately and risk server failures, or do you test everything in a lab (in our case that would be one hell of a lab if we could afford all those extra servers) and then roll out the patches once you were relatively sure they wouldn't cause problems?
We are stuggling with this all the time, and it's only getting worst.
There are tools to automate the rollout of the patches, but as far as I'm concerned the rollout of patches is the easy part, testing the patches against various hardware and applications is the real work. Hard sell to management when you tell them it's gonna take 4-5 weeks at the best to roll out the patch. But it only takes one major outage and that same management staff will be asking yout why didn't you test the patch. No win situation.
For those of you who say you haven't had a patch blow up a server, count your blessings. Either your lucky, or you haven't been patching servers very long.
-Moby
A few places I work use a tool from Sophos to push the patches onto client machines over the network.
I have emerge rsync && emerge -U world in cron.daily you insensitive clod.
You lily-livered patch posies make me shiver. What ever happened to the days of the frontier, the wild west, and discovery? Show me a guy who can get by on NT4 out of the box and I'll show you a man who can weather any storm. THAT'S what I think about patching.
Well, that's what I'd think if I had time to think. Anyone else having trouble getting into Windows Update???
It's only funny until someone gets hurt. Then, it's hilarious.
You don't choose applications based on OS, you choose OS based on applications.
If your profit-generator requires a Windows server, you run windows. Or you don't generate profit.
Think you ninnies.
"Timing the Application of Security Patches for Optimal Uptime". Steve Beattie, Seth Arnold, Crispin Cowan, Perry Wagle, Chris Wright, and Adam Shostack. Presented at the USENIX 16th Systems Administration Conference (LISA 2002), Philadelphia, PA, December 2002. Postscript. or ugly PDF.
Crispin
----
Crispin Cowan, Ph.D.
Chief Scientist, Immunix Inc.
Yes, as stated in a previous post, most if not all routers can filter traffic. It is not practical to do so in an enterprise core of any size, though.
Typically, the core routers will actually be layer-3 switches (hardware-based routers) to handle all your traffic. Unfortunatelly, layer 3 switches cannot process access lists (what you use to filter traffic) in hardware, so applying them negates any performance advantage offered by your fancy layer 3 switch.
The difference in throughput can be staggering.
Such an "Ask Slashdot" should be properly qualified. Of course, the author is talking about Windows, and of course, I'm sure lots of people have poined this out already.
But then what about Unix(like) OSes? Although Linux is free and Solaris is not, I see many instances of Linux "admins" waiting until an "official" patch / RPM / Debian package / whatever comes out rather than compiling vulnerable software themselves, just like their Solaris counterparts who don't have a choice most of the time.
So better questions might be:
How many admins depend solely on vendor patches / binary patches rather than patching themselves? And how much time does this waiting cause?
Furthermore, how many "admins" are too afraid to break things because they do not fully understand the interdependencies of their systems? This one I see a LOT.
This leads to the best question:
If you don't patch immediately, WHY NOT?
While it would be a network management challenge in order to implement, it would provide that extra piece of protection that would eliminate a great deal of network 'intrusions' from outside sources. Also, if those firewalls are configured to log and email that log to a Sysadmin, troublesome laptops could be tracked down within an hour or so of entering the network and be 'fixed' of their Trojan/Viral ailment.
If you ignore the other uses of a tool, does that make the tool less useful, or you less useful?
Through SUS, our XP and 2k workstations get updates after they're tested for a week. We have a SUS test pool with about 100 workstations on it that receives all of the updates that we can receive through SUS each Monday. After a week of testing, the production SUS server (approx 3000 clients) is syncronized with the test pool server and we get last week's updates deployed by Tuesday afternoon. Then all of the lastest updates are again "approved" for the test pool. The problem is that our network is still about 40% NT and 9x due to some legacy software that various parts of the agency can't live without or find the means to replace. These sytems are located through out the state and can take between a day and a month to get updated. Currently, the "suits" are weighing the benefits of the various patch management tools (SMS, HFNetCHk, Altaris, Patchlink, etc.), but until they manage to fumble to agreement, we're doing it all with cars and keyboards. But..... I digress. Given the choice, I'd test for a week and deploy with SUS (for newer systems) and HFNetChk for the legacy systems once I'm happy that an update wont nerf my clients applications.
I'm sure it's been documented countless times, but here are the basic steps:
First, you read the advisory. Are you running the listed software components in vulnerable versions? Maybe you have disabled the vulnerable component. Then you look at the attack vectors. Maybe the attackers would have to use protocols/ports you block at the next packet filter. Maybe you have activated special tools that thwart the attack (like that URL filter Microsoft provides).
If you determine that you are vulnerable to attacks, you examine the impact and relate it to the obstacles a potential attacker has to face (access to internal network, for example). Perhaps it's better to live with a DoS risk than to apply a hotfix in an unscheduled manner. If thinks look really bad, you have to apply the patch, but this is just a measure of last resort. If you chose not patch this time, you schedule it for the next routine maintainance. In the meantime, you can check whether problems with the patch are reported.
In my experience, the whole preparation process takes up to three hours for free software because the advisory quality is typically quite poor, and you have to browse source code and patches. Official patches might not be isolated from other functionality changes or might be just incomplete. Often, it's a good idea to look at vendor patches backporting just the fixes.
I'd believe it's somewhat faster for proprietary software because you have less information, and you don't yet look at object code diffs to better understand the problem. So you stop pretty early and rely on the vendor assassment only. Fortunately, Microsoft typically provides most of the information you need, unlike any other vendor (free or proprietary), and let's hope that the PSS team now double-checks and ensures that no attack vectors are missed.
Unix uses a system called reference counter. Each file which exists on disk has on reference counter.
Normal files, which have only one filename, have a reference count of 1.
File wich have multiple names, e.g. hard links have an increased reference count.
For example, if /bin/sh is hard link to /bin/bash . Both filenames point to the same file on disk, which haves a reference count of 2.
Another example: supose you run a program called /usr/sbin/named and you update the program with another version, you will have the following scenario:
Note: You cannot overwrite a running process program. But you can delete the filename from the directory.
DOS and NT do not allow this. (And sometimes even with files with the same name, but in an other directory!)
When upgrading, the only way to be sure you've upgraded all the machines correctly is to upgrade all the machines systematically and over as short a time-span as possible. And I mean all. Just because the machine isn't in active daily use does not mean it's not open to attack. Having security holes while being on-line and powered-up is quite sufficient.
If you get to design or modify the configuration of the network, introduce some fail-over support. It can be trivial, it doesn't have to handle the peak loads of the primary systems, it just has to cope for the few minutes a box is down.
Once you've patched all machines on the network, I would advise running a security scan over the entire network. Make sure there aren't any nasty surprises. This is to cover your back, as much as anything. If you are paid to secure XYZ's network, and the next day they discover data theft or some other breach of security, guess who is going to get the rap? You have to make sure you have done your job thoroughly.
If the network is "secure" (ie: all machines are running IPSec, SKIP or SSH, addresses are not routable, there's some good NIDS software on the firewall, HIDS software on all other machines, and proxies for external networking), then you can afford to be more relaxed.
Even if a network has only some of the above requirements, you can probably take your time. Knowing about an exploit, and even knowing about an exploitable machine, does not equal being able to do anything with that knowledge.
The idea of securing a network is to increase the mean time between knowing all the data and knowing how to use it. (You should always assume that a potential attacker knows your network at least as well as you do.)
The idea of patching a machine is to raise the difficulty of breaking in, by raising the bar of how much knowledge they need.
Finally, remember that the importance of the computer, or even the network, can be irrelevent. A decent bandwidth will attract anyone wanting to run a DDOS attack against someone else. Trust between your network and someone else's can make your machines a stepping-stone to where they want to be.
There are as many reasons why your computer might be inviting as there are crackers out there who might want to spend the time doing exactly that.
Any machine - a forgotten mail server, a programmable router, or a desktop PC of some ultra-lowly clerk - can be used to gain access to a network. It only takes one exploit on one machine, and the net is open to anyone.
However, I won't recommend wrapping your house in duct tape, and hiding in the bathroom. Nor will I recommend surrounding your PC with sand-bags. Paranoia is second only to stupidity as a cause of vulnerability.
The key to wisdom is to raise the requirements on time and knowledge to such that the probability of an undetected (and therefore unstopped) intrusion multiplied by the cost of potential damages that could result in the worst case is about equal to the cost of progressive maintenance over the same timescale.
In other words, if the probability of a successful crack increases, increase your security to match. Likewise, if the value of what is on or reachable through your computers increases, then you must also increase your security to match.
A very secure network that has valuable data isn't worth piling on further security, unless the risks increase. It costs more to go from a 1% risk to a 0.5% risk, than from a 2% risk to a 1% risk. Therefore, reduce the risk to the level you are ok with, and then just do basic maintenance.
There ar
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Agreed. I think there are two big issues to resolve before applying a patch.
The first point goes without saying: if you trust your company network to someone else's QA...
The second one is, in many ways, more serious than the first. Sure, it's smart to to update your desktop boxes, but they're relatively safe from most things anyway if they're sitting behind a good firewall and your security procedures are half-decent. There's not much point putting a PC on someone's desk if you're going to be interrupting their work all day to patch it.
Equally, you can't just patch and restart a critical server that's in use 24/7 on a frequent basis. Presumably you have some sort of back-up/redundancy set-up for that kind of box, but you might have to wait a while before you can switch things around to make a patch.
I think the key thing is to balance the risks: for this patch, how much testing do I need to do myself before I consider it trustworthy, how urgent is it that the patch be installed on each candidate machine, and how much disruption to each machine's operation will result?
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Can I buy some pot from you? (Brian to Peter, A Very Special Family Guy Freakin' Christmas)
Seriously though, iTunes, iSync, iCal, iPod, iMovie, &c don't need a restart on my Mac - the only patches that need one are system fixes like upgrades, QuickTime, and Java. I don't think jaguar acually needs it, the patches are all kexts and libraries; I think Apple mandates it because OS X is a desktop OS, so uptime isn't critical, and a reboot guarantees the patches are all loaded right.
Facts do not cease to exist because they are ignored. - Aldous Huxley
I always have to laugh when the windows users in the office are complaining about their computers taking half an hour to reboot. Every time something crashes (or they just reboot because it's been a few hours since the last reboot..) they have to wait for patches/updates to be applied. Then sometimes, they have to reboot again! Everyone else on linux boxes are not interrupted by the need (or the feeling that its time) to reboot. There is something wrong with an OS when its users feel compelled to reboot and re-install periodically. (OS 9 and most windows versions seem to have this effect on people)
TallGreen CMS hosting
I run Debian unstable on our serverfarm, with cron-apt offcourse. :D
It runs every our, to make sure security patches are installed quick as hell.
But I don't think Debian understand that they need to test their packages well, before it is released and installed in Enterprise.
Packages are constantly broken, and my(about 10 000) users complain they cannot work beacuse of our servers constantly being down.
*g* , just make a call to Debian, that's what I tell em'
Is that as bad as waiting for the service and the whole OS to come back up?
CAn'T CompreHend SARcaSm?
For the most part, I'd rather have them ASAP. If they're not critical, you don't have to patch right away, and you have more time to test. If they are critical, the sooner the better, obviously.
Of course, I don't work with MS software very much, so I don't have to deal with almost daily security patches. In that case, I might want the patches more frequent than once a month - say weekly or biweekly, at least.
If you knew anything, you would know that you can install the patch and elect to reboot at a later time. You could've installed the patches during the day and rebooted all the machines at once after hours.
User boxes check for pathces nightly, if exist they get installed
Power users (5) get the patched downloaded and install at leisure.
Servers get the patched downloaded and depending on the severity get rebooted about weekly at opportune times
Linux, installed as needed.
-- Tim
TKrabec Pahh
I happily installed all the latest patches for my Redhat 8 box until one day, several months ago, on reboot (a kernel update), the box was totally hosed.
Not to play favorites, my Windows 2003 server recently crashed and burned after a patching incident, requiring a full re-install.
You should outsource your patching needs.
Sitting here with some servers that have 280+ day uptimes.
Whippie skip. I'm sitting here with some servers that have 400+ day uptimes, and at least one which is at 699 days, 16 hours, and 49 minutes as I write this post.
Yes, it's NT. Now, what's your point?
For less work than it cost you to build the Windows 2003 server, you could have installed MONDO.
To restore a machine that's been backed up with mondo, you do this:
1) Boot mondo CD.
2) Wait for restore to finish (mondo can call for additional CDs, or load files from elsewhere on your local network, or even from Red Hat's site for OS files)
3) Reboot.
You're done! On one of my large servers, with 128 GB of files, it takes a while. But much faster than DLT (and I've found DLT hardware to be so failure-prone I don't even use it anymore, personally -- I prefer Mammoth2 and AIT.)
Sounds like you're blaming your failure to admin your linux box correctly on linux itself, and then making a somewhat bogus claim on behalf of windows.
Once you get a little more *nix experience, you'll be able to do a better job. The big problem with *nix is the harsh learning curve.
Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.
That was true but now you can use Network quarantine areas on Windows 2003. My server checks when they login and if virus updates and patches etc...are not applied it shunts them to a highly restricted quarantine server. The only comms they have then are for virus updates and patches.
Right, I want to install a patch for Windows from Windows Update. So i do, and it comes up "do you want to reboot your computer"! This is now acceptable proceedure for most, but why? With debian, when I do a apt-get dist-upgrade, it updates a hell of a lot, including security updates, and the server is down for seconds, jsut so it can restart the service. Why can Windows not do this? Ridiculous! I've only once had to reboot my Linux box - and that was to upgrade the kernel. Does Microsoft need to take a page out of the UNIX ideal?
"Remote attacker can get full control of your machine..." is something I read in patch descriptions on a monthly basis. It simply doesn't get any more straightforward than this.
Could you provide a factual example to #1?
I patch all my companies and clients systems minutes to hours after I recieve the notification of a patch.
Of course I run Debian GNU/Linux everywhere so I don't have to worry about silly things like rebooting to apply a security patch to an email server.
I run windows xp pro and I usually check windows update at least once a week. I keep my virus defs updated too. Ironically, this is "proactive" security measures.
The windows patches I download are usually the critical updates and some of the "recommended updates." I am doubtful of the driver updates because the current NVIDIA driver wasn't too stable. I don't enable automatic updates, but I do that for my parents' and sister's computer because like most people they don't understand what patch security is.
I haven't had any real problems with patches screwing up my computer, except for that NVIDIA driver. But I did take comfort in Window's driver rollback that allowed me to the older driver that was stable.
I think that this system up update patches at one source makes things a lot easier than finding patches for windows 95 like back in the day. But obviously if they base system was more stable and secure, I wouldn't have to update as frequently.
Ooooo! An "Ask Slashdot" style ad! I wonder what the going rates are for those?
*cough*
Sorry, my cynicism's acting up a bit today. Don't mind me.
</troll>
At least on my home box with w98SE. Recently I decided I want it up to date, downloaded all the pending patches, installed them as I was supposed to (MANY reboots) and it resulted in a system that crashes after 3 mins of using the explorer (the tool to browse your disk!), can't load MS Word, can't launch Notepad (!) and works -nearly- stable as long as I use strictly non-MS software and avoid anything that was shipped with windows (luckily start menu worked). I thought "screw this", re-checked security on my firewall, formatted windows partition, reinstalled Windows from CD and disabled Windows Update using some tweak tool. And to make things even better, I switched off mostly everything that could be switched off from system run-time services. Now I have just the date in system tray, just "My Computer" on desktop, 2 tasks on tasks list (Explorer and Systray), startmenu and quicklaunch bar to that - and I got the most stable of Windows I ever had!
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
It's a home web server. My wife would fire me.
I run something called RedHat Linux 9. I don't have an account with RedHat, I downloaded the ISO's off of the internet. I update my system with something called yum (yellowdog update maintainer). I 'set it and forget it'(tm). It may take several hours for it to get it's turn to download patches. It downloads them and updates the machine automagically. I occasionally notice whatever else I am doing is running slower than normal and catch yum in the act of updating my computer. Since its a no-brainer to update the computer, I start yum whenever I get a security alert.
runs on NT4 (sp1 iirc) because their to afraid to apply patches for fear of major system failure (2k+ employees). The risk of trying to fix what's been so long mismanaged.. forgotten.. poorly designed.. is just to great.
This site is used to summarize patches which should be applied. It stopped working about the time Microsoft issued their first "monthly" patch. Now it says, "No updates are available." for an unpatched workstation. According the monthly patch release, I can verify at least 4 patches should be applied. This gives a false sense of security to "Joe Servicepack." (I know, I know, using any Microsoft product is a security oxymoron; "military intelligence," "microsoft security").
I have tested this with about 15 separate NT4 workstations, with the exact same result.
The conspiracist says this site stopped working because Microsoft wanted it broken under NT4. It could be "SUS" is the new thing and the old thing can be left to fall into disrepair.
My company recently became a Windows-only shop, and replaced the Solaris network. Last week we had to reboot our systems three times for patches. This week we've already done it once (it's only Tuesday). The master install image for a whole product line was infected with a virus.
Oh, but we're so much more productive now with Windows than with Solaris, that I guess it's okay. I can crank out ten flimsy hyperbolic presentations with PowerPoint in the time it used to take me to write up one detailed spec in FrameMaker. That's progress!
Don't blame me, I didn't vote for either of them!
This is one of the big benefits of linux/unix: It seldomn must be rebooted after patching. :-)
Hey!!!! You asshat! I've been experimenting with transexual lesbian vegetables. Surely that has to count for something! And don't knock it. Try a Rutabaga. You might never go back.
Whare are these "patches" of which you speak?
:)
Just run a VAX/VMS system as your firewall... it's so old and obscure that no hacker will have any hope of remembering how to hack it.
TIA
Reboot after installing security updates? I do that occasionally when a kernel vulnerability is discovered. I can usually install updates without causing any interference by running
apt-get update
apt-get upgrade
No rebooting required!
Funny thing that 90% of the patches I have to do requiere no more than "service restart" with a "downtime" of maybe 2 seconds.
Maybe one should consider how much more is it worth to live by perpetuating a monopoly of worse software.
-><- no
Yeah, its kind of ironic, unless you understand there is a difference between "religion" and "church". Religion == good, church == bad. Think about it. Christ's teachings on its own is really meant to free the mind of man. This is evident if you study the teachings on your own using some text that you trust as not being terribly corrupted. Churches, on the other hand, tend to use the same teachings to instill fear or increase the power of the church collective, at the price of individual piece of mind. So, I guess with that in mind, Jefferson knew what he was talking about.
What could possibly hurt the security of the American people more than giving our own government the ability to hide its
Maybe I'm just a little naive, but Windows Server 2003 retails for like $500. Are you an MS shill or an 31337 p1r8?
company had fix for blaster decided not to install it, just waiting. Day of attacks lost 2 days complete for corporate offices, then probably a day a user over the next 2 weeks due to remote propagation. Luckily we dont do much at corporate (grin) but if this hit the stores, yikes! If you take just the salaries, this cost probably as much as 100k shortterm, plus the added new virus software and the added issues of that. Cheaper to be proactive.
I don't patch my windows until a unified update is available. Then I download it somewhere else and install it.
I think I'm still volunaberable to the blaster worm, but I don't care
.ldd can tell you which libraries are used and readelf can tell you which calls are made.
Government of the people, by corporate executives, for corporate profits.
I'm patching right now. I'll be patching from 10 P.M. to 5:00 A.M., and from 9:00 to 1:00 A.M. tomorrow night as well.
If you read about me in the paper, it'll be about me cracking Bill in the nose.
Mommy. What's a karma whore?
a new catagory should be made for obvious self advertisments.
Ah, I've always wondered about this. I mean why on earth doesn't NTFS behave 'the Unix way' meaning that files can be deleted whether they're used or not. And if we're on the subject. Why oh why does M$ NOT introduce proper (symbolic) links in the file system. And don't talk to me about reparse points. They may look like symbolic links, but aren't because deleting a reparse point also results in the deletion of the file/directory or whatever it is pointing to.
When I developed software for Solaris we could create installation areas for our software using symbolic links. Now on Windoze we have to copy the files to get the same result. Where is the logic in this? I need true symbolic links, with matching performance, and not the sh*te that comes with cygwin.
M$, do yourself a favour. Drop the next generation file system. Nobody needs it, and you're just going to make people have to upgrade their hardware. Finish what you've got first, then we'll talk.
I guess I see it as 3 options:
1) You can stay late and patch it now.
2) You can not patch it, and lose a lot of customers and exponentially more time.
3) Get a Mac.
"Politicians find new names for institutions which under old names have become odious to the people."
Ah, the dreaded Arkeia. Looks nice, doesn't it?
My employers flushed a lot of money down that rathole too... bought the enterprise version for big bucks, and it never worked right, even after six months of 5-calls-a-week to the vendor.
Nowadays I use rsync, tar, and a big server to backup everything on raid5. Each office round-robins the tarballs out to another at 2 AM for off-site backup / disaster recovery, and we keep a full set of mondo disks for all servers at each site.
No pretty dashboard, no web interface, 100% reliable backup and restore.
It doesn't, ultimately, do anyone much good to put off doing security updates (or any worthwhile system updates, for that matter) - simply out of fear of it breaking things.
I remember when MS first released Internet Explorer 6, a number of people reported problems installing the update from IE 5 or earlier. (Sometimes, it would fail to install completely -- freezing up the computer at 70% or so completion, and force you to reboot. After you did, things were really messed up, and the only good "fix" was a hard drive format and Windows re-install.)
At that time, I remember my boss being hesitant to let us upgrade the systems to IE 6 - fearing these issues. Luckily, we forged ahead anyway and rolled out the update quickly. Yes, we had a few systems that "blew up" doing the update - but it always upgraded fine after a fresh re-install of Windows, which tells me something was simply wrong with that computer's configuration to start with. If it wasn't IE 6 crashing it, it would have eventually been something else....
(Having IE 6 really ended up benefiting us, because we could do much more with setting up inherited permissions/rights from our 2000 servers with it.)
It's the nature of complex operating systems and large applications using shared libraries.... Some patches will inevitably make certain assumptions about your computer's configuration that aren't correct 100% of the time, and those exceptions will cause problems/crashes.
Any patch that really is flawed and breaks perfectly good installations on a regular basis will quickly be recalled and re-released anyway - so I say, patch early, patch often!
Where I work we wait a few weeks to hear about bugs in the patch. When it comes to applying a Microsoft patch you never know if the machine is going to come back up after a reboot. Not really trolling, it's just the truth.
....equipment often fails upon rebooting the machine. a 15-minute email stoppage can easily become a 15-hour email stoppage. Not a wise idea.
When you are writing upgrades scripts or programs, you don't want to break ANYTHING. It is better to take the benefit of the doubt, than be like Windows and say, "fuck it, lets do this".
Biggest problems with upgrades is anticipating site-specific changes. What do you do if you make a change(i.e. install a new lib which is not compatible with sendmail?) and break you're programs. Happened to me when I did Ximian and it installed it's own versions of libraries and fucked up a lot of things. To the point KDE will say: missing function in dl library, etc.
Upgrades will always be a thorn and pain in the ass: for the users who have to do it and the guys who have to write and anticipate all the little changes. Even moreso: rolling back changes are a bigger nitemare.
We use the FirstClass groupware system, the latest patch from last week (KB824141 I think) broke the client.
About a year ago one hotfix caused random bluescreens in NT4, it took me about two months to stumble on the fix (same patch, just updated).
I'm sorry if I haven't offended anyone
Unless you are replacing hardware, or the kernel itself. Why in the name of Turing would you need to reboo.. Oh, you're using Microsoft...
-- Linux, because eventually, you grow up enough to be trusted with a fork()
Take a chill pill and go see your therapist.
My clients who have switched to this have never looked back. One place has about 50 days uptime-- no reboots since I powered on the server after setting it up. I'm there for a full day of on-site support every Thursday. Since I put that server in, I spend it sitting in the server room reading or watching DVDs on my iBook, or out on the floor flirting with the attractive women in the design department. I occasionally have to fix their machines, because they're still running OS 9-- but the migration is being planned, and then I'll have even more DVD/flirting time
Another client's OS X server only has an uptime of 32 days, thanks to a power outage in their building. Since we migrated their Mac workstations to OS X in January, I've only been there to fix issues on their three Windows machines.
Yup, it's good to be a Mac admin.
On my real servers/desktops, I patch often thanks to apt-get. On my win2k3 install I had for playing condition zero and max payne 2.. I dont patch. Apparently my cdkey is 'pirated' so they refuse to provide me any updates through windows update.
While that may sound like a reasonable thing, consider the following:
You run a 600desktop company, all running win2k3 corp like you should be. One of the machines breaks, so to reinstall it your tech has to have the cdkey. Few weeks later, he gets fired. He takes the cdkey with him and procedes to leak it on IRC/usenet, thousands of people use it and it ends up blacklisted. Now your corperation can't patch until you pay 600*$MSTAX. Thanks microsoft, We need more DoS drone bots out there.
Pain lasts, kid. Its how you know you're alive. Sometimes I think this growing up thing is just pain management-TheMaxx
It is not about speed...it is about risk management. Patching is not a race. Some patches must be deployed quickly, i.e. the vulnerability poses a substantial risk to the enterprise. Other patches can be deployed more slowly (e.g. a flaw which can only be exploited from the console when the server lives in a machine room with good physical security and trusted IT staffers). Other vulnerabilities can be mitigated until the patch can be applied (e.g. new firewall rules).
Also, patches often do not fix the problem they are trying to address or produce unwanted side effects. In such cases, you must weigh risk vs. rewards. (Something that gets overlooked while your staff blindly pushes out the update to 10,000 machines in 48 hours).
Speed patching is really the wrong mindset. Remember, effective information security is all about effective risk management.
I disagree. Religion (Judeo-Christian anyway) is based on the fact that you are not in control of your own life. One of the founding tenets of religion is that whatever you do, an invisible man in the sky can come along and override you. My mom is VERY religious, and when she's confronted with a problem, she prays for either the situation to improve or for guidance. This has led her to an attitude where she'll tolerate situations that upset her and are in her control. But then again, I think that personal responsibility is becoming a thing of the past, I don't think that anyone wants to admit that they run their own life and the way it is is exactly the way they made it.
They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.
I'm sure there's probably some way to do this on a vanilla Windows install, but it might be worth mentioning that this is pretty trivial on a *NIX box. Put some remote administration commands in a shell, wrap that up in a list of hosts and patch what you can during the day, then shove the rest (possibly the majority) in cron and go home, taking your pager with you.
Remote administration and automation is really something *NIX really has down pat.
~Dalcius
Rome wasn't burnt in a day.
Am I the only person that uses LVS for redundant servers? Whenever they'res a kernel patch needed, or something to that sort, I just bring one down, let it come up, test it, then repeat. This way, I have 100% uptime. We use LVS and OSPF throughout our network for that 100%, and are damn proud of it. :P
When you say "advanced" I think you mean feature-rich. Just as a comment to that, the users I support have a very hard time learning even the basic features in their office apps (calendaring, styles in Word, even frikkin folders!). Some commercial groupware packages also run on *nix, such as Lotus Domino (Linux since 1999) which is the number two behind Exchange. Some good points, however. Still, I prefer to remain platform agnostic and take a "horses for courses" approach.
I'm sorry if I haven't offended anyone
um, just to double check... it is thrivenetworks.com (65.112.21.135) right?
1) Your analysis is based on bad assumptions so your result is way off. 2) You're a sick bastard for fucking a horse.
i have realistic ideas about how much of a target i really am. i don't usually patch at all unless i feel like i'm near the line of fire and even then, i surf around looking for the negative aspects of the patch. and when i patch, i keep track of what changes are made to my system, just in case...
short answer: not that quickly
I do agree with you... well, I guess its a difference in practice of the religion, which is what I meant by church. I was taught that God gives us the free will to do whatever the hell we want, even sin. I was also taught that no one can tell me what my relationship with God should be, because because of our free-will nature that relationship is different for everyone. So, the ultimate responsibility in life depends on one's self, but I certainly do agree that alot of people would rather leave it up to God to run their lives instead of taking ownership of your own life in Gods name. Heh.
What could possibly hurt the security of the American people more than giving our own government the ability to hide its
Or maybe those machines will all become zombies and attack the WWW. Will we have to declare war on china then?
It's like a spammers dream come true.......
It's no wonder companies are paying IT people so little these days. The person who wrote this article, is probably uncertified, may have been to college, but still desperatly lacking in higher education or common sense...
I keep one browser open to windowsupdate all the time, constantly refreshing, so I never miss an update. Why, sometimes, I even get truncated downloads because the upload on their end hasn't finished to the server yet!
Use a server operating system that doesn't need to be rebooted when patched.
Why doesn't God force his will on people? Well, can you force someone to love you?
John 8:31
To the Jews who had believed him, Jesus said, "If you hold to my teaching, you are really my disciples. Then you will know the truth, and the truth will set you free." They answered him, "We are Abraham's descendants and have never been slaves of anyone. How can you say that we shall be set free?" Jesus replied, "I tell you the truth, everyone who sins is a slave to sin. Now a slave has no permanent place in the family, but a son belongs to it forever. So if the Son sets you free, you will be free indeed.
1 Corinthians 10:23
"Everything is permissible"--but not everything is beneficial. "Everything is permissible"--but not everything is constructive. Nobody should seek his own good, but the good of others. Eat anything sold in the meat market without raising questions of conscience, for, "The earth is the Lord's, and everything in it."
Galatians 5:13
You, my brothers, were called to be free. But do not use your freedom to indulge the sinful nature ; rather, serve one another in love.
1 Peter 2:16
Live as free men, but do not use your freedom as a cover-up for evil; live as servants of God.
Gamingmuseum.com: Give your 3D accelerator a rest.
You have to reboot? That seems like a bug right there, file a report yet? ::chucke::
Patches? Patches? We don't need no stinkin' patches...
Where i used to work (F500 corp) we came up with a procedure to handle patching. The technical people
met with the data security people to evaluate risk and exposure levels. Different severities could be put into test or QA first as necessary - there was a simple flowchart that estimated times to test/implement. A recommended process/timeline to implement the patch was determined. The 'business owners' then signed off on the changes as it had the potential to affect service levels. It wasn't perfect but it worked better than the chicken-little process before it.
...for several boneheads to notify us via virus-generated e-mail that they have self-selected their names to the top of the list for the next round of layoffs. Our names being thus lowered on the list, we immediately justify our continued employment by deploying a patch that is, by that time, fully tested.
You actually think you -do- have control over the various things of life? Taxes, disease, getting hit by a car from behind, etc? Sure, you can impact those things (not paying taxes, eating right, avoiding streets), but how realistic is that, for most people? Something is still going to happen.
You obviously know very little, if anything, about religion or life in general. Do you still live at home? Religion and various faiths, for the most part, are not about "an invisible man in the sky coming along to override you". Prayer is basically the same thing as Eastern meditation methods.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
... with OS upgrades to *nix. Funny, it makes the GUI-thingy go away tho, anybody know why that happens? :)
I'd patch exploited bugs as soon as I can get my hands on the patch. I'd patch vulnerabilities as soon as I have the time and inclination (usually overnight).
WindowsUpdate? What's that?
(Subject is Fair Use.)
My personal patch policy is vastly different from my company patch policy.
Personal machines:
Windows:
Patched whenever the "Automatic Update" critter says I should.
Granted this is a lousy policy, but with a new patch every week I can't be bothered to deal with them all.
UNIX:
Source is CVS'd nightly.
A general "upgrade" build is run every 2 months to keep my systems current with minor bug fixes & other small stuff like that.
When a security advisory comes out for a service I don't use, the patch is applied whenever the next upgrade build is run.
When a security advisory comes out for a service I DO use, I pull down the source tree as soon as a patch is available, then kick off an upgrade build.
In the case of third-party software, ports and the like, the same methodology applies (if I don't run it I ignore it, if I do I patch it ASAP), but I perform the upgrade through the ports system (BSD) or by fetching and rebuilding the source for the appropriate service/package/etc.
At work, the general policy (small office situation so this is very loose) is-
Windows: If it seems OK on our personal machines after a 48 hour wait, the Windows machines in the office are patches.
UNIX: Internal machines are only patched when needed, often this amounts to "never".
External machines are patched as-needed (depending on what services they run, what OS, etc.)
I try not to leave ANY machine (personal or corporate) that is on the internet at large unpatched against any vulnerability for more than a week, even if "patched" just means "disabled service X to prevent exploitation".
/~mikeg
The Church is often bad. Politicians are often bad; Open-source advocates are often bad; <INSERT CATEGORY HERE> are often bad. That doesn't mean that Church, Politics, or anything else in themselves are inherently wrong, just badly done.
I'm really sorry if you've been done wrong to by some church; ask for guidance for a new church which is closer to the will of God; instilling fear and increasing the power of the church collective is not God's vision for the Church, so He will provide you with a better alternative than the one which has (I assume) hurt you.
Author, Shell Scripting : Expert Re
Prayer is basically the same thing as Eastern meditation methods.
well, in theory it should be. in some cases it actually is. but for many 'religious' people it's just a way of trading something for absolution. see for instance the confession thing - even the catholic church itself introduced at some time (middle ages) the idea of redeeming your sins with money - buying indulgences, how precious!
even the idea of passing the responsability - since you don't have absolute control over your life, you might as well have none when it comes to really critical things. so one says 'it's in the hands of God'. but what about small things? putting oneself completely in the hands of God is something that can be found in lots of very old Eastern spiritual writings. sadly, how many people actually do that? since it requires one to give up the vanity of 'I did that and that' for everything. on the contrary, lots of 'christians' would assume 'ownership' of acts where their control was clearly close to zero, as marks of their 'value'.
oh well, religion would bring it to your door, but it's up to you to let it inside, I guess.
NTFS has nothing to do with DOS FAT. They are two completely different beasts.
With NTFS you can in fact replace a file that is in use by performing the same operation you have to use in *nix. You just rename the old file to something else, then copy the new file. The existing processes will continue to use the renamed file, while new processes will use the new file. You can stop and restart many applications by hand, or just reboot. Many application creators just required reboots because they didn't wan't to have to deal with other applications already running. You are guaranteed sure to get a "clean" machine upon reboot. Because of Microsoft Windows Installer Service, many applications no longer require the bloody reboots.
Some applications try to put locks on the file. You can use the following "InUse" command to fix even this...
InUse
In the case of processes that can't be restarted, like the kernel itself, you must use the Windows API called MoveFileEx().
Most Win32 coders do something similar to this...
MoveFileEx(SourceFile, DestinationFile, MOVEFILE_REPLACE_EXISTING + MOVEFILE_DELAY_UNTIL_REBOOT)
See...
MoveFileEX()
Mod the parent down please?
+2
...keep old versions around just in case.
/usr/local/apache/bin/apachectl stop; sleep 1; rm /usr/local/apache; ln -s apache-1.3.28-2.8.15-0.9.7c-1 /usr/local/apache; /usr/local/apache/bin/apachectl startssl
I'm in charge of a bunch of apache servers that use ssl and were potentially affected by the recent openssl bugs.
So, rebuild with --prefix=/usr/local/apache-1.3.28-2.8.15-0.9.7c-1 (apache version, mod_ssl version, openssl version, box build id) and whatever other options you choose. Copy a test config file that listens on another port over, start it up and run a testsuite that checks pages are accessible and do the right thing to the back-end systems. Stop new server, copy live config file in place and you're ready to go.
$
All sorted with a second or two of downtime. Watch the logs closely for a day or two. If anything breaks horribly stop apache, move the symlink to the old known good version & restart it, giving you some time to debug and add new testcases so the problem doesn't reoccur next time you upgrade. Repeat until new testcases pass, put the new version live and repeat.
Why doesn't the gene pool have a life guard?
The standard out-of-the-box Microsoft XP commands are now TaskKill and TaskList. Get to know them. You should get to know all commands that are a part of your OS first, then add-in third party apps later.
+5
As a sysadmin, your job is to keep the systems running for the people who are using it.
If it's an email server, a 2-minute (wow! that's a fast estimate!) downtime will mean that users do not receive emails. If sendmail (or whatever your critical process is) doesn't need to be restarted, don't bring it down.
You spend your day working out how to best maintain your systems, so that everyone else in the company can spend their day doing the "productive" work.
If "the users" spend their time, in your perception, swapping jokes and watching movies, that's a problem for their manager, not for you.
Get a grip, and a sense of the role of IT in a company. That's the difference between a nerd and an IT professional.
Author, Shell Scripting : Expert Re
more microsoft false security. When they log in?
Most worms I know run as services so even before they log in the virus is scanning the net looking for nodes to infect. This can easily happen between the time the user has booted and when they log in. In the corporate environments I've worked in it is a normal practice to boot, and then go get coffee and come back 10-15 minutes later.. that is more than enough time for the worm to be off and running.
NT was widely blamed because of its stability, leading to poor uptime, as most servers ran a week or so.
Newer Windows are very improved, but the uptime is short anyway, as you are patching the OS all the time.
Got Pike?
In general, ssystems should be pattched as soon as possible. Most of the patches are realeased becouse there is something out there, in the wild..
Anything that is accesible from the net should be able to be taken down anyway, at least when properly planned.
Servers should be redudant, at swappable.. One of the reasons for this is, quite simply, so that you CAN do this sort of thing, without interupting customers..
Not customer wants to hear that the entire system is FUBARED, and you have to build a new system..
shesh, doesnt anyone PLAN for this sort of thing? lol
-- I'm the root of all that's evil, but you can call me cookie..
Patches? They can go on immediately. What is this "reboot" you speak of?
What, you're not running a Unix system? Why not?
Ok, not to sound like a troll - some patches do require we reboot the system, but those are usually the fairly big ones. Often I have to shut down a service temporarily. But these are fairly minor to what I see the Win32 team doing.
But even there you can run clustered, or server farms, so you can update one system at a time and stay mostly available.
Not to get too simplistic, but if your business is critical enough that you need patches immeditately, you should be putting in place enough infrastructure that you can patch fairly painlessly anyhow.
=Blue(23)
LITTLE GIRL: But which cookie will you eat FIRST? C. MONSTER: Me think you have misconception of cookie-eating process.
Generally you don't have to reboot after each patch is installed. Most patches have both a silent and a 'don't reboot' switch which allow you to chain all your patches together and then reboot (Windows 2000 and up I believe).
Secondly, most corporations are using some form of software delivery mechanism such as Novells Zen Works, CA's Software Delivery option or IBM Tivoli to distribute patches to multiple PCs. Most of these systems allow lights out distribution.
no, our IT department doesn't like the word 'patch'. it's synonymous with the word 'work'. my horror story is despite the massive campaign to patch against MS Blast, they only patched the system after 80% of our 2,000+ terminals were infected; and then had to individually remedy each infected PC directly.
1. Caps have a place.
/etc/init.d/apache restart
- At the beginning of sentences and proper nouns.
2. Have you tried running a server with Gentoo?
- I dont think so, if you had you would have realised that there are some handy features in updating a system.
-(And a solid user base whom want to help each other; you don't seem like you want to.)
3. Have you look at the versions of software in Debian-Woody(Stable)?
- If you had you would have noticed that its not that current, I have some problems in waiting for a new version of software to be realeased, security. So you need to update apache..
# emerge apache ;
Oh look im updated, I know that debian and others have similar sytems but the benifits in working from sources it time in waiting for that binary release.
Anyway I'm waffling on now..
DanB
--------------------
fb-livecd - Custom LiveCDs ?
freebox - Small Dev Hole
Have you tried running a server with Gentoo?
..of 14 year old l33t d00d h4x0rs who don't know what all those compiler switches do, but they're sure it makes them l33t if they have more of them.
/etc/init.d/apache restart
Hopefully not. People should get fired for that shit.
And a solid user base..
emerge apache ;
You did ensure that the version you're compiling from doesn't contain any additional patches and hasn't broken any mission critical function, havn't you?
Oops.
Anyway I'm waffling on now..
Whats new?
P.S: You'll find it's spelled Gentoy. Hope that helps!
If the patching process is easy, then people can patch more frequently.
OTOH, people won't be able to patch as frequent if the process is not-really-that-easy.
Most people do understand the need to patch, but many of them really don't know How To do it.
That's the gist of it, AFAIK.
Muchas Gracias, Señor Edward Snowden !
In the corporate environments I've worked in it is a normal practice to boot, and then go get coffee and come back 10-15 minutes later.. that is more than enough time for the worm to be off and running.
First off, the post is about remote log ins, not sitting in the office log ins (though you should speak to your management 10-15 minutes is a lot of wasted productivity for a cupa joe). They log into a special server which is isolated from the network, this server analyzes both their virus-defs and patches as well as any custom checking, when that check is good they are then given a one time connection (ie set up then and discarded later) connection to the network. It is basically a more advanced version of Ciscos Temporary ACLs (more advanced in that they are setup automatically, not more advanced in function).
I must post this in order to avoid seeing the number of the beast in my browser
I appreciate your insight and agree with you insofar as the same argument can be made that anything is sometimes "bad". I've not been wronged by the church myself in any way, and I am a Christian. Its just my belief that one's relationship with God and Christ is a personal one, and that I don't need to congregate with a bunch of others in order to express that belief. On Sundays I often donate my time to helping people instead.
What could possibly hurt the security of the American people more than giving our own government the ability to hide its
Whenever I get a Microsoft patch upgrade notice, I immediately drop what I'm doi
Turn off DCOM and you won't have to worry about viruses like Blaster or Welchia. Run -> dcomcnfg -> Default Properites -> Uncheck,"Enable Distributed COM Services".
My former company was a spam company, but yet it was maintained by a non-security concious boss. it wasnt until 3 months after I left (after being there for over 2 years) that they finally cracked down and got a firewall. Installing said firewall after the business was in full swing.. yea. I dont envy the monkey that took my place.
But the same standard was applied for patchs, we were told to -never- reboot certain dbs.. which HAD to have external ips, and no firewalls. Yupp, you got it, live db's with thousands and thousands of credit cards owned by a spam company... a 'sort of' big target eh?
Yea, so when the db's died due to being owned due to lack of patchs, it was no supprise that we were yelled at and held responsible by the same person who continually told us not to reboot. And if you went over his head to get permission for varios VERy important IIS patchs, you were told to reboot the server "RIGHT NOW YOU IDIOT" by the boss, due to the fact he didnt know what was going on but yet wanted to pretend he did.
In short, non IT people shouldnt be involved in the patch/security process, PHB's suck.
Personally, at my new job, i'm in charge of co-ordinating any deployments to new servers, and the change is refreshing, with the amount of firewalls, its not as a desperate situation too.
If its a major patch, 3 days to upgrade if it requires a reboot, just to notify all people working on it. If its a transparent change, possibly 24 hours and they're fully deployed.
Of course, I only work with unix now, (thank god) I dont know how the intel side handles their issues.
Possibly they pray?
Welcome to the End
If you really believed that, you wouldn't have them running Windoze.
I've found etc-update to be a total waste of time. I've figured that a lot of files: /etc/make.conf /etc/samba/smb.conf /etc/host* /etc/exports /etc/fstab
/etc/init.d doesn't even get reviewed, I don't costomize them and they only start and stop services, I overwrite the old files with the new without looking. /etc/conf.d/* gets diffed if I know I've changed the file from the default. otherwise I just overwrite.
and many more
These DONT change, there's no need to upate them , just throw away the 'new' default files.
Everything in
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
I see no point in spending hours downloading massive quantitys of secutiry updates for holes that have been in windows for years (Microsoft doesnt rewrite their code, they just patch it, give it a new box and up the price) because windows is too unstable and normaly tends to die fairly quickly (I have to reinstall everything on my entire system because win2k has decided that 30min after startup it doesnt want to run any programs). on some occasions i have had to reinstall windows up to once a week. I dont use windows any more; too buggy and insecure (154 pages of logged access attempts by my firewall in 10 minutes isnt acceptable, let alone having to update the virus scanner every day). I find linux much nicer to run. dont have to have a firewall or virus scanner
I occasionally patch windows (when i actualy use it) but i find that windows is more of a threat to its self than all the hackers and viruses in the world combined. Windows seems to be suicidal and likes killing its self or doing stupid things that requires it to be reinstalled about once every three weeks(Windows has given every single device in my system IRQ 9 which causes big probs and wont let me change it)