Slashdot Mirror


The Costs of Patching

prestidigital writes "vnunet has a brief but interesting article in which Craig Fiebig, general manager of Microsoft's security business unit, is quoted as saying "In dollar terms, patching is the most expensive security measures and keeping your antivirus descriptions up to date is the least." That seems like an important statement coming from a company who's patches are possibly responsible for 45% of traffic on some networks."

39 of 303 comments (clear)

  1. Wow...it took them this long... by Fallen+Kell · · Score: 4, Insightful

    ... to realise that it costs more to do things 2, 3, or 4 times then if they had done it right the first time...

    And that is costs more to have a new programmer look at and try to modify code that wasn't written by himself/herself...

    Amazing reality breakthrough!

    --
    We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
    1. Re:Wow...it took them this long... by Surak · · Score: 4, Insightful

      The real cost, aside from downtime, is in the integration testing of those patches. If you don't do the integration testing, the cost is potentially even HIGHER because you don't know what those patches could break. Unfortunately, doing proper integration testing means you end up way behind the curve in terms of the patch cycle, which ultimately means an even greater risk of attack.

      So you're damned if you do and you're damned if you don't.

      Hey, I know, maybe Microsoft could do this new thing called PROPER BETA TESTING, and then maybe the could get it right THE FIRST TIME!

      Nah, that'd be too easy. ;)

    2. Re:Wow...it took them this long... by vsprintf · · Score: 3, Funny

      Hey, I know, maybe Microsoft could do these new things called DESIGN REVIEW and CODE REVIEW, rather than trying to test out bugs.

      You must have missed it. After Bill declared a new focus on security, they did a code review -- one month of review for twenty years of code. The next code review is scheduled for 2022. :)

  2. Also known as... by Evil+Adrian · · Score: 3, Insightful

    This statement is also known as "an ounce of prevention is worth a pound of cure."

    --
    evil adrian
  3. Patching has saved my hundreds of dollars by Anonymous Coward · · Score: 5, Funny

    Rather than throwing away an otherwise perfectly good pair of pants, patches have allowed me to fix them and extend their life. In some cases, patches can even be fashionable. Sewing is a great skill that all geeks should learn.

    1. Re:Patching has saved my hundreds of dollars by Ravenscall · · Score: 3, Funny

      Well, it certainly is more socially acceptable than damning them

      --
      You say you want a revolution....
  4. Cost of not patching? by rhfrommn · · Score: 5, Insightful

    The difficult question is whether the costs of patching outweigh the costs of NOT patching. There's a lot to be said for "if it ain't broke, don't fix it" sometimes.

    However, with security patches usually you have no choice. The only decision for some security patches is how long do you wait before deploying it. Don't wanna be the first ones to put a bad patch on now, do we?

    --
    My motto is: Never give up - unless it's harder than you want it to be.
    1. Re:Cost of not patching? by H310iSe · · Score: 4, Informative

      Whenever deploying new patches OR antivirus DAT files (they cause havok as well) we did a full regression test of the standard desktop image.

      Fist a high level person would look at the patch (usually using install shield's application repackager), read the documentation, etc. and look for possible conflicts with the production environment. This took between 2-4 hours per patch x $60/h. The regression test took one lower-level tech about 2 days to do. We'd lump a few patches together so say 1 tech x $40/h (at least, w/ benefits, etc.) x 2 days / 3 patches per test = about $213/patch + eval ($180 per patch) = around $400 per patch to test. Deployment took another hour to write the install script (rarely did we rely on MS's installer alone), 1 hour to document and send to the regional offices and each office probably spent an hour implementing the thing. Total cost around $600 per patch for a 1,000 desktop, 11 office environment.

      Now you know.

      --
      closed minded is as closed minded does
    2. Re:Cost of not patching? by B3ryllium · · Score: 4, Funny
      Fist a high level person
      Yeah! That's right! Fight the power!
    3. Re:Cost of not patching? by pmz · · Score: 3, Insightful

      Then you'll get to have fun updating libraries whenever you want to install something, as well as patching BIND, sendmail, the kernel, etc.

      It doesn't have to be all that bad. Packages are relocatable, so unusually sensitive applications can be put into their own root directory hierarchy. Using NFS wisely can allow for one set of applications on a network (patching once and only once is quite nice). Only one or two servers on the whole network should be running Sendmail and BIND in a vulnerable mode. UNIX is also easier to pare down, so there are much fewer things that need to be patched. With a good network design, patches can be rolled out automatically over SCP, and UNIX machines tend to reboot pretty reliably, unless a patch screws up an init script.

      It is just a simple fact that UNIX is less complex than Windows. It has fewer lines of source code, more transparent modularization, strict separation between the GUI and the kernel, widely available and thorough documentation, three decades of experience behind it, almost complete scriptability, among other things. Windows, on the other hand, is as opaque as mud--there could be a golden city under there or just more mud, but we'll never know.

  5. interesting debate by ih8apple · · Score: 4, Funny

    This document was part of an interesting debate over the last year and a half between MS and Novell over whose product was more buggy (measured in terms of number of patches.)

    (Google cache version in html.)

    1. Re:interesting debate by zero-one · · Score: 4, Funny

      Yup, that document was funny. I liked this bit: "Additionally, Novell has neglected to be clear about the fact that GroupWise runs on Windows NT and Windows 2000, so patches that apply to Exchange customers also apply to GroupWise customer running a GroupWise system on Windows systems". So Microsoft are arguing that Novel haven't taken full account of the security issues due to Microsoft in a report bashing Microsoft. I am not sure that is an argument that Microsoft should be shouting about!

  6. Patching most expensive by jonfelder · · Score: 3, Interesting
    Well...patching is also one the most important things you can do with regards to security. So at least in this case the expense is justified. Although patching is annoying, until people learn how to write perfect code it is a necessity.

    IMHO getting hacked is much more expensive.

  7. NEW MATH by stratjakt · · Score: 5, Insightful

    responsible for 45% of traffic

    But spam is responsible for, what was it Taco, 60% of traffic on networks?

    I'm at 105% utilization already!

    BTW, it's just as costly, if not more, to have to rebuild your linux kernel, SSL, apache webserver, or samba installation when a bug is found there.

    Quit pretending that MS has some sort of monopoly on software bugs. "Bad code" is a patentless technique used ubiquitously.

    --
    I don't need no instructions to know how to rock!!!!
    1. Re:NEW MATH by aridhol · · Score: 5, Insightful
      Don't forget the 70% that is porn.

      Let's face it. There's no real way to know for sure what is on those wires unless you monitor them. And I don't think anybody here wants to open that can of worms.

      --
      I can't say that I don't give a fuck. I've just run out of fuck to give.
    2. Re:NEW MATH by clambake · · Score: 4, Funny

      responsible for 45% of traffic

      But spam is responsible for, what was it Taco, 60% of traffic on networks?

      I'm at 105% utilization already!


      Didn't you see that the article was about Microsoft? I'm sure there is at least SOME overlap in the spam/patch metrics.

    3. Re:NEW MATH by Pyrosz · · Score: 4, Insightful

      If your going to bash someone, make sure you are correct first. Taco did not write that comment and you didn't even read the entire comment correctly as it states "...possibly responsible for 45% of traffic on some networks." If Taco had written the comment it would not have been in Italics.

      --

      An optimist believes we live in the best world possible; a pessimist fears this is true.
  8. Nothing new there by Timesprout · · Score: 5, Insightful

    The software industry has known for years that the later you find a bug the more expensive and messy it is to resolve

    --
    Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
    What truth?
    There is no dupe
  9. Not suprising by Neophytus · · Score: 5, Insightful

    People who say 'they should have patched' do not understand the stress that installing a patch however critical on a few hundred servers, then in many cases rebooting them, can put in a commercial environment.

  10. Lamers by grub · · Score: 4, Funny


    Pff.. you lamers with your fancy-pants Windows or your free Linux or *BSDs are all clueless. I haven't patched my Apple ][+'s DOS3.3 for 20 years and it still has yet to be 0wned.

    --
    Trolling is a art,
  11. Patches by zzxc · · Score: 3, Insightful

    If MS wouldn't include so much "junk data" to keep their proprietary data secret in patches, they wouldn't be so large. And, if there was a way to do a patch "rollback", then faulty patches wouldn't bring down a system until a new fix-patch was released. (One of the recent MS patches was found to cause some machines to stop booting)

    -----------
    From Ape to Man: Evolution

  12. Re:Lamers - Oh Yeah? by freeze128 · · Score: 3, Funny

    Try to enter today's date in Appleworks.

  13. MS patches are creepy... by allanj · · Score: 5, Insightful

    I've applied my fair share of patches from MS, but lately I've become really nervous about doing so. I'm always thinking "what kind of DRM will they include in this one?". It's gotten to the point where I will NOT apply patches for anything but server products, and only reluctantly so. Call me paranoid if you wish, but I can't really shake that feeling. Hey MS, great way to promote security - making users reluctant to apply patches...

    --
    Black holes are where God divided by zero
  14. it make sense they would say this.. by geekoid · · Score: 3, Informative

    ..because one of the many new feature of server 2003 is the ability to update patches auotmatically.
    So they will use this 'cost savings' to push the new product. At the launch event, they bagged on there older products pretty damn hard.

    It's part of there latest slogan
    "do more with less".
    personally, I dln't know who this less guy is, or why I would want to do more with him. Ironically I prefer less to more.

    --
    The Kruger Dunning explains most post on /. http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
  15. Re:Well then... by Bendebecker · · Score: 3, Funny

    And maybe I should spend more time proofreading my own posts so that I don't mangle words so much!

    --
    There's a growing sense that even if The Future comes,
    most of us won't be able to afford it.
    -- Lemmy
  16. Re:I prefer Linux, but... by BlueTooth · · Score: 5, Informative

    RedHat's up2date works pretty well so long as you stick to their RPM releases of the software you want to keep updated.

    It works well for me, and all I need to stay on top of are things I build be hand (typically Webserver and its ilk plus kernel), but all my libraries stay nice and fresh.

    --
    SPAM
  17. Re:I prefer Linux, but... by Nothinman · · Score: 4, Informative
    Sometimes I wish there was the equivalent of Windows Update for Linux


    apt-get update
    apt-get upgrade


    I don't run Debian's precompiled kernels though so I don't know what the patch/release policy on them is, but for all userland things it's better than WU.

  18. Re:Lamers - Oh Yeah? by grub · · Score: 5, Funny


    Yeah and? Today is Thursday, May 1 10003.

    --
    Trolling is a art,
  19. I feel the pain by Remlik · · Score: 3, Informative

    As the only sys admin in a company of 50 desktops and 4 Win2k Servers I can fully support the notion that patching is expensive...but not for the company...for ME!

    Guess who gets to come in the office between 8 and 10pm to apply these patches to live servers...who has to wait if someone decides to work late. Who has to cross his fingers with every patch hoping that nothing else breaks...ME! And the only thing I get out of it is to be able to leave an hour or two early that friday...woot.

    Sure some things I can and do install from remote, but almost every patch requires a reboot and you just never know when a Win2k system isn't going to boot properly and require you to drive in at 1am wearing your bath robe.

    --
    Apple free since 1990!
  20. System Update Server by mr_z_beeblebrox · · Score: 4, Informative

    Microsoft has a free product out called SUS (see subject) the SUS works in conjunction with the BSA (no, Baseline Security Analyzer) to determin patch levels of 2000/XP clients and servers it then downloads all neccessary patches in a SIS (single instance storage) at the server. In this way every patch on your network is downloaded only once. If you only have four PCs this cuts update traffic by 75%. This is nearly as effective as ISA server but it is FREE. It is not as effective as coding it write the first time LOL but it is a start.

    1. Re:System Update Server by Lumpy · · Score: 3, Informative

      I get the same thing for free with linux by simply configuring a SQUID proxy to heavily cache all http traffic from update.microsoft.com. then have all the W2K boxes automatically run their updates 1 day after the master PC does this. Voila... the squid proxy caches all the updates so every other pc in my WAN get's them from the squid server lightning fast. same as the virus scan and all the other "update" sites we seem to use here.

      Works great and my mrtg graphs prove that it works to me :-)

      --
      Do not look at laser with remaining good eye.
  21. Say it ain't so! by RealAlaskan · · Score: 5, Interesting
    apt-get update
    apt-get upgrade

    That's what I do, and I'm not sure what all the fuss is about. Things get fixed, usually before I ever knew they were broken, deamons get restarted, nothing gets interrupted, life goes on ... If I took the trouble to make it a cron job, I'd never even know.

    ... Craig Fiebig, ... is quoted as saying "In dollar terms, patching is the most expensive security measures ...

    Is Mr Fiebig telling us that things don't go so smoothly if you use MS products? Or that MS can't keep up with a bunch of amatures? Do MS patches break non-MS apps? Could all this be why so many worms and viruses manage to spread across unpatched MS products? Could it be that MS patches are as bad as the bugs they fix? SAY IT AIN'T SO, CRAIG!

  22. Re:Downtime? by robbo · · Score: 4, Insightful

    I dont think a apt-get update && apt-get upgrade in cron is that hard work.

    Yikes. I don't think 'apt-get update && apt-get upgrade' in your crontab is very smart. The probability of breaking something is too high. In fact, that's the message I'm reading between the lines: virus upgrades won't break anything, so they're no problem to automate, but OS/IIS/IE patches pose a much higher probability of risking extended downtime. I don't think the situation is all that different with the Red Hat Network-- look before you leap.

    --
    So long, and thanks for all the Phish
  23. Re:I don't understand... by Zirnike · · Score: 3, Insightful
    Business application math:

    (Some patches break some applications) + (Applications being down means lost productivity, sales, possibly data, depending on the app) + (MS apps won't let you roll back the patch, so you can't recover) = Many companies feel the need to test the patches first.

    My computer at work doesn't get patched all that often (luckally it's behind multiple firewalls), because Unigraphics is very touchy (according to our support people).

    --
    I'm not shy, I'm stalking my prey
  24. Question? by Billly+Gates · · Score: 4, Interesting
    C/C++ functions like strngcopy have been known to be a cause of overflows for decades.

    Bell labs(now lucent) and various hackers have made string functions that do the same thing but are buffer safe. They are made to create more secure apps.

    My question is if gcc or visualc for that matter switched to more buffer safe libraries would it make a difference? Trusted Debian is compiled with buffer safe string functions.

    It may be time gnuc did this by default assuming all the apps could be recompiled without a problem.

    This would seem to get rid of %90 of holes in user as well as kernel space.

    1. Re:Question? by PickaBooga · · Score: 3, Informative


      The basic string copy functions in C and C++ don't keep a value for the maximum length of a string.
      (Actually, they don't even keep a value for the current length of a string, it is calculated by scanning the string and looking for the terminating null.)

      The buffer safe string libraries are not designed to be a drop-in replacement for the basic string library, because they demand more information about maximum lengths from the code using them.

  25. patches hurt people too by dwgranth · · Score: 3, Funny

    Man, I can attest to this... patches... especially ones that screw up systems not only cost time/money/bandwidth but they cost HAIR.. yes thats right... admins lose their hair b/c of the stress this makes them go through..... ::looks in the mirror:: arrhhggghh..

  26. Hmmm... by istartedi · · Score: 3, Insightful

    Well... before the knee-jerk MS-bashing starts, let's think about it.

    If you patch, you have to recompile the component, and possibly re-boot the machine or re-start the application. This is true for Linux too (unless there's a way to fast-swap kernels that I haven't heard about).

    If you update, you don't need to re-start anything.

    If you patch, you could have to patch just about anything on the system.

    If you update, you are working through one application.

    Of course, there's nothing to stop an OSS developer from writing something that just sniffs incoming data for known exploits, like a virus scanner does.

    Ahhh... but that would slow the system down.

    So I think you have to add "better performance" to the pro-patch argument.

    But then, there is probably less effort to updating, especially if it's automated. Is there any OSS system with automated patching that people are willing to trust?

    Either way, I think it's an interesting discussion. In practice, I patch.

    --
    For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
  27. Re:I prefer Linux, but... by argel · · Score: 3, Insightful
    You still have to take the machine offline for all practical purposes. You cant upgrade samba or apache in place, without interrupting service. So who cares if the downtime is for a reboot or a recompile? From the users point of view the machine is inaccessable
    You've never had to reboot a system with several SCSI drives in it, have you? The difference between cycling a daemon and cycling the box can be considerable.
    --

    -- Argel