Slashdot Mirror


Sophos Anti-Virus Update Identifies Sophos Code As Malware

An anonymous reader writes "Yesterday afternoon anti-virus company Sophos Inc. released a normal anti-virus definition update that managed to detect parts of their own software as malicious code and disabled / deleted sections of their Endpoint security suite, including its ability to auto-update and thus repair itself. For many hours on the 19th, Sophos technical call centers were so busy customers were unable to even get through to wait on hold for assistance. Today thousands of enterprise customers remain crippled and unable to update their security software." Sophos points out that not everyone will be affected: "Please note this issue only affects Windows computers."

43 of 245 comments (clear)

  1. 99.999% by jsepeta · · Score: 5, Insightful

    how many of Sophos customers are not on the Windows platform? that makes me laugh.

    --
    Remember kids, if you're not paying for the service, YOU ARE THE PRODUCT THAT IS BEING SOLD.
    1. Re:99.999% by niiler · · Score: 3, Funny

      At first I thought you meant "proof of concept" anti-virus for Linux. :-P

    2. Re:99.999% by thereitis · · Score: 4, Insightful

      Speaking of percentages, I wonder what percentage of anti-virus updates go terribly wrong like this. 0.00001%? AV companies are constantly producing new signatures, many times per day. All it takes is one mistake and you have a loose cannon and a front page news article like this one. It's impressive that there aren't more occurrences.

    3. Re:99.999% by DaveAtFraud · · Score: 3, Funny

      I'm just glad I didn't have a mouthful of coffee when I read:

      Sophos points out that not everyone will be affected: "Please note this issue only affects Windows computers."

      or I would still be cleaning coffee off of monitors, laptop, papers, etc.

      I have a couple of old Windows XP installations I can still get to when some idiot creates a web site that only works right in IE (e.g., I live in Colorado and the state has a site for doing your state income tax that doesn't work when accessed with Firefox). Ditto for software like most income tax programs. I don't otherwise use Windows. Even my work laptop is running Linux (Fedora 16).

      Cheers,
      Dave

      --
      They that can give up essential liberty to obtain a little temporary safety deserve neither safety nor liberty.
      Ben
    4. Re:99.999% by Verunks · · Score: 5, Informative

      So far, there have only been a couple 'proof of concept' viri for Linux. Nobody's figured out a way to pry any money away from us yet. :D

      but linux antivirus aren't used to protect linux, they are useful if you run a mail server or a proxy so you can clean mails and webpage before they infect a windows user, or to clean an infected windows installation, for example the kaspersky live cd is based on linux

    5. Re:99.999% by Culture20 · · Score: 4, Insightful

      What's impressive is that this got out of Sophos' testing lab and into production. I guess they must not test signatures in house at all. Congratulations, Sophos customers, you've been promoted to alpha testers.

    6. Re:99.999% by fuzzyfuzzyfungus · · Score: 4

      The trouble, in this case, is that it detects its own signature update componenets as viruses...

      Not only should this have been caught in testing(Since it would have cropped up more or less the moment the new signatures were loaded onto a live system with Sophos installed; but they hit files about which sophos presumably has intimate knowledge, this isn't some 'obscure packing/compression scheme used by legacy CAD program that seemed like a good idea in the 80's looks like a suspicious obfuscated payload' kind of thing.

      I am not impressed, though thankfully it only took me a little over half a day to fix it here...

    7. Re:99.999% by jd2112 · · Score: 5, Funny

      What's impressive is that this got out of Sophos' testing lab and into production. I guess they must not test signatures in house at all. Congratulations, Sophos customers, you've been promoted to alpha testers.

      Actually, it's an incredible show of honesty on the part of Sophos. Perhaps Symantec and McAfee will follow suit and flag their own software as malicious as well.

      --
      Any insufficiently advanced magic is indistinguishable from technology.
    8. Re:99.999% by IceCreamGuy · · Score: 2

      While this may work for you, being a careful, knowledgeable slashdotter that I'm sure you are, it would be unthinkable in a business environment. Sophos only makes business products, there is no "Sophos Home Edition," so I don't think your method really applies in this case.

    9. Re:99.999% by arth1 · · Score: 2

      They can be. The first ever virus was written for UNIX.

      Unless counting a self-replicating failure on an early Manchester machine, the first virus we know of was from 1971, and ran on TENEX on a modified PDP/10. No UNIX (or Unics).

      The first virus outside arpanet or labs infected Apple systems, by the way.

    10. Re:99.999% by JaneTheIgnorantSlut · · Score: 2

      You probably are thinking of "Sappho"

    11. Re:99.999% by RDW · · Score: 3, Funny

      What's impressive is that this got out of Sophos' testing lab and into production.

      What's really impressive is that is that it also orchestrated a DDOS attack on the Sophos tech support helpline...

    12. Re:99.999% by LordLimecat · · Score: 2

      Certainly it makes it one of the easiest to remove antiviruses, which is a pretty major AV feature in my book.

    13. Re:99.999% by isorox · · Score: 2

      My work requires av to be installed. No mention that it's files can't be chmod 000ed though :)

    14. Re:99.999% by Smallpond · · Score: 2

      I got hit by malware on Redhat years ago (the L10n worm) so it does happen.

      Anyway, I have a corporate Win 7 desktop with Sophos now and got this bug. Every few minutes it popped up a warning that I had been infected with malware. Very annoying. By the end of today it had stopped, so either IT had fixed it or it had managed to commit suicide. The one time I did get infected with malware on this PC Sophos didn't catch it and I had to download Malwarebytes and fix the registry myself.

  2. Can We Say Test our Code, anyone??? by realsilly · · Score: 4, Insightful

    This is a classic case of not thoroughly testing code and making sure you have enough variations of test machines to ensure as little pain to clients as possible.

    If I were a customer, I would be shopping for a better company.

    --
    Life takes interesting turns, but the most interest is when you're off the beaten path.
    1. Re:Can We Say Test our Code, anyone??? by MrEricSir · · Score: 3, Insightful

      If I were a customer, I would be shopping for a better company.

      Is there a better company, though? Seems like all the major antivirus vendors have had embarassing false positives like this in the past.

      --
      There's no -1 for "I don't get it."
    2. Re:Can We Say Test our Code, anyone??? by girlintraining · · Score: 3, Informative

      This is a classic case of not thoroughly testing code and making sure you have enough variations of test machines to ensure as little pain to clients as possible.

      Antivirus engines and definitions change daily, weekly at the most. Where do you suppose this "thorough testing" of code is supposed to happen? It costs time and money, and while you're busy doing that testing, the support lines are being flooded with "We've been infected by something your software doesn't protect against! What are we paying you for, anyway?" As a bonus, your competitors, who didn't decide to setup a massive lab with dozens of employees in it, testing all the typical configurations of a half dozen operating systems and the couple hundred most popular software packages of each... they already released a patch.

      Now, a software patch that causes the application to stomp on its own dick is amusing (and difficult to forgive), but demanding a massive expenditure of time and money is almost as unforgiveable. It's easy to demand best practices and ample safety margins: It's quite another thing to deliver it in a business environment. Most people in the industry, including the people at Sophos I'm sure, do the best they can with what they're given. It's pretty much the work creed of anyone in this industry -- few have the time and resources to do it right, they have to settle for 'good enough'.

      And sometimes, good enough breaks.

      --
      #fuckbeta #iamslashdot #dicemustdie
    3. Re:Can We Say Test our Code, anyone??? by Culture20 · · Score: 2

      A simple group of ~20 VMs could handle this egregious type of error. Who cares if AV X marks some specialty software with a false positive? It should at least not detect itself! Load the new sigs to the test VMs, and if they don't commit suicide after a full scan, upload the sigs to the prod download servers. At most, this costs a company ~$5,000/year for equipment and ~$40,000/year for labor. That's pocket change compared to how much the company can lose over a screw up like this.

    4. Re:Can We Say Test our Code, anyone??? by girlintraining · · Score: 4, Insightful

      That's pocket change compared to how much the company can lose over a screw up like this.

      Emphasis mine. Look, every major antivirus producer has made a similar mistake to this. Sometimes, it takes the whole operating system down with it (Symantec anyone?). Whether you agree or disagree, it's clear there are business incentives for a fast workflow process -- and as the old saying goes "Do it fast, do it right, do it cheap -- pick any two." It's obvious which ones the antivirus industry as a whole has chosen. Rather than argue over whether or not they're right, I'm pointing out why they're making those choices. Businesses aren't willing to pay a premium to avoid mistakes like this. The cost of the occasional screwup like this is less than the cost required to do all the testing and lab work that many here on slashdot seem to support.

      It's a business decision they've made, right or wrong.

      --
      #fuckbeta #iamslashdot #dicemustdie
    5. Re:Can We Say Test our Code, anyone??? by osu-neko · · Score: 2

      The fundamental problem is that, no matter what you do, your testing environment is never a perfect replication of the live, end-customer environment. It cannot be, since it's required by virtue of being a testing environment to differ so that you can test things before they go live. What happened here is, the testing environment's method of distributing updates to test differed from live (which it must if it is to be able to test definitions that aren't live yet), and the problem didn't affect the testing environment's updater. Could this have been avoiding in this particular case? Certainly. Can you invent a system that prevents this from ever happening in any case? No, that's literally impossible to do. No testing environment that fulfills the requirement of being a testing environment can exactly replicate live, and thus it cannot possibly avoid all possible cases of behaving differently than live (since it must do so), and that could include cases where something goes wrong.

      --
      "Convictions are more dangerous enemies of truth than lies."
  3. Re:QA? by MickyTheIdiot · · Score: 2

    they're running Avast free version like everyone else.

  4. Tautologies are fun by dkleinsc · · Score: 4, Funny

    Obviously, once this change had gone in, Sophos was correct to identify itself as malicious.

    --
    I am officially gone from /. Long live http://www.soylentnews.com/
  5. Re:Don't they test these things before deploying?? by jamstar7 · · Score: 2

    Considering all the people I know that still want to stay with XP no matter what, it doesn't surprise me at all.

    --
    Understanding the scope of the problem is the first step on the path to true panic.
  6. software leukemia! by scharkalvin · · Score: 2

    Let's see this isn't a virus, it's kinda like software leukemia or a software autoimmune disease.

    1. Re:software leukemia! by idontgno · · Score: 2, Funny

      It's not software lupus. It's never software lupus.

      --
      Welcome to the Panopticon. Used to be a prison, now it's your home.
  7. In other news... by MachineShedFred · · Score: 3, Funny

    The detection rate for Sophos's malware engine inched closer to 100%.

    --
    Slashdot still doesnâ(TM)t support Unicode after it was added to the HTML standard in 1997.
  8. Re:That's why I don't install AV software on my PC by asmkm22 · · Score: 5, Funny

    That's like saying you don't use condoms because you know how to pull out.

  9. Re:Which just goes to show... by localman57 · · Score: 5, Funny

    "test by eyeballing the code" has its drawbacks.

    Exactly. Sometimes code that looks useless is really pretty important. The article follow up said they removed this test from an iteration loop, since there weren't comments about what it did. Apparently the original programmers thought it obvious...

    if ( asimov_3rd_violation())
    {
    continue;
    }
    else
    {
    remove_file(filename);
    }

  10. There needs to be an award for this by phrackwulf · · Score: 4, Interesting

    Every year, we need to go down the list of software makers who have managed to totally Bork their users. The Meltdown awards. Just to distinguish between the companies that handle it well and the companies that are incompetent.

    --
    What would Richard Feynman do, if he were here right now? He'd do some math and he'd follow through!
    1. Re:There needs to be an award for this by SandyBrownBPK · · Score: 2

      YESSIR! the Slashdot Meltdown/Brick award! Let's do it!

  11. Re:That's why I don't install AV software on my PC by localman57 · · Score: 5, Funny

    My cousin used to say the same sort of thing about his know-it-all supervisor at work that was always riding him to wear safetly glasses. After he got back from disability, the guy got him a couple of tickets to Avatar in 3d, just to be an asshole.

  12. Re:That's why I don't install AV software on my PC by Anonymous Coward · · Score: 3, Insightful

    No, it's like saying you don't use condoms because you only go to bed with people you know well enough to trust them when they say they're on the pill.

  13. Re:That's why I don't install AV software on my PC by jones_supa · · Score: 2

    I would say it's like having sex without a condom with a long-term partner who you trust not to carry diseases.

  14. Malware makers take note! by erroneus · · Score: 4, Interesting

    Wanna cause problems? Add code from the various AV vendors...

  15. Re:Which just goes to show... by roc97007 · · Score: 2

    This should be obvious to any geek! What is Asimov's 3rd law? All together now: "A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws."

    I've never seen the code in question, and it's obvious to me that this means "don't delete myself".

    --
    Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
  16. Re:Which just goes to show... by localman57 · · Score: 3, Insightful

    Just so this whole thing doesn't spin out of control, the code is total bullshit I made up myself. Seemed better than just posting a comment about the 3rd law.

  17. Windows AV programs are malware by dskoll · · Score: 3, Interesting

    Just think about it. The average Windows AV program runs with sufficient privilege to wreck your system by altering or removing arbitrary files. And it gets fed multiple updates per day created by teams of workers working in a hugely stressful situation: When a new virus appears, you've got to get those signatures out NOW.

    I'm amazed people don't see this risks in this.

    1. Re:Windows AV programs are malware by Bobfrankly1 · · Score: 2

      Do you take the risk with the company paid to help you? Or the risk with random dude out to clean you out?

      Neither. I don't run Windows AV software and I don't run Windows.

      Ah. You take the other risks that I missed. Gotcha.

  18. Re:That's why I don't install AV software on my PC by TheLink · · Score: 2

    AV users have a very similar situation too. They have no infections that they or their AV software know of.

    You might assume the AV vendor is really good at spotting malware, but their job is like solving the halting problem, only without knowledge of the full inputs and program.

    I on the other hand prefer to "solve" the halting problem by ensuring the program actually halts no matter what happens- aka Sandboxing.

    --
  19. Re:Le me get this straight... by mcgrew · · Score: 2

    Measure twice, cut once!

    That's the old, craftsmanship way. These days, especially with software, it's measure with a micrometer, mark with chalk, cut with an axe.

  20. Re:How to Fix by SimplexBang · · Score: 2

    Using the identification of the client logmails : - I remotely stopped the Sophos Av service and auto update service - removed agen-xuv.ide - copied the full autoupdate folder contents to the remote pc - restarted both services After a few pcees I wrote a batch script to handle all clients Got it done after 5 hours work

    --
    Avoid your fears , or wonder at the past
  21. Notes from an effected enterprise by illtud · · Score: 4, Informative

    Yes, this was bad. The virus signature in question appears to match any software that does auto-updates (possibly trying to spot phone-home malware?) so it's flagged dozens of software packages and according to what policy you've set, quarantined or deleted the files. This includes the auto-update part of the sophos client. The flood of emails from the sophos enterprise manager package as machines were switched on this morning quickly alerted us that this wasn't good, and just looking at names of the files it was flagging was enough to see that this was a false positive. Cleanup continues.

    We've been very happy with sophos enterprise, and I'm staggered that this signature made it out the door - they should have numerous controls in place to ensure this can never happen and I await an explanation for how they failed.

    I'm not too impressed by some of the advice given in their cleanup procedure - they advise setting the policy to not scan certain sophos directories - guess where viruses may try to hide in future.

    This is an embarassing fubar which will have had a high impact on thousands of enterprises. It'll be interesting to see if Sophos come clean about the circumstances and can be convincing enough about how it's never going to happen again.