Sophos Anti-Virus Update Identifies Sophos Code As Malware

An anonymous reader writes "Yesterday afternoon anti-virus company Sophos Inc. released a normal anti-virus definition update that managed to detect parts of their own software as malicious code and disabled / deleted sections of their Endpoint security suite, including its ability to auto-update and thus repair itself. For many hours on the 19th, Sophos technical call centers were so busy customers were unable to even get through to wait on hold for assistance. Today thousands of enterprise customers remain crippled and unable to update their security software." Sophos points out that not everyone will be affected: "Please note this issue only affects Windows computers."

Comments

99.999%

[jsepeta]

how many of Sophos customers are not on the Windows platform? that makes me laugh.

Re:99.999%

[thereitis]

Speaking of percentages, I wonder what percentage of anti-virus updates go terribly wrong like this. 0.00001%? AV companies are constantly producing new signatures, many times per day. All it takes is one mistake and you have a loose cannon and a front page news article like this one. It's impressive that there aren't more occurrences.

Re:99.999%

[Verunks]

So far, there have only been a couple 'proof of concept' viri for Linux. Nobody's figured out a way to pry any money away from us yet. :D

but linux antivirus aren't used to protect linux, they are useful if you run a mail server or a proxy so you can clean mails and webpage before they infect a windows user, or to clean an infected windows installation, for example the kaspersky live cd is based on linux

Re:99.999%

[Culture20]

What's impressive is that this got out of Sophos' testing lab and into production. I guess they must not test signatures in house at all. Congratulations, Sophos customers, you've been promoted to alpha testers.

Re:99.999%

[fuzzyfuzzyfungus]

The trouble, in this case, is that it detects its own signature update componenets as viruses...

Not only should this have been caught in testing(Since it would have cropped up more or less the moment the new signatures were loaded onto a live system with Sophos installed; but they hit files about which sophos presumably has intimate knowledge, this isn't some 'obscure packing/compression scheme used by legacy CAD program that seemed like a good idea in the 80's looks like a suspicious obfuscated payload' kind of thing.

I am not impressed, though thankfully it only took me a little over half a day to fix it here...

Re:99.999%

[jd2112]

What's impressive is that this got out of Sophos' testing lab and into production. I guess they must not test signatures in house at all. Congratulations, Sophos customers, you've been promoted to alpha testers.

Actually, it's an incredible show of honesty on the part of Sophos. Perhaps Symantec and McAfee will follow suit and flag their own software as malicious as well.

Can We Say Test our Code, anyone???

[realsilly]

This is a classic case of not thoroughly testing code and making sure you have enough variations of test machines to ensure as little pain to clients as possible.

If I were a customer, I would be shopping for a better company.

Re:Can We Say Test our Code, anyone???

[girlintraining]

That's pocket change compared to how much the company can lose over a screw up like this.

Emphasis mine. Look, every major antivirus producer has made a similar mistake to this. Sometimes, it takes the whole operating system down with it (Symantec anyone?). Whether you agree or disagree, it's clear there are business incentives for a fast workflow process -- and as the old saying goes "Do it fast, do it right, do it cheap -- pick any two." It's obvious which ones the antivirus industry as a whole has chosen. Rather than argue over whether or not they're right, I'm pointing out why they're making those choices. Businesses aren't willing to pay a premium to avoid mistakes like this. The cost of the occasional screwup like this is less than the cost required to do all the testing and lab work that many here on slashdot seem to support.

It's a business decision they've made, right or wrong.

Tautologies are fun

[dkleinsc]

Obviously, once this change had gone in, Sophos was correct to identify itself as malicious.

Re:That's why I don't install AV software on my PC

[asmkm22]

That's like saying you don't use condoms because you know how to pull out.

Re:Which just goes to show...

[localman57]

"test by eyeballing the code" has its drawbacks.

Exactly. Sometimes code that looks useless is really pretty important. The article follow up said they removed this test from an iteration loop, since there weren't comments about what it did. Apparently the original programmers thought it obvious...

if ( asimov_3rd_violation())
{
continue;
}
else
{
remove_file(filename);
}

There needs to be an award for this

[phrackwulf]

Every year, we need to go down the list of software makers who have managed to totally Bork their users. The Meltdown awards. Just to distinguish between the companies that handle it well and the companies that are incompetent.

Re:That's why I don't install AV software on my PC

[localman57]

My cousin used to say the same sort of thing about his know-it-all supervisor at work that was always riding him to wear safetly glasses. After he got back from disability, the guy got him a couple of tickets to Avatar in 3d, just to be an asshole.

Malware makers take note!

[erroneus]

Wanna cause problems? Add code from the various AV vendors...

Notes from an effected enterprise

[illtud]

Yes, this was bad. The virus signature in question appears to match any software that does auto-updates (possibly trying to spot phone-home malware?) so it's flagged dozens of software packages and according to what policy you've set, quarantined or deleted the files. This includes the auto-update part of the sophos client. The flood of emails from the sophos enterprise manager package as machines were switched on this morning quickly alerted us that this wasn't good, and just looking at names of the files it was flagging was enough to see that this was a false positive. Cleanup continues.

We've been very happy with sophos enterprise, and I'm staggered that this signature made it out the door - they should have numerous controls in place to ensure this can never happen and I await an explanation for how they failed.

I'm not too impressed by some of the advice given in their cleanup procedure - they advise setting the policy to not scan certain sophos directories - guess where viruses may try to hide in future.

This is an embarassing fubar which will have had a high impact on thousands of enterprises. It'll be interesting to see if Sophos come clean about the circumstances and can be convincing enough about how it's never going to happen again.