ClamAV Forced Upgrade Breaks Email Servers

An anonymous reader writes "A couple of weeks ago Sourcefire announced end-of-life for version 0.94 of its free ClamAV antivirus package (and in fact has been talking about it for six months). The method that Sourcefire chose to retire 0.94 was to shut down the server that provided its service. Those who had failed to upgrade are scrambling now. Many systems have no choice but to disable virus checking in order to continue to process email. I am very glad I saw the announcement last week!"

Comments

Alternative

[InsertWittyNameHere]

The alternative was them not doing anything and then months later we see a story about how "ClamAV silently stops support. Virus outbreaks ensue."

Re:Alternative

[Anonymous Coward]

It's kind of an inflammatory article:

Rather than simply phase this geriatric version out (it was at least one year old, revised to versions .95 and .96 since release, and announcements about the need to upgrade had been made for six months) the development team put to halt instances of V0.94 in production

So, it's a year and two versions out of date AND they'd been saying for 6 months to move off it.. Yet still it's their fault for shutting down the server!? I'm sorry, but how much support do you want for something that's free?

Re:Alternative

[compro01]

It's quite a bit more extreme than just shutting down one of their servers. They issued a final "signature" update that literally caused each installation of that version to stop functioning.

From the announcement :

Starting from 15 April 2010 our CVD will contain a special signature which disables all clamd installations older than 0.95 - that is to say older than 1 year.

Re:Alternative

[HarrySquatter]

Would you trust an email server that is running a virus scanner that is more than a year out of date?

Re:Alternative

[ccandreva]

It's more complicated than that.

Older versions of clamd were going to crash on signatures that newer versions would accept, and they have been prevented for at least 6 months from using that type of signature. They have posted since then for people to upgrade.

When they did was publish this type of signature (has to do with length, greater than about 900bytes), where the signature itself is an error message, so when the program dumped the signature the error would be displayed.

That's all, not a kill switch as such, but using a known bug to deliver a message, rather than have it just bomb out with a hex dump when they tried to use a larger signature.

Re:Alternative

[Anonymous Coward]

Yep, and when did they post that? 6 months ago. McAfee recently gave us 2 months notice at work that pre 8.x client would no longer be supported - not a problem as 7.1 was eol ages ago - since then there's been 8.0, 8.5 and currently 8.7 which we're moving to.

No big deal for those who properly manage their systems.

Re:Alternative

[geekmansworld]

When you make assumptions, you're an ass.

I don't watch TV at work. I'm busy because I'm the only IT guy in our organization, and I do everything, on top of regular office work, on a shoestring budget.

So while you're sitting in your office preparing the budget to show your boss how many tens of thousands of dollars the new M$ Exchange system is going to cost, maybe think about how lucky you are to be able to do what you love to do full time, with a budget, and proper support staff.

Re:Alternative

[b0bby]

Honestly, for things like this that I don't have the time to do right I prefer to let someone else do them. In this case, why not route your mail through Postini or another service? I'm pretty sure that I can't hope to do a better job filtering than Google...

Re:Alternative

[Fiznarp]

I got hit by the shutdown too, however I'm not upset. If I was paying for it I would have been angry at the vendor for not notifying me. But it's a FREE antivirus service. The folks that publish ClamAV updates aren't under any obligation to keep my systems up and running. If my systems were that important, I'd pay for something with an SLA.

Re:Alternative

[CoolQ]

Uh, it HAS been filling your log files with warnings about upgrading for months, if not years. It's pretty f'ing explicit:

LibClamAV Warning: *** This version of the ClamAV engine is outdated. ***
LibClamAV Warning: *** DON'T PANIC! Read http://www.clamav.net/support/faq ***

--Quentin

Re:Alternative

[Jiro]

It may not have occurred to you that some of us only do IT for out organizations part time, and visiting the blogs of every single open-source component on our servers is not always practical.

The issue has nothing to do with your servers, really; it has to do with their servers. If you're using a free service on someone else's servers, you really can't be surprised when that service suddenly stops functioning. It's not your equipment.

And I would wager that while visiting the blogs of everything on your servers isn't practical, visiting the blogs of (or subscribing to a mailing list, or other monitoring of) everything that's on your servers but uses someone else's servers is practical

Re:Alternative

[radtea]

This such a perfect example of a loser with attitude that I deserves comment. Look at the breakdown of points, hitting every checkbox:

1) Implies that anyone who criticizes his failure to do this job is ignorant of his difficult working conditions.

2) Implies that doing is job is an unreasonable burden that no one could expect, despite other people managing it, sometimes under conditions that he has no idea how difficult they are.

3) Implies that he did absolutely nothing wrong: his configuration was not an issue--that it was right and reasonable to have his servers configured to crash on failure of this "low priority" component, like a mechanic telling you it's right and reasonable for the wheels to fall off if the radio stops working, because the radio operating correctly is a low priority.

4) Implies that he's a hero for fixing a problem he caused by his neglect and incompetence. Despite his low pay he's on call all the time, and worked for hours fixing things brilliantly and heroically, despite having mis-configured a low-priority component as a critical system whose incidental failure could crash the whole works.

5) Blames someone else who did thier job well, and for free. Accuses a supplier of a free service who have been filling his logs with messages for six months of not filling his logs with messages for six months, and then accuses them of deliberately crashing his incompetently configured servers.

6) Re-iterates how over-worked he is and how much he has to do.

7) Proclaims he's going to look for another free service to blame his next failure on Real Soon Now.

Classic, classic whiner. Your job may suck, man, and that may not be all your fault, but if you don't fix the attitude you'll be stuck in the suck for a long, long time...

Re:Alternative

[cstdenis]

The "DON'T PANIC!" was obviously the wrong message to display for something that was going to break your mail server.

Re:Alternative

[syousef]

Would you trust an email server that is running a virus scanner that is more than a year out of date?

Would you trust a company who would remotely shut off your anti-virus?

Re:Alternative

[jim_v2000]

Yes. Especially when there was six months warning that it was going to happen.

Also, I'd rather it stop working then keep working and not get definition updates.

Re:Alternative

[SunFireSpaz]

Join their announce mailing list at http://lists.clamav.net/mailman/listinfo/clamav-announce and you will be notified about these type of things.

So you had 6 months to upgrade

[gparent]

And you didn't, and now are going to complain when shit doesn't work? Go fuck yourself.

Re:So you had 6 months to upgrade

[Anonymous Coward]

go fuck yourself

uh. this is slashdot. for most of us, that is a redundant instruction.

what would have been far more offensive is

go fuck someone else

as we all know that's not possible for most of us. ...you insensitive clod.

Re:So you had 6 months to upgrade

[johnshirley]

Kinda my attitude, too. Had this affect a bunch of servers yesterday. Started researching, found the cause, and solved the problem in 30 minutes on 35 or so servers. Totally my own damned fault for not staying upgraded. Worst impact was that messages were delayed on a few mail server for half an hour and uploads to a handful of webservers threw errors because of the way I scan them. Users tried again. Problem solved.

Re:So you had 6 months to upgrade

[Culture20]

I'm effected by endless clueless customers whining that their email server broke.

While such an occurrence would prompt me into action, I doubt it would prompt me into existence. ;)

Re:So you had 6 months to upgrade

[gparent]

I really wish thieves would let you know half a year in advance when they'd steal your car.

Re:So you had 6 months to upgrade

[gparent]

http://lists.clamav.net/mailman/listinfo/clamav-announce

Re:So you had 6 months to upgrade

[masdog]

I had modded this overrated, but this really deserves a reply.

You're in the wrong place if you expect sympathy. There are a lot of other sysadmins here. There are a lot who wear all of the hats. You're not alone.

You had a poorly designed or poorly implemented mail system. That isn't clamAV's fault. It's not their fault that you didn't upgrade or check your system logs. This is no different than forgetting to pay the maintenance bill on a commercial mail gateway or hosted solution. Would you blame Symantec, McAfee, Microsoft, or CA if you didn't pay the bill and your mail stopped flowing?

The fact that you didn't follow a blog or mailing list about a critical piece of your infrastructure says a lot about you as a sysadmin. They're even on Facebook and Twitter. If you can't take the time to keep an eye on your mail gateway or antivirus product, what else aren't you keeping up on. Think about that for a few minutes, set up a Google reader account, and then start subscribing to blogs. If you have a smartphone, add Google reader to your RSS Reader. It makes good bathroom reading.

Got This Bounce This Morning

[WrongSizeGlass]

Diagnostic-Code: smtp;
451-4.5.0 Error in processing, id=02792-02, virus_scan FAILED: virus_scan: ALL VIRUS SCANNERS FAILED: ClamAV-clamd av-scanner FAILED: CODE(0x83d7540) Too many retries to talk to /var/spool/amavisd/clamd.sock (Can't connect to UNIX socket /var/spool/amavisd/clamd.sock: No such file or directory) at (eval 55) line 310.

ClamAV-clamscan av-scanner FAILED: /usr/bin/clamscan unexpected exit 50, output="LibClamAV Error: cli_hex2str(): Malformed hexstring: This ClamAV version has reached End of Life! Please upgrade to version 0.95 or later.

At least their error messages are descriptive and informative.

[clamav-announce]

[0racle]

It exists for a reason.

Re:[clamav-announce]

[entrigant]

announce lists are intentionally very low traffic. I'm subscribed to over 50, and I rarely receive more than 4 or 5 mails a week at most.

this is common

[digitalsushi]

This is what we get when we're all our own "netadmins". I'm one of them. I don't follow security lists. I don't upgrade my products. Why not? Because I'm not really a netadmin. I just have a little server that runs until it breaks. I think that's the difference between a netadmin and a fake netadmin -- a fake netadmin like me reacts. A real netadmin is proactive.

Which honestly, as pathetic as it sounds on the surface, works fairly well when your data and uptime don't matter. Because it's not pathetic because I have better things to do with my time than "run the family webserver".

No fallback ?

[morcego]

People with critical servers that don't have fallback configurations to handle this kind of thing deserve to have their servers shutdown.

I've been using 0.95 for some time now, so none of my servers were affected but, even if they were, my servers are smart enough not to interrupt the services, and to notify me.

It is really disgusting the way people build servers these days. They think all they need to do is to install a couple packages, change a couple config lines and boom, the server is ready. They are getting what they asked for when stuff like this happens.

Re:No fallback ?

[1s44c]

I had two mail servers, on two Internet connections. If either went down I'd get an alert and could fix it without mail being affected. I didn't expect both to stop processing mail at the same time. It's always the stuff you don't expect to fail that fails.

My mail was queued on DMZ mailers so nothing was lost, but it was delayed. Some of it may have been business critical.

*Correction*

[Slipped_Disk]

The method SourceFire chose to use was to encode a kill command in the ClamAV updates. If they had simply "shut down the [update] server" ClamAV would have continued to work, just without new signatures.

See their announcement at http://www.clamav.net/lang/en/2009/10/05/eol-clamav-094/

Re:*Correction*

[WrongSizeGlass]

From the link:

Starting from 15 April 2010 our CVD will contain a special signature which disables all clamd installations older than 0.95 – that is to say older than 1 year.

[snip]

We recommend that you always run the latest version of ClamAV to get optimal protection, reliability and performance.

Thanks for your cooperation!

FYI, ClamAV, DOA != cooperation.

Re:*Correction*

[Anonymous Coward]

Wow. They could have just stopped publishing updates for older versions; they do have some method of versioning, right?. Older installations could have kept chugging along using the older definitions and newer installations could get the newer definitions. But to remotely *DISABLE* older installations? I don't care if the product and service is free or not; that is pretty fucked up.

Re:*Correction*

[HarrySquatter]

What's fucked up about it? It's a huge security problem to be running an email server that is using a virus scanner whose definitions are over a year old.

Re:*Correction*

[GungaDan]

Definitions were upgraded, though, weren't they? Just the engine was a year old...

Re:*Correction*

[compro01]

The definitions were up to date (but would become out of date when they started pushing large (>980 bytes) definition updates next month, which the old version cannot handle), but the version was not.

Re:*Correction*

[petermgreen]

I personally consider use of a remote signature update system as a kill switch to be abuse of the update system.

Tisk, tisk...

[fuzzyfuzzyfungus]

Should have switched to Norton. They would have had weeks of impossible-to-ignore yellow and black pop-ups demanding their credit card number as ample warning...

Those freetards just don't understand the valuable features provided by quality proprietary software.

EOL annountment from Oct 2009

[Anonymous Coward]

End of Life Announcement: ClamAV 0.94.x
Oct 5, 2009

All ClamAV releases older than 0.95 are affected by a bug in freshclam which prevents incremental updates from working with signatures longer than 980 bytes.
You can find more details on this issue on our bugzilla (see bug #1395)

This move is needed to push more people to upgrade to 0.95 .
We would like to keep on supporting all old versions of our engine, but unfortunately this is no longer possible without causing a disservice to people running a recent release of ClamAV.
The traffic generated by a full CVD download, as opposed to an incremental update, cannot be sustained by our mirrors.

We plan to start releasing signatures which exceed the 980 bytes limit on May 2010.

We recommend that you always run the latest version of ClamAV to get optimal protection, reliability and performance.

Thanks for your cooperation!

Re:FUCK JEWS

[jDeepbeep]

FUCK JEWS

When they are exceedingly attractive, female, not married, and expressing interest, I do.

Re:so clam breaks if a remote server is down?

[mysidia]

It wasn't the server going away. They delivered an update designed to kill it

The Windows equivalent would be Microsoft Delivering a critical update with XP designed to disable windows, because you haven't updated to Vista yet.

In other words, they used the automatic update service against their own users.

From now on, my recommended course of action is that all mail administrators running clamav should REMOVE or DISABLE any automatic updates of ClamAV rules, make sure to comment out any crontab entries for freshclam.

Until the developers can either grow up and stop doing stupid shit such as abusing auto-updates to disable their own product.

Or do what they should do... include a method for automatically applying version updates.

Or force auto version update instead of disabling.

Re:It's not like they didn't tell...

[mysidia]

SUPPORT WILL END does not imply killing instances in production. It implies you stop delivering support services (such as tech support or new updates).

How would you feel if the Ubuntu folks delivered a 'security update' to Ubuntu 8.x to disable your system entirely, until you can get a chance to go install a non-EOL'd major release of your OS?

How about all those Windows Vista users who haven't upgraded to Windows 7?

Firefox 2 users who haven't upgraded to 3.

Users who are still using IE6.

Would users trust the vendors anymore with auto-updates, if they all released updates to 'kill the old product' in order to force you to manually do a clean upgrade?

Yes, they did the right thing...

[Slipped_Disk]

As someone who was bitten by the issue (yeah, I'll man up and admit it - my company's mail server went wonky for about a half hour while I upgraded) I agree -- they pretty much did the right thing.

There was plenty of notice -- The fact that many of us weren't on the clamav-announce list is OUR fault, not theirs.
A kill command may not be the most "polite" way of retiring an old version of software, but for a free service I certainly don't expect them to invest huge amounts of time and money in figuring out how to support the old stuff forever.

Debian Debs Outdated

[TypoNAM]

I just tried to update:

# cat /etc/debian_version
5.0.4

aptitude output during update:

Setting up clamav-daemon (0.94.dfsg.2-1lenny2) ...
Starting ClamAV daemon: clamd LibClamAV Warning:
LibClamAV Warning: *** This version of the ClamAV engine is outdated. ***
LibClamAV Warning: *** DON'T PANIC! Read http://www.clamav.net/support/faq ***
LibClamAV Warning:
LibClamAV Error: cli_hex2str(): Malformed hexstring: This ClamAV version has reached End of Life! Please upgrade to version 0.95 or later. For more information see www.clamav.net/eol-clamav-094 and www.clamav.net/download (length: 169)
LibClamAV Error: Problem parsing database at line 742
LibClamAV Error: Can't load daily.ndb: Malformed database
LibClamAV Error: cli_tgzload: Can't load daily.ndb
LibClamAV Error: Can't load /var/lib/clamav/daily.cld: Malformed database
ERROR: Malformed database

It appears debian repositories also need to be updated. :(

NOTE: I removed the * (star) chars from the warnings due to junk filter.

Re:Debian Debs Outdated

[iYk6]

The ClamAV package in Debian Lenny-Volatile is 0.95.3. You're using the package from Debian Lenny, which is stable, and doesn't mesh well with ClamAV, which is either the latest and greatest or broken.

Debian Volatile is meant specifically for this kind of thing.

Natalie and grits

[jDeepbeep]

Be careful, though. Natalie Portman might pour hot grits on you.

Where do I sign up sir?

Re:GODDAMMIT ALREADY !!

[Hognoxious]

With a name like ClamAV, my bet would be the Scientologists.

What the fuck Slashdot?

[wolrahnaes]

First you complain when Microsoft releases an update that won't install on compromised systems because it would break them entirely.

Now ClamAV is put in a similar position. They have three choices due to the bug in 0.94:
1. Continue supporting 0.94, flood out their update servers with full updates since incrementals won't work with that version much longer.
2. Stop supporting 0.94, leaving users who don't know to update basically unprotected.
3. Send a clear message to users who haven't updated that their antivirus solution is now broken and they need to upgrade.

To me, 3 is the obvious choice. If this was a paid solution or if it cost a fucking dime to upgrade I might see a point to complaining, but to anyone who was still using 0.94 just man the fuck up, apt-get update, apt-get upgrade, and get on with it.

This is not like Microsoft disabling XP to get you to upgrade to Vista, this is more comparable to an aircraft with faulty parts being grounded by the FAA. Those using 0.94 were doomed to a broken solution one way or another, they could not continue using it and expect it to do its job, so they needed a kick in the ass to upgrade.

Overconfidence

[gmuslera]

A lot of server stuff in linux work so well that you can even forget that it is running at all, for years. Clamav is such kind of software, you install/configure it, set the automatic signature updates, and forget that it is there. But still, some periodic checks in logs that all are working as expected is good, even if is just some artificial ignorance well applied, specially when clamav started warning on this months ago.

Misleading, yes?

[thePowerOfGrayskull]

"ClamAV forced upgrade breaks email servers" should read "Failure to upgrade despite six months warning breaks email servers" or "Inattentive server admins cause massive downtime".

Re:so clam breaks if a remote server is down?

[Lumpy]

Nice FUD. the new DB will break it anyways.. and YES microsoft does this.

They crafted a DB update that used that bug to deliver a message so the logs showed you what happened instead of a "seg fault - error in line 45867"

Re: automatic binary updates

[Slipped_Disk]

From now on, my recommended course of action is that all mail administrators running clamav should REMOVE or DISABLE any automatic updates of ClamAV rules, make sure to comment out any crontab entries for freshclam.

<SARCASM>
Mmhmm, yes. I agree 1000%. Don't update your virus signatures. Because ya know, new viruses don't get created very often. You can run with signatures over a year old and still have great protection!
</SARCASM>

Or do what they should do... include a method for automatically applying version updates.

Or force auto version update instead of disabling.

<SARCASM>
Yes, because distributing software for several versions of Free/Net/OpenBSD, each Linux distribution, Windows, Solaris, AIX, HP-UX, etc. is totally feasible for a free project.

It's not like they would have to fund the time, equipment and distribution bandwidth for that, or have to deal with irate admins screaming about how ClamAV breaks their change control policies by automatically installing binaries on production servers.

And software with automatic updates never ships an update that bricks production servers (*cough*Exchange*cough*), so this is a perfect solution.
</SARCASM>

Sometimes I really wonder what happened to the Slashdot crowd's common sense.

Volatile

[XanC]

You really should use the volatile repository. It provides updated versions of packages that are required to change (like antivirus), compiled for stable. You end up with stable + required updates.

Who uses it anyway?

[Artem S. Tashkinov]

I'd likely to be modded down by open source zealots, but using Clamav to solely protect Windows PCs from malware spread by e-mail is insane. ClamAV has one of the lowest malware detection rate amongst other commercial AV solutions. I tested my own sample of around 140 new viruses found on different Windows PCs during last six months and ClamAV could detect only 70 of them. That's ridiculous ... and fearful to say at least.