Slashdot Mirror
Critical Eye on SpamAssassin
ErrorBase writes "In this Infoworld article, Logan G. Harbaugh makes a great deal about an ancient (2.44) version of SpamAssassin comparing it with newer comercial variants.
Quote : You get what you pay for. [...] However, it took more than 10 times as long to install and configure SpamAssassin as it did any of the other products. "
Why did he not ask Kevin Railsback who had the whole thing working some while ago?)"
SpamBayes, by far.
All my incomming mail comes through SpamAssassin (cant remember which version off the top of my head), and once in a blue moon a single piece of spam will manage to find it's way through. When it does, I guess i should just applaud the spammer for being so devious.
TrollAssasin would be nice, imagine seeing posts subjects as *****TROLL***** heh
Seems like this guy did not verbalize it but that was his problem. If you know what you are doing hacking a conf file from vi is easier than a gui for sure. However, his low performance and configuration woes would have probably been handled with a easy to use graphical interface.
Aren't there tools that do this?
ACK
This was just a setup to make commercial software look better or just a incompetent reviewer. Next.
Webmin is great for setting up just about anything you can think of.
A psychopath can't tell the difference between right and wrong. A sociopath knows the difference - he just doesn't care.
"We compare a collection of recent operating systems: Windows XP Professional, Mac OS X Panther, Debian GNU/Linux 0.91".
Seriously, InfoWorld, SpamAssassin 2.44 was released in February, all the other vendors you compared were constantly updating their products to cope with the ever changing nature of spam.
John.
Great - compare generation or more older open source to fresh shrinkwrap. Who's zooming (or shilling) for who.
.01% (yes Bucko, less than 1/1000) false positives. When they implemented it several versions ago it was just as good.
My ISP (souther NH) runs SpamAssassin 2.6 - and I can tell you that at the default settings it catches 90-95% with
I've got one client where the run NO filter - some folks (the names GOTTA be on the web site) get up to 100 spams a day. IT are basically monkeys with hands. I have no idea what the CEO thinks. They wouldn't even think OS as they're a total MS shop.
I don't understand why he's so critical of a free product. I upgraded to 2.60 and it's running near flawless, and since the program is so simple, you just upgrade it, no need to change configuration options if you don't need to, you just call it from procmail.
Yeah all those GUI options look nice, but 90% of the time, why do I need to change my spamblocking settings? The Bayesian filter autoadjusts itself with little or no user intervention -- it's near transparent.
I run a mail server at home on a Linux box, with Postfix and Spamassassin 2.60. I have it configured to label mail as spam once it hits 8 points, and to automatically chuck it into /dev/null once it hits 12 (using Postfix's header_checks).
It works pretty well for me -- the mail server's only for my personal use so I don't really have to worry about irate subscribers sueing me for dropping them legit mail =p and the 8-12 point range in the spam marking gives me a chance to vet through those suspicious mails briefly before deleting them.
I've never tried any other spam filters on the server-side, so I can't really compare. I guess I'm also a bit of a Linux hacker so I don't mind tweaking all those config files along the lines of the FAQ and other hints on forums to get it to work the way I want it to.
Gan Family Homepage
Come to think of it, it seems to work out just fine.
Newt-dog
My Doctor prescribed daily nasal saline irrigation, hehe
This is likely funded by un-named virus vendors who has integrated SapmAssassin into their appliaces. Away on a vacation, I came back to find our people unaware SpamAssassin was open source. The vendor quietly forgot to mention that.
In the end, any company is going to have to put people and tools together to get a spam solutution, or outsource it. But DIY needs people time.
Don't pay vendors for SpamAssassin, it runs quite nicely on left over PCs reloaded with Linux.
I know people have been recommending SpamBayes but be warned - it is very slow to parse and move the emails. Only bother with this if you receive only a small volume of spam or have a pretty fast computer.
==> Start|Settings|Control Panel|Microsoft Office XP Professional with FrontPage|Remove
Best one yet!
He sent a long open letter to SAtalk. You can find it in the mailing list archive
well, on the first page the author already makes it pretty obvious why SpamAssassin had to come out at the bottom of the list. He is comparing version 2.44, which was included in RH9 and is thus at least 8 months old, to the latest antispam software that is regularly updated. How on earth is that an unbiased comparison? In a world where spam patters change every week, if not every day, 8 months is a generation... he even says so in his article. I'd be interested to see the results of a similar test, but with SpamAssassin 2.60 and of course with bayesian filtering and some of the other optional features enabled...
Why did he not ask Kevin Railsback who had the whole thing working some while ago?)"
He expected to get the results that he normally gets with most commercial software. Click Setup.exe, answer a question or two and it's done, up and running. Further configuration is not required though it may be desired.
The commercial vendors of Spamassassin have not improved the core product in any way. What they have improved is the packaging, the installation, the default configuration and the interface to modify that configuration. The stock SpamAssassin does not offer that although, Spamassassin setup is far more simple than some other packages out there.
versus
The first found Spamassassin easy, the second found it hard. Hmmm.
What really aggravates me is the typical "There are blacklists available that you can subscribe to, and some are updated regularly, but these are noncommercial lists with no guarantees." I'd like to see what guarantees the commercial lists come with.
Each product was tested with a different stream of mail, so the number of messages received varied, but all received enough messages to assess their capabilities.
Can you imagine someone writing "Oracle, Sybase and Postgres were compared. While the data and workloads were different, all products performed enough work to assess thier capabilities."
All the products except Brightmail and SpamAssassin allow end-users to add senders to the domain whitelist themselves.
I don't know anything about Brightmail. Spamassassin end user whitelists entries can be set up in a number of ways.
And all the products but SpamAssassin use dynamic updates to keep up with the evolving technologies spammers use to circumvent less sophisticated filters.
As aluded to in the summary, this is false with modern versions of Spamassassin, which uses Baysian filtering. (The author later says he couldn't get it working.
However, it took more than 10 times as long to install and configure SpamAssassin as it did any of the other products. [...] But just because the software is installed does not mean it will work -- filtering criteria must be added manually, and until that's done nothing is filtered out. Getting the various configuration files edited properly so that the whole package worked was not simple. Documentation was difficult to find, and not always easy to follow.
While it is true that one must be comfortable with a text editor to configure Spamassassin, thus perhaps putting it out of reach of point-and-click admins and technical journalists, I also wouldn't be prone to put my mail servers in the hands of either of those groups of people.
It looks for keywords in the subject or body of e-mails, but is frustrated by words not in the dictionary, such as "V!agra," or words that contain invisible HTML characters.
While I am not sure what tests appeared in which version, I'm pretty sure 2.44 handled off-by-one works such as V!agra. I have no idea what he's talking about when he says "invisible HTML characters", but it does seem to point to a certain technical incompetence, similar to the ostritch belief - "If I can't see you, then you can't see me."
This is not to say Spamassassin is the easiest thing in the world to deal with. I happen to love it, because of the extreme flexibility.
I just get sick of tech journos who decide that because a tool doesn't have a gui and they don't want to take the time to configure it, it sucks.
I forget what 8 was for.
[SpamAssassin] filtered only 62 percent of spam, whereas the other products produced great results, blocking 90 percent to 96 percent of all the spam they encountered with few, if any, legitimate messages blocked.
To me, this statement is pretty telling. Harbaugh must get some completely different kinds of spam than me, because, even though I receive about 60 spam mails a day (directed to my "spam" folder, so I never see them until I scan the "From:" field and then delete them), maybe one per week makes it through the filter. And seeing as how I can't even remember the last time I got a false positive, that's a pretty damn good number.
I can believe that if you receive a variety of mail and if you took no time to configure SpamAssassin other than cranking it up, maybe then it'll only catch 80% of the spam. But 62%? I'm not sure if Harbaugh is skewing the benchmarks or if he just doesn't know what he's doing.
There are some legitimate issues with SpamAssassin that might not make it ready for the enterprise, but for a handful of users, I have been more than satisfied. And the price is right.
-- "Complacency is a far more dangerous attitude than outrage." -Naomi Littlebear
You think 2.44 is ancient? Feh - Debian 'stable' is still stuck with 2.20.
Can we moderate the article at -1 Troll, please?
:)
It's just a bit too obvious that he was hoping for a severe slashdotting, driving his own numbers ("look, editor, how many people read my articles!") and the ad numbers of his paper up.
Probably submitted the story himself, too.
Assorted stuff I do sometimes: Lemuria.org
Seriously:
- The Spamassassin installation documentation could be better written IMHO.
- Why doesn't RedHat's update service offer constand updates to the current version of SpamAssassin?
- Why doesn't it (as mentioned in another post) have the most important configuratoin setups included in their overall configuration GUI?
I really wish distributions would support SA better.What? open source software having crappy and hard to find documentation?
Memo to self: if I ever spend 3 months creating free software to share, take 2 hours to write a web page showing somebody how it freaking works!
The bias apparent in this article and the crappy comparison chart aside this review doesn't even begin to touch base as a throughly researched opinion ion piece and ends up look like an advert for Brightmail.
However we do in the OS community face a UI problem. The missing rung on the ladder to mass acceptance is the absence of high quality UI that give users and indeed administrators of the point and drool variety a interface with the service they are seeking to use.
Before the Highly polished phpmyadmin I met serious resistance from admins for MySQL over msSQL based mostly on interface. The same goes for CUPS which has a web interface that I think has come of age if not achieve adult hood. The Webmin's are OK as long as you don't tinker to much or do anything slightly non-standard. I dislike Swat and am now so used to editing smb.conf I haven't even checked it;s working. I think that a lot of these services, apache, Spamassassin and X11 for example, could bare providing embedded configuration UI's if they aim to capture wider markets. Mandrakes X11 confugulator is very good.
I was going to mention the difficulty presented for admins with widely deployed Outlook when looking at these kind of solutions but then I though no only have sympathy where it is due. An I know that SpamAssassin could work seamlessly with Outlook but if users want a front end for white-listing then SpamAssassin isn't going to be your toy just yet.
Though we love the text based config file you may have to put a lot of working into configuration UI's if you want to enter the area as far as that reviewer and many sysadmins are concerned.
I knew nothing about filtering spam until I installed SpamAssassin 2.6 in a multi-user environment last week. Here are my responses:
I wouldn't recommend that my grandmother install SpamAssassin, but if you have any admin skills whatsoever, it's quite easy to use it to set up effective and useful filters. Furthermore, there are enough factual errors in the article that I'm tempted to dismiss it outright.
Of course, it's possible that it got a lot better between 2.44 and 2.6, but that begs the question, why did he install 2.44?
"The obvious mathematical breakthrough would be . . . an easy way to factor large prime numbers"
Bill Gates, 1995
I can install Spamassassin and six other applications via CPAN in the time it takes to get the syntax right for one license key.
I also like the characterization of Spamassassin as "first generation" without any supporting evidence to the fact. First generation was adding spam senders to your e-mail client's blocklist. Bayesian filtering is well beyond first generation, but spammers have learned to defeat Bayesian filtering with poison data in non-eyeball space and text obfuscation. The next generation in spam detection is to detect the Bayesian evasion features - and guess what does that!? Spamassassin (2.60).
who are those slashdot people? they swept over like Mongol-Tartars.
I've found the easiest way to implement SpamAssassin is to invoke it through MailScanner. MailScanner uses third-party virus scanners and can optionally invoke SpamAssassin as well. With the free ClamAV antivirus product, you can build a powerful open source mail scanner. Even without a virus scanner, MailScanner detects and quarantines executable attachments and other dangerous content which represent the most common types of mail-borne viruses and worms.
RedHat installs the daemonized version of SA as well as the SA Perl scripts. Using the daemon, the easiest implementation is to invoke SA in /etc/procmailrc on the mail delivery host; for mail gateways running sendmail, you need to use the milter interface. I've found the MailScanner+SpamAssassin approach much easier to configure than either of these methods, and you get virus scanning to boot!
I suspect if the reviewer had compared SA 2.60+ to the commercial products, rather than the older 2.44 version used in the review, SA would have shown better results.
I'd agree with the reviewer that one of the things SA lacks is an easy method for users to interact directly with the program. (Part of the issue has to do with security; SA runs as root. As I read the review, I wondered how the other products allow users to interact directly with the scanners without sacrificing security.) It's not easy to maintain per-user Bayesian filtering, for instance, but I generally recommend having the mail client, e.g., Mozilla, handle these tasks.
I was using version 2.44, I was able to compile and upgrade spamassassin before the number of posted replies hit 60! Can't be too hard!
Not only is this somewhat old news, it's been discussed on the spamassassin mailing list. Apparently, the article was edited so that it's more anti-spamassassin than the reviewer intended, but Mr. Harbaugh also defends his review of an older version of spamassassin as "it came with my Redhat 9" (NOT a direct a quote). He also claims it took nearly an hour to install and set up. (I counter that it took seconds to install and minutes to set up).
The current version of spamassassin is 2.60.
All my mail comes through spamassassin as well, but I am not having nearly the success you are...
.2-.5% false positive. Don't get me wrong, I am WAY happier now that before spamassassin, but if I could be getting better performace, that would be great...
I get about 60-70% of my spam correctly tagged, and about
"I'll have a Guinness, no wait, make that a Coors Light" -Grad student I work with, who shall remain anonymous...
Since then, I've downloaded a bunch of rules from The SA Custom Rule Emporium and almost nothing gets through.
If this guy had trouble, it is the fault of the documentation, not the product. Either that, or he was dumb enough not to upgrade to perl 5.8 or above, and spent forever installing modules.
He says:
Funny how when you install an old version of the product, it seems outmoded, hmmm?
Sheesh.
Pixie
don't mess with those geekgrrls
While his review was perhaps not scientifically conducted. I think there was a point to be made with the SpamAssasin blurb.
Notice that he deliberately took a standard install from RedHat 9, something some IT person (Not a tr00 g33k) might buy at CompUSA. He then tried to install the provided product. Clearly, a tr00 g33k would go and download the latest release, but keep in mind that not everyone is so comfortable with being on the bleeding edge - I believe that this was a point he tried to make. There is also the perception that the release provided with a "product" such as RedHat 9 will be up to the same standards as the OS.
While it's true the latest version has default rules and whatnot - it's quite likely that his older, more out of date version does not. In fact, going briefly to the spamassin home page the links for the 2.5 and 2.4 release documentation are broken.
The point to be made was: OSS needs to be more buttoned up. Notice that he said that he had no trouble installing redhat 9. That's becuase the installer is rather good.
Here's a nice example of a commercial guarantee. See if you can determine where it's from:
...
11. LIMITED WARRANTY FOR PRODUCT ACQUIRED IN THE US AND CANADA.
Microsoft warrants that the Product will perform substantially in accordance with the accompanying materials for a period of ninety days from the date of receipt.
YOUR EXCLUSIVE REMEDY. Microsoft's and its suppliers' entire liability and your exclusive remedy shall be, at Microsoft's option from time to time exercised subject to applicable law, (a) return of the price paid (if any) for the Product, or (b) repair or replacement of the uct, that does not meet this Limited Warranty and that is returned to Microsoft with a copy of your receipt.
Note that a) no updates or fixes are guaranteed, b) your only remedy is media replacement or a refund, and c) this choice of remedy is up to Microsoft.
I love it when people claim that you're taking a huge risk with open source software without guarantees. Microsoft says their software will work, but isn't saying that if their software doesn't work, they have to fix it.
It's better to vote for what you want and not get it than to vote for what you don't want and get it.
- E. Debs
"I installed the software on Red Hat Linux 9, with help from one of Proofpoint's systems engineers. She talked me through getting the Linux system configured properly, getting sendmail set up, and installing and configuring the Protection Server, which includes the MySQL database server for storing quarantined e-mail."
who needs a gui?
no wonder he gave spamassassin a low score. he couldnt have someone handhold him
I don't know anything about SpamBayes so I cannot comment on it at all.
POPFile is easy to use. It also performs Bayesian filtering. It is what I use.
http://popfile.sourceforge.net/
My current POPFile statistics:
Messages classified: 1,440
Classification errors: 19
Accuracy: 98.68%
saconf works for the Windows versions of spam assassin.
http://www.openhandhome.com/saconf.html
Spamagogo doesn't have quite the same setup, but it is good, and free for now.
Time for a snack.
indeed - I've been using this for a while now. No false positives, I see bits and pieces in my unsure folder - including the "Hi, heres that link you asked for http://spam.spam.spamcorp, cheers .." that Paul Graham reckons is the future of spam.
Given I get over 100 spams a day and I see non of them I am very happy with this indeed.
> I don't understand why he's so critical of a free product.
Why is there this attitude that if your project is free, then it does not matter if it is garbage. Furthermore, you are not allowed to say it is garbage, because, after all, you don't look a gift horse in the mouth. Perhaps that is why Linux is still not on the desktop. There are plenty of people who spend days configuring theirs and then post "it works for me" comments, while the rest of us silently wonder why anyone would want to spend so much time on such garbage.
To moderators. When you mod something "informative", please check the facts first. Spamassasin in RH 9 is 2.44.
Save the bandwidth. Don't use sigs!
Bayesian filtering is a bit like fuzzy-logic. Right now, it's best known for filtering spam. SpamAssassin uses a whole long list of tests and assigns +ve or -ve scores to each test that comes out positive (a bit like Slashdot's moderation).
I know someone who did a project on classifying video using Bayesian filtering. It looked at stuff like brightness, contrast, volume, basically everything they could extract from the movie file and give a value to. The concept itself is quite powerful; the difficulty is getting a list of tests that can accurately predict / classify what you have (spam/non-spam, or for video, thriller/drama/etc).
If you're interested in finding out more about actually coding Bayesian filters, you can check out the Bayes ++ project page.
Gan Family Homepage
This guy's article was a joke. Not only did he use an ancient version (in the spam world) of SpamAssassin but he either flat out lied in his article or was too lazy to seek out the truth. Hard to configure? Can't find docs? Doesn't support A B C D or E? If this guy had spent 5 minutes of his precious time doing to research on SA he wouldn't have made these flagrant lies. I don't get these people. I really don't. I CCd the Editor-in-Chief at InfoWorld, Mr. Steve Fox, as well.
Mr. Harbaugh,
This letter is in response to your InfoWorld article titled "Commercial solutions win, spam loses." In that article you portray all commercial spam solutions as winners and you portray the only open-source spam solution you reviewed as a dismal failure. I must say that as a professional in the anti-spam field I'm am truly disappointed by your incomplete and inaccurate assessment.
You start the article off quite well. Your introduction regarding two of the possible types of spam filtering is in terms that the average reader can understand. The introduction is also technically accurate, although it doesn't mention the other ways to filter spam.
You quickly take an opportunity to kick dirt on SpamAssassin by claiming it filters a fraction of the amount of spam all the commercial solutions filter. You hint at something during that statement when you said that SpamAssassin's "age showed in my tests," yet you fail to actually make it apparent to the user what the real truth is. I must ask, why did you choose to compare such an ancient version of SpamAssassin to the current versions of the four commercial products? Version 2.44 is over 9 months old. Spam filtering techniques are constantly evolving to filter a continually changing target. Comparing a 9.5 month old copy of SpamAssassin to the current version of BrightMail is like comparing a 1990 Chevy Silverado to a brand-new 2004 model. As an author and professional in the IT industry writing a column for InfoWorld, one of your goals is accuracy and fairness in reporting, is it not?
You make numerous false statements regarding SpamAssassin in your article:
1) "All the products except Brightmail and SpamAssassin allow end-users to add senders to the domain whitelist themselves... SpamAssassin allows only the administrator to add to the whitelist, with no direct access for users."
This is simply not true. SpamAssassin allows its users to add whitelist or blacklist entries to the personal preferences. It also allows its users to control the scoring for each individual ruleset with SpamAssassin's arsenal. Even the ancient version of SpamAssassin you chose to use had that simple feature. SpamAssassin also has the ability to automatically whitelist senders.
2) "Delegation of specific administrative functions is possible with all the products except SpamAssassin..."
This too is not true. As I said in response to number 1, SpamAssassin allows its users to control the scoring for each individual ruleset. This gives them the ability to disable certain rules, lessen the scores of others, and increase the scores of rules they wish had more weight. For example a user could disable the MAPS RBL DNS blacklist checks, whitelist joe@mydomain.tld, blacklist annoying-spammer@spamdomain.biz, and increase the score of the rule ALL_CAP_PORN to 2. The users can also create their own rulesets. SpamAssassin gives its users a high level of control over their spam filtering.
3) "Finally, in addition to stopping spam, all four commercial products provide content-filtering features, allowing the administrator to block incoming or outgoing e-mail that contains proprietary data, audio or video files, executables, sexually explicit words, or racial slurs. They also provide protection against DoS attacks and directory harvesting attacks."
This one baffled me at first. I'm honestly not sure why you want to compare features that have nothing to do with filtering spam. Filtering racial slurs from an email is