Seven Spam Filters Compared

Goo.cc writes "Those wondering how their spam filtering software performs in comparison to other's may want to read this article on Freshmeat, where Sam Holden performs comparative testing of various popular e-mail filters. The filters tested includes Bayesian Mail Filter, Bogofilter, dbacl, Quick Spam Filter, SpamAssassin, SpamProbe, and SPASTIC."

Comments

Good testing, but not enough samples

[TexTex]

The author makes a good attempt at comparing these products, but I don't think his samples are indepth enough to come up with real-world results.

For Bayes testing, he used 68 spam and 68 ham messages. Spamassassin for one won't even activate bayes until it's learned from 200 messages; it's not uncommon for those who regularly deal with spam management on the server side to use 5000-10,000 message corpuses to test new rule additions and to train spam.

The low number might have a slight effect if most of your mail contains similar characteristics, but I'd much rather have seen bigger numbers of samples.

Re:Good testing, but not enough samples

[cly]

I guess you wrote this after reading the first two experiments.

In the third he used 1200.

Nice way to jump the gun.

Re:Good testing, but not enough samples

[arth1]

I guess you wrote this after reading the first two experiments.

In the third he used 1200.

1273, out of which 1073 were spam. That leaves 200 non-spam messages, which isn't enough for Spamassassin's bayesian filtering to kick in, even if all messages were to be classifed as ham or spam, and not just let through.

To quote sa-learn's man page:

Another thing to be aware of, is that typically you should
aim to train with at least 1000 messages of spam, and 1000
ham messages, if possible. More is better, but anything
over about 5000 messages does not improve accuracy signif­
icantly in our tests.

The low number of emails, combined with no apparent manual reading on part of the author, makes me want to disregard this whole survey as pure drivel.

Regards,
--
*Art

What About PopFile

[MBCook]

What about PopFile? I've tried SpamAssassin and a few others, and I like PopFile the best. After a little training it's EXTREEMLY accurate. It survived the deluge of mail I've gotten in the last few days (due to virii) with flying colors.

According it it's internal statistics, it has classified 2821 messages as of the time I type this. It has made only 95 errors (often close calls, so I don't blame it). That puts it at an accuracy of 96.63%. For the record, of the e-mail I've gotten, it's 308 messages of ham, 2513 spam.

I have only been using PopFile since June 7th of this year, but it's working fantastic. The only thing I've used that's this good was Cloudmark's SpamNet, who stabbed the community in the back, so I switched to something else. I'm glad I've found PopFile, and I suggest you try it too if you're looking for something good.

PSAM

[po8]

See our PSAM project site for a refereed paper evaluating several machine learning spam filtering techniques (although not specific filters). This site also contains large standardized corpora for evaluation. The paper contains a number of tips on evaluating ML spam filters.

The /.-referenced article has some good ideas about evaluation. I particularly liked the explicit discussion of the false positives. The recommendations at the end are excellent. On the other hand, the evaluation isn't across a broad or obviously representative corpus, many of the tests are a bit odd, the ROC tradeoffs are not discussed. In particular, the evaluation set for the tests did not include enough ham to be able to accurately estimate the false positive rate: consider what would happen to the precision estimates if 0.5 were added to each of the numbers in the false positive table.

Overall, though, this was an interesting evaluation, and I'm glad that the author published it.

Re:Spamassassin and Bayes?

[numbski]

Yup. I use it all the time. Save up spam and ham in seperate folders. Then do this:

sa-learn --spam --mbox ~/mail/myspamfolder
sa-learn --ham --mbox ~/mail/myhamfolder

As I get more spam, I set it aside into a folder, and in tcsh I have this alias set:

alias spamadd 'sa-learn --spam --mbox ~/mail/got-through && rm ~/mail/got-through && touch ~/mail/got-through'

WRONG.

[imsabbel]

Of couse your baysian filter will QUICKLY learn that html tags that create invisible text are VERY common in spam and nowhere else-> problem solved
Dont forget that the filter sees more than the eye...

Five baysian filters were enough

[Sits]

Here's a quote from the article:

Also, SpamAssassin has a Bayesian classifier built in, but it wasn't used in these tests, since having five was enough.

If you reread the slightly ambiguous sentence in context you will realise he meant he had evaluated five baysian filters and felt that was enough. Nothing to do with Spamassassins point system...