Slashdot Mirror
Seven Spam Filters Compared
Goo.cc writes "Those wondering how their spam filtering software performs in comparison to other's may want to read this article on Freshmeat, where Sam Holden performs comparative testing of various popular e-mail filters. The filters tested includes Bayesian Mail Filter, Bogofilter, dbacl, Quick Spam Filter, SpamAssassin, SpamProbe, and SPASTIC."
people/editors need to learn the a tag
clicky
Spam Filters
The author makes a good attempt at comparing these products, but I don't think his samples are indepth enough to come up with real-world results.
For Bayes testing, he used 68 spam and 68 ham messages. Spamassassin for one won't even activate bayes until it's learned from 200 messages; it's not uncommon for those who regularly deal with spam management on the server side to use 5000-10,000 message corpuses to test new rule additions and to train spam.
The low number might have a slight effect if most of your mail contains similar characteristics, but I'd much rather have seen bigger numbers of samples.
-Barkeep, a draft of your most hazardous brew, for the world is slowly stepping into focus, and I don't like what I see.
IMO, the best way to go with spam is to combine a heuristic filter with a text/baysian filter, in my case SpamAssassin and SpamProbe. I run them both, and it does a noticably better job than either running alone.
SpamProbe can be fooled by clever spammers who insert lots of common words in non-visible html. A Baysian filter can't really catch that, but a heuristic filter can be written to notice the pattern.
Also, set up your Baysian filter to re-learn regularly from your spam folder. SpamProbe adds a unique ID to each message, so it won't process a message twice. Therefore, you can just manually move any false negative spams into the folder, and they'll be learned from.
As was noted earlier, the set of messages given to the filters for learning was terribly small. Furthermore, SpamAssassin wasn't tested in a way useful to most as the tests in this article didn't take into account SA's Bayesian filter nor it's network-based tests (Razor, etc).
Very true. I downloaded 1600 messages with Thunderbird today (backlog) and only about 30 weren't spam. That's a huge waste of bandwidth.
"The number of Unix installations has grown to ten, with more expected." (Unix Programmer's Manual, 2nd ed.; june 1972)
How the heck could Active Spam Killer be left out? I used to get about 150 spams a day and now I get ZERO. No false positives, no false negatives.
It is an autoresponder that checks the sender against a whitelist and a blacklist. If a new e-mail is in neither, then it bounces back an e-mail asking for a confirmation that the sender is a human. Simple!
What about PopFile? I've tried SpamAssassin and a few others, and I like PopFile the best. After a little training it's EXTREEMLY accurate. It survived the deluge of mail I've gotten in the last few days (due to virii) with flying colors.
According it it's internal statistics, it has classified 2821 messages as of the time I type this. It has made only 95 errors (often close calls, so I don't blame it). That puts it at an accuracy of 96.63%. For the record, of the e-mail I've gotten, it's 308 messages of ham, 2513 spam.
I have only been using PopFile since June 7th of this year, but it's working fantastic. The only thing I've used that's this good was Cloudmark's SpamNet, who stabbed the community in the back, so I switched to something else. I'm glad I've found PopFile, and I suggest you try it too if you're looking for something good.
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
See our PSAM project site for a refereed paper evaluating several machine learning spam filtering techniques (although not specific filters). This site also contains large standardized corpora for evaluation. The paper contains a number of tips on evaluating ML spam filters.
The /.-referenced article has some good ideas about evaluation. I particularly liked the explicit discussion of the false positives. The recommendations at the end are excellent. On the other hand, the evaluation isn't across a broad or obviously representative corpus, many of the tests are a bit odd, the ROC tradeoffs are not discussed. In particular, the evaluation set for the tests did not include enough ham to be able to accurately estimate the false positive rate: consider what would happen to the precision estimates if 0.5 were added to each of the numbers in the false positive table.
Overall, though, this was an interesting evaluation, and I'm glad that the author published it.
I've been using Mailfilter for a while now and I've built a pretty comprehensive list of keywords in the subjects of spam. It seems to just pull the message headers from the server without downloading the body.
One example rule:
DENY = ^Subject:.*v[i1l!|][a4@][g8]e?r[a4@]
Then I filter whatever gets through that with SpamAssassin.
Yup. I use it all the time. Save up spam and ham in seperate folders. Then do this:
sa-learn --spam --mbox ~/mail/myspamfolder
sa-learn --ham --mbox ~/mail/myhamfolder
As I get more spam, I set it aside into a folder, and in tcsh I have this alias set:
alias spamadd 'sa-learn --spam --mbox ~/mail/got-through && rm ~/mail/got-through && touch ~/mail/got-through'
Karma: Chameleon (mostly due to the fact that you come and go).
Of couse your baysian filter will QUICKLY learn that html tags that create invisible text are VERY common in spam and nowhere else-> problem solved
Dont forget that the filter sees more than the eye...
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
If you decide to try out spamprobe or another bayesian filter, try this web interface which lets you easily reclassify mail, even those marked as spam. I found that "training" the bayesian filters was the hardest part; this definitely simplifies the process.
It wasn't mentioned in the article, but I really must plug popfile. It filters out my spam yes, but it is also a general mail categorizer. It sorts ten yahoo groups for me, personal, work, and school related emails. I know you think you could do this with rules for the emails, but for example, I get several hundred emails a day from the Harry Potter for Grownups List. Popfile can sort them into 'probably interesting' and 'probably not' for me. Very nice.
I'm using the standalone Thunderbird and it catchs everything that passes by Spamassassin. Spam is marked but never deleted, so I can go back and check. Some spam programs will delete email, which could delete a good email, unacceptable.
Basically, I'm using a mandrake linux box, imap, procmail, fetchmail and spamassassin. Easy, and I can send/receive email from my linux box, and port 25 is blocked from the Net so nobody can use me as a bouncer.
Only problem I had was, there was no complete document to set this up, I had to piece each part together.
So for anyone who wants to know, heres the quick steps.
1. I'm using mandrake, but had to update SA for the sa-learn utils. (Gotta train SpamAssassin)
2. Setup fetchmail in your personal account.
3. Setup
DROPPRIVS=YES
VERBOSE=ON
LOGFILE=/home/userac
|
4. Setup your user_prefs in your local directory for SA. (mine, but im no SA expert, but it works)
required_hits 5
rewrite_subject 0
use_terse_report 1
report_safe 1
use_bayes 1
auto_learn 1
ok_locales en
use_pyzor 1
pyzor_max 9
pyzor_add_header 1
use_razor2 1
always_add_headers 1
always_add_report 1
spam_level_stars 1
pyzor_add_header 1
skip_rbl_checks 0
#timelog_path
5. As root make sure Imap,Spamassassin is running.
6. Load Thunderbird, use Imap, use filters on x-headers.
SAProxy for Windows (Based on SpamAssassin) got the highest marks.
If you reread the slightly ambiguous sentence in context you will realise he meant he had evaluated five baysian filters and felt that was enough. Nothing to do with Spamassassins point system...
Also remember you need to feed nonspams to bayesian filters also.
Nothing to see here; Move along.
I have been using POPFile for months now, with a fairly complex setup, one of the things I like about POPFile versus the others I've seen (which are two or three bucket systems). It's classifying more than 99% accurately every month for the past three or four months (I reset my statistics around the first of every month) and has never been less than 95% accurate in a month (including its training month). For an idea of what my loads and buckets are like, this list of my buckets and the number of messages classified into them since the first of the month will help:
I've been using TB for a couple months now, and very much like it. I've used the built-in junk filtering since I first got it, and have found that it is only getting about 1/3 to 1/2 of the things already catagorized for my spam buckets, with a higher rate of false-positives than POPFile. I would like to see something more reliable, and hope updating the algorithm will help.
As complicated as my buckets may look, this system works very well for me -- with the addition of a "misc" folder that anything not classified goes into, and some filters based on the X-Classified line, almost nothing that gets into my inbox is anything other than personal email.
In addition to the above, it might be smart to create three files called "ham", "spam" and "forget":Complement with a cron job that runs sa-learn --rebuild every night.
Then, if you read your mail on the same box, and the headers doesn't say it was auto-learned, simply pipe the email to either ham or spam. If it was wrongly auto-learned as spam, pipe it to forget. If using pine, it's really easy:
| ham
Of course, if you use razor or other online services that lets you report spam, you might want to pipe some of the spam mails that weren't recognized to "spamassassin -r".
Regards,
--
*Art