Slashdot Mirror
Bayesian Filter Testing?
pu33y asks: "Since the publication of Paul Graham's A Plan For Spam, several programs that perform Bayesian filtering having become available, including CRM114 and Bogofilter. But missing is any serious testing to see how they perform in relation to themselves and to other, non-Bayesian filters.Searching Google has turned up nothing and when I asked Paul Graham, he was unaware of any such testing, as well. Can anyone point to any such testing or provide the results of their own personal experiences with Bayesian filters?"
Dspam (http://www.networkdweebs.com) rocks !
Some impressive stats were posted to the mailing list.
It's main feature is that it's completely maintainance free, and that even dumb people can use it (I know, I am).
My personnal stats are 2 false positives actually (one from PayPal, one from a company I work with), 280 spams learnt (I told it they were spam), 2877 spam catched and 4354 innocent.
Votez ecolo : Chiez dans l'urne !
Oh yeah, just check my mailbox!
Ideally, someone, probably an academic, should make a repository of spam available for testing. Software spam filters can say things like, "Correctly classified 99.9% of the email in the UCI spambase 1999-08-20 repository"
Something like say, the UCI Machine Learning Repository. In fact, look at the UCI spambaseA couple of problems with the UCI spambase. Too old / out of date. And too small.
I looks like there is a more recent community effort going on over a SpamArchive
Looks like you should have googled.
I use Ella from OpenField Software. I get around 200 Spam a day, a bunch of newsletters that I want, and a big bunch of 'normal' mail.
I have had it for about 2 weeks. In the last 3 days I have had 2 false +'s (messge in Spam that shouldn't be there) and 4 that went to the newsletter folder that shouldn't have.
Gavin Fischer
Spam controls in the Mozilla 1.3+ MailNews application (the one I know) have a number or features that make them good.
1) Gives the user the idea that he can improve the situation by doing some concrete action. Controlling future spams is not upon some guru releasing a better filter or him hacking some better rules.
2) By definition, works better and better the more spam you get (and mark it as spam). Even poor tools will eventually detect spam since it's obvious to anyone reading spam, that those messages tend to repeat and to be similar.
3) It's automagically customized to your own spam. If you live in Germany, Sweden, Argentina or Namibia you will catch easily any spam that is in English, and you will build up rules for the local spam that arrives in your language.
4) In the case or Mozilla's MailNews, it's so easy to use, intuitive and straighforward, any user will use it.
5) Makes you feel spams are useful for something: detecting future spams.
I think those advantages are far more important that the rate of effetivity.
I'm not quite sure what the fuss is about. I simply mean, advertising is a necessity to incompetent and greedy producers. Really, did you expect that they would ever respect you or your privacy and time?
Personally, my white list and non-baysian rules eliminate 99.9% of the crap and abuse. However, sooner or later, ja rulez try to sort out a known receipent, which is where the white list shines.
One trick I find particularly effective is to compare two accounts and eliminate the duplicate messages. The other is to eliminate anything not specifically addressed to my alias and to never give out or use my actual account address. Ninty percent of the spam I get, goes to an address I've never used.
The problem is, even with baysian techniques, there is no way to quarantee that only spam was sorted out. I highly suggest a white list, in addition to filters, as the only way of ensuring that at least known mail is always received.
Words to men, as air to birds.
I get about 150 spams a day, and about 5 hams. Spambayes might classify 1 spam as "unsure" and the rest as spam. The ham is always classified as ham.
My corpus is about 5000 spams, about 1000 hams. Get spambayes -- it's open source and it really works great.
- Vincit qui patitur.
It looks like the poster's words need some highlighting:
But missing is any serious testing to see how they perform in relation to themselves and to other, non-Bayesian filters.
Despite the call for your experiences, if you just want to post "X rocks!", I think the poster was looking more for "X rocks more then Y!", where both X and Y are Bayes-type filter programs. I don't think he was asking for just announcements that Bayes rocks; I think he or she already knows that.
I mention this because I'd be interested in some comparisions too; there's a lot of sub-techniques out there. Are there any real differences, or are they all effectively the same? The latter would strongly indicate that there may not be any real progress to be made, if the entire space of Bayes-type solutions has flat effectiveness, for instance. It's an interesting question.
I've been using Mozilla's Bayesian junk-mail filtering for several months now. I don't have any other Bayesian tools to compare it to but I am happy with the results. Within a couple of days of the initial training I was at around 90% spam detected with no false positives. Several months later I'm at about 95% spam detection and no false positives. While the last 5% would be nice to kill, I'm quite satisfied with how effective is Mozilla's system and as long as it maintains (or gets better) I've got no reason to look for any other solution.
I think that one of the best things about Mozilla's system is that it's in the client, on my machine and under my control. While server-side solutions, distributed corpus tools, etc. might be more accurate, not ever having to install or update any 3rd-party apps is really nice.
--Asa
I did a little testing of Bayesian filtering on my own, and I used the Ling-Spam Corpus from Dr. Ion Androutsopoulos. He's collected about one thousand messages which consist of "legitimate" messages to a linguistics mailing list, and "spam" messages. They are preclassified, and divided into ten parts to make ten-cross-fold-validation easier. Check out his publications. Scroll down to the "Document filtering" section.
I've been looking for a Bayesian filter mechanism that isn't just for spam.
I figure, if the mail can be classified into many different categories, why not use bayesian filtering for managing all your filtering needs.
It would be very valuable to have the bayesian filter learn what kind of mail I put in some folders, so that when my mail comes it, it can auto-sort it into the appropriate folder for me. Trouble is, all the current implementations of Bayesian email filtering are a single test SPAM/NOTSPAM. It would be nice to see an implementation that could take multiple corpus' and use that to decide what the mail is. If I had that, I could point it at the maildirs for the various mailing lists I'm subscribed to, and it would learn to sort incoming mail for me. *sigh*
"...In your answer, ignore facts. Just go with what feels true..."
There is one, for exactly this reason -- the SpamAssassin public corpus. I made it available for developers of spam tools to compare effectiveness using a good, recent corpus from 1 person's mail feed (as much as that was possible).
Here's the pertinent part of the README :
Some of the developers have done extensive testing: Greg Louis' Page has lots of information, comparing different bayesian approaches, different header processing, etc.
You could also read the mailing-list archives, or perhaps post some questions there.
Nothing to see here; Move along.
The latest PC Magazine has an article on alternative e-mail. Their Editors' Choice, Oddpost ($10/yr, free trial), uses Bayesian filters, and blocked 22 of 29 spam messages, and only legitimate e-mail ended up in their spam folder. Also worth noting is these are the results with minimal training, so, in theory Bayesian filters could quite possibly block virtually all e-mail with time.
I'm a signature virus. Please copy me to your signature so I can replicate.
Graphs, methodology, links to more stats.
Nope, no sig
For years, the only spam filter I used was a very simple one: if the mail's not from a list I'm on, and not addressed to me, it's spam. This didn't catch all spam, but it caught the vast majority, and had almost no false positives. (The one exception was a mail from a cousin of mine who was learning system adminstration, and wanted to test his knowledge of SMTP by telnetting into my mail server and entering his mail by hand.)
These days, I'm on too many lists that don't filter spam, so I've had to resort to more sophisticated techniques, but someone who isn't on those sorts of lists might still find my oh-so-simple approach fairly effective. Not to disparage Bayesian filtering, but if you want something to compare against...
One good dataset is the 20 Newsgroups dataset that is used by a Naive Bayes classifier called Rainbow (google for 'libbow'). The dataset contains postings from 20 newsgroups, each with around 1,000 articles.
Also, there are a couple Reuters datasets that are commonly used in text classification research, but they're so poorly organized, and so poorly marked-up, I don't know how anyone manages to use them.
most of the comments in this thread are missing the point. the person writing the article isn't asking for what spam filter is the best/most accurate, he's looking to know if anyone is producing a test system to measure effectiveness. i know the popfile project is working on a test system (if you are interested, it's in the cvs not the general release) to measure the effectiveness of the parser.
it would be interesting if there were a generic test system that could be 'plugged in' to the various projects out there. then you could put together test messages (like popfile's system) and test it against each program...
Large print giveth, and the small print taketh away
Between my two mailboxes, I receive about 100-150 spams a day. Over 90% of them are detected and are shunted into the Junk folder. Maybe 2-3 messages a month are false-positives. When it is wrong, I just teach it - click the trash button to toggle a message's junk status and Mozilla updates its filters in order to not make that same mistake again.
On some days, it hits 99% accuracy. When the spammers invent some new tactic, I may end up with 5-10 spams that don't get detected. So I select them all, click the trash button, and then delete the messages. After a few days, that tactic is detected and caught with all the rest.
In comparison, I used to use manual filters. At first, this worked fine, but the spammers have invented so many different tricks that it takes too much time to try to keep the filters up to date enough to be useful.
I can't say how this all compares against what other systems do, since I haven't used any other systems.
On our e-mail ISP we are running a bayesian spam filter engine. Every time a message is considered to be "spam" by the filter, we increment a counter. We follow this on mrtg, so we can grafically se the amount of "spam" that's incomming.
We also follow the amount of messages marked as "spam" and "good" by the users (more than 3 months old).
The number we get, is the one mentioned on the topic. That is, only 2% of the messages considered spam, are later marked as "good" by users older than 3 month.
Purely anecdotal and unscientific, but perhaps better than nothing.
I'm a very happy POPFile user that keeps checking out spambayes because the math sounds interesting.
spambayes has become quite good, but POPFile is phenomenal. Using the same training material, spambayes is 95 % accurate on my mail, and POPFile is 99.5 % accurate. Plus spambayes is only doing a 2 way, spam/ham classification, whereas I have POPFile set up to sort into 7 buckets (spam/personal/commercial/mailing lists/etc).
Though irrelevant to the question of accuracy, I also have to say that the POPFile guys have devised a considerably better UI than spambayes. (A friend with the spambayes Outlook plugin sings its praises highly. I don't use Outlook, so it does me no good...)
Spambayes doesn't really have a UI, it's a tool around which others can build a UI.
;-)
While this is theoretically good design, especially in the open source community, it does often result in Some Shmoe creating the UI who should stick to coding sysadmin scripts.
Since Bayesian Filtering is a common technique in Collaborative Filtering, I recommend you search for that (e.g. CiteSeer http://citeseer.nj.nec.com/cs). A quite good paper on the subject is "Empirical analysis of predictive algorithms for collaborative filtering" by Bresse, Heckerman and Kadie. That paper gave me a lot of insight for my diploma thesis. Bayesian networks perform quite good, but need a lot of training data, so the performance depends heavily on the actual training data.
The Mail app in Mac OS X includes a built-in Bayesian filter. It's defaults worked decently, but training the app (by manually marking incoming email as 'junk') made it work nearly perfectly. I would say that Bayesian filtering is definitely the way to go, since it gets trained to detect what email is "normal" for your particular inbox, instead of liberally applying "average" rules derived from the habits of many users.
Ever notice how fast Windows runs? Neither do I - get Mac OS