Bayesian Filter Testing?

pu33y asks: "Since the publication of Paul Graham's A Plan For Spam, several programs that perform Bayesian filtering having become available, including CRM114 and Bogofilter. But missing is any serious testing to see how they perform in relation to themselves and to other, non-Bayesian filters.Searching Google has turned up nothing and when I asked Paul Graham, he was unaware of any such testing, as well. Can anyone point to any such testing or provide the results of their own personal experiences with Bayesian filters?"

Comments

The good think about these tools

[FedeTXF]

Spam controls in the Mozilla 1.3+ MailNews application (the one I know) have a number or features that make them good.
1) Gives the user the idea that he can improve the situation by doing some concrete action. Controlling future spams is not upon some guru releasing a better filter or him hacking some better rules.
2) By definition, works better and better the more spam you get (and mark it as spam). Even poor tools will eventually detect spam since it's obvious to anyone reading spam, that those messages tend to repeat and to be similar.
3) It's automagically customized to your own spam. If you live in Germany, Sweden, Argentina or Namibia you will catch easily any spam that is in English, and you will build up rules for the local spam that arrives in your language.
4) In the case or Mozilla's MailNews, it's so easy to use, intuitive and straighforward, any user will use it.
5) Makes you feel spams are useful for something: detecting future spams.

I think those advantages are far more important that the rate of effetivity.

Spambayes!!!!

[Arkham]

I use spambayes. It's written in python and is amazingly accurate.

I get about 150 spams a day, and about 5 hams. Spambayes might classify 1 spam as "unsure" and the rest as spam. The ham is always classified as ham.

My corpus is about 5000 spams, about 1000 hams. Get spambayes -- it's open source and it really works great.

Hey everyone...

[Jerf]

It looks like the poster's words need some highlighting:

But missing is any serious testing to see how they perform in relation to themselves and to other, non-Bayesian filters.

Despite the call for your experiences, if you just want to post "X rocks!", I think the poster was looking more for "X rocks more then Y!", where both X and Y are Bayes-type filter programs. I don't think he was asking for just announcements that Bayes rocks; I think he or she already knows that.

I mention this because I'd be interested in some comparisions too; there's a lot of sub-techniques out there. Are there any real differences, or are they all effectively the same? The latter would strongly indicate that there may not be any real progress to be made, if the entire space of Bayes-type solutions has flat effectiveness, for instance. It's an interesting question.

Mozilla's Junk-mail Filters

[asa]

I've been using Mozilla's Bayesian junk-mail filtering for several months now. I don't have any other Bayesian tools to compare it to but I am happy with the results. Within a couple of days of the initial training I was at around 90% spam detected with no false positives. Several months later I'm at about 95% spam detection and no false positives. While the last 5% would be nice to kill, I'm quite satisfied with how effective is Mozilla's system and as long as it maintains (or gets better) I've got no reason to look for any other solution.

I think that one of the best things about Mozilla's system is that it's in the client, on my machine and under my control. While server-side solutions, distributed corpus tools, etc. might be more accurate, not ever having to install or update any 3rd-party apps is really nice.

--Asa

Ling Spam Corpus

[bpfinn]

I did a little testing of Bayesian filtering on my own, and I used the Ling-Spam Corpus from Dr. Ion Androutsopoulos. He's collected about one thousand messages which consist of "legitimate" messages to a linguistics mailing list, and "spam" messages. They are preclassified, and divided into ten parts to make ten-cross-fold-validation easier. Check out his publications. Scroll down to the "Document filtering" section.

Re:Not Just for SPAM

[nrosier]

Have a look at Ifile (http://www.nongnu.org/ifile); while I'm only interested in spam/no-spam filtering, I once tested this filter to filter a mailing-list. It did a pretty good job.

BogoFilter

[bobbozzo]

BogoFilter is an open-source bayesian spam filter...

Some of the developers have done extensive testing: Greg Louis' Page has lots of information, comparing different bayesian approaches, different header processing, etc.

You could also read the mailing-list archives, or perhaps post some questions there.

Re:Ja rulez

[Blkdeath]

The problem is, even with baysian techniques, there is no way to quarantee that only spam was sorted out. I highly suggest a white list, in addition to filters, as the only way of ensuring that at least known mail is always received.

With Mozilla, you get the best of both worlds. You've got Bayesian filtering with an optional whitelist component. You can select any of your address books as the source of your whitelist (default is "Personal Addresses"), so any of your friends can send you all the SPAM they want without being caught. ;)

Being optional, you can choose to disable it if, say, your friends addresses have been harvested for "Joe Job" SPAM runs. (I know one or two of mine have).

I've actually used the whitelist to my advantage when I requested a sample of a particular new type of SPAM from him so I could watch for it and mark it if Mozilla missed it.

Which brings me to the other big advantage of Mozilla/Bayesian; when SPAMmers adapt, so does it. New SPAM type? Click the trash can and it'll go away.

Nothing can really be a perpetual 100% guarantee of blocking SPAM, but IME, Bayesian filters are the best possible solution we have right now and that's why I emphatically reccomend them to all my friends, family, and customers.

Try here

[drew_kime]

From here:

I've been tracking email spam trends for a while, my personal accounts are going from 3-6 spams daily in 2001 to about 30 spams daily at present. I filter this with SpamAssassin?, so the inbox impact is pretty slight, but the traffic is becoming significant, and the trend (doubling in four months) is downright troubling.

Graphs, methodology, links to more stats.

the comments are missing the point...

[zonker]

most of the comments in this thread are missing the point. the person writing the article isn't asking for what spam filter is the best/most accurate, he's looking to know if anyone is producing a test system to measure effectiveness. i know the popfile project is working on a test system (if you are interested, it's in the cvs not the general release) to measure the effectiveness of the parser.

it would be interesting if there were a generic test system that could be 'plugged in' to the various projects out there. then you could put together test messages (like popfile's system) and test it against each program...