Bayesian Filtering For Dummies

Dynamoo writes "Bayesian filtering for spam is awfully clever stuff, touched on by Slashdot several times before. There's a very accessible article at BBC News explaining in fairly simple terms the drawbacks of current keyword-based filtering. It's slightly ironic that the BBC, through the commissioning of Monty Python, also gave 'spam' its name. Those Vikings have a lot to answer for."

Comments

More Spam!

[James Littiebrant]

I have used a bayesian filter for some time now and while it is the BEST filter type I have ever used nothing is 100% reliable. While this is the best technology for the average user it is most cirtainly not perfect. Instead I use a combination of moderate bayesian filtering and good old fasion "block sender" filtering.

Here's one I've used

[wiggys]

I set up Popfile a few weeks ago at work to stop the deluge of spam one of our POP3 accounts was getting. I've never used a spam filter before (other than the usual basic keyword-based ones) and I must say that bayesian filtering is very impressive!

I find in our case it stops 98-99% of spam dead in its tracks. There have been a few false positives, and you do need check from time to time just in case an genuine emails are misclassified, but it's surprising just how quickly the filter sorts the wheat from the chaff.

Don't expect miracles but they can save you a lot of time... what I find cool is that it learns so quickly, almost like a complicated neural net should, but it's such a simple idea. I wonder if there are any other uses for this kind of thing?

Crude but effective

[MrWorf]

I simply got to the point that I could count the number of real emails on my hands. So I reversed my previous filter. Instead of filtering spam to my spam folder, I made it default *ALL* mail to the spam folder except from certain known addresses (such as work, friends and my own domain). So far, it has only made one wrong decision, and that was because I hadn't written the email address of a friend correctly.

This is waaaaay better than any other filtermethod I've tried and requires no learning period at all :)

Re:Required Reading by E-mail Users

[dJCL]

From my understanding of his full explanation(I read it a while ago, can't remember where, dig around some) each e-mail has every word examined and given a rating from 0.01(good) to 0.99(spam), then the 15 words farthest from 0.50 are selected, some averaging is done and if the score is over some threshold(say 0.90) then it is called spam and trashed, I use spamunition for my outlook e-mail(working on moving my e-mail over to linux, hopefully soon, so I can del my windows boxen) and it can give the stats for each e-mail and it appears to use the same formula...

Part of the reason this all works is that spammers slowly change their wording over time to beat the static filters, but the baysian filter will still catch it on other parts of the message, and add the new wording to the db... the only spam that ever get throu to me now is stuff that is worded exactly like a normal e-mail, and even then they have a hard time, yet all my friends have no problems...

I think the key here is to(with this software) never delete any e-mail, spam goes to the spam folder, sort the other stuff, and stuff you wanted, but don't need, move to another folder just so allow the filter to know what to look for... I have 5200 spam e-mails saved and about 1000 legit mail saved and my accuracy level is about 99.9...

Read up on it, this stuff really does work.

Enjoy

Re:A bit of info on Bayesian filtering

[letxa2000]

A gynecologist probably wouldn't have a corpus that indicates that "sex" is a .97 spam probability. That's the great thing about Bayesian: the spam probability for each word depends on the mail and spam YOU receive. It works dang well, just as Paul Graham claims. I'm averaging 99.7% accuracy this week, and the one spam that got through was written in German.

Re:Hmmm

[letxa2000]

I can see the casual (mis)use of this technique by your average user rapidly becoming a problem - putting just one email from a legit e-mail sender into the bayesian filter could concievably snowball into a block on a lot of legit traffic under certain circumstances.

It's natural to think that is the case, but in reality it isn't. Accidentally putting one email in the wrong corpus ("good" or "spam") will not be enough to kill you. If you consistently fail to put them in the right corpus then over time, yes, things would snowball. But that'll only happen over time. A mistake now and then isn't enough to mess things up.

Re:A bit of info on Bayesian filtering

[GnuVince]

No, because if they have a lot of legitimate mails with words like "sex", "sexy", "penis", "vagina", "viagra", etc., the filter will adapt. That's the whole point. For PG, "sexy" is a sure sign of spam, but for a sexologist, it is not. You train the filter to recognize your spam. So if "sex" appears as much in your legitimate mail than in your spam, "sex" will not be considered a trace of a spam.

Bayesian filters adapt, that's why they work so well.

Re:I don't receive spam

[letxa2000]

There are in fact two big problems with Bayesian filtering (or any content-based filtering) from the perspective of an ISP or company... 1) one person's spam is another person's necessity

But that's why Bayesian advocates every user having their own Bayesian statistics. It's not a "one size fits all" for the entire ISP or company, as is the case with most keyword filters. Every user has a different set of Bayesian statistics which is why it is very difficult for spammers to get around this filter--they have no way of knowing what words are in each users' statistics.

2) you still have to waste your bandwidth and CPU before you reject it.

It's better to waste your bandwidth and your CPU than to waste the time of those receiving the spam. IMHO...

So Bayesian filters are a good tool of last resort, but there are many other tools that should be used too.

The quicker everyone uses Bayesian filters (as opposed to waiting until all the other filters are incapable of keeping up with spam) the sooner the spammers will be in trouble. I personally use both a Bayesian filter with an up-to-date blacklist of known spamvertised domains, etc. I find that, quite simply, the simple keyword filters catch spam from known spam sites and Bayesian catches the rest. But if I turned off my normal filters Bayesian would have caught it all since those spams are always assigned a high Bayesian score, too. It almost makes sense to turn off the other filters, but they can be useful if a spammer comes up with a truly unique spam and someone else has already identified the domain name. It's rare, but it can happen. So a combination of technologies is probably the best... but a combination that lacks Bayesian is a combination that could be better.

0.0001% response rates

[rippie78]

The sheer number of spam mail sent means that even tiny response rates, reportedly 0.0001%, means junk mailers turn a profit.
Are we missing a critical factor of the end user who actually responds to SPAM?
If spammers survive on 0.0001% response rate, how many people are actually clicking/buying? Are these people who provide the customers for spammers going to stop or use any sort of filters?

Re:Ironic?

[DavyByrne]

Actually, I've long wondered whether Alanis was quite clever in choosing a title for that song.

You see, none of the events she describes in the song is an example of irony, making the choice of the title "Ironic," well, ironic.

Re:Yes, we must filter out the dummies

[bluelan]

This wouldn't work.

Baysian filters for spam work because spam has a significantly different vocabulary distribution than useful e-mail. This is true because spam must deliver a commercial message and play on people's uncertainties.

Good trolls, on the other hand, look ALMOST like insightful, well written articles. The vocabulary distribution in good trolls is not significantly different than the vocabulary distribution of useful posts. So, Baysian filters would be useless, unless you come up with some smarter characteristics on which to train the filter.

You could easily develop a filter for ascii-art porno. But, those are offtopic or flaimbait, not trolls.

Re:Brief Tech Notes on Bayesian Filtering

[Ian Bicking]

Spam naturally seems to come in several categories - porn, penis enlargements, mortgages etc. However, it's unlikely that any one spam will simultaneously advertise porn and mortgages. Simply having a "spam" and a "not" category will not take advantage of distinctions such as that.

Why does it matter what category? To the user they don't care what kind of spam, merely that it's spam. And this isn't just a UI issue -- the filter is not meant to indicate authoritatively what is spam and what is not. Instead it learns what the particular user considers spam. You're only going to introduce inaccuracy if you create more categories, because the user is sometimes going to miscategorize spams (e.g., porn in penis enlargement). The user is not invested in the result of that subcategorization, so it's not a good goal for training.

Certainly there are other categorizations that are useful, e.g., work vs. private mail. Bayesian techniques can be used for further categorization, but they should only be used to categorize as far as the user cares to have their mail categorized.

Bayesian techniques for non-spam wouldn't be that useful, anyway, because non-statistical rules generally work well for everything but spam -- it's only because spammers are specifically trying to defeat non-statistical rules that we need statistical analysis. The only other place for Bayesian techniques, IMHO, is where the user can't articulate the basis of the categorization they desire (but that's probably quite common).