Slashdot Mirror
Sorting the Spam from the Ham
MrClever writes "The Sydney Morning Herald (Aust) is running an article about the merits of Bayesian filtering and a good plain-english description of how it works. Might be handy if you need to explain it to non-technophiles. The main thing that may be useful is a Bayesian spam filter written to drop straight into Outlook 2k/XP available here and written in Python by Mark Hammond."
Math buffs might enjoy reading
these pages
or browsing
this writeup
and its many links.
What happens if Slashdot runs a Bayesian filter which runs a day after the stories are posted and programs itself with all the -1 comments as "Spam" and all the +5 comments as "Ham". Then let the Bayesian filter adjust all incoming messages by up to 2 points.
I bet it'd work - and imagine if we did it to stories too! Maybe it'd reject all Taco's dupe submissions.
is a scalable popfile for larger organizations. If I could get popfile (with its super-easy-to-train/use-web-interface) that would run on my linux server, scan my IMAP mail server (well, incoming mail would actually work fine, too. I've heard they have a smtp plugin for it in cvs), and then have a popfile config page for each person, or mayby tie it into the imap/smtp server's login. THAT would rock. I've heard spamasassin does Bayesian, but I couldnt see how it was trainable (and I dont want other people on my server to read each others mail, obviously).
My own personal account is on a shared server at pair.com, and I run SpamAssassin (the perl script, can't put the spamc/d on there since I'm not root).
. zip
I have written on here before how I have saved myself a lot of hassle over the last few months by installing SA. I now stop 100+ messages a day (usually more like 140 now).
My stats tell me that since Feb, I've stopped over 15K Spam messages. Hot damn.
Where I currently work now we have Exchange and I wanted SpamAssassin on there, but we weren't getting the money approved to put it on.
So I hacked in SpamAssassin via an Exchange 2000/2003 EventSink.
If you want the code for it, feel free to grab it from http://www.cardboardutopia.com/ExchangeSpamFilter
But do note that if you have many users on your machine, you aren't going to want to use this - an EventSink on Exchange runs in serial, so SpamAssassain's Perl script (the spamc/d doesn't work under Win32) will get executed on every incoming mail, and it will have to wait until it is done before it gets the next one.
We process about 2000-5000 incoming messages a day and it does okay, but we have a very light load.
There are some odd things afoot now, in the Villa Straylight.
I don't know if I'd want it in Python, though... it does seem to be a good deal slower already than other spam filtering methods without putting it in a scripting language. Getting it in Outlook can only be good for the net (can Bayesian be applied to things like spam from Internet virii as well?)
I sat on the E-Mail policy team (a branch of the Strategic Planning team) for Miami University (Oxford, OH, not Florida) this last year (as a technical advisor, student and support desk employee. We looked at all sorts of spam solutions, as the president decided this should be a main focus (apparently the Viagra adds hit a bit too close to home for comfort ;)).
The problem in the educational market, though, is that, not being a business that can make rules and force people to live by them, educational establishments have annoyed customers (students and faculty) sometimes if any spam is blocked. (research, etc) False positives absolutely can't be tolerated. So a ranked system (spam assasian) that suggests the possibility of spam is not on the best but the only solution we have avalible. Mail will be ranked and users can make rules that trash everything but a guarenteed perfect mail, if they so desired. Or they can leave them all alone. So intelligent filtering is a necessity, not just a bennefit.
On another page, I had an odd place during this discussion of the team. I do not receive spam. (Please, don't start now). My MUOhio.edu address simply doesn't get a single piece of spam e-mail. I have had the account for two years. I have over 3000 messages in various folders. And none are spam at all. I just haven't signed up for anything with it. I put the e-mail addy on webpages too (that I author) and haven't gotten a single thing. But oh my the trash "spam" account gets 60 a day. On AOL. That blocks 80% of incoming mail. Ironically, they had MUOhio.edu blocked weeks back.
I haven't posted in so long, my sig is out of date.
I've noticed that the spam that has been getting through my Mozilla filter are the ones with innocuous sounding subjects and an embedded image.
Could this be the future of spam?
Does anyone know if any spam filters pick up on this patern or lack of pattern (after all there are no words in the body usually.)
They work pretty well for me, but nowhere near flawless. Some days I get 25 messages that go into the spam folder and only 3 in my inbox, some days I get about 10 in the spam folder and 5 in the inbox... It's a lot better than nothing. The real reason I run Mozilla for mail is the HTML rendering, which is better than any other mail client I'm aware of; The secondary reason is the bayesian filtering, and the tertiary is Enigmail, though no one I know bothers to use encryption anyway.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
The number of false positives is almost nil, and the ones that do get hit are spammy looking autogenerated reciepts from purchases I've made.
This is quite possibly the only complaint I have about spambayes, too, and it's not even that big a deal to me. After about a month of collecting spam in its own folder (named SHIT, oddly enough), it had learned enough that I was able to dial down my SpamAssassin settings (I use an old version of SA still, too, without the bayesian stuff built in -- too lazy to switch; spambayes works well enough that it's not worth it.) I check my incoming spam folder once or twice a week now, as opposed to once or twice a day when I only ran SpamAssassin at a relatively forgiving (4.5-5.5) setting.
There are a few thousand spams in SB's crap folder now; it's gotten so good that I can't really remember the last time I've had something miscategorized as spam, and of the 50-60 spams I get per day, usually only one or two make it through to my inbox, if that. Half of the time, I don't get any at all.
If you didn't have a reason for installing a Python interpreter before, now you do.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
Much of the spam that gets past it is so minimalist it cannot be blocked by a Bayesian filter. I get messages like this:
It's like someone is trying to put so little in the message, that there is nothing to filter. If only they would use the stock "We are sending you this because you opted-in on it. Click on this link to remove your address." If they used that, I'll never see the message; SpamProbe will grab it. But how could I train SpamProbe to detect the minimalist ones, without blocking everything forever?
So far I don't get too many of the minimalist ones, and I just hit delete. If it becomes widespread, I'll have to start using Vipul's Razor or something.
The other kinds of spam that get past SpamProbe are the ones that have rampant misspellings. Since none of the words are in the database, they don't match as spam terms:
I really think that I should write a filter that spell-checks an email, and rejects it if over 50% of the words with 5 or more letters are misspelled.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
Suppose
...)
1. I have a friend who uses the same kinds of words as I do and who uses Outlook (ok, an aquaintance, because friends don't let friends
2. An email virus attacks this person, snarfs up his Ham, runs a Bayesian filter on it and comes up with Spam specifically tailored for this person's aquaintances.
There's a science fiction book waiting to happen in here somewhere. If so, I own the SCOpyright on it.
about this kind of filtering is that it has to download the email content - not always as good idea, especialy in a Windows environment. Besides, I can identify spam just by looking at message header information. Sender, recipient, and subject line are nearly always enough. Plus I don't need to waste time, bandwidth, or get subjected to offensive graphics, or risk 1-pixel confirmations or getting hacked by the latest security issue. My homespun message header analysis program drops nearly all spam, and results in few legit email rejections. I score the headers based on missing recipient, sender info, keywords in subject, string match in sender email or name, punctuation count in subject line, number of contiuous spaces in subject line, plus a few other things that seem to run common in the spam I get. I can also permit certain email addresses to pass no matter the score. It's not fancy, but it works, and I never have to waste time drawing the whole content down to my local machine. What I do may not work for everyone, but it seems that in most cases it should, unless you get a lot of email from unknown (non-spam) sources - not typical for the average email user.
this is like inventing something as useful as the Knife, and using it only to attack salesmen. Why bother stopping with spam? Why not apply this filter to, say, absolutely everything? Since I just said "absolutely everything", I wont bother giving examples.
Training something to know how likely something is to be true, that sounds too useful to waste any time with on spam at all.
-- 'The' Lord and Master Bitman On High, Master Of All
First, I'd refer you to my /. Moderation Aphorism #1. Second, I'll give a serious answer to your serious observation:
I use MS Office under Crossover Office because it gives me the features I want (admittedly, one of them is the ability to share identically functional documents with Windows users) so I definitely agree with your perspective. In the case of Mozilla, there has been a great ruckus around here about spam, and I kept telling people it didn't affect me because I used Mozilla w/ Bayesian filters. Additionally, Outlook's rotten record for relaying mail worms has been a problem to me as a sys admin. Independent of the calendar/groupware features, in my immediate area, most people use Outlook as a mail client out of inertia because it came with Office and refuse to switch because of fear of the unknown rather than out of a choice based on features.
Trouble making decisions? Just flip for it.
Bayes rocks, been using it with spamassassin and it kills 99% of my spam. The problem is when some asshole spammer uses my email address in the 'From' header of his spam ... then I get scores of 'user not found' or 'virus detected' emails from legitimate mail servers ... it's not spam, but it's just as annoying. How do you guys deal with this problem?
(Score:-1, Wrong)
I use outlook because my clients use outlook (though mostly I just use the awsome web interface that fastmail.fm provides). My clients use outlook because it has great, integrated calendaring and it syncs with their various PDAs. Such is life.
I recently reviewed 7 client-side spam filters and ended up picking Spambully. It's not free and it's not perfect but for our environment (Win/outlook 2k2 w/ a weird mirapoint IMAP server and multiple PCs per user (so email needs to stay on the server)) it was the best. Very tight outlook integration (i'm a little worried about instabilities but so far it's smooth) and baysian.
But it's really just the best of a bad lot. It's great to see someone working on an open source filter that might work w/ IMAP - we can't have enough of these since right now, well, we have almost none.
closed minded is as closed minded does
Have you tried running a Bayesian filter on many messages at once? The Mozilla implementation hangs the mail app for a few seconds on a 1.4GHz Athlon when going through a hundred or so messages. Assuming Slashcode would implement it through Perl, it would be even slower. For reference, running Spamassassin with Bayes filtering (Perl scripts, not spamc) isn't exactly speedy. Going through several messages brings CPU usage close to maximum.
Bayesian filtering on comments would be too resource costly. A more plausible application would be to run stories through a Bayes-style filter, creating a profile for each story that checks each new story with previous profiles so that dupes could be reduced. But that would not be as good as having the editor looking at the current front page (as SCO stories would look similar).
You guys are a bunch of hypocrites. You don't really want spam to stop. You love spam.
Every spam thread is the same: I use X, and it blocks 98% of my spam, with no false positives! I use Y, and it blocks 99.9% -- take that! Here, I use Z + Y with these custom Perl scripts I wrote that interface with procmail and stop 101% percent of spam! It doesn't matter, because I never get ANY spam! Spam is only because people buy things in spam! What morons! Bow before me, for I am 1337!
Spam gives you something to fight. Spam gives you an excuse to solve an interesting technical problem (i.e. separating spam from ham). Spam gives you a reason to boast. Spam gives you people to dislike.
Admit it.
You love spam.
Make sense. Consider classifying with a binary tree (e.g., first divide into spam and non-spam, then divide the no-spam into personal and business, and then divide personal into two groups, and so on). If each step can be done with 99% accuracy (something my experience with Bayesian spam filtering would indicate), then you could go 5 levels deep (32 buckets, if fully populated) and have roughly 95% accuracy. Not "very slight" decrease but still quite usable,
and the cost of misclassification wouldn't be very high anyway.
I've just been migrated to Notes from Outlook. Not a happy bunny till I discovered how powerful it is with stuff like agents.
The only thing I'm missing now is a spam classification tool like popfile for notes.
Government of the people, by corporate executives, for corporate profits.