Slashdot Mirror
Sorting the Spam from the Ham
MrClever writes "The Sydney Morning Herald (Aust) is running an article about the merits of Bayesian filtering and a good plain-english description of how it works. Might be handy if you need to explain it to non-technophiles. The main thing that may be useful is a Bayesian spam filter written to drop straight into Outlook 2k/XP available here and written in Python by Mark Hammond."
Math buffs might enjoy reading
these pages
or browsing
this writeup
and its many links.
But without spam, I wouldn't get any email!
I get to do something to stop my boss from enlarging his penis anymore... It's really starting to hurt.
I went to battle MC Escher but drew a blank
What happens if Slashdot runs a Bayesian filter which runs a day after the stories are posted and programs itself with all the -1 comments as "Spam" and all the +5 comments as "Ham". Then let the Bayesian filter adjust all incoming messages by up to 2 points.
I bet it'd work - and imagine if we did it to stories too! Maybe it'd reject all Taco's dupe submissions.
is a scalable popfile for larger organizations. If I could get popfile (with its super-easy-to-train/use-web-interface) that would run on my linux server, scan my IMAP mail server (well, incoming mail would actually work fine, too. I've heard they have a smtp plugin for it in cvs), and then have a popfile config page for each person, or mayby tie it into the imap/smtp server's login. THAT would rock. I've heard spamasassin does Bayesian, but I couldnt see how it was trainable (and I dont want other people on my server to read each others mail, obviously).
And if you could, would you really want to?
bloodninja: Baby, I been havin a tough night so treat me nice aight?
BritneySpears14: Aight.
bloodninja: Slip out of those pants baby, yeah.
BritneySpears14: I slip out of my pants, just for you, bloodninja.
bloodninja: Oh yeah, aight. Aight, I put on my robe and wizard hat.
BritneySpears14: Oh, I like to play dress up.
bloodninja: Me too baby.
BritneySpears14: I kiss you softly on your chest.
bloodninja: I cast Lvl 3 Eroticism. You turn into a real beautiful woman.
BritneySpears14: Hey...
bloodninja: I meditate to regain my mana, before casting Lvl 8 Penis of the Infinite.
BritneySpears14: Funny I still don't see it.
bloodninja: I spend my mana reserves to cast Mighty of the Beyondness.
BritneySpears14: You are the worst cyber partner ever. This is ridiculous.
bloodninja: Don't shit with me biznitch, I'm the mightiest sorcerer of the lands.
bloodninja: I steal yo soul and cast Lightning Lvl 1, 000, 000 Your body explodes into a fine bloody mist, because you are only a Lvl 2 Druid.
BritneySpears14: Don't ever message me again you piece.
bloodninja: Robots are trying to drill my brain but my lightning shield inflicts DOA attack, leaving the robots as flaming piles of metal.
bloodninja: King Arthur congratulates me for destroying Dr. Robotnik's evil army of Robot Socialist Republics. The cold war ends. Reagan steals my accomplishments and makes like it was cause of him.
bloodninja: You still there baby? I think it's getting hard now.
bloodninja: Baby?
--
bloodninja: Ok baby, we got to hurry, I don't know how long I can keep it ready for you.
j_gurli3: thats ok. ok i'm a japanese schoolgirl, what r u.
bloodninja: A Rhinocerus. Well, hung like one, thats for sure.
j_gurli3: haha, ok lets go.
j_gurli3: i put my hand through ur hair, and kiss u on the neck.
bloodninja: I stomp the ground, and snort, to alert you that you are in my breeding territory.
j_gurli3: haha, ok, u know that turns me on.
j_gurli3: i start unbuttoning ur shirt.
bloodninja: Rhinoceruses don't wear shirts.
j_gurli3: No, ur not really a Rhinocerus silly, it's just part of the game.
bloodninja: Rhinoceruses don't play games. They fucking charge your ass.
j_gurli3: stop, cmon be serious.
bloodninja: It doesn't get any more serious than a Rhinocerus about to charge your ass.
bloodninja: I stomp my feet, the dust stirs around my tough skinned feet.
j_gurli3: thats it.
bloodninja: Nostrils flaring, I lower my head. My horn, like some phallic symbol of my potent virility, is the last thing you see as skulls collide and mine remains the victor. You are now a bloody red ragdoll suspended in the air on my mighty horn.
bloodninja: Fuck am I hard now.
--
BritneySpears14: Ok, are you ready?
eminemBNJA: Aight, yeah I'm ready.
BritneySpears14: I like your music Em... Tee hee.
eminemBNJA: huh huh, yeah, I make it for the ladies.
BritneySpears14: Mmm, we like it a lot. Let me show you.
BritneySpears14: I take off your pants, slowly, and massage your muscular physique.
eminemBNJA: Oh I like that Baby. I put on my robe and wizard hat.
BritneySpears14: What the fuck, I told you not to message me again.
eminemBNJA:
BritneySpears14: I swear if you do it one more time I'm gonna report your ISP and say you were sending me kiddie porn you fuck up.
eminemBNJA: Oh
eminemBNJA: damn I gotta write down your names or something
"Study your math, kids. Key to the universe." -The Archangel Gabriel
I've now lost one of my primary arguments for switching my colleagues to Mozilla!
Trouble making decisions? Just flip for it.
My own personal account is on a shared server at pair.com, and I run SpamAssassin (the perl script, can't put the spamc/d on there since I'm not root).
. zip
I have written on here before how I have saved myself a lot of hassle over the last few months by installing SA. I now stop 100+ messages a day (usually more like 140 now).
My stats tell me that since Feb, I've stopped over 15K Spam messages. Hot damn.
Where I currently work now we have Exchange and I wanted SpamAssassin on there, but we weren't getting the money approved to put it on.
So I hacked in SpamAssassin via an Exchange 2000/2003 EventSink.
If you want the code for it, feel free to grab it from http://www.cardboardutopia.com/ExchangeSpamFilter
But do note that if you have many users on your machine, you aren't going to want to use this - an EventSink on Exchange runs in serial, so SpamAssassain's Perl script (the spamc/d doesn't work under Win32) will get executed on every incoming mail, and it will have to wait until it is done before it gets the next one.
We process about 2000-5000 incoming messages a day and it does okay, but we have a very light load.
There are some odd things afoot now, in the Villa Straylight.
I use Spambayes with Outlook 2000, and it takes a little tweaking, but it works as advertised. Ahhh, the magic of mathematics. Just now, brought up Outlook, checked my mail and three little messages offering a free Sony headset, 70% off cell accessories, and a chance to take an IQ test just got tossed into my spam folder. Thanks anyway, but I think that means I just passed my IQ test.
Every so often I go in and take out some old, old spam, just to make sure my current preferences are being represented and that's all the maintenance that's required.
This is, however, the second time I've trained the filter. The first time, it incorrectly identified my FreeBSD status mails as spam, and from then on was throwing those into the Spam folder. My own fault, though, since I hadn't included any of these messages in my representative ham.
If you run Outlook, download this filter and use it. You'll be doing yourself, and a world that doesn't need fat-injected, herbally enhanced penises, a favor.
Chr0m0Dr0m!C
I don't know if I'd want it in Python, though... it does seem to be a good deal slower already than other spam filtering methods without putting it in a scripting language. Getting it in Outlook can only be good for the net (can Bayesian be applied to things like spam from Internet virii as well?)
I've been using spambayes for months now and it really is quite amazing. Now, when I get the occasionaly spam in my mailbox it's actually interesting because I want to figure out why it made it in. The number of false positives is almost nil, and the ones that do get hit are spammy looking autogenerated reciepts from purchases I've made. It's made reading email a much more enjoyable activity.
-Adam
The first discovery I'd like to present here is an algorithm for lazy evaluation of research papers. Just write whatever you want and don't cite any previous work, and indignant readers will send you references to all the papers you should have cited. I discovered this algorithm after ``A Plan for Spam'' [1] was on Slashdot.
.03% false positives [4]. It's always alarming when two people trying the same experiment get widely divergent results. It's especially alarming here because those two sets of numbers might yield opposite conclusions. Different users have different requirements, but I think for many people a filtering rate of 92% with 1.16% false positives means that filtering is not an acceptable solution, whereas 99.5% with less than .03% false positives means that it is.
Spam filtering is a subset of text classification, which is a well established field, but the first papers about Bayesian spam filtering per se seem to have been two given at the same conference in 1998, one by Pantel and Lin [2], and another by a group from Microsoft Research [3].
When I heard about this work I was a bit surprised. If people had been onto Bayesian filtering four years ago, why wasn't everyone using it? When I read the papers I found out why. Pantel and Lin's filter was the more effective of the two, but it only caught 92% of spam, with 1.16% false positives.
When I tried writing a Bayesian spam filter, it caught 99.5% of spam with less than
So why did we get such different numbers? I haven't tried to reproduce Pantel and Lin's results, but from reading the paper I see five things that probably account for the difference.
One is simply that they trained their filter on very little data: 160 spam and 466 nonspam mails. Filter performance should still be climbing with data sets that small. So their numbers may not even be an accurate measure of the performance of their algorithm, let alone of Bayesian spam filtering in general.
But I think the most important difference is probably that they ignored message headers. To anyone who has worked on spam filters, this will seem a perverse decision. And yet in the very first filters I tried writing, I ignored the headers too. Why? Because I wanted to keep the problem neat. I didn't know much about mail headers then, and they seemed to me full of random stuff. There is a lesson here for filter writers: don't ignore data. You'd think this lesson would be too obvious to mention, but I've had to learn it several times.
Third, Pantel and Lin stemmed the tokens, meaning they reduced e.g. both ``mailing'' and ``mailed'' to the root ``mail''. They may have felt they were forced to do this by the small size of their corpus, but if so this is a kind of premature optimization.
Fourth, they calculated probabilities differently. They used all the tokens, whereas I only use the 15 most significant. If you use all the tokens you'll tend to miss longer spams, the type where someone tells you their life story up to the point where they got rich from some multilevel marketing scheme. And such an algorithm would be easy for spammers to spoof: just add a big chunk of random text to counterbalance the spam terms.
Finally, they didn't bias against false positives. I think any spam filtering algorithm ought to have a convenient knob you can twist to decrease the false positive rate at the expense of the filtering rate. I do this by counting the occurrences of tokens in the nonspam corpus double.
I don't think it's a good idea to treat spam filtering as a straight text classification problem. You can use text classification techniques, but solutions can and should reflect the fact that the text is email, and spam in particular. Email is not just text; it has structure. Spam filtering is not just classification, because false positives are so much worse than false negatives that you should treat them as a different kind of error. And the source of error is not just random variation, but a live hum
Would you use the phone if you had to listen to a 10-second brothel advertisement every time you made a call?
Yes.
Definately Yes.
Is that a feature I can have added?
http://use.perl.org
Eudora 6.0 beta has spam filtering which seems to be Bayesian. It's a little slower to learn than PopFile, but it's pretty good so far, and of course integrated with the Eudora UI.
http://eudora.com/betas
I sat on the E-Mail policy team (a branch of the Strategic Planning team) for Miami University (Oxford, OH, not Florida) this last year (as a technical advisor, student and support desk employee. We looked at all sorts of spam solutions, as the president decided this should be a main focus (apparently the Viagra adds hit a bit too close to home for comfort ;)).
The problem in the educational market, though, is that, not being a business that can make rules and force people to live by them, educational establishments have annoyed customers (students and faculty) sometimes if any spam is blocked. (research, etc) False positives absolutely can't be tolerated. So a ranked system (spam assasian) that suggests the possibility of spam is not on the best but the only solution we have avalible. Mail will be ranked and users can make rules that trash everything but a guarenteed perfect mail, if they so desired. Or they can leave them all alone. So intelligent filtering is a necessity, not just a bennefit.
On another page, I had an odd place during this discussion of the team. I do not receive spam. (Please, don't start now). My MUOhio.edu address simply doesn't get a single piece of spam e-mail. I have had the account for two years. I have over 3000 messages in various folders. And none are spam at all. I just haven't signed up for anything with it. I put the e-mail addy on webpages too (that I author) and haven't gotten a single thing. But oh my the trash "spam" account gets 60 a day. On AOL. That blocks 80% of incoming mail. Ironically, they had MUOhio.edu blocked weeks back.
I haven't posted in so long, my sig is out of date.
I've got this installed for Outlook XP. Either I don't have it configured correctly (likely) or it just doesn't work well. Even using the emails in the spam folder to 'train' it, it still misses messages.
Cruising the internet on my TI-99/4A @ a whopping 300 baud!
We need one too!
Enough already. Sure it was funny the first time around.. Good chuckle second time. Now its just getting old and annoying.
In case anyone hasn't tried it yet, the Bayesian filters in the mail client of the Mozilla suite are really impressive. They have worked close to flawless for myself.
I wrote an article on how to set up SpamProbe on a server, and make it easy to train. You could also use Bogofilter or any other trainable spam filter, set up the same way.
I get at least 100 spam messages a day now, and I only see about a half-dozen or so. SpamProbe deals with the rest, and I don't have any problems with false positives. (SpamAssassin thinks that ads for LinuxWorld Expo are spam, but as I have it trained, SpamProbe doesn't.)
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
I use PopFile. What I like about it is that it easily lets me use multiple personalities in Eudora, Outlook or any other mail client. Nice web based interface and a very active development community.
You can run it locally on Windows or Linux. But, you can also set it up on a server and then use it to filter e-mail from multiple client machines. That's what I like about it. I have a home machine in my basement office but also upstairs in the TV room. Unlike plug-ins that only work locally, I can have my reclassification decisions apply to multiple client machines.
Right now, they do not have multiple user capabilities so that my wife and I can both use the same instance and not have our classifications interfere with each other. However, you can set up multiple instances bound to different ports. The developers list multi-user capability as a priority.
Worth checking out along with the other choices.
For an article in an "IT tech" section of a paper, this is really very weak.
It really doesn't do much more than precis Paul Graham's arguments, then ends in a blatant plug for just one Outlook addon.
I suppose if there are still people in the column's audience who haven't heard this all before, and it gets the message out that spam can be effectively filtered, it's a minor goodness.
I've noticed that the spam that has been getting through my Mozilla filter are the ones with innocuous sounding subjects and an embedded image.
Could this be the future of spam?
Does anyone know if any spam filters pick up on this patern or lack of pattern (after all there are no words in the body usually.)
Having used the spam filtering built in to Mozilla for the last six months, I can testify to its effectiveness. In very little time at all, I'd trained it to send 95% of the filth to the spam directory and avoid doing the same for 95% of good mails. For me, not having to run a "middle man" piece of software was a real boon.
However, my life isn't totally spam free, as I find that I become neurotic about those 5% false positives that get unhelpfully moved to the spam directory, so still end up having to sift through the grot every once in a while. On the plus side, I now have a solution to my tiny cock problem, I've arranged cheaper home insurance and I have the email address of several horny co-eds who I'm assured are hungry for man juice.
http://www.davetansley.com - you proba
As I wrote only late last night, using Bayesian classification with only two categories (spam and "non-spam") is somewhat short-sighted, since if properly trained, a Bayes classifier can do a much better job than ordinary mail filtering (procmail, Mozilla or Mail.app filters, you name it).
In fact, if I had to bet on the next "killer apps", mail sorting and RSS filtering based on Bayesian classification would be right at the top of my list, based solely on the actual time-saving benefits for users. And I can't see any reason for Bayesian filtering not being included in Mozilla Mail and Apple's own (revamped) Mail.app.
I have to use Outlook at work, and after setting up Outclass (which requires POPfile) with several "buckets" to classify my corporate e-mail by project and field, I'm definetly not going back. Outlook, even with extensive use of Rules Wizard and categories, simply cannot cope with the diverse kinds of project-related e-mail I swap with colleagues, and Outclass is the only thing I could find that could deal with Exchange, PST folders and multiple Bayesian "buckets" categories.
Come on, do the right thing and tell Apple and The Mozilla Project that you want configurable Bayesian filtering on their mail clients.
Some of the departments at work have switched over to this:
http://www.canit.ca/
Haven't tried it myself, but it is apparently fairly competent.
Another liberal arts economy around the corner.
Heads up on another stock market buzz. Predictable daily fluctuations plus buying low and selling high for those that wish to capitalize on this (keep lawyers handy for insider trading abuses). Never keep to much money in the market at once, and have a good process of converting it into hard currency that is not prone to valuation swings.
I guess that last two fraudulent pushes for the most part went unpunished, we'll likely see more until we (nationally) hit a certain desaturation point.
Never keep more then 20,000 in at a time.
pawl -- Reverse engineering commercials since 1987.
I hate spam just as much as the next person, but I must admit, without it I wouldn't be the horse-sized love stud that I am. Thanks spam.
I have been using the Mozilla junk mail filter for a couple of months now. One pop mail account is one that I started using in 1996. It is a spam magnet. In the time I have been using Mozilla, it has accumulated over 12,000 spam messages. That should be plenty of training for the Bayesian filter.
Mozilla's filter does a reasonably good job at catching spam, but I still get a handful of messages every day that slip through the filter. The ones that slip through seem to be messages that have intentionally munged the spammy words with spaces, numbers, and misspellings. The spammers know that people are filtering, and they are successfully getting through the filter with their dirty tricks. Another trick spammers use is to send a message with nothing but a graphic ad. The filter doesn't have enough words to judge the the spam, so the message slips through.
I also had some 'ham' messages get filtered, so I still have the annoyance of having to check the 'junk' folder periodically for wanted messages. The filtering makes life easier, but it is still not an ideal solution to the spam problem.
personally, I use K9 and it filters out nearly 98% of my junk mail though i need to baby sit it a bit while it learns who my friendly emails are...
best of all. its free! and free of spyware too.
http://keir.net/k9.html
-- Betting on the survival of the media industry is a serious risk. I advise investing elsewhere.
Our problem is that 99% of people read email via POP, and POP only serves one mailbox per person. It is extraordinarily difficult to train everyone to use a spam filter individually, and yet installing one on the server can't work with POP's limitations. Frustrating.
Secondly, Microsoft is in the fray now. Bet any amount they will offer a authenticating email service that requires using Windows XP to work. It will work really well, you won't be able to communicate well with people who don't use it - standard tactics. They want to give people incentive to upgrade, and to stay locked in.
Much of the spam that gets past it is so minimalist it cannot be blocked by a Bayesian filter. I get messages like this:
It's like someone is trying to put so little in the message, that there is nothing to filter. If only they would use the stock "We are sending you this because you opted-in on it. Click on this link to remove your address." If they used that, I'll never see the message; SpamProbe will grab it. But how could I train SpamProbe to detect the minimalist ones, without blocking everything forever?
So far I don't get too many of the minimalist ones, and I just hit delete. If it becomes widespread, I'll have to start using Vipul's Razor or something.
The other kinds of spam that get past SpamProbe are the ones that have rampant misspellings. Since none of the words are in the database, they don't match as spam terms:
I really think that I should write a filter that spell-checks an email, and rejects it if over 50% of the words with 5 or more letters are misspelled.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
That's one of the main reasons why I switched to Thunderbird. After about a week of training, it only missed one spam email & I haven't had any false positives. It might also help that you can choose to exclude any mail from people that are in your address book from being filtered.
You might take a look at Spam Sleuth Enterprise I suspect it has what you want, since it has trainable Bayesian (individual to each user), works with any e-mail server, has a web client interface, and a lot more that you may or may not be interested in.
Bayesian filtering it's a miracle solution, but it does save about 45 seconds a day. That adds up to quite a bit over time.
If you're trying to be funny, well, sorry you're not really. If your point is that all these jokes are lame, well, yeah they may be, but if we didn't have these then where would all our internet cliches originate and flourish?
Suppose
...)
1. I have a friend who uses the same kinds of words as I do and who uses Outlook (ok, an aquaintance, because friends don't let friends
2. An email virus attacks this person, snarfs up his Ham, runs a Bayesian filter on it and comes up with Spam specifically tailored for this person's aquaintances.
There's a science fiction book waiting to happen in here somewhere. If so, I own the SCOpyright on it.
about this kind of filtering is that it has to download the email content - not always as good idea, especialy in a Windows environment. Besides, I can identify spam just by looking at message header information. Sender, recipient, and subject line are nearly always enough. Plus I don't need to waste time, bandwidth, or get subjected to offensive graphics, or risk 1-pixel confirmations or getting hacked by the latest security issue. My homespun message header analysis program drops nearly all spam, and results in few legit email rejections. I score the headers based on missing recipient, sender info, keywords in subject, string match in sender email or name, punctuation count in subject line, number of contiuous spaces in subject line, plus a few other things that seem to run common in the spam I get. I can also permit certain email addresses to pass no matter the score. It's not fancy, but it works, and I never have to waste time drawing the whole content down to my local machine. What I do may not work for everyone, but it seems that in most cases it should, unless you get a lot of email from unknown (non-spam) sources - not typical for the average email user.
Who cares indeed. Someone does, because there were a lot of little slashbots lining up to mod you down. Why promote good, funny posts? It takes so much less to just slap stuff down, right?
... so many tools. Sieg heil Taco.
ROTFL
SpamBayes is great if you're a Cloudmark Spamnet refugee like me, who left Spamnet after it went subscription. You probably have a folder full of hundreds of pieces of spam from Spamnet. SpamBayes can be trained on that folder of spam, so it can start accurately identifying spam without further weeks of training, unlike other Bayesian filtering solutions.
this is like inventing something as useful as the Knife, and using it only to attack salesmen. Why bother stopping with spam? Why not apply this filter to, say, absolutely everything? Since I just said "absolutely everything", I wont bother giving examples.
Training something to know how likely something is to be true, that sounds too useful to waste any time with on spam at all.
-- 'The' Lord and Master Bitman On High, Master Of All
I assume they are just simplifying the idea for the masses - but the article says that if "word one" shows up 60 times in spam, but twice in ham, then another message that has "word one" show up 40 times is more likely to be spam.
But technically I don't think that is how the actual process works - well, the same way - but they use sub sections.
If you only look at words, then you are really shooting yourself in the foot with bayesian analysis.
Instead it seems it would be much better to look at a block of characters at a time.
I know from my own experience playing with Markov matricies that 3 and 4 chars work fairly well in creating English text - so I would imagine that analysis would be similar.
I don't know how much the size of the chunk looked at matters as much as the fact that it looks at every character and as long as the chunk is smaller than the entire text message ideally.
So you would have a hash (dictionary if you use VB) and look at the first N characters of your e-mail and then put that in the hash and increment that value by one (in Perl: $myHash{$chunk}++).
Then you move the chunk forward one character and put that into the hash and increment it up.
That is the learning process.
Then the analysis process would be to move the chunk over the text and add up the hits and then use the final score as a way to determine how it should be dealt with.
The way SA does it is the Bayes analysis is just one part of the whole - so if the Bayes says it really is likely spam, but 5 other things say it is likely not, then depending on what you have set the Bayes trigger to worth, it could actually be decided as ham.
Anyway, I think the whole process is neat and I love playing with Markov Matricies.
There are some odd things afoot now, in the Villa Straylight.
Mark Hammond then wrote the Outlook plugin, which, admittedly, is considerably more code than SpamBayes, but not SpamBayes itself.
For the complete background on why SpamBayes is so good at what it does, and it's history, see:
- SpamBayes Background
Marc's is not the only application frontend for SpamBayes, here is a list of others:- SpamBayes Applications
No apologies for this my pedantry offered."The truth shall make ye fret" -- The Truth, Terry Pratchett
At work I have Outlook always running with the excellent bayesian FREE filter Spammunition www.upserve.com. I also do check the mailbox from home over a dial-up connection.
If I wouldn't use Spammunition, then I would spend a lot of time downloading spam messages; as it is right now, I get just the ham (several messages instead of many).
Serban
Isn't this similar to what is used in Apple's Mail.app for sorting junk mail?m l
http://www.apple.com/macosx/jaguar/mail.ht
A problem is the more classifiers you add the more likelyhood of an incorrect classification.
I've designed hierarchical bayes algorithms in attempts to deal with this situation but they add very little in the way of accuracy. It's much easier to filter mailinglists on a few keywords than it is to worry about more classifiers.
Especially irritating is when your corpus sizes are very dissimilar. If you only get 1 message a week for one corpus and 30 messages on another it can cause skewing since that 1 message is worth quite a bit more than 1 messages of the other corpus.
I've been using Spambully for Outlook for a few months now. Though it's not open source, it works quite well and has the added bonus of sending bounce messages if you want to the spam sender, as well as a challenge response mechanism to filter. There's a free trial. Just an fyi.
No, that would be Mark Hamill.
FYI, spamd is installed by default on pair.com accounts; you can call spamc from procmail. I spent most of yesterday afternoon setting it up, along with Spam Assassin and a nifty server side IMAP filtering... It's nothing revolutionary, but it's satisfying to have it setup ;)
Spam Assassin has bayes, but I've been getting 99% so far without it; expert rules work amazingly well, no need for learning. Methodology is the best way to foil spam. Have at least two/three email addresses: one public address (minimizing public exposure), one email as default reply-to (except for mailing lists) and optionally one address for close friends only. You can keep a low spam threshold then without much to worry about...
The idea of sorting the spam folder by score by injecting the rating into the subject (from this article on Reverse Spam Filtering) works wonders and it's easy to setup with procmail. If things get worse, I'll most likely be setting up temporary addresses that expire within weeks (for website contact & feedback), or a password system with password and explanation posted with contact details on my homepage)... it's almost as good as GPG/PGP for this purpose without the inconveniences for the other party.
I actually look forward to getting spam now!! hehe
I use spamnet by cloudmark. It catches everything. I can't remember the last time I had to click the "block" button. I'm very conscious of where my email ends up and I'm a hardcore advocate of email aliases. As a result, since September (last major crash), spamnet has blocked 4000 pieces while I've actively blocked only 11.
That's pretty f'n good in my book. So good, in fact, that I send all blocked messages to the "Delete" folder instead of the default "spam" folder and set outlook to permanently delete on close.
I have two concerns about this program:
--Money. They are now charging and pretty much deserve it from the average user.
--Reliability. This company could disappear tomorrow and sell off the server that has compiled spam data.
Since mathematics isn't going anywhere, I'm leaning towards switching to an open source Bayesian alternative but, as mentioned above, all my spam gets thrown out the door on contact.
What is the approximate training time of a Bayesian filter?
Laws are for people with no friends.
They didn't have email back in the 18th century when Thomas Bayes came up with this statistical method. It is simply being applied to spam, but has been used for other more "useful" purposes as well.
Actually, I think spam is a major problem and not a trivial application of statistics.
Well hot damn, didn't know it was already on the server. I looked all over the place on the server and didn't find it at all and when I wrote to them to get the okay to run the Perl script, they never mentioned it (I wasn't sure how psyched they would be if it was cpu intensive. But since they kill anything that runs over 30 seconds it doesn't matter too much).
I have a bunch of e-mail addresses and they forward around depending on where I am (less so now that I don't have a cell phone - I had filters that would forward messages to my phone as text messages if they were from certain people during certain times).
I'll have to look into the spamd - I wonder how long that has been there - I swear it wasn't when I started.
Thanks!
There are some odd things afoot now, in the Villa Straylight.
Bayes rocks, been using it with spamassassin and it kills 99% of my spam. The problem is when some asshole spammer uses my email address in the 'From' header of his spam ... then I get scores of 'user not found' or 'virus detected' emails from legitimate mail servers ... it's not spam, but it's just as annoying. How do you guys deal with this problem?
(Score:-1, Wrong)
I think Tom Mitchell did a good job in explaining the math in his book Machine Learning. It's a very pricy book, so maybe you can look for a used copy.
is a scalable popfile for larger organizations. If I could get popfile (with its super-easy-to-train/use-web-interface) that would run on my linux server, scan my IMAP mail server (well, incoming mail would actually work fine, too. I've heard they have a smtp plugin for it in cvs), and then have a popfile config page for each person, or mayby tie it into the imap/smtp server's login. THAT would rock.
Actually, I would love to have the same thing. Popfile is all Perl and open-source, so it could probably have its "guts" adapted for this use.
I've been using Popfile on my laptop to filter several POP accounts. It filters around 1000 messages a day, with about 99.4% accuracy. And the misses are usually a spam message or two that gets through. I never miss a legit message. I want to start reading my mail from an IMAP server, because when travelling I don't always have a broadband connection to download all this crap and filter it locally. Wouldn't it be great to be able to transfer my already-trained filters to this new IMAP machine?
Because every technophile here knows exactly how Bayesian filtering works, right?
I just checked the server and found spamassassin version 2.43 on there.
:)
There is no Bayes in SA until versions 2.5 and after.
I currently run 2.60 (they don't seem to have updated it in a long while now) and am too pleased with that to go back to an older version.
I will write them again and see if they are willing to upgrade the one on my server - the worst they can say is no.
Well, I suppose the worst they could say was that they are cancelling my account
There are some odd things afoot now, in the Villa Straylight.
This is the big question. Bayesian filtering has been in use for a couple of years now, and is well-proven, IMO. What's wrong with Microsoft? Why are they dragging their feet on this? They should have been shipping this with OE a couple of years ago, if not before. Not only would this have given the average user some relief, it would have slowed the recent explosion in spam itself. And it would have been so easy to do. Fuck Microsoft, one more time.
When the hell did the term "Ham" start getting used? I missed it completely 'til this article.
Not that it matters all that much, but Hormel, who has taken use of "Spam" in pretty good graces, can't be happy about this at all. It's one thing when your product is linked to a negatively-perceived other concept, but then the further implication that Ham=good, Spam=bad... hrrm.
SO YOU'RE GOING TO DIE: The Comic for Dealing with Death
However I've really gotten hooked on Bloomba -- it has good SAProxy integration. Very simple to setup compared to some of the other antispam solutions.
I also used SpamAssasin for a while, but it never seemed to do quite so well of a job. It let alot of junk get through until I set in lower, then I got some false-positives.
Have you tried running a Bayesian filter on many messages at once? The Mozilla implementation hangs the mail app for a few seconds on a 1.4GHz Athlon when going through a hundred or so messages. Assuming Slashcode would implement it through Perl, it would be even slower. For reference, running Spamassassin with Bayes filtering (Perl scripts, not spamc) isn't exactly speedy. Going through several messages brings CPU usage close to maximum.
Bayesian filtering on comments would be too resource costly. A more plausible application would be to run stories through a Bayes-style filter, creating a profile for each story that checks each new story with previous profiles so that dupes could be reduced. But that would not be as good as having the editor looking at the current front page (as SCO stories would look similar).
Now why would I care about an Outlook drop-in? Besides, the only sentence I could come up with that uses both "Outlook" and "usefull" is, "Reformating the hard drive is a usefull way to remove Outlook and other viri, that also eliminates the MS detritus that common viri feed on."
Spammunition is a great Outlook plugin that does this.
Come on!! Give some credit!
...is that they don't tie up port 8080, which you may need while doing web development locally. This isn't a huge problem (the defaults can be changed), but I wind up having to shut down Popfile when playing with Zope, for example.
The filter only works with Outlook Express. So, no need to worry about home users NOT switching to Mozilla. But, the corporate users seem to be doomed forever with the plugin for their Outlook.
Ok, now you can laugh.
Turn off html mail for Outlook and help keep them from validating your address through this method.
.reg files of their own and be able to quickly switch between viewing html and plain text mail. taah dahhh!
. 0\ Outlook\Options\Mail]0 1
. 0\ Outlook\Options\Mail]0 0
Place these two keys in
[HKEY_CURRENT_USER\Software\Microsoft\Office\10
"ReadAsPlain"=dword:000000
OR to turn it back on and view those pretty pictures
[HKEY_CURRENT_USER\Software\Microsoft\Office\10
"ReadAsPlain"=dword:000000
A Bayesian filter that reads personal ads, compares them to ads posted by women who are KNOWN to have been "easy" (on a sliding scale, configurable, ranging from "mildly slutty" to "dangerously psychotic nymphomaniac"), and returns a list of likely phone numbers.
Hell, I'd pay MONEY for a piece of software THAT good (Hmm, clickety-click, select "nymphomanic", enter search site... Ah! This one has an oral fixation! Thank you, Mr. Bayes!).
Farewell! It's been a fine buncha years!
You make a good point, but I still think i need a larger pen1s, n0 matter what you say about those spamers and their dirty market ing techniques.
You guys are a bunch of hypocrites. You don't really want spam to stop. You love spam.
Every spam thread is the same: I use X, and it blocks 98% of my spam, with no false positives! I use Y, and it blocks 99.9% -- take that! Here, I use Z + Y with these custom Perl scripts I wrote that interface with procmail and stop 101% percent of spam! It doesn't matter, because I never get ANY spam! Spam is only because people buy things in spam! What morons! Bow before me, for I am 1337!
Spam gives you something to fight. Spam gives you an excuse to solve an interesting technical problem (i.e. separating spam from ham). Spam gives you a reason to boast. Spam gives you people to dislike.
Admit it.
You love spam.
It's semi-distributed, in that users install a small plugin for Outlook, adding "block" and "unblock" buttons to the tool bar. The entire community of users works against spammers.
It works well. When I check my mail, I can watch the 50 or so spams I get daily pop into my inbox, and then promptly fly right back out again.
(Blatantly stolen from Spamnet's Learn More page)
When the message comes in, SpamNet generates a unique fingerprint of that message. The fingerprint is a one-way hash, or unique string of numbers that represents the email and can absolutely NOT be decoded.
This unique fingerprint of the message is sent to the server where it asks the database if this message is spam. The server comes back to the client with a confidence level of how sure it is that the message is spam by checking it with the other fingerprints in the database.
If the same signature has been reported to the SpamNet database, this indicates that the message is spam and it is consequently moved from the member's Inbox to the Spam folder.
If a spam slips through, the SpamFighter can use the "Block" button (vs. delete) to remove it from their inbox and report it to SpamNet to help themselves and the community. Again a unique fingerprint of the body of the message is generated and sent back to the server. Here is where TeS, or the trust system comes into play to ensure that only valid spam messages are blocked. SpamNet looks at the reputation of the person that blocked the message and depending on their individual trust rating; a confidence level is applied to that message to decide whether it should be blocked for the entire community. Each person starts with a zero trust rating and generates trust based on several factors including how accurate their reports are and the number of reports overtime. This process happens instantly taking less than 3 minutes to stop a spam message that's new to the system for the entire community.
Michael C. Hollinger
I don't want to look at the spam, ever. I want it to go to /dev/null before I even download my messages.
This is at best a band-aid and with the usual mistakes and slip-ups it hardly seems like a very good one. I mean if I have to sort through my junk box to check for mislabeled emails its not doing me so much of a favor.
All this talk about smart filtering and I'm starting to feel like you've missed the point, your still getting spam. Who cares if its semi-sorted.
Quack, quack.
I have Evo picking up my mail from server. Right now I have some simple filters, that catch the little bit of spam I get.
Is there an 'integrated' solution that works within Evolution. I heard some time back they were going to do one.
They have a command line filter. May be that can be utilized?
any experiences?
thanks
LinuxLover
Do you have some evidence that points to this? I've been tracking statistics from actual POPFile users for sometime and publishing them on the POPFile home page and there is only a *very* slight decrease in accuracy as the number of classification buckets increases. John.
That exists in various forms in various packages (that I want to see in Outlook, because that's what I use...bash as will...but it works best for me for various reasons) is:
- Only load images from HTML mail from addresses in your personal address book
and
- Whitelist/classify based on users in your address book.
If those two additional features and my Spambayes setup, I'd be very happy.
Do not fold, spindle or mutilate.
I SAID that this message WAS NOT offtopic, it was a reply to parent, who used the SAME topic. That AC had claimed FP, and I said that he/she DIDN'T. Is that offtopic, or is it flamebait? BTW, that WAS my fp about a beowulf cluster of spam.
I'm posting as AC to prevent my Positive karma from being hurt more. I was trying to get Excellent karma...
Anyway, if I had modpoints and came across that post, I would mod it down to Score:0, Flamebait, not Score:-1, Offtopic.
(plug type="blatant" subject="website")
BTW, what this post got modded to is the name of my future website, which will be at http://score-1ot.no-ip.com. It does point to something, but it's my Geocities site.
(/plug)
how about sorting the spam from the tofu and seitan
Hi, Pr0nboy. Now I know your secret. BTW, both parent and this post need modded down to -1 OT. THIS is an example of an offtopic post. BTW, this would also be either Flamebait or Troll.
/. mods aren't stupid.
I'm just making sure that
You got it. That's the whole point of POPFile... Now, if Apple would give me a job I'd happily make Mail.app the coolest mail program out there by adding mutli-bucket Bayesian mail sorting. John.
(can Bayesian be applied to things like spam from Internet virii as well?)
Yes. It all comes down to patterns and weights, like fuzzy logic.
I have one account that regularly attracts Klez-type messages. Since I installed and trained SpamProbe, not a one has ended up in my in-basket, they've all gone into my spam-bucket. The last time I had to retrain on a false-positive was a week ago, so the filtering isn't uber-strict.
1. Use Debian /etc/inetd.conf: pop3 stream tcp nowait nobody /usr/sbin/tcpd /usr/bin/hotwayd
2. apt-get install spamassassin
3. apt-get install hotway
4. Add this to your
5. Switch to Kmail
6. Menu: Settings|Configure Filters
7. Add first filter.
a. Select Match Any of the following
b. Select size 250000
c. Filter action: PIPE THROUGH spamassassin
8. Add second filter
a. Select 'Match any of the following'
b. Type 'X-Spam-Flag' (no quotes)
c. Select equals. Type 'YES'
d. Filter action: Move to folder [your spam folder]
9. It's crucial thta the second filter happes after the first (use the arrows to the left).
There you have it - a spam-free Hotmail account. Not quite setup.exe, but this is Linux after all.
I've sampled it for a bit now, has some problems with some firewalls, but works pretty well for me so far.
Spammunition http://www.upserve.com/spammunition/
...so you may not be out of arguments yet!
This is exactly what SwiftFile does for Lotus Notes.
OK, it's not the most widely used email client among Slashdot readers, but it is very extensible and this is just one example.
I DON'T LIKE SPAM! I DON'T LIKE SPAM! I DON'T LIKE SPAM!
Well, people are not working on it, you can download it now already! :-)
See http://www.pobox.com/~berend/emc/ for more details.
If I had a sig, I would put it here.
Make sense. Consider classifying with a binary tree (e.g., first divide into spam and non-spam, then divide the no-spam into personal and business, and then divide personal into two groups, and so on). If each step can be done with 99% accuracy (something my experience with Bayesian spam filtering would indicate), then you could go 5 levels deep (32 buckets, if fully populated) and have roughly 95% accuracy. Not "very slight" decrease but still quite usable,
and the cost of misclassification wouldn't be very high anyway.
I've found SAproxy to be a convenient way to run SpamAssassin with Outlook. You point your account at the locally running POP3 proxy, it filters your mail as it's coming in, prepending "*****SPAM*****" to the beginning of any suspected spam, then you add a rule in Outlook looking for that. Easy peasy (and freesy!)
I've tried multiple buckets with popfile/outclass, it gets a significant percentage of the classification correct, but it also gets enough wrong to be a serious pain in the arse when important mails get automatically misclassified into a low priority mailing list folder.
Government of the people, by corporate executives, for corporate profits.
I've just been migrated to Notes from Outlook. Not a happy bunny till I discovered how powerful it is with stuff like agents.
The only thing I'm missing now is a spam classification tool like popfile for notes.
Government of the people, by corporate executives, for corporate profits.
1) You gotta pay to send spam (there is a cost)
2) Spammers make money on the reponses they get.
3) There is a breakeven point, you need a certain response rate in order to maintain the spamming rate -- otherwise you lose money.
Anti-spamming tools help reduce the return rate (not saying we shouldn't investigate other options, just that this is something the "average joe/jane can do"). Enough people using anti spam tools will start putting a bite on the return rate, and put spammers out of business. Use the tools!!!
SpamAssassin
SAProxy (windows)
My tuppence worth:
The Geeks have won the war against spam. Most common users do not get spam in their inboxes, since it is blocked by the ISPs and business servers.
It still irritates the hell out of all Geeks, but I think it is time to announce to the world that the spam problem has been fixed and that sending spam doesn't work, so the spamvertisers can just as well stop sending it, since it just gets dumped anyway.
Spam is "chopped pork and ham" right? So you can NEVER sort the Spam from the ham - Spam CONTAINS ham!
You could sort Spam from beef, I should think.
That was classic intercourse!
Yes, I already did the no-karma-whore-bonus to mod myself down :-)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
For the Bazillionth Time, client side spam filtering does not address the problem. It's a waste of time.
The more client-side filters that are in place, the more spam will increase. It's already a cat-and-mouse game between spammers and filters. In the mean time, almost 70% of existing mail traffic is UCE. Filters don't stop that at all.
You have to stop spam at the source. You have to force spammers to act responsibly and not exploit network resources without appropriate compensation. Only when this is done, will any of us ever have any real control of the spam situation.
Client-side spam filtering is a scheme devised by companies who actually have a financial interest in spam, and the more spam there is, the more money they make. Let's get smart about this and stop acting like the rest of the country's brain dead couch potatoes.
Don't delete those messages. Tell your filter to reclassify them as spam. See, the routing and the sender are things that you really don't look at, but your filter does.
I noticed recently that my bayesian filter was blocking very minimalist messages. When I asked it what it was donig, it told me that the headers clued it in to its spam origins. I think the same guy had sent me something before.
So the long and the short of it is: trust your filter. It'll keep getting it right...
One of my favorites....
URGENT BUSINESS PROPOSAL
From:
To:
Date:
06/26/03 10:04 pm
From:Mr.Wilson Ajala
Lagos-Nigeria.
Attn:President/Ceo.
STRICTLY CONFIDENTIAL BUSINESS PROPOSALRE: TRANSFER OF US$21.5 MILLION
(TWENTY ONE MILLION,FIVE HUNDRED THOUSAND US DOLLARS ONLY).
This letter is not intended to to cause any embarrassment but just to contact your esteem self-following the knowledge of your high repute and trustworthiness.By virtue of the nature of this transaction, I solicit for your confidentiality and secrecy.I know that the transaction of this magnitude will make one apprehensive and worried but I assure you that all will be well by the end of this transaction.
I am a member of the Federal Government of Nigeria Contract Award and Monitoring Committee in the Nigeria National Petroleum Corporation (NNPC) Sometime ago, a contract was awarded to a foreign firm in NNPC by my Committee. This contract was over invoiced to the tune of US$21.5M.U.S.Dollars.This was done deliberately.The over-invoicing was a deal by my
committee to benefit from the project.We now want to transfer this money which is in a suspense Account with NNPC into any Overseas Account which we expect you to provide for us.
SHARE: -
For providing the account where we shall remit the money into, you will be entitled to 20% of the money,75% will be for me and my partners while 5% has been mapped out from the total sum to cover any expenses that may be incurred by us during the course of this transfer, both locally and international expenses.
It does not matter whether or not your company does contract projects of this nature described here.The assumption is that your company won the major contract and subcontracted it out to other companies. More often than not,big trading companies or firms of unrelated fields win major contracts and subcontracts to more specialized firms for execution of such contracts.
We have strong reliable connections and contacts at the Central Bank of Nigeria, as well as the Federal Ministry of Finance and we have no doubt that all the money will be released and transferred if we get the necessary foreign partner to assist us in this deal. Therefore,when the business is successfully concluded we shall through our same connections withdraw all documents used from all the concerned Government Ministries for 100% security.
We are ordinary civil servants and we will not want to miss this once in alifetime opportunity to get rich.We want this money to be transferred to the overseas Accounts for us, before the present Democratic Government starts Auditing all Federal Government ownedParastatals.
Please contact me immediately through my email address whether or not you are interested in this deal.If you are not, it will enable me scout for another foreign partner to carry out this deal.But where you are interested,I will require the following details from you as soon as possible in order for us to commence communication immediately as time is of the essence in this business.
1.YOUR NAME
 Â
2.PHONE AND FAX NUMBERS
Â
I wait in anticipation of your fullest co-operation.
Yours Faithfully,
Wilson Ajala.
Email: ajala8000@rediffmail.com
There's already some kind of spam filter upstream of me. I geet all these strange headers like 'perlmx-spamgauge: 67%' and a bunch of 'triggers' for the system, but it dosn't do me much good on Outlook (yeah, I know it's virusbait, but it's right there... : P) outlooks built in spam 'filters' suck ass. it gives tons of false positives.
autopr0n is like, down and stuff.
He's being helpful, damnit.
I've been running a beta of a program I found at www.inboxer.com, which, after looking at all of this dot hoopla, I checked and sure enough, it says that it is based on SpamBayes and gives credit to Mark Hammond. I got better results for blocking with their InBoxer beta version after receiving about 100 spam messages in the past two weeks, so I adjusted the sensitivity on Sunday, and I haven't seen a message since. It seems they have a control that lets you set the percentage point at which a message is considered spam. It completely knows when the man is sending me something pervy which is safe or if indeed is a penis enhancer. That my friends, is an excellent use of Bayesian
As for your indictment of spam filtering providers, could you please explain where the spamassassin devteam is making money?
My choices with regards to spam at the moment are simple. Use spamassassin or something like it, or wade through spam myself. I know which I'd prefer.
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
Agreed, it does work and I just saw the beta download on betanews.com
After many years, I've gotten pretty good at identifying spam by scanning Sender and Subject headers, so I don't really see the benefit in using a Bayesian (or any other filter). I'd rather just have a POP email client (in Windows) that will let me scan headers and remove unwanted emails on the POP server before downloading anything at all, and then only display those emails I download when I take an affirmative action to view a message body. In other words. Clicking on the header should not launch a preview pane that loads the email body.
:(
For me the answer is a well-thought out email client that assumes all email to be hostile - unlike Outlook, which assumes all email to be friendly. I've got an idea about chaining several apps together (fetchmail, Hotpop, etc.) to accomplish a lot of the functionality I want; but there's still no turnkey solution in Windows that I know of.
I find that they just get filtered fine by SB. The bounced spam still contains enough spammy tokens to get correctly filed away.
If all of the postmasters out there would setup their mail servers properly SPAM would be less of a problem.
At the very least, we would know for sure exactly where it came from and could more easily filter from those domains.
The solution to SPAM is already here. The fault for SPAM lies with the postmasters.
!!!! Postmasters unite against SPAM !!!!!
Send me mail if you're a postmaster and you want more info
cp -R
What I really want is mail stored in a monolithic file with indexed access - with the *keys* placed in these buckets. With multiple categories applied to each email so that eg. a message from your brother which contains a new joke *and* clues you about a business opportunity can get filed under "family" "jokes" and "business" all at the same time without duplicating the underlying data.
Actually while we're at it the proverbial *they* need to make this indexed data structure readable and writable by other applications. It would be nice if you could access *all* your data (contacts, post-it notes, appointment history, chat conversation logs, URL bookmarks, movie clips, everything) neatly, each item presented in the relevant format. from a single search query.
Intelligent filtering is only half useful in its current form. The real benefits won't be perceptible until we are able to use it to index our stuff in a purely subject-oriented way rather than (as now) in a format-oriented way.
go ahead and do it anyway, and use it as sample code on your application to Apple
The last peice of spam I actually downloaded from the server, cleverly enough, had the subject line "The server is down". Bastards.