Slashdot Mirror
SpamAssassin Gets a Promotion
darthcamaro writes "The folks at internetnews.com are reporting that the Spam Assassin project has been promoted to a full top level Apache Software Foundation project..the project has been in incubation for a while and it's finally made it through...the article also reveals that Apache is now using Spam Assassin themselves: 'I think spam filtering is now a critical part of the network infrastructure and Spam Assassin is a leader in the area,' said Daniel Quinlan, chairman of the Apache Spam Assassin Project Management Committee."
A well configured installation of SA got me employee of the month way back when. Sadly, UCE/UBE is/has ruined the Internet. Finally.
"To deny our own impulses, is to deny the very thing that makes us human." - Mouse
After using SpamAssassin for quite a while, it just wasn't cutting it - 75%-80% accuracy is still a lot of spam to go through and delete. I added DSpam to my mail server and my spam catching rate is now better than 99%.
DSpam also came with much better directions for integrating with Exim than did SpamAssassin. As fond as I was of SpamAssassin, they have some catching up to do.
No, Thursday's out. How about never - is never good for you?
Fixed, sorry about that.
What do you do with mail SA has flagged?
I like SA, and find it is very good for identifying around 95% of my incoming spam. However, I also have around 0.1% false positive rate, which means at some point I have to look through all the filtered spam messages and make sure none of them were legit.
I need a better tool for handling mail SA has identified as spam, either server-side or client-side. I'd like to delete anything with a score > 15, simply store anything with a score > 5, and send an auto-reply for scores between 5 and 10 indicating that the message was marked as spam and I'll probably never look at it.
A good set of procmail and formail rules will accomplish this, but my hosting company has a weird procmail setup and I'd prefer something easier to implement.
Any ideas?
Novell NetMail even supports SpamAssassin now.
http://netmail.sourceforge.net/
Filtering spam generates way too many false positives. Challenge/Response schemes are IMHO much more effective. TMDA and similar programs can be configured with whitelists for your regular mail partners, auto-whitelists for everyone who confirms their e-mail identity, and, if necessary, with blacklists too.
cpghost at Cordula's Web.
A success rate of 95% really sucks when (like me) you get just over 2,500 spams a day. That'd still mean around 125 spams a day would be getting through. (I've had the same email address since the early 1990's, back when there was no reason to keep your email address "secret.")
.qmail files. PHP is indeed good for things other than web pages; and was a little bit easier for me to maintain and deal with than Perl. The whole thing is less than 25KB of code. There is also a web backend which I use to configure it; that adds another 40KB.
;-)
Personally I do use SpamAssassin, but as an intermediate step.
First step: Check a whitelist of known senders. Deliver if the sender is on the list, AND the message originated from an IP subnet that I allow for them personally.
Second step: Scan with SpamAssassin. If the score is really high (above 20) throw it the hell out.
Third step: If the score is less than 20, and the person wasn't whitelisted, run the message through TMDA and politely tell the sender I'm not sure who they are, and I get a lot of spam, and could you please click this link to prove that you're a real person.
I've been using this three-step system for eighteen months now, and out of over one million messages that have come into my mailbox (really), exactly FOUR spam messages have made it all the way through. Apparently the spammers decided to go ahead and click on the little link, or they used a real person's return address, and when that person got they autoreply, they were too stupid to understand what was going on.
Even better, I have not received ANY indiciation that I've lost any messages; at least, no one has ever mentioned anything about an email that I didn't get.
I've got five other people at my domain using the same system, although for not quite as long (one for fifteen months, three for about a year, and one for just a month now); they have all had similar success.
So based on those numbers I'd estimate a success rate of 99.9997% for eliminating spam (which is, admittedly, COMPLETELY INSANE), and a false-positive (or at least "lost message") rate of 0% so far (fingers crossed). A few people have had to confirm their messages, of course, but I've whitelisted them as that happens.
I actually wrote all the connecting code in PHP, believe it or not, with a MySQL database as a backend. It's invoked using
The whole system took about twelve hours of programming to set up, on one Saturday.
Now, for correspondence to companies (such as Microsoft, or Amazon.com), I use a different scheme (although it's handled by the same PHP code). I create up a unique email address for each of them, which ONLY allows mail to or from that domain (for example "rptamazon@mydomain.com" only allows messages from amazon.com). Those addresses are also easily cancellable, individually, if the company starts to annoy me with spam. Basically, each email address can be assigned its own unique whitelist, and can be cancelled individually at any time, through the little web interface.
I also have a number of email addresses for things such as customer support for our company (I write computer software). I'm using the same system for those, also, but instead of checking whitelists based on the sender, I've found a simple way to do it is to check for ANY of our product names anywhere in the message body or subject. If the message doesn't mention any of them, it sends a simple autoreply back similar to that in (3) above, but mentioning that the message didn't seem to be about any of our products, but if it was, please click here, blah blah. We don't have a high volume of support messages (about one or two a day; we're a small company) but in the last year only three or four people have had to click through like that, and, honestly, their support requests were so f*cked up anyways that I'd rather it just dropped them on the floor.
Then, as a very last ste
As a side note, I don't use these email addresses for personal emails - I can hopefully trust that the people I personally send emails to are not, or are not going to become spammers.
Well, that is not a very secure assumption. Unless you know that all those people are not using an MUA/OS combination that is vulnerable to viruses or worms. Harvesting addresses is done that way nowadays...
cpghost at Cordula's Web.
not all bulk mail is spam. spam assassin gives 2.4 points if it finds anything that looks like a unique identifier for X-Sender, and another 1.4 points for anything that looks like a tracking image or tracked link.
that plus the points for any non-safe html colors or any html at all, SA effectively tags ANY bulk mail as spam!
For an end user to setup on their client (as a "junk mail" folder) thats great.. I like to have bulk mail seperated from my personal mail, but for an ISP to throw it away before it even gets to the intended recipient is fucking rediculous and should be illegal.
The only email an ISP should be allowed to discard are the ones with attached viruses or some known email worm. The only reason your customers are happy with you throwing away their email is because you don't fucking tell them.
bite my glorious golden ass.
I'd be interested in seeing the scripts you have setup for a project I'm involved with. Any thought of sharing?
Word salad I can understand (if you bayes isn't aggressively trained at least).. I don't have problems with it, but my bayes is very heavily trained. (100-300 spams a day manual training)
What I don't understand is the base64 problem.. One of the first thing SA does is decode base64. Even "rawbody" rules get base64 decoding, so really base64 encoding shouldn't make a difference at all, as SA never examines the encoded text.
As for the intentional mis-spellings of V!agr0, check out antidrug.cf (use google) or wait for SA 3.0 which includes this set of rules as a part of the standard distribution.
Disclaimer: I am the author of antidrug, and thus do have a bias here.
-Matt
Yes, I would definitely like to make this stuff publicly available; I know a lot of people would be interested. I need to find a good way to do it. I'm a bit worried about drawing needless attention to myself by releasing such a thing--for example, the system is NOT foolproof, so I could certainly see myself becoming a target for attacks and such.
:-)
Hopefully I'll find some free time later this summer (two big big programming projects I'm working on now are ending next month) and I'll see if I can take a weekend and put a site together. I'll submit it as a story to Slashdot (and if it doesn't make it, post it in my signature and leave comments about it everytime someone mentions spam here).
The unfortunate thing is that making this public will increase work for me, of course (people needing help with installations, or submitting patches, etc.), so I'd like to find a way to mitigate the work involved. I don't really know what's involved in setting up an open source project; perhaps I'll look into SourceForge and see what the deal is. Normally I write commercial software; I don't know whether or not something like this could be sold or not. Obviously, if people were paying for it, providing support and taking time away from paying projects wouldn't be as big a problem for me since I'd be compensated.
Alternatively, I've also gotten suggestions that I should keep the software to myself, and offer a paid service where my servers are the MX (mail) hosts for people's domains, giving them POP and IMAP access. I've actually been doing exactly that for my friends over the past six months or so; it's worked out well (four domains for friends currently) but I'm not sure how much the system can scale before I start running out of resources (bandwidth, CPU time, etc.). I'd really have to calculate everything carefully and work out the economics in order to do something like that as a real commercial venture.