SpamAssassin Gets a Promotion

darthcamaro writes "The folks at internetnews.com are reporting that the Spam Assassin project has been promoted to a full top level Apache Software Foundation project..the project has been in incubation for a while and it's finally made it through...the article also reveals that Apache is now using Spam Assassin themselves: 'I think spam filtering is now a critical part of the network infrastructure and Spam Assassin is a leader in the area,' said Daniel Quinlan, chairman of the Apache Spam Assassin Project Management Committee."

Comments

erm

[bruns]

Perhaps Slashdot editors might want to take an extra 20 seconds to check the spelling of the URLs they put in their stories.

spamassassin.org, not spamassasin.org

Great News!

[Anonymous Coward]

This is great news! I have been running SpamAssassin on my box for quite a while, just to filter my own mail. I recently installed it on my mother's Windows 98 box to filter her mail when she checks it with Outlook Express, and she hasn't complained about Spam since. With a bit of tweaking, its been catching 95% with no false positives. Hopefully the SpamAssassin project will keep on getting better :)

Re:Great News!

[kidlinux]

Do you use sa-learn to teach SA about new spam? I have spam tagged email dumped to a Spam folder on my imap server so I can go through it and make sure there aren't any false-negatves. I then move all the spam to a shared folder and run an sa-learn script on it nightly.

Currently I have amassed 3681 spams totalling 76 megs. I should probably empty that directory sometime :P

sa-learn makes a big difference though. Helps with the misspellings and random junk. Havn't seen a Nigerian scam come through either. In fact, I think I might see 2 spams a month or something - when the spammers figure out a new technique I guess, but just feed it through sa-learn and all subsequent spams are toast.

Here is the real link to spam assasins site

[vespazzari]

For those looking for the official spam assasin site here it is

The link in the text goes to some search page

sorting mail by spamassassin score

[David Jao]

I'd like to delete anything with a score > 15, simply store anything with a score > 5, and send an auto-reply for scores between 5 and 10 indicating that the message was marked as spam and I'll probably never look at it.

I can't speak for auto-replies, but you can do the sorting part client-side. The key is that spamassassin adds a line like "X-Spam-Level: *****" where the number of *'s is the score of the email. Almost any email client can filter mail to different folders based on headers. The unary representation of the spam score ensures that even a primitive filter can work.

For example, one popular client is Microsoft Outlook, and there are several web pages in google (such as this one) that explain how to reroute mail to specific folders depending on the spamassassin score.

Re:DSpam

[prockcore]

I added DSpam to my mail server and my spam catching rate is now better than 99%.

I haven't seen any false positive stats on dspam. It's easy to say a spam filter has a high spam catching rate, but it means nothing without a very low false positive rate.

Redirecting my mail to /dev/null gives me a 100% spam catching rate.

Re:what to do with spam after it's id'd?

[Anonymous Coward]

Sending an auto-reply on scores between 5 and 10 (or any other range) makes you part of the problem, not part of the solution.

I have a very well known address (which is why I'm posting as an Anonymous Coward :-) that receives many hundreds of messages every day. My mail server deals with about half of the spam I get. Well over half of the rest is autoreply responses from idiots who don't understand that *I* never sent that message in the first place -- the from address was forged by a virus.

The correct response to spam is to throw it away. Trying to reply to it makes the world worse, not better.

a better approach: reject the mail

[Trepidity]

If you integrate it with your mailer, you can reject the mail during the SMTP session rather than generating a separate bounce email, which would have the problems you mentioned (going to a forged from: address). As an added bonus, when you reject it during the SMTP session, you'll get taken off a lot of spam lists, since your address will look like it had delivery problems. And you still get the advantage of bounces, that legitimate mail that got rejected will end up with a bounce back to the sender informing them of it.

3.0, late-July, early August

[chathamhouse]

3.0.0pre1 was made available last week.

It will apparently take another month or so to finalize the weighting of the rules.

I've put 3.0.0pre1 on a production system that filters ~350k messages per day. With some tweaking of the RBL, bayes, and AWL rules, it is much (~10%) more efficient at tagging spam than 2.63, which I'm running on a parallel server that also sees ~350k messages/day (load balancing is your friend).

More info: http://www.au.spamassassin.org/full/3.0.x/dist/bui ld/3.0.0_change_summary

Re:what to do with spam after it's id'd?

[antsquish]

I know you mentioned procmail, but for those using Courier IMAP's maildrop, here's what I use in my ~/.mailfilter for SpamAssassin. I've just pasted the relevant sections, but it logs all deliveries, I then filter known recipients into their own folders (not shown here), then any unknown messages are filtered through Spam Assassin. Messages with a score > 10 are sent to /dev/null, while others are delivered to a spam folder.

logfile "/path/to/my/home/dir/maildrop.log"

###
### Maildrop variable substitution
###

MAILBOX="./Maildir"
DEFAULT= "$MAILBOX"
SPAM="$MAILBOX/.Spam"

###
### SpamAssassin :: filter out spam mail
###

# Filter through SpamAssassin
xfilter "/usr/local/bin/spamc"

# Handle messages marked as spam
if ( /^X-Spam-Flag: YES/ )
{
# Store messages flagged as spam in another folder; uncomment
# this during testing just in case any legit mail gets sent
# to /dev/null
#cc "./spam-store"

# Delete messages with a score of 10 or higher, filter all other
# spam messages into a spam folder
/^X-Spam-Status: yes, hits=![:digit:]+\.[:digit:]+!.*/
if ( $MATCH2 >= 10.0 )
to "/dev/null"
else
to $SPAM
}

Re:Challenge-Response schemes are more effective

[Vellmont]

I've been running SA since February, and have had a grand total of ONE false positive out of a few thousand emails. The message was from a new account, very short, and in HTML. That address has since been added to my autowhitelist. SA couple with Amavisd-new and clamav has reduced my spam volume by about 95%, and my virus emails to zero. It's a great product and I'm looking forward to 3.0.

Re:3.0?

[Brian the Bold]

Have a look at the Rules Emporium at:



I use the rules there, and even minor spam gets obliterated with no problems of catching real mail.

I recommend it!

You can even run spamassassin directly on Exchange

[AssFace]

Many people use spamassassin on unix boxes, or if they have Exchange they use SA on a unix gateway between the net and the Exchange system.
But if you are a smaller shop and don't have the resources for that, then you can run sa right on Exchange.
Here is a write up on how to do it (that particular write up is for Exchange 2003 and SA 3.0, but it will work for SA 2.x as well, and for Exchange 2000 - or any combination thereof - but it won't work on Exchange 5.5 that I know of).

Re:Bout Time!

[Just Some Guy]

I "augmented" SpamAssassin with an extremely tight Postfix ruleset. A remote server has to jump through these hoops before SA ever gets a crack at it:

1. HELO Filtering

1. Reject any connection that doesn't start with HELO or EHLO.
2. Allow any host on my LAN to continue on to step 2.
3. Reject any host not on my LAN that sends a hostname or IP of a machine on my LAN.
4. Reject non-FQDN hostnames (ala "mailserver").
5. Reject invalid hostnames (ala "432$@@112").
6. Let everyone who makes it this far continue on to step 2.

2. Sender Filtering

1. Allow authenticated senders to continue on to step 3.
2. Allow hosts on my LAN to continue on to step 3.
3. Reject non-FQDN sender domains ("foo@bar").
4. Reject unknown sender domain ("foo@imaginarydomain.com") - after all, if I can resolve their domain, then I couldn't reply to them anyway, right?
5. Let everyone who makes it this far continue on to step 3.

3. Recipient Filtering

1. Reject non-FQDN recipient domains (they'd bounce anyway).
2. Reject unknown recipient domains (same as above).
3. Allow authenticated users to send their mail and stop processing.
4. Allow hosts on my LAN to send their mail and stop processing.
5. Reject mail from anyone else that isn't to one of my domains, or one I'm an MX for.
6. Use SPF to reject spoofed email.
7. Use the relays.ordb.org, list.dsbl.org, and sbl-xbl.spamhaus.org DNS blackhole lists.
8. Greylist all email not coming in from or going out to peer MXes.
9. Pass everything else to step 4.

4. Content Filtering and Delivery

1. Use ClamAV to reject viruses. This takes a big load off SpamAssassin.
2. Use SpamAssassin to tag messages.
3. Use Cyrus's Sieve to reject high-probability spam, put medium-probability messages into a "review" folder, and filter everything else into the appropriate folders.

I reject over 95% of all incoming mail before it ever gets to SpamAssassin. This means that SA's success rate isn't as good as on other systems (since I weed out all of the obvious spam), but my mailbox is happy and shiny.

SpamAssassin is a brilliant last line of defense, but I wouldn't advise just dumping your raw incoming stream into it. Much of the useful information about a message isn't available to spamd (such as your list of local domain names, relay domains, etc.) and you should consider using a set of cheaper filters to flush out the blatant chaff.