SpamAssassin Gets a Promotion

Nice by sirReal.83. · 2004-06-25 18:06 · Score: 1

I didn't see that one coming. I have been using SA for about three years, I think... well, since whenever I heard about it anyway :)

Bout Time! by Irie+Brother · 2004-06-25 18:07 · Score: 3, Interesting

A well configured installation of SA got me employee of the month way back when. Sadly, UCE/UBE is/has ruined the Internet. Finally.

--
"To deny our own impulses, is to deny the very thing that makes us human." - Mouse

Re:Bout Time! by jest3r · 2004-06-25 18:50 · Score: 4, Interesting

Today spam assassin filtered (flagged) 19,246 incoming emails out of 20,145 total on my mail server. Absolutely no false positives since I installed it a year ago .. and only a few false negatives. I silently drop anything with a score over 13 ... my cstomers are happy .. my qmail remote queue has been happy .. spam assassin is a quality app .. spam is really not a concern anymore.
Re:Bout Time! by Mazem · 2004-06-25 20:21 · Score: 5, Insightful

Absolutely no false positives since I installed it a year ago ..
... that you know of.
Re:Bout Time! by Jacer · 2004-06-25 20:24 · Score: 3, Insightful

spam is really not a concern anymore. You mean except for bandwidth I assume.

--
--fetch daddy's blue fright wig, i must be handsome when i release my rage
Re:Bout Time! by tzanger · 2004-06-25 23:41 · Score: 3, Interesting

I do the exact same thing, but with a score of 12. Anything that trips the filter as spam gets dumped into a spam folder off the main maildir and they can use IMAP or check with webmail to see what spam they have. A cron script erases anything in the spam folder older than 2 weeks. Oh yeah, and individual users can alter their own white/blacklists and scores since I pull the username and match the scores in a postgres database. Combined with clamd and qmail-scanner, it's heaven. :-)

As for the incoming mail I found that checking against a couple of RBLs has made ALL the difference in keeping the system load down. tcpserver checks against a .cdb file filled with entries from CBL and DNSRBL. Anything matching doesn't even see the real SMTP server and thus doesn't get scanned at all.
Re:Bout Time! by paperguy · 2004-06-26 05:48 · Score: 2, Interesting

That's a 95.5% spam rate. What are your users doing to generate so much spam?
Re:Bout Time! by Anonymous Coward · 2004-06-26 10:47 · Score: 0

best guess ... porn.
Re:Bout Time! by Dr.Ruud · 2004-06-27 00:43 · Score: 1, Insightful

That bandwidth is not spilled if you make your MTA do SMTP-REJECTs, based on the high-quality-blacklists around.
Re:Bout Time! by Anonymous Coward · 2004-06-27 23:49 · Score: 0

Absolutely no false positives

That number is always hard to tell. At my current customer I set up a mailbox where users forward the false postives they find, and I suspect that I get almost all of them.

After a month of running SA in production, we are at about 0.04% false positives. That is with SA's threshold at 4.8 points with some custom rules, e.g. the full name of the company with proper spacing is scored down, important customers and partners are scored down, and some others.

That's for a company of ~ 10'000 users.
Re:Bout Time! by Just+Some+Guy · 2004-06-28 01:45 · Score: 4, Informative
I "augmented" SpamAssassin with an extremely tight Postfix ruleset. A remote server has to jump through these hoops before SA ever gets a crack at it:
1. HELO Filtering
1. Reject any connection that doesn't start with HELO or EHLO.
2. Allow any host on my LAN to continue on to step 2.
3. Reject any host not on my LAN that sends a hostname or IP of a machine on my LAN.
4. Reject non-FQDN hostnames (ala "mailserver").
5. Reject invalid hostnames (ala "432$@@112").
6. Let everyone who makes it this far continue on to step 2.
2. Sender Filtering
1. Allow authenticated senders to continue on to step 3.
2. Allow hosts on my LAN to continue on to step 3.
3. Reject non-FQDN sender domains ("foo@bar").
4. Reject unknown sender domain ("foo@imaginarydomain.com") - after all, if I can resolve their domain, then I couldn't reply to them anyway, right?
5. Let everyone who makes it this far continue on to step 3.
3. Recipient Filtering
1. Reject non-FQDN recipient domains (they'd bounce anyway).
2. Reject unknown recipient domains (same as above).
3. Allow authenticated users to send their mail and stop processing.
4. Allow hosts on my LAN to send their mail and stop processing.
5. Reject mail from anyone else that isn't to one of my domains, or one I'm an MX for.
6. Use SPF to reject spoofed email.
7. Use the relays.ordb.org, list.dsbl.org, and sbl-xbl.spamhaus.org DNS blackhole lists.
8. Greylist all email not coming in from or going out to peer MXes.
9. Pass everything else to step 4.
4. Content Filtering and Delivery
1. Use ClamAV to reject viruses. This takes a big load off SpamAssassin.
2. Use SpamAssassin to tag messages.
3. Use Cyrus's Sieve to reject high-probability spam, put medium-probability messages into a "review" folder, and filter everything else into the appropriate folders.
I reject over 95% of all incoming mail before it ever gets to SpamAssassin. This means that SA's success rate isn't as good as on other systems (since I weed out all of the obvious spam), but my mailbox is happy and shiny.
SpamAssassin is a brilliant last line of defense, but I wouldn't advise just dumping your raw incoming stream into it. Much of the useful information about a message isn't available to spamd (such as your list of local domain names, relay domains, etc.) and you should consider using a set of cheaper filters to flush out the blatant chaff.
--
Dewey, what part of this looks like authorities should be involved?
Re:Bout Time! by AmigaBen · 2004-07-01 04:35 · Score: 1

4. Content Filtering and Delivery

1. Use ClamAV to reject viruses. This takes a big load off SpamAssassin.

This should be the other way around. You should run them through spamassassin before ClamAV. ClamAV is more processor intensive, and many virus emails will get discarded by SA without having to go through Clam. Whereas you'll be clamscanning lots of virus-free spam this way around.

--
+5 Insightful, really!
Re:Bout Time! by Just+Some+Guy · 2004-07-01 06:10 · Score: 1

ClamAV is more processor intensive
Not in my experience. Maybe it's the size of my Bayes databases, but clamd uses significantly less CPU than spamd on my particular system. YMMV, of course. Anyone who blindly implements my solution without understanding it and tailoring it to their own needs is likely in for a few nasty surprises.

--
Dewey, what part of this looks like authorities should be involved?

erm by bruns · 2004-06-25 18:08 · Score: 3, Informative

Perhaps Slashdot editors might want to take an extra 20 seconds to check the spelling of the URLs they put in their stories.

spamassassin.org, not spamassasin.org

--
Brielle

Re:erm by simoniker · 2004-06-25 18:30 · Score: 3, Interesting

Fixed, sorry about that.
Re:erm by Anonymous Coward · 2004-06-25 18:54 · Score: 0, Offtopic

Do editors have to wait 20 seconds to post?

I wonder...
Re:erm by Anonymous Coward · 2004-06-26 19:23 · Score: 0

What, and break a long and proud history?

Great News! by Anonymous Coward · 2004-06-25 18:09 · Score: 5, Informative

This is great news! I have been running SpamAssassin on my box for quite a while, just to filter my own mail. I recently installed it on my mother's Windows 98 box to filter her mail when she checks it with Outlook Express, and she hasn't complained about Spam since. With a bit of tweaking, its been catching 95% with no false positives. Hopefully the SpamAssassin project will keep on getting better :)

Re:Great News! by Anonymous Coward · 2004-06-25 20:33 · Score: 0

This is really informative posting.

Sorry for trolling, but sheesh, i downloaded pr0n onto my box today, but some guy installing SpamAssassin onto his mother 'puter aint special.
Re:Great News! by NigritudeUltramarine · 2004-06-25 20:59 · Score: 5, Interesting

A success rate of 95% really sucks when (like me) you get just over 2,500 spams a day. That'd still mean around 125 spams a day would be getting through. (I've had the same email address since the early 1990's, back when there was no reason to keep your email address "secret.")

Personally I do use SpamAssassin, but as an intermediate step.

First step: Check a whitelist of known senders. Deliver if the sender is on the list, AND the message originated from an IP subnet that I allow for them personally.

Second step: Scan with SpamAssassin. If the score is really high (above 20) throw it the hell out.

Third step: If the score is less than 20, and the person wasn't whitelisted, run the message through TMDA and politely tell the sender I'm not sure who they are, and I get a lot of spam, and could you please click this link to prove that you're a real person.

I've been using this three-step system for eighteen months now, and out of over one million messages that have come into my mailbox (really), exactly FOUR spam messages have made it all the way through. Apparently the spammers decided to go ahead and click on the little link, or they used a real person's return address, and when that person got they autoreply, they were too stupid to understand what was going on.

Even better, I have not received ANY indiciation that I've lost any messages; at least, no one has ever mentioned anything about an email that I didn't get.

I've got five other people at my domain using the same system, although for not quite as long (one for fifteen months, three for about a year, and one for just a month now); they have all had similar success.

So based on those numbers I'd estimate a success rate of 99.9997% for eliminating spam (which is, admittedly, COMPLETELY INSANE), and a false-positive (or at least "lost message") rate of 0% so far (fingers crossed). A few people have had to confirm their messages, of course, but I've whitelisted them as that happens.

I actually wrote all the connecting code in PHP, believe it or not, with a MySQL database as a backend. It's invoked using .qmail files. PHP is indeed good for things other than web pages; and was a little bit easier for me to maintain and deal with than Perl. The whole thing is less than 25KB of code. There is also a web backend which I use to configure it; that adds another 40KB.

The whole system took about twelve hours of programming to set up, on one Saturday.

Now, for correspondence to companies (such as Microsoft, or Amazon.com), I use a different scheme (although it's handled by the same PHP code). I create up a unique email address for each of them, which ONLY allows mail to or from that domain (for example "rptamazon@mydomain.com" only allows messages from amazon.com). Those addresses are also easily cancellable, individually, if the company starts to annoy me with spam. Basically, each email address can be assigned its own unique whitelist, and can be cancelled individually at any time, through the little web interface.

I also have a number of email addresses for things such as customer support for our company (I write computer software). I'm using the same system for those, also, but instead of checking whitelists based on the sender, I've found a simple way to do it is to check for ANY of our product names anywhere in the message body or subject. If the message doesn't mention any of them, it sends a simple autoreply back similar to that in (3) above, but mentioning that the message didn't seem to be about any of our products, but if it was, please click here, blah blah. We don't have a high volume of support messages (about one or two a day; we're a small company) but in the last year only three or four people have had to click through like that, and, honestly, their support requests were so f*cked up anyways that I'd rather it just dropped them on the floor. ;-)

Then, as a very last ste
Re:Great News! by WebCrapper · 2004-06-25 22:21 · Score: 2, Interesting

I'd be interested in seeing the scripts you have setup for a project I'm involved with. Any thought of sharing?
Re:Great News! by walt-sjc · 2004-06-25 23:43 · Score: 1

Interesting. I've personally found that SA doesn't do well on "word salad" spams, base64 encoded spams, spams with numbers / special characters / intentional misspellings ("V!agr0"),
random word HTML ("<frank>&ltmoon>") etc. Nigerian scam spam seems to get through waaayyy too frequently.

What I have found very useful is the DNSBL's that block known spamming IP's (spamhaus.org) and all email from dynamic addresses. This cuts 95% out before SA even sees it. With a whitelist system in front and SA behind, it's quite effective, but still not 100%.

My mail server sees about 300,000 messages a day of which about 15,000 are legit. I still get about 5 spams a day that slip through to my address (which I've had for almost 10 years,) but it's managable at least. My other users report similar (usually less) amounts of spam that gets through.
Re:Great News! by slashjames · 2004-06-26 01:28 · Score: 1

Any chances that you'ld be willing to provide a download spot for your scripts? I know that I can't be the only one interested in seeing how you did this :)
Re:Great News! by kidlinux · 2004-06-26 01:38 · Score: 2, Informative

Do you use sa-learn to teach SA about new spam? I have spam tagged email dumped to a Spam folder on my imap server so I can go through it and make sure there aren't any false-negatves. I then move all the spam to a shared folder and run an sa-learn script on it nightly.

Currently I have amassed 3681 spams totalling 76 megs. I should probably empty that directory sometime :P

sa-learn makes a big difference though. Helps with the misspellings and random junk. Havn't seen a Nigerian scam come through either. In fact, I think I might see 2 spams a month or something - when the spammers figure out a new technique I guess, but just feed it through sa-learn and all subsequent spams are toast.

--
-kidlinux.
Re:Great News! by shokk · 2004-06-26 02:03 · Score: 1

PLEASE tell us where we can get these PHP scripts and the accompanying mySQL schema. And any glue or config files explaining how this runs.

I didn't see any references to ClamAV in here, but since its integration with SA is documented in other places, that can be an afterthought.

When you run your own mail server, it's easy to trump Google, Yahoo and MSN's recent multi-GB offerings. Wonder if they can top my 100GB mail account. Not that I've ever gotten more than 1GB mail worth reading in my almost 20 year history of using email.

--
"Beware of he who would deny you access to information, for in his heart, he dreams himself your master."
Re:Great News! by mkettler · 2004-06-26 02:12 · Score: 4, Interesting

Word salad I can understand (if you bayes isn't aggressively trained at least).. I don't have problems with it, but my bayes is very heavily trained. (100-300 spams a day manual training)

What I don't understand is the base64 problem.. One of the first thing SA does is decode base64. Even "rawbody" rules get base64 decoding, so really base64 encoding shouldn't make a difference at all, as SA never examines the encoded text.

As for the intentional mis-spellings of V!agr0, check out antidrug.cf (use google) or wait for SA 3.0 which includes this set of rules as a part of the standard distribution.

Disclaimer: I am the author of antidrug, and thus do have a bias here.

--
-Matt
Re:Great News! by duncf · 2004-06-26 02:19 · Score: 2

Third step: If the score is less than 20, and the person wasn't whitelisted, run the message through TMDA and politely tell the sender I'm not sure who they are, and I get a lot of spam, and could you please click this link to prove that you're a real person. ...

So based on those numbers I'd estimate a success rate of 99.9997% for eliminating spam (which is, admittedly, COMPLETELY INSANE), and a false-positive (or at least "lost message") rate of 0% so far (fingers crossed).

Yeah that is COMPLETELY INSANE. You have no idea how many legitimate messages you fail to get because the sender couldn't be bothered, or quite simply can't (i.e. automatic sender, but non-spam) click that link.
TMDA is bad.
Re:Great News! by Anonymous Coward · 2004-06-26 03:15 · Score: 0

Please,
it is quite possible to check - he just have to go through all mail that he CR's, and see who is not responding.
If everybody is responsing, who have legitimate businnes, then there is no problem.
Re:Great News! by NigritudeUltramarine · 2004-06-26 03:54 · Score: 1

You have no idea how many legitimate messages you fail to get because the sender couldn't be bothered, or quite simply can't (i.e. automatic sender, but non-spam) click that link.

Yes, I'm pretty sure I do. Like I said, I've been using this email address for almost 15 years now, and have a pretty good idea of who I correspond with. Very rarely do I get messages on my primary address from completely random people who I've never met before. It's more for personal correspondence. The idea is that since I am (conveniently) a real person, and the people I correspond with are also (conveniently) real people, I also talk to these people in the real world. So there's a backup channel of communication, and I'd certainly know about missing messages.

I am very liberal in my whitelist for people I don't know well. For example, if I correspond with Jim Michael Stevens, jmstevens@carbonred.com, who I've just met, I will whitelist all of ...

Stevens, Jim
Jim Stevens
Jim M. Stevens
Jim Michael Stevens
James M. Stevens
James Michael Stevens
Stevens, James
jmstevens@carbonred.com

And I won't associate any particular IP addresses with those until I know him a bit better (I'll do 0.0.0.0/0). Once I know his patterns, I'll tighten the whitelist for him a bit (although I'm not really sure this is necessary, today, because I actually have a number of correspondents without any IP whitelists to go with their entries, and haven't had problems with spam making it through from them, but I'm sure eventually spammers will start being more selective in their "From" addresses).

Now, I don't post my primary email address in public places anymore. So the only way for a new person to get that address is for me, or someone I know, to give it to them. Normally, I'll give people I don't know a different, temporary address first that allows ANY messages to come through (I have a whole subset of these set aside ahead of time, so I can give them off the top of my head); I'll later whitelist that address just for them, once I get messages, and eventually start corresponding with them using my regular address (whitelisting them first). That also prevents the situation of a double-verification deadlock, if someone I write to is ALSO using a similar system. Since I initially always correspond with an "open" address, I am guaranteed to receive their verification request.

So that just leaves if someone I don't know gets my email address from someone I know. In that case, it seems unlikely that if they went to the trouble of writing me a message, that they wouldn't take the two seconds to click the link in the return message (which states politely that if they don't, there message will end up being deleted).

I've actually gotten messages from my friends' moms or grandparents ... who have had NO problems figuring out the system and clicking on the links. So it can't be too complicated to use. And I'm not too worried about people running email software that can't handle hyperlinks directly (an old version of PINE for example), since anyone using such "obtuse" software these days will also be smart enough to know how to copy and paste.
Re:Great News! by NigritudeUltramarine · 2004-06-26 04:14 · Score: 3, Interesting

Yes, I would definitely like to make this stuff publicly available; I know a lot of people would be interested. I need to find a good way to do it. I'm a bit worried about drawing needless attention to myself by releasing such a thing--for example, the system is NOT foolproof, so I could certainly see myself becoming a target for attacks and such.

Hopefully I'll find some free time later this summer (two big big programming projects I'm working on now are ending next month) and I'll see if I can take a weekend and put a site together. I'll submit it as a story to Slashdot (and if it doesn't make it, post it in my signature and leave comments about it everytime someone mentions spam here).

The unfortunate thing is that making this public will increase work for me, of course (people needing help with installations, or submitting patches, etc.), so I'd like to find a way to mitigate the work involved. I don't really know what's involved in setting up an open source project; perhaps I'll look into SourceForge and see what the deal is. Normally I write commercial software; I don't know whether or not something like this could be sold or not. Obviously, if people were paying for it, providing support and taking time away from paying projects wouldn't be as big a problem for me since I'd be compensated. :-)

Alternatively, I've also gotten suggestions that I should keep the software to myself, and offer a paid service where my servers are the MX (mail) hosts for people's domains, giving them POP and IMAP access. I've actually been doing exactly that for my friends over the past six months or so; it's worked out well (four domains for friends currently) but I'm not sure how much the system can scale before I start running out of resources (bandwidth, CPU time, etc.). I'd really have to calculate everything carefully and work out the economics in order to do something like that as a real commercial venture.
Re:Great News! by ajs · 2004-06-26 06:09 · Score: 1

Apparently the spammers decided to go ahead and click on the little link, or they used a real person's return address, and when that person got they autoreply, they were too stupid to understand what was going on.

I respond to those all the time. I politely send a "please don't auto-reply to forged spam" message. It's not my fault that your anti-spam solution is stupid enough to re-define an email reply to mean that you should accept forged mail.
Re:Great News! by NigritudeUltramarine · 2004-06-26 06:52 · Score: 1

I respond to those all the time. I politely send a "please don't auto-reply to forged spam" message. It's not my fault that your anti-spam solution is stupid enough to re-define an email reply to mean that you should accept forged mail.

I'm sure I'm not the only one who has just one word to say in response to that ... Huh?

Seriously, if you would "reply" to the confirmation autoreply, you'd just get another email back saying, again, I don't know who you are, and I get a lot of spam, so please click this link if you want your message (this time, your "reply") to be delivered. Your "reply" would not get through, nor would the original (forged) message be released for delivery by you "replying" to the message.
Re:Great News! by ajs · 2004-06-28 00:43 · Score: 1

Ah, I didn't realize you were talking about a different system. Most of them are reply-based. Since I read my home mail using mutt, "clicking" is a meaningless term, and I would have ignored your mail. Out of curiousity, does your "click here" blurb trip SpamAssassin's click-through tests? It's quite possible, you'll still get through, but raising the odds of your "I'll never see your mail unless you see this" message getting trashed by spam filters seems like a bad plan.
Re:Great News! by Mysticalfruit · 2004-06-29 03:28 · Score: 1

Or... you could strip out all your personal information and either make those 76megs available for others to train their spamassassians or make the SA database available...

--
Yes Francis, the world has gone crazy.
Re:Great News! by NigritudeUltramarine · 2004-06-30 12:37 · Score: 1

Out of curiousity, does your "click here" blurb trip SpamAssassin's click-through tests?

Indeed, the message ends up with a negative score in SpamAssassin, because it has proper "In-Reply-To" and "References" headers and such. And since it quotes the text of the original message, at the bottom, it gets through any Bayesian filters and such they have as well, unless their message was very spammy in the first place. (In which cast it's their own fault, not mine.)

Here is the real link to spam assasins site by vespazzari · 2004-06-25 18:09 · Score: 4, Informative

For those looking for the official spam assasin site here it is

The link in the text goes to some search page

--
"Alcohol, cause of, and solution to, all of life's problems" -Homer Simpson

3.0? by lorcha · 2004-06-25 18:15 · Score: 1

Anyone know when Spamassassin 3.0 is going to be released? Some spammers seem to have outsmarted 2.63. I'm really excited to see what changes they have made to up the ante in teh war on spam

--
"Avoid employing unlucky people - throw half of the pile of CVs in the bin without reading them." -- David Brent

Re:3.0? by Brian+the+Bold · 2004-06-25 22:03 · Score: 4, Informative

Have a look at the Rules Emporium at:

I use the rules there, and even minor spam gets obliterated with no problems of catching real mail.

I recommend it!

--
-- BtB

And how. by the+HIM · 2004-06-25 18:16 · Score: 1

"I think spam filtering is now a critical part of the network infrastructure"

DSpam by Pinball+Wizard · 2004-06-25 18:21 · Score: 5, Interesting

After using SpamAssassin for quite a while, it just wasn't cutting it - 75%-80% accuracy is still a lot of spam to go through and delete. I added DSpam to my mail server and my spam catching rate is now better than 99%.

DSpam also came with much better directions for integrating with Exim than did SpamAssassin. As fond as I was of SpamAssassin, they have some catching up to do.

--

No, Thursday's out. How about never - is never good for you?

Re:DSpam by Anonymous Coward · 2004-06-25 18:25 · Score: 0

i don't mind deleting by hand
spamassassin makes such cute little headers
Re:DSpam by Anonymous Coward · 2004-06-25 18:53 · Score: 5, Interesting

DSpam 3.0 is definitely not easy to set up. Add to that there is a database that needs to be set up on the back-end, and lots of configure flags at compile-time, plus permissions issues, etc. etc.
It's also not very easy to understand how it works, or configure your mail client to easily train it, or to configure procmail how to properly call it (there are a lot of command-line flags as well).

That being said, IT IS WORTH IT. A properly set up and trained DSPAM filter will SOLVE your spam problem. Training time usually takes about 2 weeks and the results are fantastic after that.

You can also set it up a number of ways - server-side, user-side, with postfix or another mail server, with procmail or without. Relay or not. It's up to you.
Re:DSpam by rasjani · 2004-06-25 19:28 · Score: 1

I'd say that SpamAssassin is neither very easy to configure.

It has a lot of perl module depencies and integration of with MTA was documented quite poorly when i tried to install S.A.

--
yush
Re:DSpam by prockcore · 2004-06-25 19:54 · Score: 2, Informative

I added DSpam to my mail server and my spam catching rate is now better than 99%.

I haven't seen any false positive stats on dspam. It's easy to say a spam filter has a high spam catching rate, but it means nothing without a very low false positive rate.

Redirecting my mail to /dev/null gives me a 100% spam catching rate.
Re:DSpam by Wakkow · 2004-06-25 20:21 · Score: 1

How was spamassassin configured on your mail server? I get better than 99% with spamassassin and so far only one known false positive for the year I've been using it.

However, I might be switching over to Exim soon from qmail, and I'll be sure to check out dspam.
Re:DSpam by fyonn · 2004-06-25 20:35 · Score: 3, Interesting

I've only had dspam installed for a week or so but my stats are as follows: I've taught it 43 spams (ie from a database of nothing, 43 got through and I've trained on them) and 1 false positive (an itms reciept)(again taught to the system) and since then it's been pretty damn good. it's flagged 632 spams and let 730 innocent spams through correctly.

I've got my system set to deliver spam to a spambox which I check nightly for false positives.

and the docs say that I ought to have alot more training before it's up to standard. it's already better for me than SA was.

dave
Re:DSpam by Cato · 2004-06-25 20:43 · Score: 1

Did you turn on SpamAssassin's Bayesian filtering? I found that this is generally good as long as you train it on enough ham and spam. SpamAssassin uses Bayesian filters, rule-based filters, black lists and Razor style services, so it's generally proof against spam that gets through several of these defences. I get something like a 99% hit rate these days.
Re:DSpam by Huge+Pi+Removal · 2004-06-25 21:17 · Score: 2, Interesting

I have to say I had the same problem with SA missing a lot (mind you, I have yet to upgrade to newer versions), and Dspam solved it. Having said that, I still use SA as a "first pass", and delete any mail with a score of >9 or so (I would put it lower, but any false positives and users would complain). This leads to less mail in the dspam quarantine.

It's a bugger to set up with Procmail, but if anyone wants a peek at my config file, just e-mail... One thing I did do was forget about that whole "forward spam to this e-mail address" thing: just too much trouble for users. Instead I created a special IMAP folder into which users could save spam, then a simple script corpus-feeds the contents of that folder into Dspam each night.

Oliver.

--
- Oliver

The right to bear arms is only slightly less stupid than the right to arm bears...
Re:DSpam by paleblueeyes · 2004-06-26 00:06 · Score: 1

I'm on a mailinglist averaging about ten messages/day.
Some time ago the list operators set up spamassassin. Judging from some of the posts to the list they realy gave it some effort and most list members are very satisfied with the result.

Two months ago I enabled DSPAM for mail coming through the list to me. This was realy just a matter of telling it to use a shared dictionary which had been trained by another accounts' mail.

I just went and checked the spambox for mail from that list (no need to do that normally and indeed there were no false positives): it had 20 spam mails which SA (+ additional measures taken by the list ops) apparently had missed.

Maybe someone could try the other way around and see what SA catches after DSPAM.
Not me though. The recent stories about SA make me want to stay away from it even more than before. It all seems so tedious and complex compared to DSPAM which I haven't had to touch after installation.

Alas, by writing this I wasted more time than DSPAM has saved me by filtering the 20 messages :-)
Re:DSpam by dmaxwell · 2004-06-26 02:03 · Score: 1

I have to admin a groupware server and a Spamassassin milter that I've placed in front of it. I throw out anything that scores more than 15 points and tag anything that scores over five. Autowhitelisting and the built-in Bayes filter are enabled. I've even dropped in a selection of third party rules.

This all works pretty well but a trickle of spam still gets through to the end users. I've set up conferences on the groupware box where users can drag missed spam and mistagged ham to for training. Here is the kicker. I have a population of 130 or so of which maybe 7 actually cooperate in feeding me mail to train the Bayes filter (or tweak the others). For everyone else, whatever spam gets through is just something that happens to them (thank goodness for clamav which the milter also runs). Entreaties to feed me training material go unheeded.

I'm interested in further refining this setup with DSPAM or CRM114. It seems to me that neither of those will work without active cooperation from the users for training material. How do you get around that? For all of SA's flaws, it can work even for apathetic users.

And yes this is necessary apathy or no. The powers that be don't like public school teachers getting penis enlargement ads.
Re:DSpam by laitcg · 2004-06-26 02:10 · Score: 1

That being said, IT IS WORTH IT

I agree and in my case, it is much better than SA. I have incorporated it into p3scan[1][2] also. Now you have the option of using SpamAssassin or DSPAM when checking your pop3 mail.

[1] http://p3scan.sourceforge.net
[2] In the development branch.

--
When you want a computer system that works, just choose Linux. When you want a computer system that works, just, choose
Re:DSpam by Chief+Typist · 2004-06-26 03:42 · Score: 3, Interesting

The best feature of DSPAM, in my opinion, is that the SPAM never leaves the mail server.

The bad messages go into a quarantine on the server and can be reviewed by the end user using a web-based interface (looking for false positives.) In the press of a button, that quarantine can be emptied, freeing up disk resources on the server.

Other SPAM solutions (like SpamAssassin) mark the message and continue with delivery. What's the point in downloading the SPAM to your mail client just to throw them away?

-ch
Re:DSpam by Anonymous Coward · 2004-06-26 04:49 · Score: 0

One other problem I had with SA is that (because its perl-based), after training with 1500+ spam & innocent mail, it could only process 8 mails a second on a Dual-Xeon 3.2Ghz.

I have DSPAM doing about 60+ mails a second ;)
Re:DSpam by ghettoboy22 · 2004-06-26 05:44 · Score: 1

Yes, the whole spam-user@domain reporting thing is too confusing for non-techs. I also created an IMAP folder for all users on the mail server, which my Postfix/Amavisd-new/ClamAV/DSPAM gateway nfs maps to said IMAP folder, then cron feeds every hour into DSPAM. Works quite well. So well I'm afraid to upgrade DSPAM for fear of breaking everything!!!
Re:DSpam by Anonymous Coward · 2004-06-27 09:43 · Score: 0

Well, in fact, SpamAssassin relies on Procmail or some such mail processing system to work out the details. It's trivial to tell procmail to leave it at the server.

I suppose it might be some work getting/building the browser interface though.
Re:DSpam by kalidasa · 2004-06-27 23:33 · Score: 1

If you scour around for good filters and such, you get get SA up to 99.7% or so in my experience.
Re:DSpam by blosphere · 2004-06-29 21:03 · Score: 1

Yup, and then you have to take into accout that spamassasin uses perl. Give it enought mails per second and it will kill your system. Not so with dspam.
Also DSpam was breeze to set up on my freebsd. Just insert the ready-made shema in the database, put in some configure options which are very well documented, change the LDA in the postfix config, add an alias into alias list and you can start training.
Then the fun starts.
Start thinking large-scale. Use inoculation groups that vaccinate groups of users filters before they even catch spam. Put up some honeypot addresses/proxies that feed to your filters, further vaccinating the masses. You can even put people on different groups based on their behaviour (sales and engineering people tend to get different type of spam) and pretty soon you have quite robust system. Then you can present a nice corpus of spam for new users so they never have to train their filters from the bottom up.
I got it up with postfix&mysql-dspam-cyrus&sasl&mysql combo in 3 hours, fed first spam in 10 minutes and it detected the first one by itself the next day after feeding 30 spams. No false positives and accuracy over 99.999%. Got what I asked for, now I'm going to see how it handles 100K+ users.

Re:DSpam seems okay but not for relay hosts by cbreaker · 2004-06-25 18:26 · Score: 1

There was no clear way to integrate dspam with my postfix gateway as with spamassassin. Lots of instruction on setting it up with a local mail handler but not so much for a relay host.

Maybe there's a way to do it but I couldn't take the time to figure out a good way to get it done.

--
- It's not the Macs I hate. It's Digg users. -

If Only It Was For Real by Nom+du+Keyboard · 2004-06-25 18:30 · Score: 5, Funny

If only it truly assassinated spamers.

--
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."

Re:If Only It Was For Real by FireBreathingDog · 2004-06-25 18:50 · Score: 1

Yeah, I hate spamers.

--
Shame on Google.
Re:If Only It Was For Real by Anonymous Coward · 2004-06-25 19:16 · Score: 0

Like a pig ninja.
Re:If Only It Was For Real by Douglas+Simmons · 2004-06-25 19:22 · Score: 1

me too!
Re:If Only It Was For Real by Mattygfunk1 · 2004-06-25 20:43 · Score: 1

Feature requests go to SourceForge not Slashdot. :)
--
Dependra Web Hosts
Re:If Only It Was For Real by Anonymous Coward · 2004-06-25 21:47 · Score: 0

I think This guy has the right idea.

i'm going to become rich and famous after i invent a device that allows you to stab people in the face over the internet

The problem is... by Lord_Slepnir · 2004-06-25 18:30 · Score: 4, Funny

See, i'm not interested in Assassinating Spam. Now if there was a SpammerAssassin, then I'd be all over using that.

what to do with spam after it's id'd? by Hollins · 2004-06-25 18:32 · Score: 5, Interesting

What do you do with mail SA has flagged?

I like SA, and find it is very good for identifying around 95% of my incoming spam. However, I also have around 0.1% false positive rate, which means at some point I have to look through all the filtered spam messages and make sure none of them were legit.

I need a better tool for handling mail SA has identified as spam, either server-side or client-side. I'd like to delete anything with a score > 15, simply store anything with a score > 5, and send an auto-reply for scores between 5 and 10 indicating that the message was marked as spam and I'll probably never look at it.

A good set of procmail and formail rules will accomplish this, but my hosting company has a weird procmail setup and I'd prefer something easier to implement.

Any ideas?

Re:what to do with spam after it's id'd? by dasunt · 2004-06-25 19:03 · Score: 4, Insightful

I need a better tool for handling mail SA has identified as spam, either server-side or client-side. I'd like to delete anything with a score > 15, simply store anything with a score > 5, and send an auto-reply for scores between 5 and 10 indicating that the message was marked as spam and I'll probably never look at it.

Procmail can do it, but please reconsider the auto-replies. What happens if I'm pissed at bob and decide to sent out 1m spams with the return address of bob@example.com? More common, what about viruses that forge headers?

I would consider auto-whitelisting instead.
Re:what to do with spam after it's id'd? by Twirlip+of+the+Mists · 2004-06-25 19:10 · Score: 3, Interesting

I need a better tool for handling mail SA has identified as spam, either server-side or client-side.

Yes, you sure do.

Odds are that this doesn't apply to you, but the Mac OS X mail program, Mail, does a brilliant job. It recognizes the YES or NO header that SpamAssassin adds to filtered messages and, depending on your preferences, filters accordingly. By default it merely flags spam messages with a little trash-bag icon and leaves them in your inbox. At the flip of a switch, you can have the program automatically move spams into a Junk folder that (again, depending on your prefs) can be automatically emptied every week or month or day or whatever.

If your mail program doesn't already do this, then your mail program sucks. ;-)

--

I write in my journal
Re:what to do with spam after it's id'd? by Anonymous Coward · 2004-06-25 20:04 · Score: 5, Informative

Sending an auto-reply on scores between 5 and 10 (or any other range) makes you part of the problem, not part of the solution.

I have a very well known address (which is why I'm posting as an Anonymous Coward :-) that receives many hundreds of messages every day. My mail server deals with about half of the spam I get. Well over half of the rest is autoreply responses from idiots who don't understand that *I* never sent that message in the first place -- the from address was forged by a virus.

The correct response to spam is to throw it away. Trying to reply to it makes the world worse, not better.
Re:what to do with spam after it's id'd? by Cato · 2004-06-25 20:46 · Score: 2, Interesting

Auto replies would also get your address marked as 'confirmed valid' i.e. able to receive emails, even if you don't read the spam, so you'll probably just get even more spam.
Re:what to do with spam after it's id'd? by antsquish · 2004-06-25 20:57 · Score: 4, Informative

I know you mentioned procmail, but for those using Courier IMAP's maildrop, here's what I use in my ~/.mailfilter for SpamAssassin. I've just pasted the relevant sections, but it logs all deliveries, I then filter known recipients into their own folders (not shown here), then any unknown messages are filtered through Spam Assassin. Messages with a score > 10 are sent to /dev/null, while others are delivered to a spam folder. logfile "/path/to/my/home/dir/maildrop.log" ### ### Maildrop variable substitution ### MAILBOX="./Maildir" DEFAULT= "$MAILBOX" SPAM="$MAILBOX/.Spam" ### ### SpamAssassin :: filter out spam mail ### # Filter through SpamAssassin xfilter "/usr/local/bin/spamc" # Handle messages marked as spam if ( /^X-Spam-Flag: YES/ ) { # Store messages flagged as spam in another folder; uncomment # this during testing just in case any legit mail gets sent # to /dev/null #cc "./spam-store" # Delete messages with a score of 10 or higher, filter all other # spam messages into a spam folder /^X-Spam-Status: yes, hits=![:digit:]+\.[:digit:]+!.*/ if ( $MATCH2 >= 10.0 ) to "/dev/null" else to $SPAM }
Re:what to do with spam after it's id'd? by KjetilK · 2004-06-25 22:28 · Score: 1

This is something that you need to do at the MTA level, so unless you control the MTA, there isn't a lot you can do.
In your situation, I would simply suggest saving messages above 5 to a special folder and forward messages above 12 to Dave Null.
As others have mentioned, never, never send auto-replies or configure your MTA to send bounce messages.
If you want to reject messages, you have to do it in the SMTP dialogue with the spammer or his raped relay. If you accept the message first, or your ISP does it for you in your case, it is too late allready: Any generated bounce will go to whoever was forged into MAIL FROM: or whatever. And that's really nasty.
But if you control the MTA, then Exim 4 is pretty good at this. If you want the excrutating details, check out Marc Merlins page on using SA with Exim
But there is actually a much simpler version that can do rejection at SMTP-time for messages over a certain threshold and let message between this and another threshold marked as such, using exiscan-acl. You'll find some detailed instructions there.

--
Employee of Inrupt, Project Release Manager and Community Manager for Solid
Re:what to do with spam after it's id'd? by Daniel_Staal · 2004-06-26 03:20 · Score: 1

For a little more complicated, here's my spamassassin-relevent rules from .procmailrc (Note I've got actions I'm not currently using.): :0fw | spamc # Marks extreme spam, and handls one of several ways. :0 * ^X-Spam-Level: \*\*\*\*\*\*\*\*\* { #First report it as spam to the athorities. :0c | spamassassin -r # :0 # For complete deletion... # /dev/null # :0fw # Remove markup... # | spamassassin -d # :0 # For sending to the FTC # ! uce@ftc.gov :0 # For when checking highspam... $MAILDIR/.caughtspam.highspam/ } :0 * ^X-Spam-Status: Yes $MAILDIR/.caughtspam/

--
'Sensible' is a curse word.
Re:what to do with spam after it's id'd? by Crudely_Indecent · 2004-06-26 05:16 · Score: 1

Out of roughly 3000 users of my mail server, I've determined that no 2 users will wish the same fate for their spam. Some users have chosen to be removed from filtering completely, while others want the messages marked but not deleted, while still yet others wish to have the score lowered.

In response, I've set up qmail + vmailmgr + f-prot + horde projects + spamassassin. Users now have the ability to turn on/off spam filtering, increase/decrease spam sensitivity, keep/delete identified spam messages, manage white/black lists per account, auto-reply, forward, etc..., etc...

Unfortunately, there isn't a one size fits all mail-server/spam solution. With a little research and a lot of work you can put together a system that acommodates the needs of a majority of different usage styles.

Now...if I could only teach my network users how to right click......

--

"Lame" - Galaxar

3.0 will be released... by Anonymous Coward · 2004-06-25 18:36 · Score: 0

as soon as it learns to speak french

http://article.gmane.org/gmane.mail.spam.spamass as sin.devel/26614

What's the big deal? by FireBreathingDog · 2004-06-25 18:46 · Score: 4, Funny

Everyone on Slashdot always seems to be complaining about spam. I don't see what the big deal is. I enjoy receiving e-mail from people and companies I don't know. Each morning when I run my e-mail program, it starts downloading, and the unexpected e-mail is a pleasant surprise that brightens my day. Well, a few hundred pleasant surprises that is, and they brighten my day in the same way that stepping in a pile of dogshit brightens my day. A few hundred times. So what the fuck? Why are all you whiny bitches on Slashdot always complaining about spam? Don't waste your time writing or deploying spam blockers. Enjoy life. And relax. Assholes.

--
Shame on Google.

Re:What's the big deal? by prockcore · 2004-06-25 19:58 · Score: 1

I enjoy receiving e-mail from people and companies I don't know.

I have a gmail account and I haven't gotten a single email in a week. Not one. Kind of makes me want to post my email address to a news group.

Kind of.

Novell NetMail by Anonymous Coward · 2004-06-25 18:47 · Score: 1, Interesting

Novell NetMail even supports SpamAssassin now.

http://netmail.sourceforge.net/

You people need to stop being so cynical by Enlarge+Your+Penis · 2004-06-25 18:55 · Score: 5, Funny

I don't employ Spamassassin or any other spam blocker. As a result, I now have a penis that will make her scream, hot lesbian schoolgirls lusting after my every move, a wide range of generic drugs, 2 PhDs and a completely clean credit record

A step up from living in your parent's basement and whacking off to an inflatable doll, right?

I'd stay and chat, but I have to get back to a Nigerian man about a bank transfer

Re:You people need to stop being so cynical by cryms0n · 2004-06-25 19:28 · Score: 4, Funny

I am no expert on inflatable dolls, but I think you are supposed to make love to them, not whack off looking at them.
Re:You people need to stop being so cynical by Chris+Siegler · 2004-06-25 20:05 · Score: 1

When I first read this comment at Score:1 I didn't get the sarcasm, probably because I DO use SpamAssassin. Oh the irony.
Re:You people need to stop being so cynical by Anonymous Coward · 2004-06-25 21:00 · Score: 1, Funny

Dude, you're slipping. Weren't you going to refinance your home?
Re:You people need to stop being so cynical by zsau · 2004-06-25 21:39 · Score: 1

Lesbians lusting after someone with a penis? How odd.

--
Look out!

problem solved by Anonymous Coward · 2004-06-25 18:55 · Score: 0

call this guy

sorting mail by spamassassin score by David+Jao · 2004-06-25 19:08 · Score: 4, Informative

I'd like to delete anything with a score > 15, simply store anything with a score > 5, and send an auto-reply for scores between 5 and 10 indicating that the message was marked as spam and I'll probably never look at it.

I can't speak for auto-replies, but you can do the sorting part client-side. The key is that spamassassin adds a line like "X-Spam-Level: *****" where the number of *'s is the score of the email. Almost any email client can filter mail to different folders based on headers. The unary representation of the spam score ensures that even a primitive filter can work.

For example, one popular client is Microsoft Outlook, and there are several web pages in google (such as this one) that explain how to reroute mail to specific folders depending on the spamassassin score.

Don't worry by KalvinB · 2004-06-25 19:20 · Score: 4, Funny

they'll get it when they post the story again.

Ben

--
Work Safe Porn

Spam... I just don't get it. by NemosomeN · 2004-06-25 19:36 · Score: 1

I intended that to mean "understand," but really, I don't get any. Just have a regular (Well, recently upgraded to ad-free) mail.com address, and I get exactly 0 spam emails. Don't even know specifically of any spam-blocking software.

--
I hate grammar Nazi's.

Re:Spam... I just don't get it. by Anonymous Coward · 2004-06-25 19:38 · Score: 0

Shoulda previewed. I don't know of any spam-blocker used by mail.com.
Re:Spam... I just don't get it. by cpghost · 2004-06-25 20:54 · Score: 2, Funny

Publish your addy on /. (or anywhere else), wait a few days, and have fun!

--
cpghost at Cordula's Web.
Re:Spam... I just don't get it. by NemosomeN · 2004-06-27 16:29 · Score: 1

I anticipated that response soon enough, heh. I think not being stupid is one reason for not getting spam. Though I doubt spambots would pick it up here (Slashdot is not a great place to mine for emails, most of them are obscured).

However, I've seen an article (On /.?) from a guy who bought one of those lists to look at it. Tons of duplicates, many "abuse@*," "admin@*," etc. I can imagine many have NOSPAM in them. I wouldn't be surprised if /. was just a source of screwed up email addresses in lists.

I await the day that Yahoo et al. start automatically removing NOSPAM from incoming mails...

--
I hate grammar Nazi's.

this means by Anonymous Coward · 2004-06-25 19:38 · Score: 0

YOU

ive tried everything by Anonymous Coward · 2004-06-25 19:46 · Score: 0

not even DSpam will block these things
i keep getting messages from some company called CERT that say to upgrade windows stuff
i don't even have windows
nothing blocks these BASTARDS

a better approach: reject the mail by Trepidity · 2004-06-25 20:12 · Score: 2, Informative

If you integrate it with your mailer, you can reject the mail during the SMTP session rather than generating a separate bounce email, which would have the problems you mentioned (going to a forged from: address). As an added bonus, when you reject it during the SMTP session, you'll get taken off a lot of spam lists, since your address will look like it had delivery problems. And you still get the advantage of bounces, that legitimate mail that got rejected will end up with a bounce back to the sender informing them of it.

--
10 PRINT CHR$(205.5+RND(1)); : GOTO 10

Get the owner, not the dog..... by Univac_1004 · 2004-06-25 20:13 · Score: 5, Insightful

Spam Assassin, while a very clever program, is as misdirected as the "Canned Spam" legislation. It has no effect on the real economics of spam: who pays for it.

Somebody is paying for the spamming, and we know exactly who it is. The URL of that organization is prominently displayed in every item of spamail. It is the advertiser.

The advertiser is right there out in the open, easy to locate. If they're not, the spam isn't doing its job, and wouldn't have been sent. And easy to locate means easy to go after, easy to sue, to fine, DoS or whatever.

Dinging the advertisers, and dinging them hard, will instantly put the spammers out of business.

Spamming can be eliminated without blocking, white lists, or anti-spoofing RFC's. Just go to where it's pointing.

To draw an [ugly, graphic] picture: a dog comes and poops on sidewalk in front of my house, and I step in it. Yelling at the dog is going to be only moderately successful, building a poop filter is difficult, messy, and leaky (as Spam Assassin demonstrates) . Following the dog's leash and fining the owner is what works.

The owner doesn't bring the dog back since s/he doesn't want to pay another fine.

No owner, no dog, no spam.

Get the owner.

Kill the spam.

Re:Get the owner, not the dog..... by Anonymous Coward · 2004-06-25 21:50 · Score: 2, Insightful
The advertiser is right there out in the open, easy to locate. If they're not, the spam isn't doing its job, and wouldn't have been sent. And easy to locate means easy to go after, easy to sue, to fine, DoS or whatever.
1. Send out spam pointing to competitor's website
2. Watch them get sued/fined/DoSed/whatever
3. Profit!
Re:Get the owner, not the dog..... by Anonymous Coward · 2004-06-25 23:06 · Score: 1, Insightful

When I read the grandparent, I thought the same thing.

But I strongly doubt this would happen a lot. No, I am not saying to just go with it. I am saying that these URLs make an excellent indication at who to suspect.

The $ passed should be traceable (look under Marketing in the books). Larger businesses must be sticklers for accounting while smaller ones probably can't afford the "service" (and yes, I know about Enron and Worldcom).

The accounting trail would put the nail in the coffin, but not the spam by itself.
Re:Get the owner, not the dog..... by Chatmag · 2004-06-26 00:57 · Score: 2, Insightful

What I had suggested in other posts regarding spam is this:

Let the FBI actually buy something from a spammer, trace the money, as its being bought with a CC, then prosecute whoever cashes the CC transaction. They do buys for drug busts routinely, so why not.

--
Pete Carr Owner Chatmag.com
Re:Get the owner, not the dog..... by dubl-u · 2004-06-26 05:12 · Score: 1

Spam Assassin, while a very clever program, is as misdirected as the "Canned Spam" legislation. It has no effect on the real economics of spam: who pays for it.

I'm not sure why so many people on Slashdot think this, but when you have a good idea, it doesn't mean the other ones are bad.

Your idea is good. The cops should indeed go after spammers exactly as you say. However, that won't get all of them. One recent spam I got was sent from Brazil, advertises a site hosted in China, and transfers the money through a company in Latvia. The beneficiaries are probably American, but they may be a little hard to find.

In the meantime, things like SpamAssassin do indeed change the economics of the situation. If people use just a 95%-effective filter, that means the spammers have to send 20x the amount of mail for the same exposure. And to regularly get past the filters, they have to spend much more time crafting their messages and building software. Or they have to pay somebody who specializes in it. Either way, income goes down and costs go up.
Re:Get the owner, not the dog..... by Anonymous Coward · 2004-07-10 14:18 · Score: 0

Or kill the dog.

New Website then? by Anonymous Coward · 2004-06-25 20:13 · Score: 0, Insightful

Does this mean they finally have to sort out that god awfull tip of a web site then?

Sounds Good by thing2b · 2004-06-25 20:23 · Score: 0, Offtopic

Sounds Good...

--
Webmaster of Infoweb

3.0, late-July, early August by chathamhouse · 2004-06-25 20:31 · Score: 4, Informative

3.0.0pre1 was made available last week.

It will apparently take another month or so to finalize the weighting of the rules.

I've put 3.0.0pre1 on a production system that filters ~350k messages per day. With some tweaking of the RBL, bayes, and AWL rules, it is much (~10%) more efficient at tagging spam than 2.63, which I'm running on a parallel server that also sees ~350k messages/day (load balancing is your friend).

More info: http://www.au.spamassassin.org/full/3.0.x/dist/bui ld/3.0.0_change_summary

Re:3.0, late-July, early August by AKnightCowboy · 2004-07-09 03:10 · Score: 1

I've put 3.0.0pre1 on a production system that filters ~350k messages per day. With some tweaking of the RBL, bayes, and AWL rules, it is much (~10%) more efficient at tagging spam than 2.63, which I'm running on a parallel server that also sees ~350k messages/day (load balancing is your friend).
Just out of curiosity, that sounds like you're running it on your mail gateway. How do your users set their spamassassin options to adjust their filter settings, or do you just give everyone a global setting?

Know what? by Anonymous Coward · 2004-06-25 20:53 · Score: 0

I'm with you 99 %.

Challenge-Response schemes are more effective by cpghost · 2004-06-25 20:59 · Score: 1, Interesting

Filtering spam generates way too many false positives. Challenge/Response schemes are IMHO much more effective. TMDA and similar programs can be configured with whitelists for your regular mail partners, auto-whitelists for everyone who confirms their e-mail identity, and, if necessary, with blacklists too.

--
cpghost at Cordula's Web.

Re:Challenge-Response schemes are more effective by Vellmont · 2004-06-25 21:35 · Score: 4, Informative

I've been running SA since February, and have had a grand total of ONE false positive out of a few thousand emails. The message was from a new account, very short, and in HTML. That address has since been added to my autowhitelist. SA couple with Amavisd-new and clamav has reduced my spam volume by about 95%, and my virus emails to zero. It's a great product and I'm looking forward to 3.0.

--
AccountKiller
Re:Challenge-Response schemes are more effective by Anonymous Coward · 2004-06-25 22:35 · Score: 0

No, TMDA is a pain in the ass when viruses/worms, etc forge your address and you get TMDA challenges for messages you didn't write. I personally permban users of TMDA that send me challenges for messages I didn't write.
Re:Challenge-Response schemes are more effective by daBass · 2004-06-26 02:51 · Score: 1

You've been using the wrong filter, it seems.

I filter 97%+ of Spam (~175/day) and the only false positives I get are from companies just begging to be filtered. Ie: Easyjet.com, in the (massive HTML) conformation of my itinerary trying to sell, sell, sell me hotels, car rentals and the lot.

Those companies should realise that Spam is a problem end even though they are not spammers, they should keep in mind their messages will be sent through a filter, so better make sure they don't look like it.

I have never missed a truly, genuine, personal email with SpamAssassin, nor has anyone I know.
Re:Challenge-Response schemes are more effective by jdowland · 2004-06-26 05:49 · Score: 3, Insightful

Challenge/Response is fundamentally broken. For more information, take a look at some discussions on the topic from debian-user: here's one. There's a few google-harvested discussions on the topic too.
Re:Challenge-Response schemes are more effective by cpghost · 2004-06-26 08:27 · Score: 0

The arguments used against C/R systems in those discussions all boil down to users' inability to use such a system correctly.

Take for instance mailing lists: if you subscribe to such a list, you should, of course, whitelist that list. Moreover, most C/R systems have built-in safeguards so that they won't send a confirmation message to the list. If you, as a list member, reply to a question, you should answer publicly to the list, so that everyone can profit from your knowledge. Such replies will make it through the whitelist, because they come from the list itself. If, OTOH, you decide to write to me privately, well, that's tough. If you value your own reply, you won't object confirming your address anyway before proceeding.

Another point is if I sent you a message out of the blue. Of course, it would be rude if you had to confirm your address just because you've replied to my message. Most C/R systems allow me to set up my MUA in such a way that it either auto-whitelists every e-mail address I send mail to (so they won't have to confirm when they reply), or I can send them time-limited tagged addresses.

The only real problem with C/R systems is when you subscribe to a service, and that services' confirmation messages come from a completely different domain. It is gross from them to do this, but it happens every now and then. So I always check the tail of the pending queue when I didn't get an expected confirmation message from one of those (IMHO deliberately misconfigured) servers.

The last point is that spammers could set up an autoresponder to auto-whitelist themselves. Well, that almost never happens. It would be great if the increased use of C/R systems forced them to do so, because we could then track those spambags and sue them. As long as they cowardly hide behind fake addresses, C/R is (IMHO) the way to go.

If you take all this into account, C/R systems are absolutely GREAT. Used in combination with regular anti-spam configuration in postfix (or sendmail, qmail, exim, ...), they are very effective.

--
cpghost at Cordula's Web.
Re:Challenge-Response schemes are more effective by cpghost · 2004-06-26 09:47 · Score: 2, Insightful

We shouldn't feed the trolls (eh. ACs), but I'll bite anyway, because it's a valid argument.

You also ban all innocent bystanders than send you regular 550: no such user bounces, right? TMDA messages are exactly like bounces if you think of it. They appear automatically generated on purpose. It's a piece of cake to filter them if you dislike 'em. It's not like spam which tries to deceive you.

Now, trying not to be too caustic, backscatter is a fact of life. If you really want to avoid this completely, you have to follow a strict whitelist policy. Some people actually do this, and if you must, go ahead, block all TMDA users. It's your decision to allow/disallow users (legitimate or illegitimate), bots, or spammers to access your network. That's exactly what TMDA is all about: putting the recipient, not the sender, in control.

OTOH, it's up to TMDA's users to decide how they control their own networks. If it helps stem the spam tide (and it does extremely well!), it will be used. Sending innocent bystanders a 550: No such user or a TMDA confirmation message with a list of full headers is qualitatively the same; perhaps even better, because if you belong to some spam busters brigade, you're free to use those headers to RBL the initial offender, dynamic IP zombie or whatever.

Instead of whining about backscatter, fix SMTP or your legislation (or both). In the mean time, C/R systems are the only alternative to content-based filtering. If you combine C/R and C/B systems, you also reduce the amount of TMDA bounces. Permbanning only helps the spammers by intimidating potential TMDA users and slowing down a more widespread adoption of C/R systems (which would also dry up the spam stream substantially). But, as said, you're the recipient, and you're free to do whatever you like. It's your resources. Make good use of them.

--
cpghost at Cordula's Web.

I prefer my method - sacrificial subdomains by anti-NAT · 2004-06-25 21:18 · Score: 1

Not perfect, then again, spam prevention methods never are.

What I do

--
The Internet's nature is peer to peer - 20050301_cs_profs.pdf

Re:I prefer my method - sacrificial subdomains by cpghost · 2004-06-25 21:30 · Score: 2, Interesting

As a side note, I don't use these email addresses for personal emails - I can hopefully trust that the people I personally send emails to are not, or are not going to become spammers.

Well, that is not a very secure assumption. Unless you know that all those people are not using an MUA/OS combination that is vulnerable to viruses or worms. Harvesting addresses is done that way nowadays...

--
cpghost at Cordula's Web.

It is an odds game by anti-NAT · 2004-06-25 21:43 · Score: 1

The people I trust with my "hidden" personal email address are also people I trust to run a patch MTA/MUA. Of course, they also have my personal phone number, if they need a different way of contacting me, when email doesn't work.

If I don't trust them to run a patched MTA/MUA, then they'll get one of my "special" ie. sacrificial subdomain email addresses, rather than my actual personal one.

A key point, which I didn't realise when I wrote the original text at the URL, is that nothing is permanent, including my physical address, my phone number or even my name. Once you accept that all these things have different levels of temporalness, you can then, in the specific case of spam, start creating and using email addresses that have differing levels of life expectency. My (current) personal email address isn't ever going to be permanent, as I probably won't have nosense.org as a domain for the rest of my life.

Once you get over the idea of a permanent, rest-of-life email address, you can then start to be a bit more creative with spam mitigation techiques.

--
The Internet's nature is peer to peer - 20050301_cs_profs.pdf

Re:It is an odds game by cpghost · 2004-06-25 22:10 · Score: 1

nothing is permanent, including my physical address, my phone number or even my name.

Now that makes sense ;) It's actually a great idea.

Imagine the repercussions on the legal system if we could switch IDs in real life every now and then! Change your bank account, diplomas, contracts... would be quite difficult. Unless you have a secret unique ID stored on some government computer. Connect to that server, and change your public name anytime you like (after paying a fee?).

$ ssh <my_unique_id>@idmgmt.idserver.gov Passphrase: <what-was-it-again?> Sorry.

Oh, that was too optimistic. You'll have to use some closed-source, drm-protected, proprietary software to access a nice flashy GUI where you can manage your public IDs.

:-)

--
cpghost at Cordula's Web.

throws away ANY bulk mail by gfody · 2004-06-25 22:05 · Score: 4, Interesting

not all bulk mail is spam. spam assassin gives 2.4 points if it finds anything that looks like a unique identifier for X-Sender, and another 1.4 points for anything that looks like a tracking image or tracked link.

that plus the points for any non-safe html colors or any html at all, SA effectively tags ANY bulk mail as spam!

For an end user to setup on their client (as a "junk mail" folder) thats great.. I like to have bulk mail seperated from my personal mail, but for an ISP to throw it away before it even gets to the intended recipient is fucking rediculous and should be illegal.

The only email an ISP should be allowed to discard are the ones with attached viruses or some known email worm. The only reason your customers are happy with you throwing away their email is because you don't fucking tell them.

--

bite my glorious golden ass.

Re:throws away ANY bulk mail by hyperlinx · 2004-06-25 22:46 · Score: 1, Interesting

Now that emails are used in court as evidence, ISPs or Webmail providers should never auto delete emails....ISPs should at least offer a link to a quarantine folder and let u choose to delete or not...Webmail services could generate an automatic folder called SPAM? that users could review occasionally if they want to rule out any misfiled emails. With the results others are getting with SpamAssassin, I would appreciate some filtering especially for my Webmail account...The filters currently used by several of my Webmail account providers certainly don't catch anywhere near the percentage of spam like those reported here.

--
In /.space, no one can hear you SCREAM!
Re:throws away ANY bulk mail by khuber · 2004-06-25 23:18 · Score: 1

not all bulk mail is spam
Which bulk email isn't spam?
Re:throws away ANY bulk mail by 0x0d0a · 2004-06-26 00:28 · Score: 2, Insightful

but for an ISP to throw it away before it even gets to the intended recipient is fucking rediculous and should be illegal.

Thank Microsoft. ISPs could easily just add a header line and let the user filter on it, but Outlook Express is crippled from Outlook in that it can't match on arbitrary header lines, forcing ISPs to delete or leave alone.

I agree that SA is great client-side, which is how I use it. The problem is that it isn't plug-and-play on even *IX, and it's not trivial to set up on the client side on Windows.

--
May we never see th
Re:throws away ANY bulk mail by KjetilK · 2004-06-26 00:29 · Score: 1

Never been a problem here, and my parents are on quite a few mailing lists where they get these kinds of messages. I'm still at 0.01% false positives.
The point is, there are also many rules that give negative scores for especially hammy messages. So, messages can have a lot of very spammy things about them, but as long as they also have a lot of hammy things about them, you're ok.
Also, be sure to train your bayes database well with a lot of ham, it'll help a lot too.

--
Employee of Inrupt, Project Release Manager and Community Manager for Solid
Re:throws away ANY bulk mail by Alexis+de+Torquemada · 2004-06-27 10:20 · Score: 1

not all bulk mail is spam. spam assassin gives 2.4 points if it finds anything that looks like a unique identifier for X-Sender, and another 1.4 points for anything that looks like a tracking image or tracked link.
that plus the points for any non-safe html colors or any html at all, SA effectively tags ANY bulk mail as spam!
I don't agree with all of its default settings either, but then, it's simple to adjust the scores in any way you like.

--
I love C++
Re:throws away ANY bulk mail by Anonymous Coward · 2004-06-28 01:54 · Score: 0

Mailing lists you are subscribed to.
Re:throws away ANY bulk mail by Anonymous Coward · 2004-06-28 02:31 · Score: 0

(AC because I'm moderating)

Many ISP's use subject line modification to get around outlook's inability to match using arbitrary header lines.
Re:throws away ANY bulk mail by Just+Some+Guy · 2004-06-29 01:28 · Score: 1

Any of the 25,000 copies of the weekly newsletter that one of my clients sends to everyone who followed the sign-up procedure on their website.

--
Dewey, what part of this looks like authorities should be involved?

It's all about training.. by xmedh02 · 2004-06-25 23:57 · Score: 1

Well, I still use SpamAssassin, some 30 spams a day, no false positive, and 1 false negative in a month or so. Why is it so successful? I have spent a lot of time training it's Bayesian filter on both ham and spam. That's the key to success for any spam filter nowadays..

You can even run spamassassin directly on Exchange by AssFace · 2004-06-26 01:47 · Score: 3, Informative

Many people use spamassassin on unix boxes, or if they have Exchange they use SA on a unix gateway between the net and the Exchange system.
But if you are a smaller shop and don't have the resources for that, then you can run sa right on Exchange.
Here is a write up on how to do it (that particular write up is for Exchange 2003 and SA 3.0, but it will work for SA 2.x as well, and for Exchange 2000 - or any combination thereof - but it won't work on Exchange 5.5 that I know of).

--

There are some odd things afoot now, in the Villa Straylight.

Good for them by Seven001 · 2004-06-26 03:19 · Score: 1

SpamAssassin has been a life saver. A few months back my server started taking a spam pounding. My personal spam alone went from just a couple a day, to at least 15 a day (I know thats not much for some of you, but it is for me, I've always been smart about where I give my email address. Unfortunately, I have to give an address that shows on domain whois databases and thats mostly where they get me). So I installed MailScanner, SpamAssassin, and ClamAV. I have it set to tag email only, but that works fine, I just filter spam to a spam folder in OE and occasionally take a quick glance to see if it caught any non-spam emails before I empty it. So far it has only caught one email I wanted to received, and it looked so much like one big ad that I understood why.

I'll never need Spam Assassin by fleener · 2004-06-26 03:25 · Score: 1

See, I've realized that spammers get my address because of my own behavior. I changed my address and started protecting my it responsibly, and have not received spam in two years. Count 'em. Two years. Ya'll are whores playing fast and loose with your personal information and then crying when you get herpes.

Re:I'll never need Spam Assassin by Anonymous Coward · 2004-06-26 05:47 · Score: 0

Some people are on unmoderated mailing lists.
Re:I'll never need Spam Assassin by fleener · 2004-06-26 08:44 · Score: 1

#1 Few spammers monitor mailing lists, simply because there are more efficient methods of collecting addresses.
#2 Listserv moderation has nothing to being protected from spammers.
#3 The issue is whether a mailing list is archived on a web site, not protected behind a password. If you join such a listserv, don't whine about the spam. When you engage in risky behaviors, expect consequences.
Re:I'll never need Spam Assassin by cpghost · 2004-06-26 09:19 · Score: 2, Insightful

You're just plain lucky. It's a fact of life that at least one of your email pals will use Windows, and store your emails in an Outlook or Outlook Express mail folder. Some days later, your pal will catch a worm or virus, and this little spam helper will harvest all those addresses, including your beloved, "protected" addy.

--
cpghost at Cordula's Web.
Re:I'll never need Spam Assassin by fleener · 2004-06-26 13:53 · Score: 1

Oh, I do get virus and worm infected e-mails. However, none of those infections have resulted in true spammers getting ahold of my address.
I don't worry about the infected e-mails because my filter is 100% effective at screening them out. I really don't consider infected e-mails to be spam because the purpose and intent is entirely different than spam-for-profit.
Re:I'll never need Spam Assassin by cpghost · 2004-06-26 19:48 · Score: 1

The problem is not that you can screen/filter viruses on your side, it's that your pals won't on their side; and once a virus on your pals' machine gets your address (from their addressbook or mail folder content), it will send this address to harvesting servers, which collect all new addresses. It is then just a matter of time before you start getting tons of spam too. You didn't disclose your address, but some virus on your pals' machine did it. You have absolutely no control over this, once your email leaves your computer and is stored on someone else's more vulnerable machine.

--
cpghost at Cordula's Web.
Re:I'll never need Spam Assassin by fleener · 2004-06-27 02:21 · Score: 1

>It is then just a matter of time before you start getting tons of spam too.
I guess time is on my side then. I get plenty of infected e-mails. Probably 200 Beagle variants alone. None have resulted in my address being harvested by spammers. I believe you are mistaken about viruses and worms harvesting addresses. (Got some documentation?) By definition, I should be spammed out by now.
The problem with using virusses and worms to harvest addresses is that "the powers that be" will analyze the infection and identify where the addresses are being sent to. Right off the bat, we have to be talking about an overseas criminal, but even then, those people would be tracked down, or the destination storage area deactivated. The U.S. government is effective in those respects. Giving your virus or worm such an easy track-back mechanism is akin to a masked bank robber giving a bank teller his bank account number to deposit the stolen money.
Re:I'll never need Spam Assassin by cpghost · 2004-06-27 05:53 · Score: 1
The problem with using virusses and worms to harvest addresses is that "the powers that be" will analyze the infection and identify where the addresses are being sent to.

Don't overestimate their investigative prowess. It is actually quite easy to send back the addresses where they can be collected:
- Post them on USENET
- Put them on spammer's P2P networks (some zombies are great at this)
- Hide them in spam itself.
As for the last point: you can add the addresses as plain text in the spam payload, or you can even cryptographically encrypt them and disguise that as bayesian filter fodder.

Whatever method the virus uses, spammers can collect the information in all anonymity and, therefore, impunity.
--
cpghost at Cordula's Web.
Re:I'll never need Spam Assassin by fleener · 2004-06-27 09:23 · Score: 1

> spammers can collect the information in all anonymity
You've provided no evidence. Sure, anything is possible, but that's hardly an argument. Show me evidence that spammers are using worms and viruses to harvest addresses from infected users' address books. If they were, by all rights, I should be swamped with spam.
Re:I'll never need Spam Assassin by cpghost · 2004-06-27 11:07 · Score: 1

Yes, I didn't provide evidence, because the only evidence I have is source code from a grey-hat workshop experiment which establishes a stealth P2P virus/worm-based network (in a lab environment) which does exactly that harvesting stuff I was talking about. This self-spreading, self-organizing and self-healing network code must never be released in the wild for reasons you certainly understand.

Coding this is very easy for experienced people with shellcoding, sockets-api and openssl experience. It's not difficult to find programmers with this skill set. I'm sure that the black hat community is using similar networks since many years now.

So, sorry, you'll have to take my word on this or not. It's your call ;)

--
cpghost at Cordula's Web.
Re:I'll never need Spam Assassin by fleener · 2004-06-27 12:42 · Score: 1

Yes, I take your word for it. You have convinced me my inbox is loaded with virus and worm-induced spam. I don't know how I ever got by without a spam filter. I must go out and buy a spam filter now. Must buy. Musy buy. Brains. Must eat brains.
Proof is in the pudding.
Re:I'll never need Spam Assassin by Pendersempai · 2004-06-29 04:08 · Score: 1

Look; sometimes, with your non-disclosure strategy, you'll get lucky. It looks like you did. Congratulations.

Other times, maybe your hosting company will let slip your email address. Maybe one of your friends will send you one of those goddamn Yahoo E-cards, and Yahoo will have your address to sell at will. Maybe one of your friends will fill your address in one of those "Forward this article to:" fields on a commercial news site; do you trust the site? Maybe one of your friends will normally send an email to you, and HER hosting company will sell your address. Perhaps that's not likely if she uses Earthlink or AOL, but what about addresses like Hotmail? Yahoo mail? Do you trust Microsoft and Yahoo? Or, here's a good one: maybe a spammer will dictionary-spam your email server and record each address that DOESN'T bounce.

Remember, once your address gets on a single mailing list, it doesn't matter how diligent or lucky you are in the future. Marketers regularly share and conflate lists.

Even if you use your email address minimally -- by which I mean sending email to friends and accepting email from friends and nothing else -- you might well get snagged by one of the above tricks.

And, of course, some people don't have the luxury of using an address that minimally. Some people need to supply a contact link, or print business cards, or in some way advertise.

But even if that's not the case, it's still largely out of your control. So, once again, congratulations that you got lucky, but don't assume that those of us who didn't are somehow promiscuous with our personal data.

hey, did anyone else... by ShadowRage · 2004-06-26 04:11 · Score: 1

think of some monster inside a giant larval sac when they said incubating?

Re:hey, did anyone else... by tehcyder · 2004-07-02 00:54 · Score: 1

Counselling not working out, huh?

--
To have a right to do a thing is not at all the same as to be right in doing it

Re:DSpam seems okay but not for relay hosts by ghettoboy22 · 2004-06-26 05:40 · Score: 1

I'm no *nix expert, but I finally figured out how to do DSpam 2.10 on a Postfix/Amavisd-new gateway. Email me com dot akghetto at gdl (letters not backwards, just the words) and I might be able to help.

My big mistake- left DNS record public by Anonymous Coward · 2004-06-26 05:51 · Score: 0

Over about a year, my spam levels had been gradually reducing from 250/day (for a single email address!) down to six or seven, as I had been bouncing spam diligently - I know it's a pain for faked senders, but it seemed to effectively discourage the spammers better than just dropping it.

Recently I renewed my domain name at Network Solutions. I made the mistake of not paying the extra $9/year to keep the record private. Within a couple of days my spam levels went to about 50-100 per day. I'm a cheapskate, and I didn't feel like I should have to spend money for this, but now I wish I had.

Personal spam filter? by Gorimek · 2004-06-26 05:58 · Score: 1

Is there a good spam filter I can install on my personal computer? I guess it has to involve setting up a mail server on my machine just for my incoming mail, but I'd be ready to look into doing that if it would help.

My ISP is using SpamAssasin, but I have no influence over the setting they use, and it lets far too much through.

I assume a server solution is better than a filtering client. Besides, it would make it possible to filter for both computers in the household.

I'm on Mac OS X.

Re:Personal spam filter? by texspeed · 2004-07-01 04:51 · Score: 1

OK - I'll bite...

If you're using MacOS X and NOT using the supplied Mail application - why not? It has some pretty sophisticated (and effective) spam filtering features and is dead simple to use.

Unless you have a very high volume of email, it will likely do the job for personal use. After training a bit, I've had one false positive in about 2 years of use.

Re:You can even run spamassassin directly on Excha by aardvarko · 2004-06-26 06:42 · Score: 1

It can also be run with godforsaken GroupWise with the addition of a piece of software known as "Guinevere" and a lot of elbow grease; trust me, I've had the pleasure of doing it.

OT: Spam Cannibal by gilgongo · 2004-06-26 10:50 · Score: 1

As it seems now obligatory to mention anti-spam systems whenever a /. story mentions spam, I thought I'd add the following:

Please have a look at Spam Cannibal

It's an interesting concept that if correctly deployed (big "if") by even a relatively few admins around the world, could really make a difference to the amount of spam on the net. It can also protect hosts against DoS attacks of various kinds.

Don't get me wrong, I'm not astroturfing this (much...). It has flaws - there are those who think blacklisting is a bad idea, and I can see their point of view on that - but I just think Spam Cannibal needs more visibility as an approach.

--
"And the meaning of words; when they cease to function; when will it start worrying you?"

WOOT WOOT!!!! by script14 · 2004-06-26 13:23 · Score: 1

Nice finally. Makes me want to run web mail now.

Two Words by Alexis+de+Torquemada · 2004-06-27 08:16 · Score: 1

Joe Jobs

--
I love C++

How do you configure a site-wide SA filter? by Anonymous Coward · 2004-06-27 11:20 · Score: 0

How do you configure SpamAssassin for a mail relay of about 200 users?

I have SA running on a Linux server (sendmail) as a mail relay receiving mail from the Internet, and forwards the mail to multiple Exchange servers of about 100 people each. The internal users don't have accounts on the sendmail server.

What's the best way to implement SA in this situation? A common Baysian database doesn't work very well since each user has different definition of what "spam" is, and loading everyone's Inbox into the database isn't a very good option.

Re:Huh? by daringone · 2004-06-28 03:00 · Score: 1

Try turning on the Bayes filter and be amazed :-) Spamassassin + SARE Rulesets + a well trained Bayes filter = 99+% effectiveness for me.

Blacklists by GetPFunky · 2004-06-28 04:28 · Score: 0

I blacklist *@*.* and it's quite effective at getting rid of spam. I have a 100% spam filter. It also filters my e-bills as well as a added feature.

SA and filtering by nuggz · 2004-07-08 05:33 · Score: 1

My one email account uses SA, but does not auto delete.
Evolution just files those with high rankings into another folder.

I don't think auto deleting emails is a good idea.

Yahoo does a similar thing, throws suspect spam into a bulk folder. Quite often I find stuff (website registrations) in there.

Cost of spam by nuggz · 2004-07-08 05:37 · Score: 1

There are 3 main costs to spam
Server - CPU/Storage
Bandwidth
User time to read/sort it

User time is expensive.
Bandwidth and servers are relatively cheap
Email filtering saves the most valuable resource, this helps limit the damage of it. Nobody is saying this is the ideal case, but it is an effective tool.

SA is the only way to go. by plutoiddiamonds · 2004-07-09 03:10 · Score: 1

I was so impressed ever since I heard about SA.

Slashdot Mirror

SpamAssassin Gets a Promotion

168 comments