DSPAM v3.6 Released

Nuclear Elephant writes "After six months of development, DSPAM v3.6 has been released. The most notable change is the series of new features added to make an anti-spam gateway appliance possible (Knoppix anyone?). Version 3.6 also includes a highly accurate alternative to Bayesian filtering known as Markovian discrimination, based on Bill Yerazunis' research. Other significant enhancements include trusted sender whitelisting, integrated Clam Antivirus and LDAP support, a centralized spam training alias, and a new dependency-free storage driver. Much of the documentation has also been rewritten to make installation easier. A change log and release notes are also available. Slashdot has recently featured a review of the author's book, Ending Spam and an interview as well."

Comments

Comparison to other tools

[Puramoca]

It would be interesting to compare this version to other spam filters and see how it measures.

Re:Comparison to other tools

[gvc]

TREC's Spam Track will evaluate several spam filters. There's also a toolkit for do-it-yourself comparison.

Although DSPAM is not an official participant at TREC, three configurations will be evaluated for comparison - with tum, toe, and teft training modes. Zdziarski reported some of the preliminary results in his interview, but complete and comparative results won't be available until TREC in November.

Re:Comparison to other tools

[pushf popf]

While it's great that it learns and makes decisions about the "spamminess" of various incoming items, the most reliable method I've found so far is Greylisting.

The moment I installed and started GLD (gasmi.net), the spam simply stopped. It was like flipping the "nospam" switch on. The spam just stopped. No false positives, no missed spam, nothing.

Every now and then I get unwanted email, but at least now it's from an actual, identifiable SMTP server, not a spam-bot.

It's an amazing improvement from implementing a really elegant concept.

Get SCUBA Diving Water conditions the coastal US waters at: bupkis.org

Windows and Exchange.

[Jaruzel]

I know I'm going to get mauled over this quesiton... but has anyone compiled it on Windows 2003 server ?

For practical reasons I don't have linux in my test lab, and I'd like to have DSpam on my Webserver which is running IIS6 and Windows 2003 Server.

I can see I need to run it in SMTP mode with a relay to my Exchange box, but I don't want to waste my time trying to compile it (using Visual Studio), if someone already knows it wont work.

-Jar.

Re:Windows and Exchange.

[myspys]

from the FAQ (http://dspam.nuclearelephant.com/faq.shtml#1.15)

Q. Does it work with Windows?
A. v3.2 is the first to include a Windows build supplement, which includes the necessary Visual C++ project files and portage to compile the agent and tools under Windows. Check out the win32/ directory in the source tree for more information. Win32 support is still unofficial, but seems to work well. Of course getting it compiled is one thing, getting it integrated is another. It's probably best to build it under Cygwin using the general distribution.

Too late

[mordors9]

But the great news is this product is no longer needed. After all the FBI has put a stop to all of that: http://www.detnews.com/2005/technology/0510/16/B01 -349738.htm (For those that are easily confused, the comment was tongue in cheek)

Try DSPAM

[ajs]

I'm a long-time proponent of and rare contributor to SpamAssassin, and I'll continue to be, but fighting spam is much like fighting disease: you have to diversify your defenses. DSPAM is a nice package, and is very well designed. I've spoken to the author in the past, and he has an excellent understanding of the complexities of the issue (as opposed to the legions of people who seem to think that spam filtering should be easy, given the right algorithm).

As far as I'm concerned there are two tools for spam filtering: DSPAM and SpamAssassin. Try them both. See what fits your needs. My impression is that SpamAssassin provides more knobs and buttons and is more easily extended by the casual user, but DSPAM can be lighter weight. Both are highly accurate, with very low false positive rates.

Re:Try DSPAM

[gvc]

I use Spamassassin with a special user configuration file and I train it systematically. In this configuration it works pretty well (much, much, better than out-of-the box). But Bogofilter and Popfile work about as well. As does just the Bayesian component of Spamassassin, ignoring all the other cruft. DSPAM, on the other hand, doesn't work at all well for me.

Re:Try DSPAM

[gvc]

If a user has to go into the SPAM box and double check that no mistakes have been made then the system is worse than not having any SPAM checking at all.

Not true. First, if the user's mailbox is cluttered with spam, the user is more likely to overlook good mail. More likely than a good spam filter. Second, it is way easier to scan a list of predominantly spam for occasional good mails (and vice versa) than to have everything jumbled together. Third, spam filters are good enough that one does not need normally to look through the quarantine list. Instead it can be searched if and when email goes missing. Almost all spam that is misclassified by a filter is weird in some way - cold call, internet transaction, advertising. Generally one of two mitigating circumstances holds: (1) there is a secondary social mechanism whereby the missing mail will be noticed and retrieved [e.g. nobody assumes that a cold call is delivered, and a reply to an internet transaction would be expected]; (2) the user doesn't really care about the email [e.g. advertising from their frequent flyer plan].

I've found greylisting to be the best solution so far

Greylisting "works" only because spammers aren't on to it yet. And it is intrusive - adding delay and risk of non-delivery. Greater risk, I posit, than the risk of using a spam filter.

Linux Router

[Stavr0]

I know I'm going to get mauled over this quesiton... but has anyone compiled it on Windows 2003 server ? (Release the hounds!)

How about getting it compiled into a Linksys WRT54G router firmware i.e Sveasoft firmware?

Re:Linux Router

[op00to]

DSPAM, as it's running in my cluster, is using way more ram than the WRT54G physically has. Probably not a good idea to run it on that little box.

Re:hiding your address

[Bogtha]

Though this is only possibly with PHP, ideally running on a Debian system, it's the most important language to learn in the universe.

What kind of fuckwittery is this? No, plenty of languages can code a simple contact form handler, the platform you run it on is pretty irrelevant, and PHP is by no means "the most important language to learn in the universe". It's a pretty typical scripting language, not the magic you make it out to be.

Re:hiding your address

[BigJim.fr]

> The best defense against spam is never to type your
> personal address anywhere on the internet.

Hiding your address does not work because some viruses collect addresses from your correspondents addressbook. Your address will percolate to spam lists, it is only a matter of time. If like me you have kept your adress for many years, you absolutely need some form of spam defense.

OpenBSD port

[chrysalis]

The OpenBSD port can be downloaded from ftp://ftp.00f.net/misc/port-dspam-3.6.0.tar.gz

Re:curious about MD

[Nuclear Elephant]

Below are some tests I ran with a pre-release version of DSPAM on a test corpus. As you can see, Markovian discrimination is significantly more efficient than any Bayesian methods and Chi-Square. Markovian showed slightly more (4 more than the top contender) false positives, but it also caught 100 more spam... some additional tuning, tweaking, and most importantly, training, can easily get this down to a very low error rate.

Bayesian (burton)
TP: 785 TN: 1003 FN: 218 FP: 4 SC: 4 IC: 0
SR: 78.27% IR: 99.60% OR: 88.96%

Chi-Square (multiword)
TP: 801 TN: 1005 FN: 202 FP: 2 SC: 0 IC: 0
SR: 79.86% IR: 99.80% OR: 89.85%

Chi-Square (single Word)
TP: 794 TN: 1003 FN: 209 FP: 4 SC: 2 IC: 0
SR: 79.16% IR: 99.60% OR: 89.40%

Bayesian (graham)
TP: 833 TN: 1002 FN: 171 FP: 4 SC: 4 IC: 0
SR: 82.97% IR: 99.60% OR: 91.29%

Bayesian (graham-burton)
TP: 838 TN: 1000 FN: 166 FP: 6 SC: 4 IC: 0
SR: 83.47% IR: 99.40% OR: 91.44%

Markovian discrimination (burton)
TP: 950 TN: 996 FN: 54 FP: 10 SC: 0 IC: 0
SR: 94.62% IR: 99.01% OR: 96.82%

Re:hiding your address

[MasTRE]

> Other than annoying whitelists, there is no anti spam warez that is bulletproofly reliable.

Yeah yo, no bulletproofly reliable warez yo!

> ...just set up a simple form and use simple php to make it convenient for them to...

Make it convinient to root your server, yo! Yeah, yo! Bulletproofly warez, yo!

> Though this is only possibly with PHP...

Yeeeeaaaah, buddy! Warez, yo!

NOT!

Whatever TF this guy is smoking, you lemmings shouldn't mod it +4/Informative. It's a crap post.

Re:curious about MD

[Nuclear Elephant]

4FPs for 100-something more TPs? Heck yeah. At least for me.. But keep in mind these are just preliminary training numbers with 1000 messages in each corpus. After real-world training, any of these approaches will be much more accurate.

Not an advertisement...

[pabl0]

... but it'll sound like one: I recently converted from a rather involved anti-spam defense utilizing SpamAssassin with Razor, Pyzor, and several RBL checks. I spent a fair amount of time selecting RBLs that worked the best and tweaking SA test scores whenever I got false positive/negative messages. I even had all sorts of validity checks turned on in the MTA to block out badly formed messages and the like.

I replaced all those defenses with: DSPAM. And I'm seeing better results out of the box than I ever did with a multi-layered SA-based solution, even after a lot of time tweaking.

A quick anecdote: When I converted, I opened up a bunch of previously blocked spamtrap addresses, just to get some good training material for the filter. I've long since passed my initial training threshhold but haven't even bothered to block the spamtraps again because I never see the spam. At the risk of sounding like I'm bragging, I literally don't have a spam problem anymore, and DSPAM is entirely responsible for that.

Now, I'm not necessarily advocating that you give up all your custom defenses and switch to DSPAM. (I've turned off all my other filters, but I haven't removed them completely.) There's always a chance that an ingenious spammer will find a weakness in DSPAM setups, but I can testify to the fact that DSPAM is "scary good" as of right now. Training the filter is a simple matter of dropping misclassified messages (and there aren't many) into an IMAP folder.

If what you have is working for you, stick with it. But if you're looking for a low-maintenance, high accuracy filter, you should definitely give DSPAM a shot.