DSPAM v3.0 RC1 Spam Filter Released

How is this a YRO? by magefile · 2004-05-31 06:16 · Score: 3, Insightful

I don't get it.

Good filter by vegetasaiyajin · 2004-05-31 06:18 · Score: 5, Informative

I am using this filter and after some training it is very effective. Especially useful is the inoculation feature, which you can use to register a spam only address to spam sending sites so that it trains faster.

--

My heart is pure, but make no mistake, it's pure evil

Another one for the arms race... by Anonymous Coward · 2004-05-31 06:19 · Score: 5, Interesting

How much more complex will spam filters have to get to gobble up all the CPU on the mailserver or mail client machine?

I'm all for throwing technology at the problem, but I hope people still realise that having a complex (and effective) spam filter does not take away the millions of megabits of traffic wasted on UCE when it's in transit.

Re:Another one for the arms race... by AntiOrganic · 2004-05-31 06:29 · Score: 4, Insightful

I'm all for throwing technology at the problem, but I hope people still realise that having a complex (and effective) spam filter does not take away the millions of megabits of traffic wasted on UCE when it's in transit.
If people stop receiving spam, and therefore the morons among us stop giving money to spammers by buying their crap, and thus remove all semblances of profits obtained through spamming, there won't really be much incentive to spam anymore, will there?
Re:Another one for the arms race... by WormholeFiend · 2004-05-31 06:33 · Score: 4, Funny

what I really like about this arms race is that news stories about "how bad spam is" are becoming a regular feature in numerous media outlets...

what that means is that the opinion of the silent majority is being moved toward "angry mob" status, which, I believe will lead to the downfall of the Spam Kings.

so if anyone is interested, I'm planning on opening an online store specializing in torches and pitchforks...
Re:Another one for the arms race... by Senior+Frac · 2004-05-31 06:39 · Score: 3, Insightful

If people stop receiving spam, and therefore the morons among us stop giving money to spammers by buying their crap, and thus remove all semblances of profits obtained through spamming, there won't really be much incentive to spam anymore, will there?

Boy, that's a losing battle you propose. The spammer only needs one sucker out of 10 million to stay in business (since he steals his advertising costs). Yet, the defending network must educate all 10 million not to buy from spammers, an impossible task.
Re:Another one for the arms race... by Kwil · 2004-05-31 07:04 · Score: 5, Funny

Hey, you should send out some email about this service, I bet people would love the chance to buy in.

Why, I think I know a place where you can send email to up to 2 million addresses for only...

--
That Jesus Christ guy is getting some terrible lag... it took him 3 days to respawn! -NJ CoolBreeze

But will it? by ajiva · 2004-05-31 06:21 · Score: 4, Funny

But will it find out who sent the SPAM and hurl them into the Sun? Until I get this feature, I don't think it'll be perfect :)

Re:But will it? by dheltzel · 2004-05-31 07:26 · Score: 3, Funny

I want a button on my mail client called "Retaliate", that will hunt down the sender, use various cracking techniques to take over their system, send back a copy of all their personal data, and subtly corrupt any email addresses it finds in any files or databases on the system. Optionally, it would locate some illegal content off the internet and copy it all over the filesystem, then send the IP address and other identifying info to the appropriate government agency.

That would make it fun to get Spam!

Innovation - Statistical Hybrid Filter by ospirata · 2004-05-31 06:22 · Score: 4, Insightful

DSPAM has a strong focus on providing better data to already existing algorithms (Bayesian, Chi-Square, etcetera) Combination algorithms work inherently well, but depend on the quality of data. Some of the approaches deployed in DSPAM towards this goal include Chained Tokens, Inoculation Groups, Classification Groups, advanced de-obfuscation techniques, and a new noise reduction algorithm called Bayesian Noise Reduction. The goal is to incorporate processing algorithms that can withstand the long haul of ever increasing message complexity. So far we're doing a great job.

The idea of combining more than one anti-spam heuristic is not new. But one thing that cant be denied is that all methods are just complementar to Bayesian analysis, that can reach up to 95% precision by itself. Chi-Square, itself, can reach up to 85% precision

Yay, we fixed spam! by Anonymous Coward · 2004-05-31 06:24 · Score: 5, Insightful

Look! We came out with this great filter so nobody else gets spam! This solves the problem of spam once and for all! Even though spam is still clogging our networks and wasting bandwidth, this filter will solve all of our problems.

With all the time spent on making spam filters, why don't we spend that time working out a new protocol for email transfers, one that would not be able to spoofed, or spend that time installing server side programs that put a small time delay between messages as well as bandwidth restrictions for all outgoing mail?

Re:Yay, we fixed spam! by Senior+Frac · 2004-05-31 06:50 · Score: 4, Interesting

With all the time spent on making spam filters, why don't we spend that time working out a new protocol for email transfers, one that would not be able to spoofed,

Because there's nothing wrong with SMTP. SMTP already has extensions to allow authentication but it still requires a central authority to say "He is Senior Frac, we verify it." No one will trust such an authority even if it was scalable enough. If you think spam is caused by a lack of authentication, you're sadly misinformed. The cause is a lack of responsibility by the sending networks to enforce proper behavior of their users.

or spend that time installing server side programs that put a small time delay between messages as well as bandwidth restrictions for all outgoing mail?
These technologies exist. Unfortunately, most that install them stop monitoring them. Such work is considered a resource hog which the ISP would much rather spend on signing up new customers. Bandwidth restrictions on a customer who is running their own MTA makes things much more complex and much less scalable.

Spam is in our culture to stay by Jotaigna · 2004-05-31 06:24 · Score: 4, Insightful

unless mail sending protocol is redesigned(for example,in a way you have to have your fingerprints recognized when you type it) we will have to face the fact SPAM will be in our daily news. Soon slashdot will put an article where the best 3 spam filters are compared, like a normal review.

--
"The quality of life is inversely proportional to the number of keys on your keyring."

Re:Spam is in our culture to stay by linuxbaby · 2004-05-31 07:04 · Score: 4, Informative

Spam filters compared, here. This article was linked from Slashdot a few months ago. Good info, too.

Is it easy to setup? by Trashman · 2004-05-31 06:27 · Score: 4, Insightful

I tried to setup spamassasin a couple of months back and I found it to be too much of a hassle to setup. Could someone who used both spamassasin and dspam comment on easy or difficult it is to setup dspam?

--
Do not read this .sig

Re:Is it easy to setup? by petabyte · 2004-05-31 07:02 · Score: 3, Insightful

Are you piping it to spamassassin or spamc? If you have that much email it might make sense to run the spamd server (which is basically just spamassassin running all the time so you don't have to wait for it to start) and pipe the message to spamc to do its magic (the filter works the same). My advice is if you are really getting that much email, use spamd.

Also it is posible to train spamassassin in evolution fairly easily. All you have to do are change two of the labels in evolution to "Ham" and "Spam". Then write 2 filter rules, 1 that says if its labeled "Ham" pipe it to sa-learn --ham; and another for "Spam" that does sa-learn --spam. Then you just change the label on the email you want to be spam, and apply filters to the message. There's a site on the web that has screenshots to go along with this but I can't find it at the moment.
Re:Is it easy to setup? by humungusfungus · 2004-05-31 07:27 · Score: 3, Informative

Easy is a relative term, but I think it's safe to say that you found spamassasin a hassle, you will not have an easy time with DSPAM.

Like most good server-side software, it requires a moderately good understanding of it's general operation and at least a passing familiarity with its command line arguments and such. Having a handle on how to make your MTA do whatever you want, and the willingness to do some reading of faqs, mailing lists etc doesn't hurt either.

In short, it's does take some mucking around to tweak it all out properly. Also of note, if you intend to use the cgi pictured in the screenshots, you should know something about setting up a webserver with proper exec priviledges for cgi.

If you're thinking about using it only for yourself, I would recommend a cleint side solution like Mozzila Thunderbird or Eudora (win32 only) instead. They both have bayesian spam filtering built in and they're *really* easy to set up.

--
No sig.

It takes a lot to train by smartin · 2004-05-31 06:33 · Score: 4, Informative

Warning, it seems to be designed more for high volume use than individual sites. I've fed dspam almost 3000 spams and it is still only catching 80%, does seem to be getting better though.

--
The difference between Canada and the USA is that in Canada healthcare is a right and gun ownership is a privilege.

Re:client-side versus server-side anti-spam by ubiquitin · 2004-05-31 06:34 · Score: 5, Interesting

When you run your own mail server, or administrate a mail server for a large number of people, server-side anti-spam filters and countermeasures start making a lot more sense. Do the math on a company with 100 employees (at $25/hr) who check mail twice a day and spend 5 minutes each time hassling with anti-spam measures in client-side mail apps. In this scenario, a seamless anti-spam solution is worth conservatively $400 per day, or $100k/year not counting bandwidth savings. There are definitely cases when client-side filtering makes sense, but if you can handle it at the server, email-based business methods scale better.

--
http://tinyurl.com/4ny52

Content based filters and spam by Senior+Frac · 2004-05-31 06:35 · Score: 3, Interesting

I have not actually used DSPAM, but have just read the specs.

Yawn. Yet another, albeit well designed, content-based filter. While content-based filters are a valuable tool, let's not forget that the spam problem is one of anti-social behavior and consent and has nothing to do with content. Using content as a factor in deciding what is spam or not spam will always be flawed. Even if you tweak your favorite filter from 99% to 99.9%, the spammers can just up the ante by sending more. Scaling up costs them little on an individual basis. It saddens me to see really brilliant people put great amounts of work into a project whose underlying premise is flawed.

Re:Content based filters and spam by Matt2k · 2004-05-31 07:13 · Score: 3, Insightful

Just because it is not perfect does not mean it is flawed or undeserving of effort. 99.9% accurate is 99.9% better than 0%

a True AntiSpam measure by CodeTRap · 2004-05-31 06:49 · Score: 4, Insightful

would be to publicly humiliate/boycott the companies that use the spammers services. Like drug dealers, as long as there is a market, the spammers will be around. Remove the demand, and the suppliers will eventually move onto selling something else.

If you can't kill the leeches because the water is too murky, then boil off the pond!

--
CodeTrap (www.codetrap.net)

So how does this help me reduce the ... by Skapare · 2004-05-31 06:52 · Score: 3, Insightful

So how does this help me reduce the amount of bandwidth and server resources used by spammers who continue to try sending spam to me and my users?

--
now we need to go OSS in diesel cars

Does it still mess up mail contents? by lintux · 2004-05-31 07:05 · Score: 3, Interesting

I wanted to try DSPAM some time ago, but I stopped as soon as I read that DSPAM puts an ID string in every mail it processes. In the mail body, that is. I have no problems with a program that adds headers, but it should leave the message body alone.

Does DSPAM do that now? Can't find anything about it...

Re:Compared to other OSS projects by Anonymous Coward · 2004-05-31 07:06 · Score: 5, Informative

As far as I know, the main difference is DSPAM does not use weighted filter rules at all like SpamAssassin's hybrid approach does - DSPAM is designed to purely rely on analysis of spam's properties (Bayesian, etc).

The other cool thing about DPAM is that it is designed to let users add/modify their own spam database - every email DPAM processes is tagged with an identifier, and is logged in a server-side database. If a delivered email is in fact spam but wasn't tagged as such, the user can then forward the email to the designated spam-sorting address, and DSPAM will automatically update that user's spam corpus (eg, because it's tagged with an identifier, you don't have to worry about the user forwarding the full headers, as the server already has that info on file).

AFAIK you can't do that with SpamAssassin.

Unless you have false positives and such. by khasim · 2004-05-31 07:13 · Score: 3, Interesting

I'm the one running the spam filter (SpamAssassin) at work. Overall, it has been VERY popular with everyone else. They don't receive the most obnoxious sex spams any more.

On the other hand, there are a few false positives that reduce the overall savings in your post. I auto-delete anything about 10 and flag anything above 5.

But the end users still have to look through the flagged stuff to see if there are any false positives. Then they drop them into the false positive folder. The users also have to identify all the missed spam and drop that into the spam folder.

It's still work for them so the costs aren't as clear as in your post. But the non-tangible benefits are also important.

I think we're at the point of dimishing returns on simple scanning processes. I think we need to look at actively seeding the spammer's lists with false names and tuning the spam filters with those.

Re:Mozilla/Firebird work well for me ... by iconian · 2004-05-31 07:16 · Score: 3, Interesting

I find that the spam letters that do get through T-Bird's junk mail filter are the ones padded with random strings of letters. My guess is that T-Bird is able to identify the spam words (eg: debt consolidation, enlargement) but the mispelled words (eg: peni5) are unknown to T-bird. So T-Bird makes the conservative decision not to mark the e-mail as spam. I figure a simple filter criteria that requires the correct spellings for at least half the words in the body (for unknown senders) should get rid of this problem. Anyone care to enlighten me if such a rule is in T-bird or is in the works? At the very least, this will have the side effect of encouraging people to at least spellcheck their e-mails before sending. :)

DSPAM ID by XanC · 2004-05-31 07:20 · Score: 4, Informative

DSPAM uses the ID string because people send corrections by forwarding mail to a certain address. Other filters require you to move mail to a Spam folder, but that requires a fairly specific configuration (you must use IMAP, filtering mail gateways are difficult, etc).

You can configure DSPAM to not use the ID, but this requires users to "bounce" the incorrect e-mails instead of forwarding them (as forwarding strips the headers).

Is the ID really that inconvenient?

Obligatory by jonfelder · 2004-05-31 08:16 · Score: 3, Insightful

Your post advocates a (*) technical ( ) legislative ( ) market-based ( ) vigilante ( ) lack of an approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.) ( ) Spammers can easily use it to harvest email addresses (*) Mailing lists and other legitimate email uses would be affected ( ) No one will be able to find the guy or collect the money ( ) It is defenseless against brute force attacks ( ) It will stop spam for two weeks and then we'll be stuck with it (*) Users of email will not put up with it (*) Microsoft will not put up with it ( ) The police will not put up with it ( ) Requires too much cooperation from spammers (*) Requires immediate total cooperation from everybody at once (*) Many email users cannot afford to lose business or alienate potential employers ( ) Spammers don't care about invalid addresses in their lists ( ) Anyone could anonymously destroy anyone else's career or business Specifically, your plan fails to account for ( ) Laws expressly prohibiting it (*) Lack of centrally controlling authority for email ( ) Open relays in foreign countries ( ) Ease of searching tiny alphanumeric address space of all email addresses ( ) Asshats ( ) Jurisdictional problems ( ) Unpopularity of weird new taxes ( ) Public reluctance to accept weird new forms of money (*) Huge existing software investment in SMTP (*) Susceptibility of protocols other than SMTP to attack ( ) Willingness of users to install OS patches received by email ( ) Armies of worm riddled broadband-connected Windows boxes ( ) Eternal arms race involved in all filtering approaches ( ) Extreme profitability of spam ( ) Joe jobs and/or identity theft ( ) Technically illiterate politicians ( ) Extreme stupidity on the part of people who do business with spammers ( ) Dishonesty on the part of spammers themselves ( ) Bandwidth costs that are unaffected by client filtering ( ) Outlook and the following philosophical objections may also apply: (*) Ideas similar to yours are easy to come up with, yet none have ever been shown practical ( ) Any scheme based on opt-out is unacceptable ( ) SMTP headers should not be the subject of legislation ( ) Blacklists suck ( ) Whitelists suck ( ) No-lists suck ( ) We should be able to talk about Viagra without being censored ( ) Countermeasures should not involve wire fraud or credit card fraud ( ) Countermeasures should not involve sabotage of public networks (*) Countermeasures must work if phased in gradually ( ) Sending email should be free (*) Why should we have to trust you and your servers? ( ) Incompatiblity with open source or open source licenses ( ) Feel-good measures do nothing to solve the problem ( ) Temporary/one-time email addresses are cumbersome ( ) I don't want the government reading my email ( ) Killing them that way is not slow and painful enough Furthermore, this is what I think about you: (*) Sorry dude, but I don't think it would work. ( ) This is a stupid idea, and you're a stupid person for suggesting it. ( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!

Problems with DSPAM by gonz · 2004-05-31 08:28 · Score: 4, Informative

I've been using DSPAM for about three months. A few criticisms:

First, by default DSPAM wants to run as the "root" user and usurp delivery of e-mails. (With Exim, they actually want it to recursively reinvoke the mail server for actual delivery!) It took quite a bit of configuring to get it to work like SpamAssassin from procmail.

This software is somewhat buggy, so running DSPAM as root would also introduce security concerns. For example, I'm using 2.10.6 because the 3.0.0 compiled and installed with no problems, but failed to classify anything. (Even with several hours of gdb tracing I was unable to determine why). Another bug is that if I run the "--falsepositive" on an e-mail that's lacking the "!DSPAM" signatures, the message should be ignored, but apparently this is not the case because the statistics counters are incremented.

From the FAQ:
"Q. Does DSPAM support whitelists?
A. DSPAM doesn't have a whitelist manager, rather whitelisting is an automatic function of DSPAM's Bayesian filtering mechanism."

This is crazy -- the whole point of whitelists is for when the Bayesian filtering fails! And DSPAM does fail. Twice now I've had to reset my database because the classifications were wrong and training wasn't helping. All I can say is I'm glad I've got procmail to rescue the important e-mails.

I think one source of my problems was that the default training mode ("train on everything") causes incorrect learning when you fail to report a false positive. This was a big problem for me, since I get around 700-800 spams/day. While false negatives are easily caught, the false positives go unnoticed unless I happen to wonder why someone never responded, and invest some time to search my spam folders. (I'm still trying to figure out exactly how to deal with this problem. E.g. maybe I could have it challenge the sender with Turing Test or something.)

I will say that DSPAM's basic technology is quite good. It's just that the software still has a "prototype" feel, and I'd caution you to do some experiments before unleashing it on your users. (For example, there's no manpage, and there isn't even a command-line option to print out the current version number!)

-Gonz

Obligatory by jonfelder · 2004-05-31 08:37 · Score: 4, Funny

Your post advocates a ( ) technical ( ) legislative ( ) market-based (*) vigilante ( ) lack of an approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.) ( ) Spammers can easily use it to harvest email addresses ( ) Mailing lists and other legitimate email uses would be affected (*) No one will be able to find the guy or collect the money ( ) It is defenseless against brute force attacks ( ) It will stop spam for two weeks and then we'll be stuck with it ( ) Users of email will not put up with it ( ) Microsoft will not put up with it (*) The police will not put up with it (*) Requires too much cooperation from spammers (*) Requires immediate total cooperation from everybody at once ( ) Many email users cannot afford to lose business or alienate potential employers ( ) Spammers don't care about invalid addresses in their lists ( ) Anyone could anonymously destroy anyone else's career or business Specifically, your plan fails to account for (*) Laws expressly prohibiting it ( ) Lack of centrally controlling authority for email (*) Open relays in foreign countries ( ) Ease of searching tiny alphanumeric address space of all email addresses (*) Asshats (*) Jurisdictional problems ( ) Unpopularity of weird new taxes ( ) Public reluctance to accept weird new forms of money ( ) Huge existing software investment in SMTP ( ) Susceptibility of protocols other than SMTP to attack ( ) Willingness of users to install OS patches received by email ( ) Armies of worm riddled broadband-connected Windows boxes (*) Eternal arms race involved in all filtering approaches (*) Extreme profitability of spam (*) Joe jobs and/or identity theft ( ) Technically illiterate politicians (*) Extreme stupidity on the part of people who do business with spammers (*) Dishonesty on the part of spammers themselves ( ) Bandwidth costs that are unaffected by client filtering ( ) Outlook and the following philosophical objections may also apply: (*) Ideas similar to yours are easy to come up with, yet none have ever been shown practical ( ) Any scheme based on opt-out is unacceptable ( ) SMTP headers should not be the subject of legislation ( ) Blacklists suck ( ) Whitelists suck ( ) No-lists suck ( ) We should be able to talk about Viagra without being censored ( ) Countermeasures should not involve wire fraud or credit card fraud (*) Countermeasures should not involve sabotage of public networks (*) Countermeasures must work if phased in gradually ( ) Sending email should be free (*) Why should we have to trust you and your servers? ( ) Incompatiblity with open source or open source licenses ( ) Feel-good measures do nothing to solve the problem ( ) Temporary/one-time email addresses are cumbersome ( ) I don't want the government reading my email ( ) Killing them that way is not slow and painful enough Furthermore, this is what I think about you: (*) Sorry dude, but I don't think it would work. ( ) This is a stupid idea, and you're a stupid person for suggesting it. ( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!

Slashdot Mirror

DSPAM v3.0 RC1 Spam Filter Released

31 of 182 comments (clear)