DSPAM v3.0 RC1 Spam Filter Released

Nuclear Elephant writes "DSPAM v3.0 RC1 is now available for download, with a stable release scheduled for June 13. DSPAM has appeared on Slashdot and in Wired News in the past for its high levels of accurate spam filtering. v3.0 is the product of three solid months of work. Some of the highlights include a very sleek redesigned interface, PostgreSQL support, many mathematical enhancements, and support for many of Gary Robinson's algorithms (such as Chi-Square, Geometric Mean Test, and Robinson's technique for combining P-Values)."

Comments

How is this a YRO?

[magefile]

I don't get it.

Re:How is this a YRO?

[Trashman]

maybe because It's your right (in the US at least) to not get spam? look at the recent legislation that's been passed.

Innovation - Statistical Hybrid Filter

[ospirata]

DSPAM has a strong focus on providing better data to already existing algorithms (Bayesian, Chi-Square, etcetera) Combination algorithms work inherently well, but depend on the quality of data. Some of the approaches deployed in DSPAM towards this goal include Chained Tokens, Inoculation Groups, Classification Groups, advanced de-obfuscation techniques, and a new noise reduction algorithm called Bayesian Noise Reduction. The goal is to incorporate processing algorithms that can withstand the long haul of ever increasing message complexity. So far we're doing a great job.

The idea of combining more than one anti-spam heuristic is not new. But one thing that cant be denied is that all methods are just complementar to Bayesian analysis, that can reach up to 95% precision by itself. Chi-Square, itself, can reach up to 85% precision

Yay, we fixed spam!

[Anonymous Coward]

Look! We came out with this great filter so nobody else gets spam! This solves the problem of spam once and for all! Even though spam is still clogging our networks and wasting bandwidth, this filter will solve all of our problems.

With all the time spent on making spam filters, why don't we spend that time working out a new protocol for email transfers, one that would not be able to spoofed, or spend that time installing server side programs that put a small time delay between messages as well as bandwidth restrictions for all outgoing mail?

Re:Yay, we fixed spam!

[AbbyNormal]

Exactly! This is merely more duct tape to solve a leaking pipe problem. Look what happened with telemarketers! It became an epidemic, people got ticked off, and the FCC created the "Do not call list". Sure a few evil bastards found ways around it, but nevertheless, the amount of calls went down drastically. The same principles that kept telemarketers in business are the same that keep SPAM in business. There are a few morons that actually buy stuff from spam. I believe legislation is actually needed. Before everyone jumps out of their seat with statements like: "governments don't have the time/energy to go after every spammer", I say you are right.

But what about the technically minded crowd? If a solution was delivered that could help speed up the nailing (or shutting down of zombie machines), and it was as simple as a "Spamcop" post, why wouldn't it work? Trace it back to the companies who peddle their wares in the SPAM messages and hammer them out.

Technical solutions will never stamp out SPAM. Its similiar to the plight that the RIAA is facing. Come up with a "Secure" method of safeguarding media, and a few hours later it is broken. The arugments differ in that most people just want to listen to their own media and are not out to make a buck.

Spam is in our culture to stay

[Jotaigna]

unless mail sending protocol is redesigned(for example,in a way you have to have your fingerprints recognized when you type it) we will have to face the fact SPAM will be in our daily news. Soon slashdot will put an article where the best 3 spam filters are compared, like a normal review.

Is it easy to setup?

[Trashman]

I tried to setup spamassasin a couple of months back and I found it to be too much of a hassle to setup. Could someone who used both spamassasin and dspam comment on easy or difficult it is to setup dspam?

Re:Is it easy to setup?

[petabyte]

Are you piping it to spamassassin or spamc? If you have that much email it might make sense to run the spamd server (which is basically just spamassassin running all the time so you don't have to wait for it to start) and pipe the message to spamc to do its magic (the filter works the same). My advice is if you are really getting that much email, use spamd.

Also it is posible to train spamassassin in evolution fairly easily. All you have to do are change two of the labels in evolution to "Ham" and "Spam". Then write 2 filter rules, 1 that says if its labeled "Ham" pipe it to sa-learn --ham; and another for "Spam" that does sa-learn --spam. Then you just change the label on the email you want to be spam, and apply filters to the message. There's a site on the web that has screenshots to go along with this but I can't find it at the moment.

Re:Another one for the arms race...

[AntiOrganic]

I'm all for throwing technology at the problem, but I hope people still realise that having a complex (and effective) spam filter does not take away the millions of megabits of traffic wasted on UCE when it's in transit.

If people stop receiving spam, and therefore the morons among us stop giving money to spammers by buying their crap, and thus remove all semblances of profits obtained through spamming, there won't really be much incentive to spam anymore, will there?

Re:Another one for the arms race...

[Hannes Eriksson]

How much more complex will spam filters have to get to gobble up all the CPU on the mailserver or mail client machine?

It already is. At 500+ users and 200 pieces of junk mail a day, that is already more mail than there are seconds in the same period. Would you think the new spam filters use less than 1 cpu second per mail? I hope you have a bad-ass mainframe for your companys spam filtering...

Re:Another one for the arms race...

[Senior Frac]

If people stop receiving spam, and therefore the morons among us stop giving money to spammers by buying their crap, and thus remove all semblances of profits obtained through spamming, there won't really be much incentive to spam anymore, will there?

Boy, that's a losing battle you propose. The spammer only needs one sucker out of 10 million to stay in business (since he steals his advertising costs). Yet, the defending network must educate all 10 million not to buy from spammers, an impossible task.

Re:Another one for the arms race...

[Karamchand]

Problem is many organisations aren't allowed not to deliver email which is probably spam. So they have to deliver it, probably only tagged (with an extra X-Header or some phrase added to the subject). I.e. people still receive spam they just have the possibility to trash it easier - which stupid people, i.e. people buying things advertised in spam emails, won't do anyway.
Sucks, eh? :-/

Re:Mozilla/Firebird work well for me ...

[Senior Frac]

Wonderful, if you just want to stop seeing the spam. I, however, would enjoy not having to pay for it's delivery. This is the ostrich method of spam fighting.

a True AntiSpam measure

[CodeTRap]

would be to publicly humiliate/boycott the companies that use the spammers services. Like drug dealers, as long as there is a market, the spammers will be around. Remove the demand, and the suppliers will eventually move onto selling something else.

If you can't kill the leeches because the water is too murky, then boil off the pond!

Re:a True AntiSpam measure

[keraneuology]

Nothing will eliminate all spam. Period.

That said, there are certain courses of action that would be quite effective against certain types of spam. For example, consider the dozen or so mortgage applications that arrive each day with specific promises along the lines of "$200,000 mortgage for $350 regardless of credit". If I were to reply some loan officer somewhere would presumably call me back. If said officer were required by law to give me $200,000 at $350/month you can believe that word would quickly get around that this particular lead source was no good and that particular individual would be forced to change tactics.

To step up the pressure a bit, a law that revoked the license of any lender who purchased leads from any company that did not include a specific phrase - "this is an unsolicited attempt to procure a loan application" for example - would mean that lenders either use bulk emailers who provide a clear and consistent way to identify (and /dev/nul) to generate leads or lose their entire income. Ditto goes for insurance underwriters.

For all spams that are not outright frauds there is a stationary target ultimately providing the goods/services. They are not hard to identify, nor is it difficult to regulate them at the federal level (Article I, Section 8). For most insurance and lending organizations they are already subject to a myriad of of regulations. It won't matter that the spam is sent from zombie blind dates in China who have pockets full of herbs to give you many big large p.e.n....is pr0n, the underwriters who use companies that engage in fraudulent advertisements are easy to find, and have nice, deep pockets to pay all kinds of fines.

So how does this help me reduce the ...

[Skapare]

So how does this help me reduce the amount of bandwidth and server resources used by spammers who continue to try sending spam to me and my users?

Re:So how does this help me reduce the ...

[Anonymous Coward]

Use DSPAM's source address reporting tool to blackhole spammer IPs.

Re:Great!

[GigsVT]

Be careful what you wish for.

My mail hosting used to out and out block spam, and their filter wasn't very well maintained so it blocked lots of legitimate mailing list mail (like Securityfocus and NANOG).

They've went to tagging mail now instead of dropping it, which is a lot better.

ISP/mail server based blocking isn't really a good idea, even with ultra-conservative blocking, you'll still block legitimate emails.

Re:Content based filters and spam

[Matt2k]

Just because it is not perfect does not mean it is flawed or undeserving of effort. 99.9% accurate is 99.9% better than 0%

Obligatory

[jonfelder]

Your post advocates a

(*) technical ( ) legislative ( ) market-based ( ) vigilante ( ) lack of an

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
(*) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
(*) Users of email will not put up with it
(*) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
(*) Requires immediate total cooperation from everybody at once
(*) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
(*) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
(*) Huge existing software investment in SMTP
(*) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

(*) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) No-lists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
(*) Countermeasures must work if phased in gradually
( ) Sending email should be free
(*) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(*) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!

Obligatory

[jonfelder]

Your post advocates a

( ) technical (*) legislative ( ) market-based ( ) vigilante ( ) lack of an

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
(*) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
(*) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
(*) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
(*) Asshats
(*) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
(*) Extreme profitability of spam
( ) Joe jobs and/or identity theft
(*) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
(*) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

(*) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
(*) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) No-lists suck
(*) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
(*) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
(*) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(*) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!