Slashdot Mirror

Ive been playing with DSPAM which seems very good. They claim a 99.991% accuracy. Apparently this is 10 times more accurate then a human. But Ive heard that most anti-spam solutions are very good.

(And just for the record, I've had 13 false negatives and zero false positives in that same interval using dspam)

The Motorola v710: Verizon's New Crippled Phone
http://www.nuclearelephant.com/papers/v710.html

Just got the v710 with Verizon Wireless and am horribly disappointed after I got over the novelty of having the phone connect to my car. The "obeymoto" voice command is the only good thing about this phone. Otherwise it's heavy, expensive, the camera is abysmal, the UI is awkward, and the bluetooth is crippled. I don't want all those extras (SMS, MMS, games, songs, movies, ringtones, etc.), just want a good phone to talk on!! The main reason I bought the v710 was to use hands-free in my car without a cumbersome headset, but that doesn't work very well. Already I've missed some incoming calls since neither the car nor the phone announces incoming calls when they're connected via bluetooth. The lack of bluetooth obex implementation to transfer the phone book is a deal breaker. Does Verizon really expect us to spend the rest of our lives manually reentering our phone books everytime we buy a new phone?? Without obex, "bluetooth handsfree" is no different than plugging a headset directly into the phone. Wait, it's worse since you can't use obeymoto voice command over bluetooth, and the phone doesn't ring for incoming calls when bluetooth is active. Too bad I can't plug the car's microphone and speaker "handset" directly into the phone. After reading Jonathan's interview with Verizon's Brenda Raney, I'd be shocked if Verizon ever enables obex on the v710. It's easy for Verizon's sales and service reps to promise upgrades and fixes so that people don't cancel their contracts within the 15 day grace period. But after that interview we all know how Verizon does their math. The Verizon v710 is for suckers. I'm cancelling.

If you didn't catch this from the nuclear elephant article, he's got a reward pot going for anyone who can provide a hack to enable OBEX on the phone. I think this is a great idea... I would love to see Verizon lose control of this thing. I almost bought one of these things just to be able to sync my address book with bluetooth, and at the last minute my intuition (or experience with Verizon/Moto) saved me.

Your definition is a good one. But it still doesn't make this product the first - or revolutionary. Sendmail created the 'milter' interface many years ago precisely to make this kind of rejection of unwanted mail possible. There are many sendmail milters written in many languages. The most popular being C, Perl, Python in that order. I run a Python milter which removes Windows executables (except DOC and XLS), checks SPF, and checks content with DSPAM wrapped for Python. Of the 40000 spams a day we get, nearly all are rejected before SMTP DATA. Those flunking content check are rejected before the connection closes - except when addressed to a 'screener', in which case it goes to a spam mailbox. Screeners have the task of providing feedback to the Bayesian filter - relieving others in the company of the burden.

It's my belief that the most likely source of the birth of Artificial Intelligence will be the SPAM filter.

Think about it - we now have software that "learns' what you like.

Sorry, but anything that "learns" fits a definition of intelligence - using past results to predict future outcomes. Note that I'm not saying "self aware" or "conscious", simply "intelligence".

As we move forward, we'll see more and more intelligence on the part of the spammers, and the warring factions of intelligence will likely provide massive financial and political impetus to build ever more intelligence solutions - thus AI is born.

The problem with other vehicles for developing AI is simply the budget. With SPAM, everybody has a direct, financial incentive to develop it, so development will definitely happen!

What's ironic is the RIAA and MPAA are making more money with filesharing running free than without it. There have been many articles already posted on slashdot pointing out this overlooked fact. There is an unscientific poll going on which gives some insight.

http://www.nuclearelephant.com/projects/sharing/

Check it out. I've personally purchased a few thousand dollars in DVD movies (read box sets) that I wouldn't have without P2P. Basically it's given me the ability to see if something sucks and should avoid it (or purchase it).

Speaking of, can't someone set up a poll or tally of the amount of people who use piracy as "try-before-buy"?

Wish granted.

Abstract: I've often been asked why an intellectual type guy such as myself would believe in Jesus - a figure most Americans equate to a good bedtime story, or a religious symbol for people who need that sort of thing. Read essay

How exactly is this news ? It seems that the author of the neural network idea didn't do his homework - e.g. DSPAM includes neural network as an experimental classifier already. And compared to the proposed C# solution, DSPAM is a widely used and mature product already.

Regards, Jan

Well, take it with a huge grain of salt:

He says: I am a Born-Again Spirit-Filled Heterosexual Serious-About-God Christian (TM)

The document linked from the parent really tells a lot. He says:

Jesus Elicits a Reaction

When people swear, do you ever hear the words "Buddha Damn" come out of their mouth? Or how about "Allah Christ"? Instead, all of the profanities we hear which involve deity revolve around Jesus Christ.

...this guy seriously believes the earth is a scant 10000 years old. And he dismisses all evidence to the contrary without a throuogh explanation. I can't help but wonder if he treat's other people's research with the same disregard.

He appears to be a bible-thumper.

One of my favourite quotes from this article is "there's more evidence to support the existence of Jesus than there is Julis Caesar". !!

Personally I find "Nuclear Elephant"s writing ridiculous. Read this article about how terrorists are going to use data centers for their next attack.

Here's a response from the DSPAM author.

:w

And here is a very long and detailed response on the DSpam site by Jonathan A. Zdziarski himself..

In an effort to ensure that we had not misinterpreted the installation instructions, we ran CRM-114 in a train-on-everything configuration and, as predicted by the author, the result was substantially worse.

As with CRM114, we trained DSPAM only on misclassifications, as suggested in the documentation.

CRM-114 and DSPAM exhibit substantially inferior performance to the other filters, regardless of threshold setting. Both exhibit substantial learning through outthe email stream, leading us to conjecture that their performance might asymptotically approach that of the other filters. From a practical standpoint, this learning rate would be too slow for personal email filtering as it would take several years atthe observed rate to achieve the same misclassification rates as the other systems.

This is interesting considering the harsh words the DSPAM author directs towards SpamAssassin in the DSPAM FAQ. In contrast, I think, the SpamAssassin developers say they are interested in testing the "dobly" noise reduction technique that DSPAM employs, see SpamAssassin bug 3078.

It's unforchunately that DSPAM was left out of this very good quality report. I have personally used SpamAssassin, SpamProbe and DSPAM

After using each for a couple months at a time, I found DSPAM to be by far the most effective (after it was properly trained)

DSPAMS claim "DSPAM (as in De-Spam) is an extremely scalable, open-source statistical hybrid anti-spam filter. While most commercial solutions only provide a mere 95% accuracy (1 error in 20), a majority of DSPAM users frequently see between 99.95% (1 error in 2000) all the way up to 99.991% (2 errors in 22,786). DSPAM is currently effective as both a server-side agent for UNIX email servers and a developer's library for mail clients, other anti-spam tools, and similar projects requiring drop-in spam filtering. DSPAM has been implemented on many large and small scale systems with the largest systems being reported at about 125,000 mailboxes." was quite accurate for me


Also check out some priceless photos Priceless Photos

It works really great and Spam is not an issue for me anymore (I had more than 50 Spammails in my inbox AFTER mozilla-thunderbird filtering...)

This looks to me like Windows '95, which is really Apple '86. Sorry, I don't mean to flame, but honestly - why would anyone want to pay for that when you can make your desktop look so much better with free, GPL'd products?

SpamAssassin is a good start. If you're really wanting to reduce false positives, consider bringing
dspam into the mix. "DSPAM presently peaks at 99.985% accuracy, which is ten times more accurate than a human being and is presently being used on implementations as large as 125,000+ mailboxes." bogofilter is another advanced project in the same functional space.

Otherwise your weights will be all wrong.

Equal parts ham and spam will yield good spam catching. RTFAQ.

Since when did Verisign ever care about privacy? Isn't this the parent company that OK'd the sale of millions of email addresses from whois records? I'm not sure if Verisign is in my corner.

A landmine system would be relatively easy to implement - you set up a few hundred landmines and block any customer IP who sends a spam to a landmine. It's similar to honeypots, although you treat the accounts like mines where even a single email will get an address temporarily blacklisted. Once blacklisted, you can shut off port 25 for that IP, disconnect their session for 30 minutes, or do whatever you want. The Streamlined Blackhole List server could be used to create a landmine database with a spread of 1 to instantly identify new hosts.

SpamAssassin's pretty heavyweight; a purer statistics based system like dspam is probably more suitable for large scale systems like this; you don't want a perl script chugging over every single email for seconds at a time. I wouldn't be suprised if they needed 20 mail servers if they were using SA...

We all remember the Twilight Zone episodes about the Earth being dominated by cold, unthinking, uncaring machines that take over and establish an Orwellian rule, or the Terminator who speaks of "Skynet" becoming self-aware.

Yet this is not how machine intelligence has been evolving. Think Google, think Visa's fraud detection system, think the learning DSPAM which simulates learning so effectively.

Machine intelligence will not replace human intelligence - like the cerebrum augmenting the "lizard brain" at the base of the brain, our extrasomatic information technologies and machinery augment, enhance, and extend the capabilities of our cerebrum.

What's interesting is that the evolution of machine intelligence has hit a point where it surpasses, or will soon surpass, the thinking performed in our collective cerebrums.

We are becoming the Borg of Star Trek tales, armed with our cell phones, Internet access terminals, (I write this from a Hotel 2 blocks from Disneyland, in Anaheim, CA with my Linux laptop - where are you?) telephones, pagers, walkie-talkies, and other inter-personal communication devices.

And I find no end of fascination in watching the birth of this machine intelligence, and watching it become so tightly integrated with our humanity.

I can now perform in minutes research that just a few decades ago would have taken hours or days. This is the result of machine intelligence, and it will get far better, not worse.

Sit back and enjoy the ride - it's going to be a fun one!

You want to implement both. SPF detects envelope forgeries before you have wasted much bandwidth. You can then use right hand side blacklists on sender domains. Yes, spammers too are adopting SPF. This is OK - those who like spam have something other than instinct to warn them when they are dealing with a scammer instead of a spammer. Those who hate spam can ignore it more efficiently.

Domain Keys validates the message headers. It protects you against forgeries by users in the same domain - e.g. a spammer on yahoo forging an innocent party on yahoo. SPF can also detect envelope sender forgeries from the same domain in conjuction with SES (Signed Envelope Sender) - which adds a crypto cookie to the local part.

You should implement SPF first. It is simpler, and eliminates most forgeries before SMTP DATA. SPF requires sepcial consideration for forwarders (SRS - Sender Rewriting Scheme) or whitelisting.

DK is a good addon for large ISP domains like yahoo and aol, but is broken by forwarders or mail processing tools that modify the body. For instance, my DSPAM bayesian filter adds "tags" to messages.

Dobly DSPAM

You can bet if any of the companies have ties to Diebold or Symantec, that this will be a pro-censorship lobby, trying to pass laws similar to the DMCA to outlaw private citizens from having access to things like exploit code or diagnostic tools.

I'm glad these guys are doing great things to combat spam, but when the submitter of the article stands to benefit from posting of the article on Slashdot, then full disclosure (not stealth disclosure) is warranted. No surprise that the "donate" link is right up at the top of their page.

Jonathan, don't get me wrong. I really appreciate what you're doing here. But failure to disclose your relationship of the project you're promoting is on the level (though not the same extent) as the deception that spammers employ.

-S

I'm glad these guys are doing great things to combat spam, but when the submitter of the article stands to benefit from posting of the article on Slashdot, then full disclosure (not stealth disclosure) is warranted. No surprise that the "donate" link is right up at the top of their page.

Jonathan, don't get me wrong. I really appreciate what you're doing here. But failure to disclose your relationship of the project you're promoting is on the level (though not the same extent) as the deception that spammers employ.

-S

DSPAM implements a Dolby-type noise reduction algorithm called Dobly

Despite the musical reference on the DSPAM site, I figured some people still won't get the joke. So here it is:

JEANINE: You don't- you don't do heavy metal in dobly, you know, I mean...it's--
NIGEL: In what??? In what???
JEANINE: In dobly...
NIGEL (GRINS): In doubly!?! What's that?
DAVID: She means Dolby, alright? She means Dolby, you know? You know perfectly well what she means.

--from the movie "Spinal Tap"

Possibly too late in the thread to be read, but things I would like to see to make it an IE killer:
* SVG support
* Privoxy available as a plug-in

For Thunderbird:
* spam-assasin and dspam available as plug-in options

Phillip.

I use Dspam, which puts adjacent pairs into the dictionary as well. The random English words don't phase it. However, the spammers could start using a Markov generator to defeat it.

Damn straight.

Although, one thing needs to stay clear: Linux is only secure if you know what the hell you're doing. 51% of all known successful root compromises occur under Linux. (Linux has more than 51% of the market share, IIRC, so it's not a very fair comparison. If anybody has market share data, please provide it so we can look at ratios.)

I prefer running Linux, of course. At least I know I can secure it.

> 1) false positive rate

Most filters are down to below 0.03. DSPAM is down to 0.01% and lower with some of our users.

> 2) false negative rate

That's the accuracy I was referring to; 99.9% catch rate.

On the server:

rblsmtpd (DNS-based block lists) in front of qmail

DSPAM filtering pre-delivery

SpamCop for the ones who make it through.

I'm planning to add SpamCop reporting for the messages that DSPAM catches and there is also ongoing development in the project that will log IP addresses of machines delivering SPAM for local RBL use.

Speak for yourself. I haven't gotten a spam in months, although my quarantine box has caught thousands. My kids aren't going to know what spam is because they'll never see one.

Is this really necessary to post a different article every day of someone in our field who agrees spam is bad? It's like there's a template for every article on slashdot about spam:

CEO of [NAME] reported today that SPAM is [GOOD|BAD] and recommends [LEGISLATION|CRACKING DOWN|PRODUCT].

There are enough freely available working solutions out there now that work with Mr. Allman's product (such as DSPAM and BogoFilter) where we really have gotten to the point where we can quit complaining about spam and actually succeed in the high 99% at stopping it. If everyone quit whining and installed one of these tools, nobody would get spam, and the spammers would be out of business.

That's because you're using the wrong anti-spam tool. You need an advanced self-teaching anti-spam tool like DSPAM to keep up with the spammers.

Ximian evolution (the client I use) gives you this option too - they took it a little further and let me disable it only for users not in my addressbook. On the other hand, I haven't gotten a spam in months to really test it with (avid DSPAM user)

Give it a week for advanced Bayesian statistical filters (like DSPAM) to learn the new patterns and then you'll never see them again. Honestly, I don't know why people are still complaining about spam - our system has caught 99.9% of all my spams (about 50-60 per day for me, 150 per day for my wife). Why waste cycles when there are free solutions that work.