Slashdot Mirror
Two Spam Filters 10 Times As Accurate As Humans
Nuclear Elephant writes "The authors of two spam filters, CRM114 and DSPAM, announced recently
that their filters have achieved accuracy rates ten times better than a human is capable of. Based on a study by Bill Yerazunis of CRM114, the average human is only 99.84% accurate. Both filters are reporting to have reached accuracy levels between 99.983% and 99.984% (1 misclassification in 6250 messages) using completely different approaches (CRM114 touts Markovan, while DSPAM implements a Dolby-type noise reduction algorithm called Dobly). If you're looking for a way to rid spam from your inbox, roll on over to one of these authors' websites."
Once Email Spam is eliminated, then IM spam will begin...
Jeff
can this be used with Spamassasin, or is a stand alone program? Does it need something like Amasis to run?
CB
free ipod and free gmail!
Well, it certainly sounds better than the pay-per-email "postage" idea. If postage hasn't stopped snail spam, why would it stop e-mail spam?
I reached the conclusion of "two filters better than humans" by using two sequential filters:
server side spamassassin, and a couple of simple procmail recipes. They have kept almost all the SPAM away.
However, it is good to see such good techniques becoming available and we can hope to see them as straight forward usable tools.
So, when will mozilla/TB (or your favourite server side or client side filter) get them?
S
No, humans are not 100%.
If you see a strange name in your inbox with an odd title, that might be a Nigerian businessman, or it might be your long lost Nigerian brother.
I recently tried to order a t-shirt from this guy for a band he used to be in. I found his band because we have the same (semi-uncommon) name. So, he got an email From: himself. I had to send him two emails because he deleted the first one assuming it was spam.
I ordered some RAM for my dad a while back. He gets 200 spam emails a day (email addy in resume & web page), and he deleted the confirmation email from the RAM vendor. The RAM never shipped, and it took us a week to figure out that there was a problem.
People make mistakes all the time. Why is this an unexpected result? People are jackasses. This should be obvious.
There are no trails. There are no trees out here.
I order all kinds of stuff online, wouldn't the receipt emails look like spam? My current spam solution is very simple:
1. display my email online as little as possible
2. use a number of addresses that all filter into one account, then filter by the sent-to address... this has turned up some VERY interesting results, for instance. I used dellorders@mydomain.com for an order from Dell, and NEVER used it or even typed it anywhere again, and started get spam about 6 months later, and I mean the nasty stuff, no just innocent stuff from Dell resellers...
3. i built a rudementary filter that looks for viagra,free,debt,enlarge, etc... if the sender is not in my address book, and the email contains these words, it is sent to a "check these out" folder...
How might a spam filter help me out without zapping confirmation type emails?
Cloud City Digital: DVD Production at its cheapest/finest
And if the study posted about is accruate, of those 1% that are left, you will (if you're a perfectly average person) accidentally delete 0.16% of good messages. Surely you've deleted a valid message by accident before? I do it regularily, deleting 25 spam messages with a single good one embedded in it when I just woke up before I had my coffee is not a good thing ;)
At the very least, if you were given the same data as these tests, that would be true. Consider if you *didn't* use popfile - how many spams would you be deleting every day, and how many good messages would be accidentally deleted? I know that if I had to manually delete the two or three hundred spams interspersed with good messages, my false-positive rate (the percentage of good mail I accidentally deleted) would skyrocket.
So just be glad you've got popfile. Not only do you not have to go through as much spam, but you're also more accurate while going through the little you must.
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
I find it interesting that an algorithm that was originally for image noise reduction found it's way to Machine Learning through a company whose purpose is to impliment noise reduction in audio. From my Googling, I think this is the first time anyone has used Baysian Noise Reduction in Machine Learning. Does anyone know otherwise?
6000 over what period?
This represents 8 days worth of spam for me. Yes, ~800 per day.
My address has been valid for 10 years. Why should I change it? Bogofilter is currently letting 2-3 per day into my inbox. I generally check for false-positives, but as the training has progressed, I am finding none anymore.
I plan to implement a single-shot, one try notification sender. I.e., if the mail gets classified as spam: lookup the mx record for the envelope return address, if it's nonexistent, lookup the a record. Make a connection and try to deliver a message indicating their message (include subject reference) was identified as spam, include a way for them to reliably get a message through to me. If any of the smtp exchange or address lookup fails, just forget it, they're probably not real anyway.
People are jackasses.
Hence we have spam in the first place.
KFG
That actually makes humans much more accurate. We can eliminate many of the messages just by looking at the subject.
The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?
Karma: It's all a bunch of tree-huggin' hippy crap!
Before I used a spam filter, I once missed a very important message whose subject line was something to the effect of "URGENT - DON't REBOOT THIS MORNING." That was a bad one to miss.
Of course humans make mistakes, and it is entirely possible for an automated or semi-automated system to be more accurate than a human alone.
The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?
I believe that humans can be 100% accurate (or thereabouts) if they read the *ENTIRE* message, however that's exactly the point - if you have to read an entire message to tell that it's spam, the spam has succeeded.
Their number probably concerns how people can tell without reading the entire message whether or not the message is spam. My brother accidentally deleted a few messages I had sent to him, however if he had read them fully he would have known they were legit.
Cheers,
Justin
Dolby noise reduction works by filtering a spectrum into a bunch of bands, each of which are compressed (in an audio sense, not in a digital sense), and recorded to tape. On playback, they go through an expander...how does that concept translate to spam filtering? It can't be "dolby-type", that doesn't make any sense...
ZuluPad, the wiki notepad on crack
I dunno. I'm running CRM114 now, and it's taking something like 1.5 seconds to identify emails. I am on a slow machine though, which used to do SpamAssassin at around 4 seconds, and inaccurately to boot. CRM114 is a big improvement, and if it trains well after the first fortnight I'll kiss TMDA goodbye.
Karma: It's all a bunch of tree-huggin' hippy crap!
Having such a powerful statistical spam filter is definitely a luxury. I have no difficulty believing the accuracy values presented here. I have had experience with spamprobe, CRM114, bogofilter, spambayes, and spamassassin and all of these do an amazing job to the point where spam no longer exists (for you).
Which leads to me plug a little project called WPBL that uses exactly these types of statistical spam filters to spot spam sources in a distributed fashion. Each project member uploads hourly the IPs they see relaying spam and non-spam, where the 'decision' is made by these extremely reliable filters. This effectively converts your regular mail account into an intelligent spam-trap that feeds a central blocklist.
The more members we get, the better we can identify active spam sources around the world. This information is then used by some sites for quite large-scale blocking. Since you're doing all this filtering processing anyway, why not also share "what you learn" (the IPs that are spamming you)?
If this grabs your interest, read up on the reporting scripts or alternatively, the open WPBL data upload protocol if you want to code your own report generator. Bandwidth usage is minimal.