Slashdot Mirror
Two Spam Filters 10 Times As Accurate As Humans
Nuclear Elephant writes "The authors of two spam filters, CRM114 and DSPAM, announced recently
that their filters have achieved accuracy rates ten times better than a human is capable of. Based on a study by Bill Yerazunis of CRM114, the average human is only 99.84% accurate. Both filters are reporting to have reached accuracy levels between 99.983% and 99.984% (1 misclassification in 6250 messages) using completely different approaches (CRM114 touts Markovan, while DSPAM implements a Dolby-type noise reduction algorithm called Dobly). If you're looking for a way to rid spam from your inbox, roll on over to one of these authors' websites."
I'm sorry, Dave... That Nigerian guy looks suspicious and I can't let you send him money.
------- "A true friend stabs you in the front." -Eliot
Comment removed based on user account deletion
Once Email Spam is eliminated, then IM spam will begin...
Jeff
I presume they mean more accurate than a human that was only looking at the subject line? I fail to see how someone could misclassify an email after they'd already opened it unless it was some kind of marathon testing, which would be totally unrepresentative of any real life situation. Once you're getting 6,000 messages, it's time to reach for "Delete All" and change your address, methinks
Am I crazy or is that nowhere near "10 times better"?
Support the First Amendment. Read at -1
can this be used with Spamassasin, or is a stand alone program? Does it need something like Amasis to run?
CB
free ipod and free gmail!
If your email is indistuinguishable from spam by a human, perhaps the problem isn't the receiver. It's the sender.
Forgive me if I don't feel any pity that some moron's email gets filtered to the junk bin because I couldn't discern it from spam.
I have been pwned because my
Isn't the rough defintion of SPAM "Anything I don't want in my mailbox"? If that's the case, isn't the human score going to be 100% (at least for the intended recipient)?
"Prepare for the worst - hope for the best."
Just enter a valid email address, and hit submit!
I haven't been 100% accurate.
I received an email from my sister-in-law from her work, and the address looked suspicious (one of those weird-looking "letter and number" jumbles.
I deleted it. It happens.
Humans sometimes make mistakes, that's where the inaccuracy comes from.
Well, it certainly sounds better than the pay-per-email "postage" idea. If postage hasn't stopped snail spam, why would it stop e-mail spam?
but can you identify spam before opening it 100% of the time? Now, I realize that the mail program is looking at the actual data as well, which gives it an advantage, but on the other hand, how else can IT detect spam?
I write code.
Human=99.84
.16% and the machine only missues .016% hence the machine is 10 times better.
New proggie=99.984
So the human misses
I suppose it depends how you're defining spam. Perhaps the ultimate spam messages that don't get past them are capable of passing a turing test... hence fooling those gullible human recipients into thinking that it isn't even spam!
Fortunately, soon we will all be able to use the superhuman spam-detection capabilities of these filters to save us from ourselves. Imagine all of those pesky e-mails from your 'friends' getting caught by your spam filter before they even impinge upon your consciousness.
It'd be a wonderful world.
"We have to go forth and crush every world view that doesn't believe in tolerance and free speech." - David Brin
...and only one locked pod bay door per 6250, I like those odds.
If you read the post, it quotes a study and says humans are only accurate 99.84% of the time.
:)
Kinda makes you wonder how they can know the filters are right though.
(please don't reply telling me how)
Probably used those same people who open viruses as test subjects.
that i'm 100% accurate.
maybe some of those people just dont know where their 'del' key is, or what it does...
20 mil and I will! Learn Esperanto with 20M others.
Does this mean that if I use the 2 together, i get a 99.99999728% accuracy? Awesome! THat means it would takes months for me to see a single error!
Beware of he who would deny you access to information, for in his heart he dreams himself your master.
1 in 6250?
;)
Who wants to bet that they only sent two 'spam' and one of them was disguised well?
I reached the conclusion of "two filters better than humans" by using two sequential filters:
server side spamassassin, and a couple of simple procmail recipes. They have kept almost all the SPAM away.
However, it is good to see such good techniques becoming available and we can hope to see them as straight forward usable tools.
So, when will mozilla/TB (or your favourite server side or client side filter) get them?
S
I still think it is the best 'filter' available, since filtering is a lookup into a database of 'good senders' http://www.knowspam.net
it's not that humans are not as accurate, it's that 1 in X times we really do want a mini camera or free porn. It is what seperates us from those cold, heartless machines.... mini cameras and porn....
My Machine outhinks me!!"
I've seen better stories in Highlights for Children
Obviously you've never seen someone new to the internet sit in front of their computer. Lots of people don't know what popups are. Lots of people read some spam not knowing what it is. To these people, a computer is merely an interesting string of sensations.
Know what I like about atheists? I've yet to meet one that believes God is on their side.
SpamAssassin is a single approach. It looks at a bunch of features, then combines them linearly and compares the result against a threshold function. It's a relatively simplistic method, compared to these two. Not hard to see how more sophisticated methods could do better.
I received an email from my sister-in-law from her work
Yeah, so did I. The subject line was "I want you so bad."
I deleted it. Turned out the message was genuine. I'll never forgive myself...
The coolest voice ever.
I'm also sure that Yahoo's "SpamGuard" was great when they first introduced it. Now, It catches roughly half of all the spam I get. Why? Because people have figured out how it works and taken advantage of it. The same will happen with any content-recognition-based spam software. In the extreme case, even if a piece of software were 100% accurate at saying "This piece of e-mail looks like spam," then spammers would just make their e-mails look exactly like e-mail from one of your buddies. How could software ever tell the difference between:
Hey, dude, check out this website I found. There are some hot naked chicks and stuff. Sweet.
Signed,
Your Buddy
and
Hey, dude, check out this website I found. There are some hot naked chicks and stuff. Sweet.
Signed,
SpamKiddy
Even a human can't tell the difference. The only real difference is who they're from.
Perhaps they mean that Human A is reading email intended for Human B and attempting to classify the email as spam or not spam. I wouldnt be surprised if a computer could do a better job at that sort of task. Besides Im sure Human B wouldnt want Human A reading that cyber sex chat log.
[*Bing* -- mail from VP of sales pops into my inbox. Subject: "Making money fast!"]
[*Bam* -- I hit delete, thinking "Stupid Spam!"]
Ahh, shit! Lookie, a human screwed up.
The filter would have actually examined the message and probably decided that it was legitimate.
If these filters can hit 99.99% with those, I'd be quite impressed.
What I'm listening to now on Pandora...
It's really easy to design an effective solution when the problem is purely mechanical or natural. As long as you're working with spammers who don't adapt, you can slice through their shitstorms very effectively.
But when a single solution becomes mainstream, spammers will adapt to it. Bayesian filters tend to work very well, but now spammers are adding sprawls of randomly generated green-light text to offset the filter's score.
Google found an excellent way to rank websites, but then it became widespread enough that webmasters began to game the system it had created. It's been playing catch-up ever since.
Once the adversary begins to adapt, we lapse into the same cat-and-mouse game of technological barriers and counter-barriers that we've seen so many times before.
Quite simple:
With 10 messages (after automatic spam detection) humans are 100% accurate.
With 1,000 messages, (before automatic spam detection)
humans are less than 100% accurate.
The experiment was done on 5849 messages.
Remember; one thing computers are good at is doing boring things repeatedly.
No, humans are not 100%.
If you see a strange name in your inbox with an odd title, that might be a Nigerian businessman, or it might be your long lost Nigerian brother.
I recently tried to order a t-shirt from this guy for a band he used to be in. I found his band because we have the same (semi-uncommon) name. So, he got an email From: himself. I had to send him two emails because he deleted the first one assuming it was spam.
I ordered some RAM for my dad a while back. He gets 200 spam emails a day (email addy in resume & web page), and he deleted the confirmation email from the RAM vendor. The RAM never shipped, and it took us a week to figure out that there was a problem.
People make mistakes all the time. Why is this an unexpected result? People are jackasses. This should be obvious.
There are no trails. There are no trees out here.
I order all kinds of stuff online, wouldn't the receipt emails look like spam? My current spam solution is very simple:
1. display my email online as little as possible
2. use a number of addresses that all filter into one account, then filter by the sent-to address... this has turned up some VERY interesting results, for instance. I used dellorders@mydomain.com for an order from Dell, and NEVER used it or even typed it anywhere again, and started get spam about 6 months later, and I mean the nasty stuff, no just innocent stuff from Dell resellers...
3. i built a rudementary filter that looks for viagra,free,debt,enlarge, etc... if the sender is not in my address book, and the email contains these words, it is sent to a "check these out" folder...
How might a spam filter help me out without zapping confirmation type emails?
Cloud City Digital: DVD Production at its cheapest/finest
I think these guys are trying to put the focus on the server side of things where they emphasize greater speed and efficiency in eliminating spam from a large number of accounts as opposed to a single one. Just out of curiosity, do Thunderbird and iMail use similar filtering techniques with their junk mail controls?
No, imaginary humans with infinite time and dedication are 100%. But real humans are not. The percent goes down with time and dedication continuously, so I really don't understand what this 99.84% means.
Spam is what is defined by humans as Spam.
To determine the accuracy of a spam detector, it is necessary first to come up with a sample of what is or isn't Spam. (I'd assume a human would do this?) So the best result we can get be evaluating humans is how often they agree with the result of the initial label.
This figure probably won't be 100%. People have slightly different concepts of what mail is requested vs. unwanted, and what is advertising or useful information. So there is a valid possibility of disagreement.
That doesn't mean humans can't do the job accurataly. (After all, if they couldn't, then the initial human-made labels would themselves be wrong and any data based on them meaningless!)
If the training data is labeled with the same criteria as the test data, it is obviously possible that a trained system can acheive results which more closely agree with the test data. They are being trained on similiar data. But that doesn't mean that the system is MORE accurate at detecting spam than humans. It means that the system agrees with a particular human (or set of humans) more than other people do in a labelling of spam/non-spam.
For all we know, the evaluators idea of spam is "wrong".
And if the study posted about is accruate, of those 1% that are left, you will (if you're a perfectly average person) accidentally delete 0.16% of good messages. Surely you've deleted a valid message by accident before? I do it regularily, deleting 25 spam messages with a single good one embedded in it when I just woke up before I had my coffee is not a good thing ;)
At the very least, if you were given the same data as these tests, that would be true. Consider if you *didn't* use popfile - how many spams would you be deleting every day, and how many good messages would be accidentally deleted? I know that if I had to manually delete the two or three hundred spams interspersed with good messages, my false-positive rate (the percentage of good mail I accidentally deleted) would skyrocket.
So just be glad you've got popfile. Not only do you not have to go through as much spam, but you're also more accurate while going through the little you must.
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
If you have no spam filters, then classifying email amounts to "delete, delete, delete, delete, down-arrow, delete, delete, down-arrow, delete, delete, whoops!" That one mistake just dropped your average to 90%. Frankly, I'm amazed humans scored as well as they did.
Results of new spam filters cannot help but to be bogus... The true test of a filter is how well it works *after* all the spammers know how it works and try to circumvent it.
Enjoy science fiction? "Turing Evolved" - AI, Mecha, Androids and rail-gun battles. What more could you want?
Hasn't anybody noticed the obvious Spinal Tap reference?
...you couldn't hear the...
Jeanine: You know, it might have been better if the, uh, album had been mixed right.
David: Well I suppose you could cry about that, of course it's true. I mean it's true.
Jeanine: It wasn't...it was mixed all wrong, wasn't it?
Nigel: It was mixed wrong?
Jeanine: Yeah....
Nigel: Were you there?
Jeanine:
Nigel: How do you know it was mixed wrong?
David: But she's...she's heard the...she's heard the record.
Jeanine: No, but I've heard the album.
Nigel: So you're judgement is that it was mixed wrong.
Jeanine: You couldn't hear the lyrics all over it.
David: You don't agree that you can't hear the
vocals?
Nigel: No, I don't. I do not agree. No.
David: Well I think maybe....
Nigel: It's interesting that she's bringing it up.
David: Well she'd like to hear the vocals.
Nigel: I mean it's like it's me saying, you know, you're using the wrong conditioner for your hair.
David: Don't be stupid.
Jeanine: You don't, you don't do heavy metal in
dobly, you know, I Mean...it's
Nigel: In what??? In what???
Jeanine: In dobly...
Nigel: In dublin!?! What's that?
David: She means Dolby, alright? She means
dolby, you know? You know perfectly well what she means.
This spam filter goes to 11!
DSPAM implements a Dolby-type noise reduction algorithm called Dobly
Despite the musical reference on the DSPAM site, I figured some people still won't get the joke. So here it is:
JEANINE: You don't- you don't do heavy metal in dobly, you know, I mean...it's--
NIGEL: In what??? In what???
JEANINE: In dobly...
NIGEL (GRINS): In doubly!?! What's that?
DAVID: She means Dolby, alright? She means Dolby, you know? You know perfectly well what she means.
--from the movie "Spinal Tap"
programs out perform humans all the time. Where am I? my GPS knows. What was that person's number? my PDA knows. What is 2365 times 8675309? just use a calculator: 20517105785. Wow, I was just out performed three times in the space of a minute.
A CRM114 plugin for SA is available, thanks to Devin Nate:
d =2 301
http://bugzilla.spamassassin.org/show_bug.cgi?i
I find it interesting that an algorithm that was originally for image noise reduction found it's way to Machine Learning through a company whose purpose is to impliment noise reduction in audio. From my Googling, I think this is the first time anyone has used Baysian Noise Reduction in Machine Learning. Does anyone know otherwise?
Best phrase I've read all week... Oh, yeah, it's only Monday! This one will probably hold me over 'till Friday, though. ;-)
If humans don't have 100% accuracy, who/what is defining what spam is?
client/server-side filtering does NOT solve the problem!
The biggest problem with spam is the invasion of third party computers on the Internet. The ILLEGAL activity spammers perpetrate by breaking into machines, forging headers and hijacking servers.
Any filtering method does not address this most serious problem, and even if you do not see any spam in your inbox, you're still paying for the bandwidth and system resources these spammers steal.
Stop with the filtering algorhythms and take some of that energy and contact your local Attorney General, DA and FBI and demand that they prosecute these people who are BREAKING THE LAW.
I know a guy who has a Korean grad student who doesn't speak English very well. He manages to produce subject lines for the messages he sends that get him blocked by spam filters nearly all the time. Not his fault really, but it happens.
-"It seems like you're trying to exploit a security hole. Would you like help?"
People are jackasses.
Hence we have spam in the first place.
KFG
Don't worry, I can forward you the one she sent me. Sounds like the same email.
I remember when legal used to mean lawful, now it means some kind of loophole. - Leo Kessler
I work with some people who use their computer every single day. Have had an email address for years, who still buys what they read in an email. Photoshop for $50...sure! Herbal viagra...why not?
Well, she always has a big smile on her face, maybe there's something to this spam thing.
"Wisdom is not a product of schooling but of the life-long attempt to acquire it." -Albert Einstein
How can you be sure that you've never deleted an important email as spam?
Matt Slot / Bitwise Operator / Ambrosia Software, Inc.
Not to sound like a litigation whore, but ...
I wonder if it would be possible to sue these spammers for interfering with a business transaction. Granted, the amount in question here is minimal, but just the possibility that a spammer could be found liable for this might deter some of them.
If that doesn't work we should sign up every megacorp CEO on every spammer list possible, and hope s/he misses an important memo costing megacorp millions. Then megacorp could sue spammer into oblivion.
Lots of people don't know what popups are.
Uh, sure they do. Popups - that's like those porn storms, isn't it? Some people say it only happens with IE and Windows, but I talked to my service provider and they told me 'just pull the power plug out of the wall when that happens'.
Easily fixed.
That actually makes humans much more accurate. We can eliminate many of the messages just by looking at the subject.
The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?
Karma: It's all a bunch of tree-huggin' hippy crap!
Saying an algorithm is x% accurate is not sufficient, because there are two kinds of errors: false acceptance of spam, and false rejection of non-spam. Personally, I'd settle for 90% false acceptance if I knew the false reject rate was 100% rather than have a program that was 99% at both.
Also, I wonder how many people have actually looked at CRM114 and tried to use it.
The really interesting thing about CRM114 is the windowed polynomial hashing technique used although there's some evidence that it can work just as well (if not better) on a much smaller window of only two tokens. I'm hoping someone will do a full exploration of the idea for SpamAssassin's Bayes module.
That's an easy one. The computer is 10 times better at recognizing what it has decided is spam. We humans are lucky to even be in the same league.
Now that you understand that, you're one step close to being "computer literate".
If we humans are only 99.84% accurate, then 0.16% of the time we will incorrectly think the email is real and buy viagra ? I don't think so.
I read the email and delete it. Exactly the same as the spam filters do it, only MORE accuratly. I think the tests applied would have been between a human reading the header of an email and deciding whether to open it or not verses the spam filter making the decision for us. BUT the spam filter makes its decision by opening the email. Therefore to have a proper comparision I should be allowed to open the email as well before I make the decision. Therefore I am 100% accurate.
if
I bet it allows messages from General Jack D Ripper or any email that contains the secret phrase "purity of essence", "peace on earth" or "precious bodily fluids".
-- http://www.swcp.com/~hudson/
Presumably they must use a superhuman who has 100.00% accuracy.
Karma: It's all a bunch of tree-huggin' hippy crap!
The thing with spam is that it's supposed to be a way for somebody to make money... i.e. they are trying to sell you something, be it directly or indirectly. I can't think offhand of an email I have recently received that could be misconstrued as trying to sell me something. From that simple viewpoint, spam can never look exactly like regular mail, because it has a different purpose.
Before I used a spam filter, I once missed a very important message whose subject line was something to the effect of "URGENT - DON't REBOOT THIS MORNING." That was a bad one to miss.
Of course humans make mistakes, and it is entirely possible for an automated or semi-automated system to be more accurate than a human alone.
Maybe it's the %0.16 of people who are responding to spam.
CRM is actually quite a acinating product. It's like a super grep where you can match against blocks of text instead of just lines. It also has some logic operators and such. I think there is a quote on his web site that refers to it as "grep bitten by a radioactive spider" and it's true.
You can use it for lot more then spam processing, it's a really neat all purpose tool.
The best way to support the US war effort is to continue buying American products.
In addition, every user will get special discounts on software, mp3s and computer parts with my partners, and two FREE MP3'S every month.
There are also special savings on 100% all-natural and effective male enhancement products. A portion of the rebates will go towards a $100000 fund needed to get 100,000,000 dollars (ONE HUNDRED MILLION DOLLARS) from Liberia into an account in Switzerland. If you provide your social security number (SSN) and your checking and savings account number you will get part of the ONE HUNDRED MILLION US DOLLARS. Only the first 100 people will qualify, so hurry up and don't miss this offer!
--I am Sun Tzu of the Borg. Resistance is feudal.
If there is no universal bottom line of what Spam is, we can never manage it.
I think 'unsolicited request for money from a for profit oranization' will fit into everybodies base definition. Some people will expand on it, but we need a defined place to start.
The Kruger Dunning explains most post on
No, humans aren't 100% and yes, you can test for that. Try a thought experiment: fill a bin with 50,000 red balls and 50,000 blue balls. Ask a human to sort them all. The result probably won't be 100%, but you can still check the result and figure out how accurate the human is without relying on a superhuman ability to tell the balls apart. Same thing for spam: if you start with a known training set, you can test humans to see how well the spam is identified by manual sorting.
I'm not surprised a filter beat the human, considering the study used a sample of 5849 messages. As the sample size increases, the filter's accuray will increase, and the human's will decrease. Furthermore the higher the spam/real ration, the better the filter will do in comparison to a human trying to sort at a reasonable speed. The reason being humans tend to skim, and rairly actually read entire subjects, much less messages. Give a human 5000 messages and an hour and he will probably make some mistakes. On the other hand, in 10 messages, the human will probably be 100% correct. Most email filters rely on this already, as they tend to err on the side of caution. With the bulk of the spam taken out, it is not a burden to have the human check the iffy bits. Furthermore the type of email can mislead humans. A business-type email sent to someone's personal email is much more likely to be mistaken as spam, and vice versa. The main disadvantage of automated filtering is people generally have an idea of when a really important e-mail is going to come (the type that false positives are completely unacceptable) and who it will be from.
The post quotes "a study" which gives the 99.84% figure. In fact, the 99.84% figure is mentioned in the one paper as "the human author's measured accuracy as an antispam filter...on the first pass". This is what we who understand statistics call "nonsense". An individual human had an estimated accuracy of 99.84% when looking at one particular sample set of data, once. This is not a meaningful number, and it sure as heck ain't "a study".
I have never deleted an email I meant to keep
How could you possibly know? You deleted it!!
Write boring code, not shiny code!
Then megacorp could sue spammer into oblivion.
Or more likely, megacorp fires it's mail administrators for being incompetent and goes on about it's business.
By your definition, every spam message is a mistake for the spam filter because it "reads" all of them (at least to the same extend as it "reads" any non-spam email). The filter is more accurate because it is fast enough to be more thorough than any human can possibly be expected to be. If we could thoroughly analyze hundreds of emails in a matter of seconds, we would have no need for spam filters. We have spam filters because we don't have the time (or the patience, for that matter) to be as careful as a filter.
I talked to my service provider and they told me 'just pull the power plug out of the wall when that happens'.
Ok, now the screen dimmed a little and I heard the hard drive spin down, but the pop ups are still a comin! Oh, and something about "battery level at 98%" or something.
I'm not a journalist, but I play one on slashdot
would love to rtfm, but I want a fairly simple answer to this, how can I do a 30 minute job of integrating this into the mozilla mail client, or does it have to be tied into the server itself? I was wondering if this was a quick, easy fix, or if it is an all weekend type of project.
Most likely it will take at least as long as reading an article. So you might as well not bother.
Or your dad is an idiot who doesn't know how to route his email.
But I was only contesting the great-grandparent poster, who said that humans are by definition 100% accurate.
While my dad may be an idiot, he is also human. I am correct, great-grandparent poster is incorrect, and you are off topic. As far as I can tell, I've never deleted an email I meant to keep either. But you and I aren't the only people worth discussing.
There are no trails. There are no trees out here.
There goes my bussines idea. I wanted to start a bussines that puts humans in an eastern europe contry to sort corporate e-mail.
Now I have to think again about putting humans to decorticate sunflower seeds, it's cheper than all those machines.
The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?
I believe that humans can be 100% accurate (or thereabouts) if they read the *ENTIRE* message, however that's exactly the point - if you have to read an entire message to tell that it's spam, the spam has succeeded.
Their number probably concerns how people can tell without reading the entire message whether or not the message is spam. My brother accidentally deleted a few messages I had sent to him, however if he had read them fully he would have known they were legit.
Cheers,
Justin
If every individual human has an accuracy of 99.983%, then two independent humans have an accuracy of 1 - .00017^2 or 99.99999711%. This would allow ample accuracy to judge the computer, except that it's not true[1]. A better answer is the one you suggest: humans must judge spam from subject/author alone, whereas computers get to look at the whole message. Humans reading the whole message, and possibly even following included links, responding, etc., can be assumed to have full accuracy, within epistemic bounds. Indeed, merely re-checking your work, etc. - being consciously more diligent than the average spam-sorter - should insure your accuracy is better than average.
As for how accuracy was actually judged in this particular study, I suppose you would have to read the article for that. I haven't, myself...
[1] It assumes the probability of error is equal for every message, which is obviously not true (i.e., that error is random rather than systematic). The real accuracy of two humans in concert is surely much lower; OTOH, it is still sure to be much, much higher than the accuracy of a single human.
But the computer reads the entire message, so it's not really a fair comparison, is it? How many more lines of information was the computer allowed to look at to create its superior result?
Karma: It's all a bunch of tree-huggin' hippy crap!
Current spam filters may be "10x" better than humans, current spam filters may be terrible on future spam.
Filters beating spam and spam beating filters is a continuous arms race. In the limit, optimal spam filtering is equivalent to solving NLP (natural language processing); Unless you build a filter that can fully understand the text (syntax, semantics, pragmatics, world knowledge, the whole shebang), an adversary can always construct spam to defeat your filter.
You mean you've never noticed this before? Idiots are some of the happiest people I know.
Please give this man a drink of grain alcohol and rainwater.
ObKubrick: In 2001: A Space Odyssey, one of the pods was marked with the designation CRM-114. And in Clockwork Orange, Alex is injected with serum 114. I suppose CRM-114 is to Kubrick as THX1138 is to Lucas.
Dobly, on the other hand, is from This is Spinal Tap , a mispronounciation of "Dolby" by David St. Hubbins's girlfriend:
Not to mention that it probably avoids trademark infringement (though I wouldn't put it past Dolby Labs or Thomas Dolby to raise a stink).
Maj. Kong
Shoot, a fella' could have a pretty good weekend in Vegas with all that stuff.
Good question! We're working on this problem, among other things, at the PSAM project. We have a project to produce high-quality benchmark corpora for spam filter testing. Watch that space for ongoing work, or e-mail us an offer to pitch in and help---we could use it!
When these factors are considered, I think it's quite possible to write software that in the long run has a higher success rate than a human who has better things to do than filter his mail all day.
Dolby noise reduction works by filtering a spectrum into a bunch of bands, each of which are compressed (in an audio sense, not in a digital sense), and recorded to tape. On playback, they go through an expander...how does that concept translate to spam filtering? It can't be "dolby-type", that doesn't make any sense...
ZuluPad, the wiki notepad on crack
...are still the only real solution to the issue of trust, reputation, and accountability on the Internet. We need it for so many other things in addition to guaranteeing email legitimacy.
If every user or at least every server had a key and we all signed each others keys creating a web of trust and only accepted signed and trusted mail the spam problem would be solved. I really dislike the way SSL certificates are handed out. A central CA is a very bad idea due to the cost and browser lock-in issues etc. With GPG and web of trust if you want to run a mail server you need to talk to a friend who is already running one and get them to sign your key. Perhaps we could even use DNS to propagate and cache the keys and sigs. If you sign a key that turns out to be a spammer you better revoke that signature fast before the person upstreeam from you revokes yours. Problem solved. Now if only we could get the big guys to go along with it...
What you're planning has already been done, it's called TMDA, and it's not such a good idea. You're going to send out 800 "challenge" emails per day - have you given any thought to how many of those will be genuine addresses, but have nothing to do with the spam you receive because they just happen to be the joe-job victim? These kind of challenge/response systems may slighlty alleviate your own suffering through spam, but at a cost to all those unfortunate enough to have had their email addresses faked. And if the sheer impoliteness of such net behaviour doesn't put you off, note that you're using up more of your own bandwidth to send out such challenges
If any of the smtp exchange or address lookup fails, just forget it, they're probably not real anyway
It would make a lot more sense to make these kind of checks when you're receiving the email in the first place. Reject at the SMTP level - you never accept and process the spam in the first place
My next sig will be ready soon, but subscribers can beat the rush
How would it know if I consider brunettes non-spam but blondes spam? I did opt-in for one of those email categories, but not the other.
From what I gather of Pinker's theory is that language is implemented by a dedicated module in the human brain. This module is just neurological hardware, operating entirely by physical means, and does not invoke any sort of deus ex machina; therefore, what it does is an algorithm.
The language module does invoke other parts of the brain, such as general knowledge; however, there's nothing in the process that depends on it being in a human brain. Given that cognition is a physical process, one could postulate a computer program that could achieve the same results, even if drawing on a very large database of cultural information. The suggestion that language is "innately human" sounds a bit too much like carbon chauvinism, the belief that intelligence is an exclusive property of carbon-based life.
I dunno. I'm running CRM114 now, and it's taking something like 1.5 seconds to identify emails. I am on a slow machine though, which used to do SpamAssassin at around 4 seconds, and inaccurately to boot. CRM114 is a big improvement, and if it trains well after the first fortnight I'll kiss TMDA goodbye.
Karma: It's all a bunch of tree-huggin' hippy crap!
Yes, but it is meaningfull nonetheless. If you just think that it's very likely that after reviewing 650 messages, you may have missed one email that you thought was spam, then the "study" is right. I don't care if the number is 900 or 400 emails. Those 400 mails are making me lose a _lot_ of time, and if I value my time, I am losing a lot of productivity, and also missing an important email.
.99 accuracy, then it's a real time saver, and if it only makes a mistaque every 2000 emails, then SURELY I will be more accurate than me. That depends of course, on how much spam you do get. I get arround 20 to 1 ratio of spam to real meat, and I get arround 100 spam messages a day. I can't spend 1 hour a day cleaning spam with 99,9% accuracy, so I am forced to quick sweep. This thing could make me regain the time, and the false positives would mean i even make less mistakes than manually.
If the program can have a
The important things is how accurate the antispam tool is, and how accurate I am (ratio of spam to meat, and how much a miss costs me). How much other people make mistaues is not really that important. Everybody knows how much time they have, and how much spam to meat they have, and thus, it's very likely that if they don't have a LOT of time to waste, they will be making a mistake for every 200 to 600 spam messages.
unfinished: (adj.)
Umm, Fetchmail + procmail on your local machine?
Not sure exactly why you need a pop3 proxy involved, just use Fetchmail to deliver locally, run things through procmail.
Set your local mailserver (sendmail/qmail/postfix/exim/whatever) to use your ISP's SMTP server as a smarthost, and it'll send everything it doesn't recognize as local off to them to handle.
Having such a powerful statistical spam filter is definitely a luxury. I have no difficulty believing the accuracy values presented here. I have had experience with spamprobe, CRM114, bogofilter, spambayes, and spamassassin and all of these do an amazing job to the point where spam no longer exists (for you).
Which leads to me plug a little project called WPBL that uses exactly these types of statistical spam filters to spot spam sources in a distributed fashion. Each project member uploads hourly the IPs they see relaying spam and non-spam, where the 'decision' is made by these extremely reliable filters. This effectively converts your regular mail account into an intelligent spam-trap that feeds a central blocklist.
The more members we get, the better we can identify active spam sources around the world. This information is then used by some sites for quite large-scale blocking. Since you're doing all this filtering processing anyway, why not also share "what you learn" (the IPs that are spamming you)?
If this grabs your interest, read up on the reporting scripts or alternatively, the open WPBL data upload protocol if you want to code your own report generator. Bandwidth usage is minimal.
That should explain why Dubya's always smiling even when he's trying to be serious.
$5 / month hosted VPS on linux = awesome!
I say get a bunch of honeypots and do the test again.
.
A human doesn't have to determine if it's spam simply by the title.
The human should have all the advantages these filters have body / header / ip
Cheers
Well if the human was given the chance to read the body text as well like the filters do, then they would be 100% able to delete their own spam.
DRACO-
Consider yourself blessed if you are sneezed on by a dragon and only get wet, it could have been a fireball.
statistics.
This headline is misleading. I refuse to RTFA, because I imagine the "10 times as effective" figure comes from the article itself.
Come on, folks. The figures do, in fact, show a 10 times increase in effectiveness between humans and these filters. But what the heck does that mean? I have to question the studies. How did they come up with this 99.84% figure? Does it mean that one person will mis-classify about 16 emails in 10000 (a small number indeed)? Or did one or two outliers taint the data?
The important thing here is that we're comparing three averages. Were the conditions between the trials the same? Were the humans given time limits? Were the accounting methods accurate? Were the spam messages the same?
It's quite possible that these averages were bounded by possible error quantities (they should have been!) and that these were tossed when reporting the numbers to us. This was so that a startling result (10 times as effective as a human) could be shown in a headline. It's all about coming up with a flashy "fact".
It's very easy to make numbers say what you want them to say, so I'd be a little wary of running around to your friends "citing" this 10x improvement figure without doing some deep delving into the processes involved in arriving at the number.
I write spam filters for a living, and I promise you that they can eliminate many of the spams just by looking at the subject too.
Of course, so can I. Now, since I write the filter based on my human judgement of what constitutes spam, which is more accurate?
Not comparable. The job of a junk mail filter is to drop things I don't want to read. It is trying ot match my evaluation, not to match a semi-objective criterion like red or blue.
If I read 1000 messages and say which I wish I hadn't read, then I am 100% accurate by definition.
Of course, if they are really talking about a pure spam filter -- ie one which identifies unsolicited commercial email -- then they can be more accurate than me, but at an uninteresting, perhaps even counter-productive, task:
I may get unsilicited commercial email I do want to read one day. Almost happened once (I had inadvertantly signed up for it, so it was not really unsolicited, and I didn't actually buy the piece of kit they had on special offer that week, but was tempted). I also get stuff I don't want which isn't spam (notably email from virus infected machines).
The referenced study seems to be a very sloppy job from this POV. They don't define what their criterion of sucess is, and to the extent they put in a hand waving attempt it is clearly nonsense:
`Unsolicited' does not imply `not desired'. If they don't tease those two apart, they can't get interesting results for real world applications. Eg, someone mailing my work address with a commercial proposition may well be a very welcome unsolicited commercial email._O_
.|< The named which can be named is not the true named
We don't need to trust the *person* sending the mail. It would be sufficient to trust the machine that is doing so.
Look at http://spf.pobox.com/ which is sufficient. With SPF, you know that if you are getting SPAM saying it is from @ultraviolet.org, then it really is from @ultraviolet.org (or at least someone who ultraviolet.org trusts).
Your solution requires a certain level of technical proficiency (setting up and managing the key) of *all* participants. SPF's solution only requires technical proficiency from those who manage DNS settings and those who manage email servers (in particular the person who manages your email server).
Also, what about *stolen* keys? And who handles key checking? SSL certificates are restricted to a few root signers, but you don't want a central certificate authority. PGP/GPG work well because they only involve small numbers of people. In general, you know the person directly. Occasionally it will be a friend of a friend message. What do you do when the chain is 10 or a 100 or a 1000 keys long? How long will it take for you to find out that 978 has since revoked their signature for 977 (counting in steps from you, i.e. you are 0 and 1000 is the original signer of this chain)? Or how long will it take you to verify all 1000 keys if you try to do it real time (i.e. when you get the message)?
If humans are 99.84% accurate and these filters are ten times as accurate, wouldn't that make these filters 998.4% accurate or am I missing something?
Sounds like POPfile was what you were actually looking for!