Slashdot Mirror
Two Spam Filters 10 Times As Accurate As Humans
Nuclear Elephant writes "The authors of two spam filters, CRM114 and DSPAM, announced recently
that their filters have achieved accuracy rates ten times better than a human is capable of. Based on a study by Bill Yerazunis of CRM114, the average human is only 99.84% accurate. Both filters are reporting to have reached accuracy levels between 99.983% and 99.984% (1 misclassification in 6250 messages) using completely different approaches (CRM114 touts Markovan, while DSPAM implements a Dolby-type noise reduction algorithm called Dobly). If you're looking for a way to rid spam from your inbox, roll on over to one of these authors' websites."
I'm sorry, Dave... That Nigerian guy looks suspicious and I can't let you send him money.
------- "A true friend stabs you in the front." -Eliot
Comment removed based on user account deletion
Once Email Spam is eliminated, then IM spam will begin...
Jeff
It's hard to believe that a single approach like this is better than SpamAssassin. I wonder hot is compares?
I presume they mean more accurate than a human that was only looking at the subject line? I fail to see how someone could misclassify an email after they'd already opened it unless it was some kind of marathon testing, which would be totally unrepresentative of any real life situation. Once you're getting 6,000 messages, it's time to reach for "Delete All" and change your address, methinks
Am I crazy or is that nowhere near "10 times better"?
Support the First Amendment. Read at -1
can this be used with Spamassasin, or is a stand alone program? Does it need something like Amasis to run?
CB
free ipod and free gmail!
If your email is indistuinguishable from spam by a human, perhaps the problem isn't the receiver. It's the sender.
Forgive me if I don't feel any pity that some moron's email gets filtered to the junk bin because I couldn't discern it from spam.
I have been pwned because my
That has to be some stupid people it is comparing to.
The day that something programmed out performs a human just goes to show how bad the World is coming to... although there was that Chess game that beat the World's Champion.. even though that was a different story. =/
"Instant gratification takes too long." - Carrie Fisher
Isn't the rough defintion of SPAM "Anything I don't want in my mailbox"? If that's the case, isn't the human score going to be 100% (at least for the intended recipient)?
"Prepare for the worst - hope for the best."
Just enter a valid email address, and hit submit!
Would someone like to explain how a program (even if it's right 99.something% of the time) is more accurate than me (100%)?
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
How does one test a program like this that's more acurate the humans?
I haven't been 100% accurate.
I received an email from my sister-in-law from her work, and the address looked suspicious (one of those weird-looking "letter and number" jumbles.
I deleted it. It happens.
Humans sometimes make mistakes, that's where the inaccuracy comes from.
Accuracy of the SBPH/BCR classifier has been seen in excess of 99 per cent, for 1/4 megabyte of learning text. In other words, CRM114 learns, and it learns fast .
Great, someone finally came up with a spam filter that learns.
Well, it certainly sounds better than the pay-per-email "postage" idea. If postage hasn't stopped snail spam, why would it stop e-mail spam?
but can you identify spam before opening it 100% of the time? Now, I realize that the mail program is looking at the actual data as well, which gives it an advantage, but on the other hand, how else can IT detect spam?
I write code.
Human=99.84
.16% and the machine only missues .016% hence the machine is 10 times better.
New proggie=99.984
So the human misses
If I say it is spam, I'm not reading it... and I am deleting it.
Any software that tries to stop me is removed via because it is faulty.
--Phillip
Can you say BIRTH TAX
I suppose it depends how you're defining spam. Perhaps the ultimate spam messages that don't get past them are capable of passing a turing test... hence fooling those gullible human recipients into thinking that it isn't even spam!
Fortunately, soon we will all be able to use the superhuman spam-detection capabilities of these filters to save us from ourselves. Imagine all of those pesky e-mails from your 'friends' getting caught by your spam filter before they even impinge upon your consciousness.
It'd be a wonderful world.
"We have to go forth and crush every world view that doesn't believe in tolerance and free speech." - David Brin
...and only one locked pod bay door per 6250, I like those odds.
One doesn't ever seem like enough. Like blacklist + spamassassin. how come you can never get to the links in the spam anyway, what's the point ? :)
If you read the post, it quotes a study and says humans are only accurate 99.84% of the time.
:)
Kinda makes you wonder how they can know the filters are right though.
(please don't reply telling me how)
Probably used those same people who open viruses as test subjects.
that i'm 100% accurate.
maybe some of those people just dont know where their 'del' key is, or what it does...
None of my mail is spam! I take the penis enlargment and brest enhancement very seriously.
20 mil and I will! Learn Esperanto with 20M others.
Does this mean that if I use the 2 together, i get a 99.99999728% accuracy? Awesome! THat means it would takes months for me to see a single error!
Beware of he who would deny you access to information, for in his heart he dreams himself your master.
1 in 6250?
;)
Who wants to bet that they only sent two 'spam' and one of them was disguised well?
I reached the conclusion of "two filters better than humans" by using two sequential filters:
server side spamassassin, and a couple of simple procmail recipes. They have kept almost all the SPAM away.
However, it is good to see such good techniques becoming available and we can hope to see them as straight forward usable tools.
So, when will mozilla/TB (or your favourite server side or client side filter) get them?
S
Would'nt accuracy differ from user to user? For a user who receive almost no spam and likes to keep his mail clean wouldnt the anti-spam learn to delete stuff that is just being cleaned and is not spam?
And also i'll be the one to judge its accuracy as ONLY I know what my spam is.
Lord of the Binges.
I still think it is the best 'filter' available, since filtering is a lookup into a database of 'good senders' http://www.knowspam.net
it's not that humans are not as accurate, it's that 1 in X times we really do want a mini camera or free porn. It is what seperates us from those cold, heartless machines.... mini cameras and porn....
I use sa, and still get about 200 spams a day. Every once in a while, while deleting spam, I accidentally open one up. I imagine I have deleted non-spam mail too.If you count this human error, these methods could actually be more accurate than humans.
...days! Yee haw!
"A great democracy must be progressive or it will soon cease to be a great democracy." --Theodore Roosevelt
My Machine outhinks me!!"
I've seen better stories in Highlights for Children
Obviously you've never seen someone new to the internet sit in front of their computer. Lots of people don't know what popups are. Lots of people read some spam not knowing what it is. To these people, a computer is merely an interesting string of sensations.
Know what I like about atheists? I've yet to meet one that believes God is on their side.
I received an email from my sister-in-law from her work
Yeah, so did I. The subject line was "I want you so bad."
I deleted it. Turned out the message was genuine. I'll never forgive myself...
The coolest voice ever.
I'm also sure that Yahoo's "SpamGuard" was great when they first introduced it. Now, It catches roughly half of all the spam I get. Why? Because people have figured out how it works and taken advantage of it. The same will happen with any content-recognition-based spam software. In the extreme case, even if a piece of software were 100% accurate at saying "This piece of e-mail looks like spam," then spammers would just make their e-mails look exactly like e-mail from one of your buddies. How could software ever tell the difference between:
Hey, dude, check out this website I found. There are some hot naked chicks and stuff. Sweet.
Signed,
Your Buddy
and
Hey, dude, check out this website I found. There are some hot naked chicks and stuff. Sweet.
Signed,
SpamKiddy
Even a human can't tell the difference. The only real difference is who they're from.
Perhaps they mean that Human A is reading email intended for Human B and attempting to classify the email as spam or not spam. I wouldnt be surprised if a computer could do a better job at that sort of task. Besides Im sure Human B wouldnt want Human A reading that cyber sex chat log.
I am testing a dup filter for slashdot stories.
It is 99.9% accurate.
It is 99.9% accurate.
It is 99.9% accurate.
It is 99.9% accurate.
It is 99.9% accurate.
It is 99.9% accurate.
It is 99.9% accurate.
Table-ized A.I.
Sure I can. No one knows My Email. I regard it all as spam. therefore, I only have email so that I can complain about spam.
Yes, we all agree that being better than a human is damn near impossible.
Great.
Still better than the pay-for-email thing.
Honestly, who wouldn't rather delete an email or two
a week about their penis than pay for every message they send?
If pay were required for email, a new kind of electronic mail would develop.
"Once robots outlaw humans from detecting spam only spam detecting outlaws will be human robots." or something like that.
Okay, so someone let 1 or 2 go during a test of over 6000 emails. I'd like to see their faces when the testers told them that their mother telling them to enlarge their penis was spam. I'd actually like to see that email that they thought was legitimate but in fact some nigerian asking for $5000 to "buy" $1,000,000
[*Bing* -- mail from VP of sales pops into my inbox. Subject: "Making money fast!"]
[*Bam* -- I hit delete, thinking "Stupid Spam!"]
Ahh, shit! Lookie, a human screwed up.
The filter would have actually examined the message and probably decided that it was legitimate.
No matter what, in the end, the human CANT be wrong... right?
Nah, wrong.
At least, I think it's wrong. Either way, one of us is wrong, so I must be right, because you said that humans can't be wrong and I said that you're wrong about that. Right?
"We have to go forth and crush every world view that doesn't believe in tolerance and free speech." - David Brin
If these filters can hit 99.99% with those, I'd be quite impressed.
What I'm listening to now on Pandora...
It's really easy to design an effective solution when the problem is purely mechanical or natural. As long as you're working with spammers who don't adapt, you can slice through their shitstorms very effectively.
But when a single solution becomes mainstream, spammers will adapt to it. Bayesian filters tend to work very well, but now spammers are adding sprawls of randomly generated green-light text to offset the filter's score.
Google found an excellent way to rank websites, but then it became widespread enough that webmasters began to game the system it had created. It's been playing catch-up ever since.
Once the adversary begins to adapt, we lapse into the same cat-and-mouse game of technological barriers and counter-barriers that we've seen so many times before.
Combine the two together to get enlarged breast-shaped penises, or penis-shaped breasts.
Quite simple:
With 10 messages (after automatic spam detection) humans are 100% accurate.
With 1,000 messages, (before automatic spam detection)
humans are less than 100% accurate.
The experiment was done on 5849 messages.
Remember; one thing computers are good at is doing boring things repeatedly.
What kind of stats is this? I would guess that the selection of what mails to receive he user makes would be the definition of accuracy here.
No, humans are not 100%.
If you see a strange name in your inbox with an odd title, that might be a Nigerian businessman, or it might be your long lost Nigerian brother.
I recently tried to order a t-shirt from this guy for a band he used to be in. I found his band because we have the same (semi-uncommon) name. So, he got an email From: himself. I had to send him two emails because he deleted the first one assuming it was spam.
I ordered some RAM for my dad a while back. He gets 200 spam emails a day (email addy in resume & web page), and he deleted the confirmation email from the RAM vendor. The RAM never shipped, and it took us a week to figure out that there was a problem.
People make mistakes all the time. Why is this an unexpected result? People are jackasses. This should be obvious.
There are no trails. There are no trees out here.
Ah, but if you had opened the email, thereby giving both you and the program the same criteria with which to classify as spam or not, then you wouldn't have deleted it.
Based just on subject line I'd be tempted to say that a computer would also have classified your example as spam.
I order all kinds of stuff online, wouldn't the receipt emails look like spam? My current spam solution is very simple:
1. display my email online as little as possible
2. use a number of addresses that all filter into one account, then filter by the sent-to address... this has turned up some VERY interesting results, for instance. I used dellorders@mydomain.com for an order from Dell, and NEVER used it or even typed it anywhere again, and started get spam about 6 months later, and I mean the nasty stuff, no just innocent stuff from Dell resellers...
3. i built a rudementary filter that looks for viagra,free,debt,enlarge, etc... if the sender is not in my address book, and the email contains these words, it is sent to a "check these out" folder...
How might a spam filter help me out without zapping confirmation type emails?
Cloud City Digital: DVD Production at its cheapest/finest
I think these guys are trying to put the focus on the server side of things where they emphasize greater speed and efficiency in eliminating spam from a large number of accounts as opposed to a single one. Just out of curiosity, do Thunderbird and iMail use similar filtering techniques with their junk mail controls?
No, that just means that the human defined the rules that it and the computer will follow. However, the computer will always follow those rules, while the human won't. People often will sometimes delete a mail, perhaps not recognizing an address from a valid sender, thinking it was spam. This would be a failure on the human part. Or perhaps they might consider opening up an email and reading it when it is in fact spam, a failure.
No, imaginary humans with infinite time and dedication are 100%. But real humans are not. The percent goes down with time and dedication continuously, so I really don't understand what this 99.84% means.
Spam is what is defined by humans as Spam.
To determine the accuracy of a spam detector, it is necessary first to come up with a sample of what is or isn't Spam. (I'd assume a human would do this?) So the best result we can get be evaluating humans is how often they agree with the result of the initial label.
This figure probably won't be 100%. People have slightly different concepts of what mail is requested vs. unwanted, and what is advertising or useful information. So there is a valid possibility of disagreement.
That doesn't mean humans can't do the job accurataly. (After all, if they couldn't, then the initial human-made labels would themselves be wrong and any data based on them meaningless!)
If the training data is labeled with the same criteria as the test data, it is obviously possible that a trained system can acheive results which more closely agree with the test data. They are being trained on similiar data. But that doesn't mean that the system is MORE accurate at detecting spam than humans. It means that the system agrees with a particular human (or set of humans) more than other people do in a labelling of spam/non-spam.
For all we know, the evaluators idea of spam is "wrong".
But I don't wanna have to do any of it by hand, so I'm gonna add my own recipes... so I don't see any see the stuff that isn't spam to the machine (not in the .0016% that it 'misses') but that I consider spam. That's a human using a tool to do something...
Nothing will ever be 100%, but the asymptote can get smaller and smaller the more closely the user and machine are working together for this.
As far as Mozilla integration... mozilla's just reading from my mail spool, so why would I want my MUA consuming resources that procmail will use more efficiently, silently in the background?
Some people won't run procmail, or run some OS that isn't compatible. I understand. But that's like not wearing a seatbelt: do so at your own risk and possible injury.
To understand recursion, you must first understand recursion.
And if the study posted about is accruate, of those 1% that are left, you will (if you're a perfectly average person) accidentally delete 0.16% of good messages. Surely you've deleted a valid message by accident before? I do it regularily, deleting 25 spam messages with a single good one embedded in it when I just woke up before I had my coffee is not a good thing ;)
At the very least, if you were given the same data as these tests, that would be true. Consider if you *didn't* use popfile - how many spams would you be deleting every day, and how many good messages would be accidentally deleted? I know that if I had to manually delete the two or three hundred spams interspersed with good messages, my false-positive rate (the percentage of good mail I accidentally deleted) would skyrocket.
So just be glad you've got popfile. Not only do you not have to go through as much spam, but you're also more accurate while going through the little you must.
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
I need a filter to prevent anti-SPAM hype from getting into my brain's inbox!!
If you have no spam filters, then classifying email amounts to "delete, delete, delete, delete, down-arrow, delete, delete, down-arrow, delete, delete, whoops!" That one mistake just dropped your average to 90%. Frankly, I'm amazed humans scored as well as they did.
Results of new spam filters cannot help but to be bogus... The true test of a filter is how well it works *after* all the spammers know how it works and try to circumvent it.
Enjoy science fiction? "Turing Evolved" - AI, Mecha, Androids and rail-gun battles. What more could you want?
Don't care, as I don't use IM...
In walking, just walk. In sitting, just sit. Above all, don't wobble.
-- Yun-Men
Hasn't anybody noticed the obvious Spinal Tap reference?
...you couldn't hear the...
Jeanine: You know, it might have been better if the, uh, album had been mixed right.
David: Well I suppose you could cry about that, of course it's true. I mean it's true.
Jeanine: It wasn't...it was mixed all wrong, wasn't it?
Nigel: It was mixed wrong?
Jeanine: Yeah....
Nigel: Were you there?
Jeanine:
Nigel: How do you know it was mixed wrong?
David: But she's...she's heard the...she's heard the record.
Jeanine: No, but I've heard the album.
Nigel: So you're judgement is that it was mixed wrong.
Jeanine: You couldn't hear the lyrics all over it.
David: You don't agree that you can't hear the
vocals?
Nigel: No, I don't. I do not agree. No.
David: Well I think maybe....
Nigel: It's interesting that she's bringing it up.
David: Well she'd like to hear the vocals.
Nigel: I mean it's like it's me saying, you know, you're using the wrong conditioner for your hair.
David: Don't be stupid.
Jeanine: You don't, you don't do heavy metal in
dobly, you know, I Mean...it's
Nigel: In what??? In what???
Jeanine: In dobly...
Nigel: In dublin!?! What's that?
David: She means Dolby, alright? She means
dolby, you know? You know perfectly well what she means.
This spam filter goes to 11!
DSPAM implements a Dolby-type noise reduction algorithm called Dobly
Despite the musical reference on the DSPAM site, I figured some people still won't get the joke. So here it is:
JEANINE: You don't- you don't do heavy metal in dobly, you know, I mean...it's--
NIGEL: In what??? In what???
JEANINE: In dobly...
NIGEL (GRINS): In doubly!?! What's that?
DAVID: She means Dolby, alright? She means Dolby, you know? You know perfectly well what she means.
--from the movie "Spinal Tap"
A CRM114 plugin for SA is available, thanks to Devin Nate:
d =2 301
http://bugzilla.spamassassin.org/show_bug.cgi?i
I find it interesting that an algorithm that was originally for image noise reduction found it's way to Machine Learning through a company whose purpose is to impliment noise reduction in audio. From my Googling, I think this is the first time anyone has used Baysian Noise Reduction in Machine Learning. Does anyone know otherwise?
ObSpamError: I almost deleted an e-mail from a friend with the subject "Free Beer". It was only because no spammer could guess that well that I took a second look at the sender.
What I would like to know is, if the spam filter is more accurate than humans, then by definition, how have they detected 1 misclassification? maybe there were two misclassifications and they detected only 1? by definiton, they are worse off than the filter itself.....
Best phrase I've read all week... Oh, yeah, it's only Monday! This one will probably hold me over 'till Friday, though. ;-)
If humans don't have 100% accuracy, who/what is defining what spam is?
We are after all, what we eat. I feel sorry for the guy if he's read so much marketing-speak that he writes like a spammer now...
client/server-side filtering does NOT solve the problem!
The biggest problem with spam is the invasion of third party computers on the Internet. The ILLEGAL activity spammers perpetrate by breaking into machines, forging headers and hijacking servers.
Any filtering method does not address this most serious problem, and even if you do not see any spam in your inbox, you're still paying for the bandwidth and system resources these spammers steal.
Stop with the filtering algorhythms and take some of that energy and contact your local Attorney General, DA and FBI and demand that they prosecute these people who are BREAKING THE LAW.
It always humors me when there are any stories about spammers on Slashdot. I think that the "War on Terror" is, in some ways, similar to the "War on Spam". Terrorists want to kill us; spammers want to spam us. Terrorists and spammers use unconventional warfare. People don't like being terrorized, nor do they like being spammed. Both are unwanted facets of humanity. What I don't understand is why everyone wants to fight spam using the legal system (fines, penalties, law suits, etc.), but to combat Terrorism we must "win the people over". If God has an Inbox, he should get some filtering software to send all Muslims to the Trash Can. Hahahahahahahahaha. "Are you sure you want to delete this Muslim?" "Yes." LOL OMFG ROFLMAO
Not that all Muslims are Terrorists, of course, but come on! Randomly checking old grannies in wheelchairs at the airports? WTF?! Get a Motherfucking Clue (TM) and just check out every Arab-looking guy! Also, the French are not welcome.
In Slashdot, the moderators filter you!
MOUNT TAPE U1439 ON B3, NO RING
by the whole 0.16%!!!
(the first small step toward fame, I hope)
I agree with you, I expected Humans to rate an even 90% at best...
True. In fact mine reads and replies to Slashdot posts for me.
I know a guy who has a Korean grad student who doesn't speak English very well. He manages to produce subject lines for the messages he sends that get him blocked by spam filters nearly all the time. Not his fault really, but it happens.
-"It seems like you're trying to exploit a security hole. Would you like help?"
don't forget that sorting through over 6000 emails in one sitting is bound to be tiring for a human... there's bound to be some misclassification as the human gets bored and frustrated... the computer can easily sort through 6000 emails with the same attention to detail at the 6000th email as on the 1st... not so for the human
People are jackasses.
Hence we have spam in the first place.
KFG
Don't worry, I can forward you the one she sent me. Sounds like the same email.
I remember when legal used to mean lawful, now it means some kind of loophole. - Leo Kessler
I work with some people who use their computer every single day. Have had an email address for years, who still buys what they read in an email. Photoshop for $50...sure! Herbal viagra...why not?
Well, she always has a big smile on her face, maybe there's something to this spam thing.
"Wisdom is not a product of schooling but of the life-long attempt to acquire it." -Albert Einstein
Most people don't recieve hundreds of pieces of junk mail everyday. Spammers can make money with only a VERY small percentage of the recipients actually responding. If you send spam to a million people and only 0.01 % buy your product you still sold 100 units of your product. If it cost a tenth of a cent to send each email then you would need to make at least $10 per unit under the current economic model to have it still be profitable.
Mine is smart enough to log in first, and is patched with humor-1.22-irony.wit :-p
Hammer of Truth
Not to sound like a litigation whore, but ...
I wonder if it would be possible to sue these spammers for interfering with a business transaction. Granted, the amount in question here is minimal, but just the possibility that a spammer could be found liable for this might deter some of them.
If that doesn't work we should sign up every megacorp CEO on every spammer list possible, and hope s/he misses an important memo costing megacorp millions. Then megacorp could sue spammer into oblivion.
Lots of people don't know what popups are.
Uh, sure they do. Popups - that's like those porn storms, isn't it? Some people say it only happens with IE and Windows, but I talked to my service provider and they told me 'just pull the power plug out of the wall when that happens'.
Easily fixed.
That actually makes humans much more accurate. We can eliminate many of the messages just by looking at the subject.
The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?
Karma: It's all a bunch of tree-huggin' hippy crap!
Pretty much how you'd expect: by making fewer mistakes.
This seems a little like suggesting that a sewing machine isn't more accurate than a human seamstress, because it doesn't *know* about its misstitches. Comprehension isn't the issue. I have deleted legitimate email as spam by knee-jerk, and the math above (which leads to 1 in 625 deletions as in error) seems perfectly legitimate to me.
Or are you one of those that thinks that mathematical models aren't useful if they don't have a deep love for and brotherly understanding of what they're approximating?
StoneCypher is Full of BS
Saying an algorithm is x% accurate is not sufficient, because there are two kinds of errors: false acceptance of spam, and false rejection of non-spam. Personally, I'd settle for 90% false acceptance if I knew the false reject rate was 100% rather than have a program that was 99% at both.
Lots of people read some spam not knowing what it is. To these people, a computer is merely an interesting string of sensations.
Can you release this quote under the GPL, do you think?
Yes but your computer has the time to read through a hundred spam emails to get to the 3 non spam emails. Do you?
Also, I wonder how many people have actually looked at CRM114 and tried to use it.
The really interesting thing about CRM114 is the windowed polynomial hashing technique used although there's some evidence that it can work just as well (if not better) on a much smaller window of only two tokens. I'm hoping someone will do a full exploration of the idea for SpamAssassin's Bayes module.
Comment removed based on user account deletion
If we humans are only 99.84% accurate, then 0.16% of the time we will incorrectly think the email is real and buy viagra ? I don't think so.
I read the email and delete it. Exactly the same as the spam filters do it, only MORE accuratly. I think the tests applied would have been between a human reading the header of an email and deciding whether to open it or not verses the spam filter making the decision for us. BUT the spam filter makes its decision by opening the email. Therefore to have a proper comparision I should be allowed to open the email as well before I make the decision. Therefore I am 100% accurate.
if
I bet it allows messages from General Jack D Ripper or any email that contains the secret phrase "purity of essence", "peace on earth" or "precious bodily fluids".
-- http://www.swcp.com/~hudson/
Presumably they must use a superhuman who has 100.00% accuracy.
Karma: It's all a bunch of tree-huggin' hippy crap!
The thing with spam is that it's supposed to be a way for somebody to make money... i.e. they are trying to sell you something, be it directly or indirectly. I can't think offhand of an email I have recently received that could be misconstrued as trying to sell me something. From that simple viewpoint, spam can never look exactly like regular mail, because it has a different purpose.
Before I used a spam filter, I once missed a very important message whose subject line was something to the effect of "URGENT - DON't REBOOT THIS MORNING." That was a bad one to miss.
Of course humans make mistakes, and it is entirely possible for an automated or semi-automated system to be more accurate than a human alone.
Maybe it's the %0.16 of people who are responding to spam.
What do I want in my inbox? I get a few dozen "job opportunities" a day. I'm unemployed right now, yet I've still learned to dump the majority of those without looking at them. Sometime their might be a legitimate opening in one of them and I will dump it. Making me less than 100% accurate, because I deleted an email that I didn't want.
Filters at least get most of the spam I get. (In fact most of those opportunities are things I signed up for not realizing they were not only bogus, but also gave no [obvious] way to get off their list) Back when I got 100+ a day I went through my inbox with the big delete button. Most than once I hit delete, yes I'm sure..., then looked up and watched something I think I wanted disappear. However when you have 110 new emails and 100 are spam, I don't have the patience to go through and read them all.
Since the MIT Spam Conference took place, I've been wondering if new ideas would be implemented as a result. And low and behold, not one but two innovative new approaches to Spam filtering!
This is more than I had hoped for. Thanks to all involved!
I want to avoid terms like "war" or "arms race", but it's good to know that every once in a while the "good guys" take a big step forward. Hopefully the "bad guys" wont catch up to quickly.
Cheers,
Andre
If we are to believe Noam Chomsky or Steven Pinker (not saying I necessarily do), the study of linguistics shows that language is innately human and at present no mathematical formula can interpret its meaning.
The Sparse Binary Polynomial Hash is a generalization of the Bayesian filter and as such cannot detect "spam" any more than any group of people can all have the same interpretation of Shakespeare or Milton.
As Supreme Court Justice Potter Stewart said about pornography in "Jacobellis v. Ohio" (1964) -- "I know it when I see it." Neither humans or computers can define spam any more than they can define pornography.
Considering that 5% voted for >100% in the recent poll.
To these people, a computer is merely an interesting string of sensations.
If only FuFme weren't jokeware. :-(
Karma: It's all a bunch of tree-huggin' hippy crap!
If each one is near-perfect, and they use completely different strategies, why don't we parse all messages through one first, and then through the other one, that way you'll get higher accuracy rates. Only the ones where there's a discrepancy get shown to the user.
__________________________________________
Take comfort in your ignorance.
Grandmaster Plague
Maybe it's because those humans were braindead /. moderators
I find that this parallels my own experience to a great extent. I have found that using a good spam filter with a whitelist is indeed more accurate that my own ability to filter email manually.
Once I reached the conclusion that this was the case it made a heck of a lot os sense to use a spam filter.
I have never deleted an email I meant to keep, therefore I must be a SuperGenius(TM W.E.Coyote)!
Or your dad is an idiot who doesn't know how to route his email.
CRM is actually quite a acinating product. It's like a super grep where you can match against blocks of text instead of just lines. It also has some logic operators and such. I think there is a quote on his web site that refers to it as "grep bitten by a radioactive spider" and it's true.
You can use it for lot more then spam processing, it's a really neat all purpose tool.
The best way to support the US war effort is to continue buying American products.
In addition, every user will get special discounts on software, mp3s and computer parts with my partners, and two FREE MP3'S every month.
There are also special savings on 100% all-natural and effective male enhancement products. A portion of the rebates will go towards a $100000 fund needed to get 100,000,000 dollars (ONE HUNDRED MILLION DOLLARS) from Liberia into an account in Switzerland. If you provide your social security number (SSN) and your checking and savings account number you will get part of the ONE HUNDRED MILLION US DOLLARS. Only the first 100 people will qualify, so hurry up and don't miss this offer!
--I am Sun Tzu of the Borg. Resistance is feudal.
If there is no universal bottom line of what Spam is, we can never manage it.
I think 'unsolicited request for money from a for profit oranization' will fit into everybodies base definition. Some people will expand on it, but we need a defined place to start.
The Kruger Dunning explains most post on
No, humans aren't 100% and yes, you can test for that. Try a thought experiment: fill a bin with 50,000 red balls and 50,000 blue balls. Ask a human to sort them all. The result probably won't be 100%, but you can still check the result and figure out how accurate the human is without relying on a superhuman ability to tell the balls apart. Same thing for spam: if you start with a known training set, you can test humans to see how well the spam is identified by manual sorting.
I'm not surprised a filter beat the human, considering the study used a sample of 5849 messages. As the sample size increases, the filter's accuray will increase, and the human's will decrease. Furthermore the higher the spam/real ration, the better the filter will do in comparison to a human trying to sort at a reasonable speed. The reason being humans tend to skim, and rairly actually read entire subjects, much less messages. Give a human 5000 messages and an hour and he will probably make some mistakes. On the other hand, in 10 messages, the human will probably be 100% correct. Most email filters rely on this already, as they tend to err on the side of caution. With the bulk of the spam taken out, it is not a burden to have the human check the iffy bits. Furthermore the type of email can mislead humans. A business-type email sent to someone's personal email is much more likely to be mistaken as spam, and vice versa. The main disadvantage of automated filtering is people generally have an idea of when a really important e-mail is going to come (the type that false positives are completely unacceptable) and who it will be from.
(please don't reply telling me how)
Kinda makes you wonder how they could get from San Diego to Los Angeles in under three hours. Please don't reply telling me how.
(Extra joke for southern californians: wait for the first person to say "the highway!" Laugh, rinse, repeat.)
StoneCypher is Full of BS
When the Markovan is rockin', don't come a-knockin'...
Ah yes. Misspelling an adjective describing the general procedure, and *then* mistaking that misspelling for the actual *name* of the algorithm. Always nice to see the intelligence of submitters at work!
sorry dude, but i work at Dolby and well, this is probably trademark infringement. I had to send it to our trademark/IP folks. You will probably be hearing from someone soon.
Again, nothing personal, but we have to look out for this kinda stuff. Trademark is a big deal here.
At best, they may just have you take that logo down, at worst, you will have to change all of it.
Yeah, I screwed up the numbers. So sue me, I'm dyslexic.
KFG
How do you know your training set is correct?
The last time I checked, the plural of corpus was corpora (or corpuses), not corpi. CORPI?! What was he thinking!
Its pretty difficult to set up.
Like, it refers to setting a secret password somewhere, but I can't find it!
Gah!
*bashes head on wall*
I don't think you need to worry about that. Your right to quote that phrase should be protected by fair use rights. Even RMS doesn't release quotes under GPL.
The post quotes "a study" which gives the 99.84% figure. In fact, the 99.84% figure is mentioned in the one paper as "the human author's measured accuracy as an antispam filter...on the first pass". This is what we who understand statistics call "nonsense". An individual human had an estimated accuracy of 99.84% when looking at one particular sample set of data, once. This is not a meaningful number, and it sure as heck ain't "a study".
I have never deleted an email I meant to keep
How could you possibly know? You deleted it!!
Write boring code, not shiny code!
Then megacorp could sue spammer into oblivion.
Or more likely, megacorp fires it's mail administrators for being incompetent and goes on about it's business.
All who are harping on about human spam detection rates, the article states:
"By comparison, a human
is only about 99.84% accurate in filtering spam and nonspam, so any of these filters
is more effective than a human "correspondence secretary"."
So, they define "human" to be a secretary, not an uber geek.
I would love to rtfm, but I want a fairly simple answer to this, how can I do a 30 minute job of integrating this into the mozilla mail client, or does it have to be tied into the server itself? I was wondering if this was a quick, easy fix, or if it is an all weekend type of project. While I'm on the subject of mail, what is a good all in one mail bundle with webbased interface that isn't opengroupware or ms exchange for php/apache under unix?
Sig: I stole this sig.
DSPAM does pretty well. As of version 2.8.1 if you receive at least 10% good email it'll happily classify most spam away. As of version 2.10 it should be possible to lower that to near 0 (2.10 implements whitelisting, which saves the extreme end cases where certain addresses (such as john@anydomain.com) receives on the order of 1 good email : 1000 spams.
Can anyone explain how these two filters work?
The white papers are far too complicated for simple minds like mine.
I talked to my service provider and they told me 'just pull the power plug out of the wall when that happens'.
Ok, now the screen dimmed a little and I heard the hard drive spin down, but the pop ups are still a comin! Oh, and something about "battery level at 98%" or something.
I'm not a journalist, but I play one on slashdot
Hrm, I just installed this, and I'm anxious to try it out.
I'll just post my email address here, and wait a few minutes...
The humans are making errors like accidentally hitting the wrong button. The filters are accidentally classifying things as not SPAM. These are not the same kinds of errors. At that level of accuracy individual humans cannot be taken as good baselines.
Or your dad is an idiot who doesn't know how to route his email.
But I was only contesting the great-grandparent poster, who said that humans are by definition 100% accurate.
While my dad may be an idiot, he is also human. I am correct, great-grandparent poster is incorrect, and you are off topic. As far as I can tell, I've never deleted an email I meant to keep either. But you and I aren't the only people worth discussing.
There are no trails. There are no trees out here.
There goes my bussines idea. I wanted to start a bussines that puts humans in an eastern europe contry to sort corporate e-mail.
Now I have to think again about putting humans to decorticate sunflower seeds, it's cheper than all those machines.
The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?
I believe that humans can be 100% accurate (or thereabouts) if they read the *ENTIRE* message, however that's exactly the point - if you have to read an entire message to tell that it's spam, the spam has succeeded.
Their number probably concerns how people can tell without reading the entire message whether or not the message is spam. My brother accidentally deleted a few messages I had sent to him, however if he had read them fully he would have known they were legit.
Cheers,
Justin
Ever run a car into a tree?
Me neither, but somebody has.
And I bet they thought staying
on the road was as easy as
deleting spam.
If every individual human has an accuracy of 99.983%, then two independent humans have an accuracy of 1 - .00017^2 or 99.99999711%. This would allow ample accuracy to judge the computer, except that it's not true[1]. A better answer is the one you suggest: humans must judge spam from subject/author alone, whereas computers get to look at the whole message. Humans reading the whole message, and possibly even following included links, responding, etc., can be assumed to have full accuracy, within epistemic bounds. Indeed, merely re-checking your work, etc. - being consciously more diligent than the average spam-sorter - should insure your accuracy is better than average.
As for how accuracy was actually judged in this particular study, I suppose you would have to read the article for that. I haven't, myself...
[1] It assumes the probability of error is equal for every message, which is obviously not true (i.e., that error is random rather than systematic). The real accuracy of two humans in concert is surely much lower; OTOH, it is still sure to be much, much higher than the accuracy of a single human.
Huh... so would three spam filters be 100 times as accurate? I never thought of running them in series before. Cool! :)
But the computer reads the entire message, so it's not really a fair comparison, is it? How many more lines of information was the computer allowed to look at to create its superior result?
Karma: It's all a bunch of tree-huggin' hippy crap!
I'm glad these guys are doing great things to combat spam, but when the submitter of the article stands to benefit from posting of the article on Slashdot, then full disclosure (not stealth disclosure) is warranted. No surprise that the "donate" link is right up at the top of their page.
Jonathan, don't get me wrong. I really appreciate what you're doing here. But failure to disclose your relationship of the project you're promoting is on the level (though not the same extent) as the deception that spammers employ.
-S
--- What parts of "shall make no law", "shall not be infringed", and "shall not be violated" don't you understand?
Current spam filters may be "10x" better than humans, current spam filters may be terrible on future spam.
Filters beating spam and spam beating filters is a continuous arms race. In the limit, optimal spam filtering is equivalent to solving NLP (natural language processing); Unless you build a filter that can fully understand the text (syntax, semantics, pragmatics, world knowledge, the whole shebang), an adversary can always construct spam to defeat your filter.
Feed me, Mandrake!
You're missing the point! Humans aren't 100% accurate on classifying JUST on the subject and From lines, but I don't see how we aren't when reading the body.
You mean you've never noticed this before? Idiots are some of the happiest people I know.
Lots of people don't know what popups are.
Yes, we call those people "surfers who don't use Internet Explorer" (seeing as pretty much every other browser has options to kill them).
SCNR
I'll just stick with my 99.9% accuracy with Popfile, it acts as a proxy so everything happens seamlessly, and the buckets are awesome for sorting your mail...
How long was the training period? Does that count too? Many filters can maintain complete 100% accuracy over finite periods of time (4,000 - 10,000 messages ) once they're trained - such as...
Which one is better? Does anyone have any comments on the actual article?
-Gonz
ObKubrick: In 2001: A Space Odyssey, one of the pods was marked with the designation CRM-114. And in Clockwork Orange, Alex is injected with serum 114. I suppose CRM-114 is to Kubrick as THX1138 is to Lucas.
Dobly, on the other hand, is from This is Spinal Tap , a mispronounciation of "Dolby" by David St. Hubbins's girlfriend:
Not to mention that it probably avoids trademark infringement (though I wouldn't put it past Dolby Labs or Thomas Dolby to raise a stink).
Maj. Kong
Shoot, a fella' could have a pretty good weekend in Vegas with all that stuff.
I totally agree. I reached that threshold today, having more than 1000 messages, with only 1 non-spam for about 4 days worth of email.
Also, I think that the articles' ratio of spam to non-spam 1935/5849=33% was way out of proportion.
Specifically, for the period of November 1 to December 1, 2002, a pre-trained mailfilter.crm (as distributed as CRM114 version 2002-11-26 ) processed the authors live incoming mail stream, a total of 5849 messages (1935 spam, 3914 nonspam) , with only 4 false acceptances, 0 false rejections, and 2 messages considered "not humanly classifiable".
My messages to spam ratio is more like (100 messages) 98 spam, 2 nonspam 98/100=98% at the most. My question is, how does the author keep his spam ratio so low? If you update the study from 2002 to 2004, I'm sure those numbers will be vastly different.
My Journal.
However my telepathy skills are not so good so thats not such a great idea.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
if you had asked me, I'd say balls to you!
Good question! We're working on this problem, among other things, at the PSAM project. We have a project to produce high-quality benchmark corpora for spam filter testing. Watch that space for ongoing work, or e-mail us an offer to pitch in and help---we could use it!
When these factors are considered, I think it's quite possible to write software that in the long run has a higher success rate than a human who has better things to do than filter his mail all day.
This is an unfair comparison in the first place, unless their spam filter only looks at things like Subject, From, and Date. Their filters in all likelihood also analyze the full body of the message; if a person read the body of the message to do that analysis himself, it would make the process of determining spam/not-spam moot, wouldn't it?
Dolby noise reduction works by filtering a spectrum into a bunch of bands, each of which are compressed (in an audio sense, not in a digital sense), and recorded to tape. On playback, they go through an expander...how does that concept translate to spam filtering? It can't be "dolby-type", that doesn't make any sense...
ZuluPad, the wiki notepad on crack
On our main campus there will be a pilot program to use DSPAM. Although I've heard it requires more user training, Brightmail doesn't need any or very little feedback to block 80% of the SPAM.
...are still the only real solution to the issue of trust, reputation, and accountability on the Internet. We need it for so many other things in addition to guaranteeing email legitimacy.
If every user or at least every server had a key and we all signed each others keys creating a web of trust and only accepted signed and trusted mail the spam problem would be solved. I really dislike the way SSL certificates are handed out. A central CA is a very bad idea due to the cost and browser lock-in issues etc. With GPG and web of trust if you want to run a mail server you need to talk to a friend who is already running one and get them to sign your key. Perhaps we could even use DNS to propagate and cache the keys and sigs. If you sign a key that turns out to be a spammer you better revoke that signature fast before the person upstreeam from you revokes yours. Problem solved. Now if only we could get the big guys to go along with it...
The fact that you know you deleted a non-spam message indicates that you eventually got the message anyhow.
Ergo - you are still right.
This issue is a bit more complicated than you think.
my addy has been on the net since about 1988 and so I get more than 500+ spam per day, maybe even close to 1000 on a bad day.
So many emails say "Hi" or "hey" or other subjects I use. It get's harder to see what's going on esp. if your not using a 'Nix based reader
So I can tell you i'm not accurate in picking out the good ones, esp the newest spam which uses realistic sounding email names. I'd be eager to try out one of these filters.
http://www.hawknest.com/
No, I'm not missing the point.
My dad, and the other guy with my name, BOTH could have viewed the message body. We might be better than machines when reading the message body, but given the 200 spams my dad gets every day, he chooses not to. So it doesn't matter.
Even with an autopreview window: My dad will see 20 new spams, shift-select all of them, and delete without viewing. This is learned behavior for him.
Yes, you are correct: the artificial intelligence embodied in these anti-spam solutions are not more accurate than a human who actually reads the message body. But they are still better than humans at sorting mail.
There are no trails. There are no trees out here.
Insightful?
Where the hell did it say these programs were designed to force you to read email?
Ironically, the word ironically is often used incorrectly.
obviously, because humans can recheck the ones that they thought were spam but the computer thought wasn't, or visa versa. Humans aren't 100% accurate in a single pass. but won't generally get the same thing wrong, pass after pass.
Mine is 100% accurate. If it doesn't have a certain sequence of words at in the topic, at the start or after the Re:, then it's spam.
My question is was the mistake a blocked or deleted "real" message? or a spam thet slipped through. In business, I'm far more worried about false positives than false negatives.
I am become Troll, destroyer of threads
Uhm, the *authors* of the spam are 100% accurate in identifying their messages as *spam*. Likewise for the authors of legitimate messages.
has anyone made windows binaries of CRM114?
What you're planning has already been done, it's called TMDA, and it's not such a good idea. You're going to send out 800 "challenge" emails per day - have you given any thought to how many of those will be genuine addresses, but have nothing to do with the spam you receive because they just happen to be the joe-job victim? These kind of challenge/response systems may slighlty alleviate your own suffering through spam, but at a cost to all those unfortunate enough to have had their email addresses faked. And if the sheer impoliteness of such net behaviour doesn't put you off, note that you're using up more of your own bandwidth to send out such challenges
If any of the smtp exchange or address lookup fails, just forget it, they're probably not real anyway
It would make a lot more sense to make these kind of checks when you're receiving the email in the first place. Reject at the SMTP level - you never accept and process the spam in the first place
My next sig will be ready soon, but subscribers can beat the rush
Well...I suppose if you just say the computer will read so much to be this accurate and you will read so much to be so accurate, it really is a fair comparison because it assumes you both make your judgement based on how much of the email you'll read.
Besides, you could argue in the other direction and say that since the computer evaluates the email in a fraction of a second, you should also..
Twenties Retirement
How would it know if I consider brunettes non-spam but blondes spam? I did opt-in for one of those email categories, but not the other.
With an accuracy rate of 99.84%, humans will miss about 10 out of every 6250 messages. If three people run through the same set of messages, they will most likely get different messages wrong. In the cases where humans disagree, the researchers can take a closer look and determine what is or is not spam.
I do it regularily, deleting 25 spam messages with a single good one embedded in it when I just woke up before I had my coffee is not a good thing ;)
So do I, which is why I have a separate mail folder into which absolutely everything gets saved, always. If I realise I've deleted something, I can always go back and get it.
Helps to have a colocated server with a pile of RAIDed disk in it for that though - once it's on the server it doesn't cost any more to keep a copy.
I dunno. I'm running CRM114 now, and it's taking something like 1.5 seconds to identify emails. I am on a slow machine though, which used to do SpamAssassin at around 4 seconds, and inaccurately to boot. CRM114 is a big improvement, and if it trains well after the first fortnight I'll kiss TMDA goodbye.
Karma: It's all a bunch of tree-huggin' hippy crap!
Yes, but it is meaningfull nonetheless. If you just think that it's very likely that after reviewing 650 messages, you may have missed one email that you thought was spam, then the "study" is right. I don't care if the number is 900 or 400 emails. Those 400 mails are making me lose a _lot_ of time, and if I value my time, I am losing a lot of productivity, and also missing an important email.
.99 accuracy, then it's a real time saver, and if it only makes a mistaque every 2000 emails, then SURELY I will be more accurate than me. That depends of course, on how much spam you do get. I get arround 20 to 1 ratio of spam to real meat, and I get arround 100 spam messages a day. I can't spend 1 hour a day cleaning spam with 99,9% accuracy, so I am forced to quick sweep. This thing could make me regain the time, and the false positives would mean i even make less mistakes than manually.
If the program can have a
The important things is how accurate the antispam tool is, and how accurate I am (ratio of spam to meat, and how much a miss costs me). How much other people make mistaues is not really that important. Everybody knows how much time they have, and how much spam to meat they have, and thus, it's very likely that if they don't have a LOT of time to waste, they will be making a mistake for every 200 to 600 spam messages.
unfinished: (adj.)
One filter good, two filters better.
Good, inexpensive web hosting
There goes my bussines idea. I wanted to start a bussines that puts humans in an eastern europe contry to sort corporate e-mail.
A year into that project you'd get an email: "Progess is good. Spam was filtered. Send more cans!"
The power of Christ compiles you!
I suppose if the comments on it are good, I'll have to try it eventually. At the moment my current emails are only a year old and I'm just relying on Mozilla's filters...which oddly enough have had trouble because I haven't gotten enough spam for them to train on!
Twenties Retirement
I for one welcome our new superhuman overl....
Eh, nevermind...
And that, my friends, is how you get to the magic number--3670 posts--on Slashdot: converse with yourself. No need to create another account, just type your initials at the end of each post and respond to every Anonymous Coward that mentions anything you're talking about. You can even try going back to old threads and posting AC so that you can respond without the fear of someone finding out your dirty secret.
Having such a powerful statistical spam filter is definitely a luxury. I have no difficulty believing the accuracy values presented here. I have had experience with spamprobe, CRM114, bogofilter, spambayes, and spamassassin and all of these do an amazing job to the point where spam no longer exists (for you).
Which leads to me plug a little project called WPBL that uses exactly these types of statistical spam filters to spot spam sources in a distributed fashion. Each project member uploads hourly the IPs they see relaying spam and non-spam, where the 'decision' is made by these extremely reliable filters. This effectively converts your regular mail account into an intelligent spam-trap that feeds a central blocklist.
The more members we get, the better we can identify active spam sources around the world. This information is then used by some sites for quite large-scale blocking. Since you're doing all this filtering processing anyway, why not also share "what you learn" (the IPs that are spamming you)?
If this grabs your interest, read up on the reporting scripts or alternatively, the open WPBL data upload protocol if you want to code your own report generator. Bandwidth usage is minimal.
The way to get a high post count is to post a lot over an extended period of time.
Film at 11.
3671
KFG
That should explain why Dubya's always smiling even when he's trying to be serious.
I say get a bunch of honeypots and do the test again.
.
A human doesn't have to determine if it's spam simply by the title.
The human should have all the advantages these filters have body / header / ip
Cheers
The greatest achievement Spam ever made was proving to the world it didn't exist...
Well if the human was given the chance to read the body text as well like the filters do, then they would be 100% able to delete their own spam.
DRACO-
Consider yourself blessed if you are sneezed on by a dragon and only get wet, it could have been a fireball.
I just tried installing CRM114 with no success. The documentation is confusing to me. Perhaps someone can help me out.
.css files should be in the same directory as
/myscript.php
#1.
From the docs it says:
> In either case your
> your mailfilter will "run" in (yes, this can be changed, but that's
> an advanced topic).
What does this mean? What is the path to my mailfilter? I have qmail on my
system.
#2.
I'm being told to edit mailfilterconfig.crm from the docs. However this file is not found
anywhere in the source folder nor could I find it anywhere on my system
after installing CRM114. Where do I get this file?
#3.
Currently I'm piping my emails to a support ticket script. In the
'/var/qmail/alias/.qmail-support' file I have this line:
|/usr/local/php/bin/php
How would I go about having CRM114 filter the mails and then still have the
mails piped to the support ticket script?
eTrade SUCKS
I think it's not *right* (umm... there gotta be a better word?!) to compare the accuracy of human filter and computer filter, because, in general, human don't act as a filter.
Human defines what is spam. The filter, on the other hand, does its best to classified them based on the information it's given...
Yes, you are correct: the artificial intelligence embodied in these anti-spam solutions are not more accurate than a human who actually reads the message body. But they are still better than humans at sorting mail.
But that's not the point! You could hire someone to read your email and classify as Spam or not spam, and I doubt they would EVER mess up. Do you disagree with this?
THAT'S the point.
statistics.
This headline is misleading. I refuse to RTFA, because I imagine the "10 times as effective" figure comes from the article itself.
Come on, folks. The figures do, in fact, show a 10 times increase in effectiveness between humans and these filters. But what the heck does that mean? I have to question the studies. How did they come up with this 99.84% figure? Does it mean that one person will mis-classify about 16 emails in 10000 (a small number indeed)? Or did one or two outliers taint the data?
The important thing here is that we're comparing three averages. Were the conditions between the trials the same? Were the humans given time limits? Were the accounting methods accurate? Were the spam messages the same?
It's quite possible that these averages were bounded by possible error quantities (they should have been!) and that these were tossed when reporting the numbers to us. This was so that a startling result (10 times as effective as a human) could be shown in a headline. It's all about coming up with a flashy "fact".
It's very easy to make numbers say what you want them to say, so I'd be a little wary of running around to your friends "citing" this 10x improvement figure without doing some deep delving into the processes involved in arriving at the number.
I dont think your math is entirely correct. The whole thing fails because the two persons have to determine and make a combined decision. So, if both persons make the correct guess, that a true spam is spam, it seems logical to think it is.
:)
But if you have two persons, you actually increase the risk that someone makes an incorrect assumption about the spam. Ie, you have two chances of failure. So even if you reduce the probability that _both_ persons would incorrectly classify a mail as spam, you increase the probability that one will. So how do you deal with that situation ?
True spam, A(spam),b(!spam) -> decision ?
True !spam A(spam, b(!spam) -> decision ?
The problem that occurs can be reduced by having an additional person, and a majority voting process in the end. The chance that a majority of voters would misclassify the mail is very low. You could also stay put with your two algorithms and have a "revote", assuming that the algorithm is not statical (ie will come up with the same decision all the time).
This leads us to the key issue. The algorithms are most likely statical. That means
A: You must have two different algorithms in your case
B: Revote doesnt work
So i'd suggest setting up three different spam filters a middleman voting system. Or you could just click "delete" yourself
Probable impossibilities are to be preferred to improbable possibilities.
Aristotele
How am i supposed to increase my penis size now?
I figured out how this works...
as I get TONS of spam... maybe 700 or so per legit email (joking)...
and I save a few of them...
all that the computer would have to do is delete all of them to achieve a higher accuracy rate!
of course that could also be considered a 100 percent failure rate.. but
Please use [ informative / summarizing ] SUBJECT LINES
Flame me here
Well, the math isn't wrong, but there is an unstated assumption, that whenever the two disagree, it will be possible to check and determine who is right. What multiple judges allows is better -detection- of errors, not in itself correction of errors. A majority scheme with an odd number of judges would indeed allow correction as well.
However, you've only actually increased the probability of error in any sense if you count a disagreement as an error - but there is no more reason to count a disagreement as an error than there is to count it as a correct judgment. It is, in fact, distinct from either. A disagreement implies no conclusion, so it cannot be right or wrong.
Be nice if the figures given (and no, of course I haven't RTFA) specified rates for false positives and false negatives. While a human may have 0% false negatives, if you have friends who write dodgy subject lines, you may well have the odd false positive.
It's "delete, delete, delete, delete, down-arrow, delete, delete, down-arrow, delete, delete, whoops!, up, undelete, expunge"
I use Fire as my IM client. If someone wants to go on my whitelist, they have to use a non-IM method to communicate that fact to me. Some people might not like that, but I'm quite happy with it.
I write spam filters for a living, and I promise you that they can eliminate many of the spams just by looking at the subject too.
Of course, so can I. Now, since I write the filter based on my human judgement of what constitutes spam, which is more accurate?
Statistics may be a "pure mathematical construct," but the application of statistics to the real world requires verification that the assumptions hold and the conditions remain unchanged.
While it is technically true that the detection of spam accuracy given 10 messages would on average be very slightly less than 100%, it is very reasonable to assume the accuracy would be higher for people looking through 10 messages/day rather than 100s/day.
The condition of only evaluating 10 e-mails is fundamentally different than sorting through "a total of 5849 messages (1935 spam, 3914 nonspam)" in a month or an average of 190+ a day. Extrapolating the author's result to a much lower message load is unwarranted.
Once Email Spam is eliminated, then IM spam will begin... ...did email spam get eliminated sometime in the 90s, and noone told me?
SPAM goes where the people go... on Usenet, on Email, on WWW, on IRC, on IM, on P2P progs. The person that invented a spam-free service would be a very rich or very popular man, probably both.
Kjella
Live today, because you never know what tomorrow brings
Not comparable. The job of a junk mail filter is to drop things I don't want to read. It is trying ot match my evaluation, not to match a semi-objective criterion like red or blue.
If I read 1000 messages and say which I wish I hadn't read, then I am 100% accurate by definition.
Of course, if they are really talking about a pure spam filter -- ie one which identifies unsolicited commercial email -- then they can be more accurate than me, but at an uninteresting, perhaps even counter-productive, task:
I may get unsilicited commercial email I do want to read one day. Almost happened once (I had inadvertantly signed up for it, so it was not really unsolicited, and I didn't actually buy the piece of kit they had on special offer that week, but was tempted). I also get stuff I don't want which isn't spam (notably email from virus infected machines).
The referenced study seems to be a very sloppy job from this POV. They don't define what their criterion of sucess is, and to the extent they put in a hand waving attempt it is clearly nonsense:
`Unsolicited' does not imply `not desired'. If they don't tease those two apart, they can't get interesting results for real world applications. Eg, someone mailing my work address with a commercial proposition may well be a very welcome unsolicited commercial email._O_
.|< The named which can be named is not the true named
I don't know what all the fuss is about with this spam thing. Sure it's a pain responding to all that email, but now I've got a 10 foot penis!
Remember; one thing computers are good at is doing boring things repeatedly.
Does that mean computers/robots will never learn to masturbate?
"We have a project to produce high-quality benchmark corpora for spam filter testing"
So... people are selling the contents of their hotmail accounts on ebay as "spamfilter test data"?
a horrible place
Yes, I do disagree with you. Because eventually, that person you hired would learn that they didn't need to read the text of some of the email, and they would start deleting messages without reading them (or without reading them carefully).
People make mistakes.
There are no trails. There are no trees out here.
puts humans? as as opposed to what, donkeys?
We don't need to trust the *person* sending the mail. It would be sufficient to trust the machine that is doing so.
Look at http://spf.pobox.com/ which is sufficient. With SPF, you know that if you are getting SPAM saying it is from @ultraviolet.org, then it really is from @ultraviolet.org (or at least someone who ultraviolet.org trusts).
Your solution requires a certain level of technical proficiency (setting up and managing the key) of *all* participants. SPF's solution only requires technical proficiency from those who manage DNS settings and those who manage email servers (in particular the person who manages your email server).
Also, what about *stolen* keys? And who handles key checking? SSL certificates are restricted to a few root signers, but you don't want a central certificate authority. PGP/GPG work well because they only involve small numbers of people. In general, you know the person directly. Occasionally it will be a friend of a friend message. What do you do when the chain is 10 or a 100 or a 1000 keys long? How long will it take for you to find out that 978 has since revoked their signature for 977 (counting in steps from you, i.e. you are 0 and 1000 is the original signer of this chain)? Or how long will it take you to verify all 1000 keys if you try to do it real time (i.e. when you get the message)?
If humans are 99.84% accurate and these filters are ten times as accurate, wouldn't that make these filters 998.4% accurate or am I missing something?
I can just imagine:
"Please take a seat here.
Thank you for volunteering for the 'Spam Deletion Turing Test'.
We would like you to delete any spam that comes into this in-box...
mail will be arriving at around 10-20 messages per minute."
5 minutes pass
10 minutes pass
"Sir? Could you tell me how long this will take?"
"Another 6 and 1/2 hours."
o-O
You quitting proves that the karma kap worked. The most annoying of the whores shut up. --CmdrTaco
And he's never cared, apparently.
Can you truly miss something you never had?
Never confuse volume with power.
Anyone know how I could use either of these with my copy of MS Outlook or Exchange 2000 server at work?
What would happen if a spam filter were applied to /. comments?
-- When did Ignorance Become a Point of View?
Why the hell would you have a cyber-sex chat log mailed to you?
Much like the rule stating there are few things less funny than a "funny" IRC log you did not participate in, I can think of few things more sexless than a cyber-sex chat log received via e-mail...
The Humblest Mollusk on the Net
...however if he had read them fully he would have known they were legit.
....
Maybe if you didn't start off your emails by talking about cheap online drugs, work from home make $!000, Discounts for Diabetics , Can I Enhance it bigger, Esgic D1scounts, Ch.eap Microsoft Adobe Corel Autodesk OEM software for sa.le! dcunrisnjt kld
Then he wouldn't have to read them fully
Sorry got carried away, I jsut started cut and pasted subject lines from my spam folder... How does any of that look like real email? dcunrisnjt kld???
DJMD - The fourth man - Planetary
I thought SPAM was unwanted bulk email. So you are saying the programs are 10 times more accurate in knowing if the email is unwanted by me than I am?
. .
M ACHINE: Move along
Or are they saying it is 10 times more accurate in identifying bulk email - whether it is wanted or not?
Wanted bulk email != SPAM.
What I want is a filter that round-files anything that "calls home" or otherwise sends information when opened. I can slog though the rest and make my own determination as to whether it is wanted or not.
MACHINE: This email is like other email you did not want, so you don't want it either.
ME: Wait a minute, I DO want to read that.
MACHINE: No you don't. I am more accurate than you, so you don't want to read it.
Me: Yes...master...I...don't...want...to...read...it.
MACHINE: These are not the 'droids you want.
ME: These...are...not..the...'droids...we...want...
ME: Move...along...
Acts of massive stupidity are almost never covered by warranty. --me.
if you have to read an entire message to tell that it's spam, the spam has succeeded.
I thought the spam succeeded if I actually sent money to nigeria...
Are you telling me that they profit from me wasting my time? Gasp!
http://use.perl.org
Check the spam tools here
If humanity defines what Spam is, how can a machine be better?
A blog I run for the wealth
Hell I can tell 100% of the time when I have TWO blue balls. I'll bet you a million bucks that I would be 100% accurate if I had 50,000 of them! OUCH! Damned dick-teases!
---- It puts the lotion on its skin or else it gets the hose again. It does this whenever it's told.
the average human is only 99.84% accurate. Both filters are reporting to have reached accuracy levels between 99.983% and 99.984%
So to be ten times more accurate, the filters would have to be 998.4% accurate.
"My spam filter gives 110%"
The further question is, if humans aren't as accurate as the computer, how are they measuring the accuracy at all? That is, how do they know that the 1 in 6250 messages is wrong, if a human, known to be inaccurate, was testing for accuracy?
You're joking right? I give my students math problems all the time and they regularily give me back inaccurate results. I can still correct those results. Yet, I also make mathematical errors from time to time. Sometimes, I can even correct my own results. This is a normal part of testing.
I have a lot of opinions about Cyborgs and Architects
I've met this guy at a few parties. Cool chap, lots of fun, easy going.
He told me about the "Beyond Baysian" filtering they were putting in place a while back.
The name comes from "Dr. Strangelove". Gotta love it.
"Live Free or Die." Don't like it? Then keep out of the USA
How can a spam filter be more accurate than humans?
>>>>>
Filter out 100% of their email and ignore the false positives?
That and the fact that ever so rarely, you cannot rule out something as spam by the subject alone, and thus must click on it (though this isn't quite fair, since the computer reads 100% of the email, and you only read those that you're not sure to be spam...)
People buy products from spammers, hence why there continues to be spam - it's profitable.
If people buy products from spamming vendors, it follows that people aren't 100% effective at determining spam. Yes, you could argue that if the person was interested in the product, then it wasn't spam to them, but when they get that bogus penis enlargement supplement that doesn't work, and THEN realize that the message was for a product they really didn't want (because it doesn't work as advertized), you can see then that they made a mistake that a spam filter could have saved them from. If every spam product worked as advertized, I think that very little of it would be spam since almost everybody would want these miracle products (especially since it seems that all of the penis enlargement products have the handy benefit of enlarging breasts if consumed by females).
Hence one of the reasons that a spam filter (human or otherwise) can't be 100% accurate, as it can't accurately predict if the product will work as advertised.
Of course, this is heavily dependent on your exact definition of spam. Does spam=UCE? Does spam= ANY unsolicited email, commercial or otherwise? Does spam = anything you don't feel like reading, solicited or not? Or one of myriad other possible definitions. Most spam filters define spam as either UCE (word lists) or things you don't want to read (bayesian). But what about the bad news that your brother has died? I didn't want to read that, but I should, and so I would want it to get through. So in effect, what spam filters are TRYING to do is sort out what you SHOULD read from what you SHOULDN'T read, regardless of the combination of commercial, solicited, or pleasant. And for the reason given above concerning ordering a bum product, since many people are gullible and/or unknowledgeable, there are many things they THINK they should read that they shouldn't (or vice versa if they get overzealous when deleting what they think is spam after seeing only the subject and sender). Spam preys on both of those (gullibility and lack of knowledge in a particular area) to entice people to buy products that aren't worth the asking price by a long shot. A filter might be able to do a better job of weeding these out than the average person (i.e. a person might buy a penis enlargement product or believe the Nigerian scam, but the filter would know better), but neither is ever likely to be 100% (the filter might let through something you subscribed to that offers to sell you a legitimate product from a legitimate vendor, but still turns out to be a lemon, or even fail to let through a TRUE deal). In some cases you can consider a spam filter to be like an expert system, providing expertise in email deception practices to the average computer user that doesn't even know that "From:" headers (among others) are easily forged.
Sorry for the rough edges - I'm rambling a stream of conciaousness without editing due to lack of time (gotta get to work!)
Oh, was that my outside voice?
Well you could develop RDF Schema for porn spammers that would include attributes as hair colour, nationality, age, soft, hard, images, video, etc. on the other hand you could just go to a password site and get the same thing for free.
Any sufficiently advanced man is indistinguishable from God
Ok, I think we'll just plain disagree here then :)
I thought this was "insightful". A program can't be "100%" accurate in identifying spam unless it knows _everything_ about the user.
If you correct all your own results, then you are 100% accurate in the end.
Karma: It's all a bunch of tree-huggin' hippy crap!
"4, Informative?"
Goddamnit! This is spam!
Obviously, the moderators are not superhuman.
you almost had it. here's the procedure for future reference.
SUBJECT=`subject of the story`
ACTION=`change $SUBJECT is subject to`
SUBJECT=`strip_definite_article $SUBJECT`
ACTION=`third person singular case of $ACTION`
JOKE=`echo In soviet russia, $SUBJECT $ACTION you!`
In building these "corpora" of spam, aren't you trying to hit a moving target? Spammers are always evolving their techniques to avoid filters; as the identifying characteristics of "spam" therefore change constantly, I fear that your corpora will result in filters that do an excellent job of blocking last year's spam.
That that is is that that that that is not is not.
Part of what we're trying to do is establish a methodology for semi-automatically building good benchmark corpora. So ideally, if we think the spam stream has changed substantially, we should eventually be able to mostly just stuff in more messages, push a button and get a good current benchmark corpus. At least, that's the ideal.
In any case, we believe that everyone should be running filters customized for their personal current ham and spam streams. Our corpora are not intended to be used as training data for your filter. However, they should help you estimate generic properties of that filter, such as quality of its learner.
> but can you identify spam before opening it 100% of the time?
Not with certitude, no. About 2% of the time I have to look at the message
body to be sure. Nevertheless, this nonsense about humans only being 99.8%
accurate is based on the *average* human, and that figure is dragged way down
by a relative few who lack any kind of discernment at all, and a somewhat
larger minority whose accuracy is less than what it ought to be because they
are careless.
Cut that out, or I will ship you to Norilsk in a box.
The "right" and "wrong" in this situation also geatly depends on morale applied when judging his action.
hany
Most of my spam (4878/4891 = 99.73% so far this year) arrives at one e-addr which I no longer use. [The rest (13/4891 = 0.27%) has arrived at another non-existant e-addr (looks like a message_id) - probably used by one spammer.]
None of the spam arrives at my active e-addrs - as soon as it does, I've got a sure fire moan at the source: each sign-up that requires an e-addr get an individual, traceable one. (The only problem is my main, family one: if that gets spammed, it implies that it's been leaked by a friend, possibly by a worm/virus.)
A rose by any other name would smell as sweet;
A chrysanthemum by any other name would be easier to spell
so, you're only 98% accurate? You're below average and dragging it down. Get with it.
I write code.
> so, you're only 98% accurate?
No, but I achieve my accuracy by knowing when the headers alone aren't
enough to be sure and examining the body in those cases.
Cut that out, or I will ship you to Norilsk in a box.