Slashdot Mirror
Response to Gordon Cormack's Study of Spam Detection
Nuclear Elephant writes "In light of Gordon Cormack's Study of Spam Detection recently posted on Slashdot, I felt compelled to architect an appropriate response to Cormack's technical errors in testing which ultimately explain why one of the world's most accurate spam filters (CRM114) could possibly end up at the bottom of the list, underneath SpamAssassin. I spend some time explaining what is a correct test process and keep my grievances simplified about the shortcomings of Cormack's research."
I usually frown when I see many of these so called studies offering conclusions, several of which differ radically from my own experience. There recent Java/C++ performance one was a classic example. It gets annoying when a pro MS result is immediately decried as marketing FUD because it just cant be better and a pro Linux result is taken gospel truth here on /. Usually I tend to take all results with a grain of salt or just plain ignore them and focus on the debate around them.
The benifit of these studies though is that fantical crap aside informed people will usually take the time to interpret results or suggest corrections/improvements that actually benifit developers and improve their knowledge base more than any information provided by the actual study.
Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
What truth?
There is no dupe
This guy seems a little harsh and just a bit jealous of the success of Gordon Cormack's article. I'd like to know what makes his opinion any more valid than Gordon's.
Information on his professional career was very hard to find on the site.
This just seems like a flame because his software(dspam) didn't perform well in the test.
Haven't you ever Googled something? Haven't you ever input data into a computer? (The use of the word input as a verb is, of course, the result of verbing, and it's now considered acceptable usage.) In recent years it has become common in English to "verb" nouns. In fact, I just did it. English, like any other language, evolves over time.
To deny this fact makes you just another prescriptivist language maven, completely disconnected from reality and any sense of the advancement of human language.
Folks, don't listen to this dinosaur. He's not insightful, he's simply living in the past.
There's really very little to be said in favor of Jonathan A. Zdziarski's "defense". I guess it just amounts to him wanting to sell his product. Of course, I remember when CRM114 first came out, it was subject to some very dubious--or often simply incoherent--claims. It's pretty clear Zdziarski is in quite a bit over his head... not quite as bad as the amateurs who discover their own "breakthrough" encryption techniques, but tending in the same direction.
As near as I can tell (I skimmed, admittedly, I didn't read every word carefully), his defense amounts to "please don't test the different filters because..." Fill in what feature of the test MUST not be the same as the CRM114 users who get 99.95% accuracy. This is precisely the meaning of "special pleading" in rhetoric. Also the same argument about "if only he had tried the latest-and-greatest (even though we made our wild claims before that version came out, too)."
Cormack &alia make a reasonable best effort to test several tools; and as with any test, they make certain assumptions, and choose certain methodologies. Frankly, I find that a lot more useful that "just trust us, ours works best...but we can't quantify what 'works' means."
FWIW, I wrote an empirical study of different spam filters, way back shortly after the Paul Graham buzz:
I know my study is based on quite old tool versions by now. But AFAIK, it's one of the few that actually came at the comparisons from an unbiased viewpoint. Most figures are based on the "experiences" of the strongest proponents of a given tool (or occasionally from a strong detractor). I had/have no agenda for or against any particular tool, I was just curious.
Buy Text Processing in Python
Well said. HOWEVER, I have to agree with the poster who pointed out that using "architect" as a verb in the context of writing is a little out of place. If we're going to help the language grow, let's at least do so in useful ways. "Architect a solution to an engineering problem", sure, "architect a whiny, defensive rebuttal", no. If we're going to make it a verb let's at least have it relate somewhat to the noun.
Actually, I was trying to be Insightful, not Funny.
RTA?
;-)
Read the article, then post!
There's really very little to be said in favor of Jonathan A. Zdiarski's "defence?"
Now, I could start posting how ignorant that statement is, but then I'd just be rewriting Zdiarski's article. Cormack's entire test was flawed - He used SpamAssassin (95% accuracy) to create his 'ham' corpus. He used software versions that were 6+ months old. Even the email address he used for testing is incredibly unique and atypical! (He uses an address that he's had for 20+ years; One that has been posted all over the WWW numerous times. An address that has many forwarders pointing to it. How is that typical in any way??)
Ok, go read the article (don't just 'skim' it, as you mentioned), then come back and tell me why you believe he is only trying to 'sell' his product.
Please back up your claims with some evidence this time
I'm not happy about this, first he says that this account has a abnormally high spam ratio and then says that a normal user can have 60%. Where do we get these figures from I would like to know as my average is pushing up against 100%. I don't think that there is such as thing as an average user, some people seem to get nearly no spam and the rest of us get almost complete spam.
Reviewing todays inbox reveals around 200 emails, of which 8 were legit. You do the maths, I would be making progress if it was only 81%.
Oh boy he goes on and on, if ever you wanted to cut out the spam in an article...
His main points (at least the ones I agreed with):
1. No training period, many features only turn on after lots of real emails have been processed. Fair enough.
2. No purge window, stale emails get purged over time (e.g. 4 months), but in a test everthing is shoved through at once (in minutes) and so nothing gets purged. Again fair.
The rest of it complains about the tester, or complains that it was less than ideal conditions & settings for the particular filter.
We call that 'the real world' here.
Sys admins are not experts in configuring filters.
Also he should realise that any new filter gets a better rating than the dominant filter. Spammers try to defeat the most popular filter of the day. So sure a new filter might perform better than an existing one *initially* simply because the spammers are targetting it. Until it becomes dominant and then the spammers adjust the spam to defeat the new dominant filter.
So in the real world the data set will always be unusual because the spammers make it that way.
Unfortunately it seems like the author is too intent on slamming Cormack for his review to fit my description of an "Excellent Review". I wish he had toned this down as he could still have delivered the same technical message in a more credible fashion.
"Excellent counterattack" might be more fitting.
"You mean like any other normal person who might be wanting to use such a product?"
...nevermind, I don't need to say anything else.
And to that, I would say... Someone writing an article for publication in a peer-reviewed journal should become experienced in their area of research before attempting to publish their results!
For example, I'm sure you don't have much experience with Nuclear Magnetic Resonance imaging - And you might or might not have experience with X11 forwarding. But unless you are fluent with both of those topics, I would not expect you to attempt to publish a paper in a peer-reviewed journal discussing those topics!
(Like I did, last December)
However, for the sake of presenting some evidence to back up what I'm saying here, I'll take your example of Consumer Reports.
From their site: CR has the most comprehensive auto-test program and reliability survey data of any U.S. publication; its auto experts have decades of experience in driving, testing, and reporting on cars.
For the love of Cthulu, people, "architect" is a noun, not a verb.
Languages are dynamic, not static. If enough people begin to use 'architect' as a verb, then it is a verb. I have a strong hunch that 20 years from now, the verb form of architect will appear in Merriam-Webster...
// TODO: Insert Cool Sig
Exactly - what's the point if you have to re-check it anway?
That is the main reason I don't use any spam filters.
Without a filter I can check emails as they come rather than create myself a "homework" of having to check 50 messages at once...
Irritation is a perfectly reasonable reaction. It is not, however, constructive to vent the irritation in response.
Somehow came not long after this:
Something I learned from girlfriend #4: validate feelings. Yes, the Nuclear Elephant was hurt. He's right to be hurt. But no, lashing out is not adult, it is not constructive.
To characterize other researchers as ignorant, wagon-jumping glory hounds with poor self-control does not encourage cooperation.
The ultimate point where I lost patience was where he claimed that the results were invalid because they didn't conform to accepted, real world knowledge. The study was empirical; it shows something, based on how it was set up; and what it shows is valuable.
But without knowing how the test was set up, how can you trust the test's so-called empirical results?
In medicine, research results aren't generally trusted unless 1) the study was sound, e.g., double-blind and 2) a separate team has recreated equivalent results using the published methodology. If, as Zdziarski says, Cormack is not making his config files available, then that alone should be a reason not to blindly accept the study's results. The methodology is unknown.
I can see not publishing the mail messages - in medicine, for example, you don't want to re-use the same test subjects from the first test, so there's no point to it as well as the privacy issue - but the config files? What possible reason could there be for not making them available?
English does evolve, and good writers sometimes repurpose words to great effect. Alas, judging by the rest of the reviews here, our hero is NOT a good writer -- having built a shoddy and ramshackle outhouse, he proudly crowns himself the architect of it.
As for all those people who shout "prescriptive grammarian!", I often suspect they're just too lazy to learn to write well, and have decided that claiming that rules are passe is an effective workaround.
Everybody's a libertarian 'till their neighbour's becomes a crack house.
I remember going through the CRM114 installation docs, and vividly remember the 20 or so steps that I had to go through, and after about 3 or 4 hours of trying to get it installed, I finally gave up. I think part of the goal of software design is to make your software so that people will be able to quickly install and use it. The author of this program lost sight of this important point. I'm not going to sit there and reverse engineer some esoteric codebase just to get it working, and I'm sure alot of other people feel the same way. Therefore, I use SpamAssassin among other things, and it works really well and was quick and relatively painless to get working. I didn't have to go through their source code to figure out how to get it installed.
The Article was necessary. It comes down to this glaring fact: ...."
".... If you use a tool that is only 95% accurate to prepare a test for tools that are 99.5% accurate, then the lesser tool will appear to outperform the better tools whenever the better tools are correct.
"He cannot scratch his itch because he cannot reach it."
You don't have to be a developer. As I said you can start a campaign to ask for donations, you can write letters to companies asking for sponsorship, you can donate some of your own money, you can try to get like minded individuals together to solve the problem.
OPEN SOURCE DOES NOT WORK UNLESS YOU CONTRIBUTE.
" Rather, you should acknowledge that the area is weak and that more focus needs to be given there in the future."
More focus needs to be given by who? Are you saying I should grab random programmers off the street and yell at them until they write a GIS program for me?
evil is as evil does
I have noticed that black lists are indeed effective. Many spammers now use "bullet proof" spam hosts, so they use static domain names. However, there has been an marked rise in zombie systems sending spams. These are systems that are infected by viruses and then used as spam hosts. Since these systems come on line rapidly (when they are infected) and then drop out (when they are cleared of the virus or booted off their ISP) it seems unlikely that black lists will help.
At least in the spam stream I see, there is more than 1-2 percent of the spam flow from zombies. The best technique seems to be to use a black list first and then content filter.
An a related topic in the parent post:
In a previous post, in another discussion, I also suggested that the sophistication of spam filters like SpamAssassin, which use several algorithms to filter spam, would consume lots of system resources. Another poster wrote that these tools do not consume much in the way of processor and memory resources. This seems counter intuitive, but I don't have any contrary evidence.
No, you can scan your spam folder in seconds, because you will recognise the subject lines. The duration is not comparable. When you have a folder for spam, any non-spam sticks out, but if you need to think looking at alternating spam and non-spam messages, you spend more time thinking.
You are being MICROattacked, from various angles, in a SOFT manner.
While obviously Cormack and Lynam are central to this discussion, it's depressing that this is +4, Informative when instead they obviously resent any serious questioning of their work. Is there a '-1, Wussy' moderation?
... and then RESPONDING, and using his work in your errata?
"We shall not respond" -- huh? Pull the log out of your ass guys. Like it or not, he's got legitimate beefs with your study. What's more, he's got cred: dude puts SERIOUS effort into GPL'd software that helps people, so his input is relevant and valid. Get over it.
Besides, his questioning of your credibility are neither 'ad-hominem' or irrelevant. Claiming that it is betrays a decidedly unprofessional sensitivity to criticism. as he points out, it is more than legitimate to question the credentials of the tester when interpreting results -- UNLESS the test has been repeated. 'Ad-hominem' attacks means irrelevant insults, whereas he's merely questioning your approach and relevant experience. don't go public with your stuff if you don't like the heat.
How about instead, you address his most damaging points:
- put all of your configuration data and any other information required to re-run the test online, immediately. there is absolutely no reason to resist this. you might want to explain why you haven't already.
- your errata is so far entirely due to his corrections. professional class would merit gratitude for his attention. try it on for size. after all this is supposed to be a *review* period yes?
- he directly questions the use of human error-checking. is he right? wrong? i don't know but it's a damn interesting question, and one your response does not address.
- finally, what's up with saying you won't respond
there are more problems here but you get the gist. you guys get paid to do this so do it right.