Reviewing Anti-Spam Offerings

Joel Snyder writes "Just finished looking into the innards of 40+ anti-spam products at Network World. The biggest, ugliest, and most comprehensive look at this market that's ever been done. Conclusions: lots of great products to choose from at the top (a dozen or more); a few stinkers in the bunch; and it's basically impossible to review Spam Assassin, which is unfortunate."

Comments

SpamAssassin?

[ajs]

They say, "Although a few well-meaning souls volunteered to be the contacts for SpamAssassin, when it came time to test no one would step up to the plate and represent the product at a level that would make it competitive to the other enterprise-focused vendors."

I can only wonder what it was that they asked and who they asked. There are several companies that provide products based on SA, and the developers are very responsive.

I'll have to look in more depth later and see if any of the products they reviewed were SA-based.

Still, a review that does not cover common open source implementations such as DSPAM and SA is not a review that I would put much stake in.

Worthless accuracy table

[Ekman]

The way they reported the results was pretty bad. The left two columns ranked products by false positives, while the right two ranked products by spam caught. It is very difficult to look at this table and get a sense of which products performed the best. For example, the top product for false positives, BorderWare at 0.04% looks very impressive until you look at the other column and see that it only caught 88%. It's easy to have a low false positive rate when your catch rate is low, too.

At minimum, they should have taken the false positive rate, added it to the percent missed and ranked by that. Doing so sends BorderWare into the middle of the pack where it belongs, and more likely winners rise to the top. (Postini and MailFrontier). Pretty shoddy reporting when the end reader has to take your numbers and plug them into a spreadsheet to make any sense out of them.

They could have also weighted the two error rates, but deciding on weights would be pretty subjective. Some might think false positives should be weighted higher, while others might think the opposite. Ranking them without weights would have been an acceptable compromise.