Slashdot Mirror
Reviewing Anti-Spam Offerings
Joel Snyder writes "Just finished looking into the innards of 40+ anti-spam products at Network World. The biggest, ugliest, and most comprehensive look at this market that's ever been done. Conclusions: lots of great products to choose from at the top (a dozen or more); a few stinkers in the bunch; and it's basically impossible to review Spam Assassin, which is unfortunate."
From deep within the article:
"Although these tests were conducted with the assistance of Borderware, we where careful to ensure results where fair and objective."
So, that would be why borderware's product got the #1 position?
Mine isn't in the list.... http://www.mxlogic.com
I have said it before on here, but I use Mx-logic.com to filter e-mail before it even gets to my mail server (as their filtering is in-line). They run multiple concurrent virus scanners, and you can set all policies related to attachments, sizes, virus scanning, quarantines, SPAM (deny, accept, etc, for different "levels" of probability).
It's really efficient. I haven't gotten a virus in any attachments and maybe just 2-3 SPAM messages / month (down from 100+ / day). It also does cool stuff like remove the imbedded tracking images from SPAM HTML messages (should one get through), etc. No, I don't work for them. I used to quarantine messages and review it weekly (that were medium / high probability spam), now I trust their service so much I just deny receipt to my mail server of any Medium+ probability SPAM
We are one consciousness experiencing itself subjectively. Back to you with the weather, Bob!
RTFA:
We also reached out to the SpamAssassin community (see "What about SpamAssassin?"), but couldn't find someone who could act as a representative for support and configuration assistance. However, two commercial vendors, Roaring Penguin (on Unix) and NoSpamToday! (on Windows) sent products that exposed their SpamAssassin cores.
They have a whole page discussing this.
I never thought I'd get to use it... but... RTFA jackass. Don't just see a question and post something about it. Answer: http://www.nwfusion.com/reviews/2004/122004spamsid e6.html
I just upgraded my server to the latest version 3.0.1 of spamassassin and the difference is amazing. I haven't had one piece of spam get through to my inbox today. And from what I can tell, there are no false positives yet. Unless you think that Darcy really wants me to come over and check out her new webcam.
They say, "Although a few well-meaning souls volunteered to be the contacts for SpamAssassin, when it came time to test no one would step up to the plate and represent the product at a level that would make it competitive to the other enterprise-focused vendors."
I can only wonder what it was that they asked and who they asked. There are several companies that provide products based on SA, and the developers are very responsive.
I'll have to look in more depth later and see if any of the products they reviewed were SA-based.
Still, a review that does not cover common open source implementations such as DSPAM and SA is not a review that I would put much stake in.
Using Thunderbird greatly cuts down on the amount of spam you see in the inbox. After using for only about a month, 90% of spam was automatically deposited in the "junk mail" folder. Surely this isn't as good as a paid spam-prevention service, but its free :)
Maybe it's just me and I'm one of the few lucky people in the world, but out of 5 regular email addresses that I use on a daily basis, I rarely if ever recieve spam, and during the workday, watching mailserver logs, the only people in my company getting silly amounts of spam (to me, one or two messages a day is just a minor annoyance) are people who click every popup and put their email addresses in every form available. If it wasn't for the built in spam filtering of Kerio Mail server, which is what we use here, it would probably be impossible for them to get any real work done, as out of 200 people, these 5 or so get more spam directed towards them than the rest of the company gets regular emails. Some common sense goes a long way in avoiding spam.
The mere appearance of SA, though, is impressive because those trade rags rarely include anything open source (partly due to marketing opportunity for commercial, paying companies).
Jerry http://www.syslog.org/
...is to treat your e-mail address like you treat other personal, abusable personal information.
Do what I do: create a Yahoo (or some other free e-mail) account and use that address for all questionable forms you fill out.
I've had the same address now for almost three years now and receive about five spams per week, at most.
There's a Mercedes gap too. I want one and can't afford one, but it's not government's job to do anything about it.
The buying guide is useful just for putting all the contenders together. But don't believe the claims until you test them. Barracuda, for example, touts the capability of millions of messages a day, but we are sending our second test unit back because it just can't handle a modest load of real world mail. Their 600, for example, claims it can process "25 million messages per day" but that assumes it is rejecting 95% of the mail -- that's nowhere in their fine print.
I certainly do get a lot of false positives with thunderbird's spam-controls, and would really like an interface through which I can view the filtering logs (words, frequency, etc) that thunderbird must be creating.
A reporting feature (even if thunderbird just exports a database csv file) would provide more value to me. I'd also like to be able to transfer my thunderbird spam filtering profile to new installations (after reformatting, for example).
A lot of other packages (e.g. spamassassin) support some of these, but I see no reason that thunderbird couldn't try to include some of them too.
The force that blew the Big Bang continues to accelerate.
If you're going to review things for the enterprise, then you need to keep in mind the requirements of an enterprise. Very few large businesses are willing to trust a product that doesn't have some sort of obvious support structure behind it. If the reviewer could not find a solid support structure for it, then it isn't suitable as an enterprise spam solution.
This sig has been temporarily disconnected or is no longer in service
"However, two commercial vendors, Roaring Penguin (on Unix) and NoSpamToday! (on Windows) sent products that exposed their SpamAssassin cores. Although neither met our false-positive threshold for inclusion in the top 12 finalists (probably because of difficulty of tuning Bayesian engines and neural networks in a test lab setting), we were very pleased to have them participate in the project."
Still, a poster that does not RTFA before making such a comment is not a poster I would put much stake in.
I don't think I've seen any false-positives since about 0.7 - but it does miss some emails now and then, so it's not really 100% success rate, but really, what is?
But I do see your point... however, you also have to understand that with Thunderbird, you're not really running a separate application to filter your spam (or running anything on your server for it) - this is just a free email client that does it's own filtering.
Though as I said, I'm quite satisfied. And of course, your mileage may vary.
Thunderbird's anti-spam is nice, but I wouldn't call it excellent, at least from my experience. I've been using the junk mail feature since 0.7 or thereabouts, on a mail account that gets anywhere from 10 to 30 messages a day, 90% of which are spam. When I recently downloaded 300 or so messages, I still had 25 junk mails that it didn't flag. After several months of training, I'd think it should be more effective than that.
Easy. A Postfix server running Postgrey and Anvil. Before mail ever hits a mailbox most spam (and a lot of viruses too) are weeded out. It can protect against distributed dictionary attacks.
The world's burning. Moped Jesus spotted on I50. Details at 11.
This is a spam filtering service that I use, In 52 weeks 22,624 spam messages out of 93,714 have been blocked before entering my users inbox. The nice thing about this service for us is our IT dept is very under-staffed and makes it useful to have someone else worry about it. The do our anti-virus scanning as well and am proud to report that they have stopped all 5213 infected messages before even touching my server. Very worth while service if you are in a under-staffed situation like I am.
If you block spam you'll never increase the size of your penis.
"We invited every anti-spam vendor in our online Buyer's Guide to participate"
And what is there "online Buyer's Guide"? - a pay for inclusion directory!
Between that and their #1 choice helping them with the review process - I have serious questions as to the value of this report
. Accurately simulating a bunch of different anti-spam systems all getting the same e-mail is a bit of a trick. If one of the major players is helping set the rules - its way to easy for them to stack the deck.
What he's really saying is that they couldn't find anyone willing to PAY them to review SpamAssassin on Apache. That's about what passes for "comprehensive reviews" these days.
Though it's a small project, bspam is an excellent Bayesian filter for *nix... I tried bogofilter and some others but nothing jived with my qmail/procmail/pine setup as nicely as bspam.
RBL (list.dsbl.org : bl.spamcop.net : blackholes.mail-abuse.org : sbl-xbl.spamhaus.org : multihop.dsbl.org : cbl.abuseat.org) + greylistd == average 0 spam in inbox/day.
What I like best about this approach is that you reject most of the spam at SMTP-time without accepting it. If I could I'd add spam-assassin-on-SMTP to the end of the chain, but my server is tight on memory :-(
(Unfortunately there's a bug somewhere between the debian greylistd and python whereby the daemon shuts down on me all the time, but I've lodged a bug report and hope to get some help tracking it down.)
Belief is the currency of delusion.
A well-designed RBL blocks 95+% of spam and consumes less resources than all the other solutions. Plus it has the added benefit of stopping virus and worm propagation, phish e-mails and lots of other scenarios where unauthorized SMTP relays operate.
I see no reason to use client or server-side products that analyze the mail content, when this slows down mail service and reliability. RBLs, blocking mail based on the legitimacy of the source address has proven to be the most effective method of curtailing spam, and unlike all the other solutions, this one aversely affects spammers by not allowing them to consume your resources.
If you're in the business of making money off selling spam products, I can see your support of these various half-way solutions, but otherwise, the best way IMO is to employ RBLs at the server level and slowly work towards SMTP whitelisting. I contend this is an inevitability if the authorities don't start prosecuting spammers for their illegal computer tampering.
Flame suit on, if they can't even get Spam Assassin working... why should I trust them to be knowledgable enough to truly provide a unbaised and effective review of Anti-spam solutions?
To get a junk mail filter for my real life mailbox that auto sorts into my real life recycle bin.
GFI got a horrible review last year. The product they submitted was a pure 'word checker' (i.e., if you've got Viagra anywhere, you're spam) and so their false positive rate went through the roof. They also had some horrible heuristics, such as "if you're not on the "to:" line, it must be spam." My experience is that it was architected for a small office where you can tune it out the wazoo. They have since (I have heard) fixed their product, but they were so heavily burned by last year that they didn't want to come and play this year. I can't really blame them; once burned, twice shy. But we'll never really know, will we?
Where's SpamAssassin?
Although a few well-meaning souls volunteered to be the contacts for SpamAssassin, when our marketting department contacted them regarding advertising no one would step up to the plate and shell-out for print ads like the other enterprise-focused vendors.
The one product that I am familiar with is Barracuda, as we run that where I work. They claim that Barracuda doesn't support SSL for management, which is dead wrong. In fact it's very simple to _force_ the Barracuda to use SSL for this purpose.
It's only one point, but they make a fairly big deal out of it.
Spammers will Spam you if they can Guess or Get your Email Address so the trick is to make it hard for them to get it.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
FROM TFA:
The short answer is that no one submitted it, but of course there's more to it than that. This year we reached out to the SpamAssassin community and asked them to participate. Although a few well-meaning souls volunteered to be the contacts for SpamAssassin, when it came time to test no one would step up to the plate and represent the product at a level that would make it competitive to the other enterprise-focused vendors.
Interest in SpamAssassin is understandable. In the small-business market, the open source SpamAssassin dominates many anti-spam systems. When well tuned and integrated by a value-added reseller (VAR) that knows what it is doing, it turns out to be a very effective system. SpamAssassin users routinely report 100% spam reduction and 0% false positives (although these self-reported statistics are probably biased), and are generally overjoyed with the results.
Advertisement:
By itself, SpamAssassin is little more than the software implementation of an interesting idea: apply statistics, neural networks and Bayesian probabilities to the problem of classifying mail as spam or not. Train the engine by giving it desirable and undesirable mail, and it can tell you for each new message what pile it most resembles. It turns out to work astonishingly well, especially in small businesses where mail flow is very homogeneous. SpamAssassin's Bayesian engine even redefines the meaning of spam by letting you say, "This is the mail I want," and "This mail I don't want." SpamAssassin also mixes other tools into its scoring system, such as DNS-based blacklists and collaborative scoring, as well as more traditional keyword searches and formatting tests.
The key to SpamAssassin's success, though, is a smart VAR or IT person installing it. SpamAssassin requires a significant amount of integration work to make an enterprise-class installation succeed. Without a GUI, database, quarantine, anti-virus scanner, policy or per-user configuration, SpamAssassin is a great tool for those who want to build their own anti-spam system, but is in no way a solution by itself.
This doesn't mean that SpamAssassin wasn't well represented in our test. The important core of SpamAssassin, a Bayesian engine, was recognizable in at least one-third of the products we tested and might well have been hidden in the guts of more. The strategy of combining multiple tests to identify spam is in nearly all modern, anti-spam products, including SpamAssassin.
The difficulty in testing or recommending products that require heavy engine training, or ones based on trained neural networks, is that companies with many employees have very diverse mail flows, and the training will likely generate false positives or negatives across large numbers of users. For example, a multinational company might have many employees who don't read or speak Italian, and might train all their Italian mail as spam - something that would upset the Milan and Rome offices. Or imagine IDG, which owns many publications, all which have specialized vocabularies. No one set of training mail would work for the different communities.
Products that successfully include a Bayesian recognizer, such as SpamAssassin, do so by considering it as one factor in the larger cocktail of spam identification. By weighting the Bayesian verdict with other information, vendors have followed the trail that SpamAssassin blazed and made it enterprise-ready.
Actually, the #1 selling enterprise anti-spam device (the Barracuda line) is a SpamAssassing core device.
[RIAA] says its concern is artists. That's true, in just the sense that a cattle rancher is concerned about its cattle.
The way their testing was conducted, they probably had to overlook spam filters that are embedded in proprietary email services but if you are only interested in getting all your mail and none of the spam, google is doing a great job.
My gmail account has had 2 false positives out of 500 messages. Given the vulnerability to having your address fall into unknown hands that is inherent in Google's viral marketing technique for promoting the product, I would bet LOTS of other GMAIL users have the large number of spams coming in...even on new accounts where they have been careful who they gave the address too. I get about a dozen spam items a day but when one of the sh!theads sells his address list to the next spammer, I can get a burst. Bottom line: ZERO spams in my inbox...none...not any. The Bayesian stuff that spammers try to circumvent, the spoofed headers...so far none of it fools Google. And since it buffers the spam in its capacious 1Gb-per-account holdings, I have 30 days to check for false positives at my liesure.
Questions?
1. what vulnerability?
when you accept a google gmail invitation, no matter how many hands it has gone through, Google posts a notification of your new address to the original giver of the invite...who could be some spammer you never met....happened to me.
2. any pattern to the false positives?
not sure...only have two data points. Those two items were email alerts from newspaper subscriptions which tend to be crambed with ad text and ad links...in which case, gmail is clearly trying to do me a favor and I appreciate the effort.
SLASHDOT: news for people who can't concentrate on work or have no life at all and got tired of yelling back at the TV.
RTFA. Postini was in it, both in the big table and in the Dirty Dozen finalists.
Thanks for the compliment... because, you see, I first used the methodology in 2003, in the original Network World test (see http://www.nwfusion.com/reviews/2003/0915spam.html ).
m demo.html)
Or, you could go back to February, 2003, and see the same methodology being prototyped at the Demo conference (http://www.nwfusion.com/reviews/2003/0224antispa
Let's see: Feb 2003: 2 products.
Sept 2003: 16 products, with 4 top overall performers.
Dec 2004, 36 products, with 12 top overall performers.
And Network Computing? 23 products with 10 finalists, in between my two reviews for Network World.
Yeah, I'm feeling like what Network Computing does in between my reviews makes me a copycat...not.
What are you, a NWC ad salesman? Or just a bit clueless yourself?
> It's economy at work, you pinky commies
My fine capitalist customers pay to get email, not to get unwanted bulk advertising, much of it fraudulent, and a lot of it in fact coming from computers that have been made into zombies by worm writers breaking the law.
The world's burning. Moped Jesus spotted on I50. Details at 11.
At minimum, they should have taken the false positive rate, added it to the percent missed and ranked by that. Doing so sends BorderWare into the middle of the pack where it belongs, and more likely winners rise to the top. (Postini and MailFrontier). Pretty shoddy reporting when the end reader has to take your numbers and plug them into a spreadsheet to make any sense out of them.
They could have also weighted the two error rates, but deciding on weights would be pretty subjective. Some might think false positives should be weighted higher, while others might think the opposite. Ranking them without weights would have been an acceptable compromise.
The only thing I can say about RBLs is that you need one that is an amalgam of others. This is the same theory that drives SpamAssassin: you may be able to fool one, but you can't fool them all.
I am doing testing with SenderBase and it gives any IP address a -10 to +10 score. Pick your own false positive/false negative threshold and you can slice out a big chunk of garbage. But SenderBase is not generally available except through a web interface. It's gone through a couple million messages of ours with one false positive.
I know that Symantec/Brightmail and Postini both have their own 'reputation-based' services as well that seem to work.
What I don't know of is any RBL that is itself an amalgam of other RBLs, returning a score (as opposed to a "go"/"no-go" answer). My own luck with RBLs before SenderBase was so poor that I basically discounted them as either (a) not helping enough to be worth the effort or (b) too many false positives.
A number of the products that I looked at had "RBL voting:" they lookup things in more than one RBL, and if they meet a threshold you set ("must appear in 2 RBLs..."), then the message is marked as spam. Others consider the RBL as a component---if it's in an RBL AND has "Viagra" and a URL in it, then it's probably spam.
I think that either a combo-RBL or RBL-voting has to be the way to go.
They seem to have gotten a lot better in the past couple of years.
Not only does it allow you to cut off spam, it gives you traceable addresses that can be used to see who leaked email to spammers. And it's perfect against phishing attempts.
--
Mad science! Robots! Underwear! Cute girls! Full comic online! http://www.girlgeniusonline.com/
You did not read the article. From the Who got left out or opted out page:I'd say that given these two statements, their motives are impeccable. They did review SpamAssassin-based products. They did not review SA on it's own because there was no way to make it fit with their methodology. There were many other products that also got left out for these reasons, and their reasons make sense.
You are full of shit.
There are no trails. There are no trees out here.
http://assp.sourceforge.net