I'm sure that if they're serious about actually showing that the statistics are useful then we can find 10 random sites who are willing to be 'ethically hacked.'
The astonishing thing is that most people who will read this press release just don't get it, and the depths of their not getting it are even more astonishing...
I am challenging the conclusion, not the data. I believe that they think that they have found vulnerabilities. I suspect they have found a lot of lousy code. No surprise here. 70%, sure. I'll bite off on that number. I'm not arguing with that.
But there is a huge difference between turning a vulnerability into a breach. Let me give you an example. A lot of Cross-Site Scripting attacks let you steal cookies. So they probably found those. But the question is: when you have a cookie, what can you do with it? Can you steal important data? Can you turn that cookie into a breach? Good web sites that use them also tie cookies to your IP address, which means that if you steal my cookie, you got nothing but crumbs. So the point is not that there are these vulnerabilities, but that they have done nothing to show whether these vulnerabilities are truly breachable and able to get an attacker real useful data.
Same for things like directory listing. You can do that to my web site. Is that a security problem? No, in fact, I turned it on specifically. If I didn't want people to read it, I wouldn't have put it on the friggin' web server.
Is a web site that's susceptible to an SQL injection attack hackable? Depends on where you get to inject the code. I'm sure that someone who put their mind to it could take a web site like, say, slashdot, and inject some SQL. Then they might be able to... well, they could read all those posts that are on the web site. Except they wouldn't be nicely formatted, but real men write HTML with vi anyway. Maybe they could store or corrupt data with the injection, and maybe they couldn't. Maybe (and this is most likely) they could cause the script to blow up. Is that "hacking" a web site? Hell, I get script explosion errors from web sites WITHOUT hacking them.
Is being able to view a script a security vulnerability? it depends. It depends on the web site. The script. The webmaster's intentions.
What percentage web sites actually have data that's worth anything?
So the point is not that they've found a lot of theoretical issues, but whether they've actually found security issues. And the only way, in my mind, to see whether they have is to see if the issues can be exploited. If they can, I'll pay up. If they can't be exploited, then all they've done is made long lists of things that don't matter from a security point of view.
Reacting to spam as an electronic vigilante seems to cause much more collateral damage than actually solve any problems. Most of us have been victims of unintentional Joe-job attacks, and that's bad enough... how's a couple million misdirected emails in a week to help your to-do list?
The same thing is true of people who go after spamvertised web sites and the ilk.
We used to run a public-access traceroute server specifically to help people diagnose network problems. It got picked up by an anti-spam site and started doing 10s of thousands of traceroutes a day.
Then, the "error rate" began to get to me. Calls at 3AM from morons who couldn't figure out what the hell they were doing, but had somehow had my name show up in some traceroute they were doing. Threatening letters (from more idiots), and even the occasional DoS attack.
Just got too much and I basically had to cripple the resource because of the false positives.
Extremely bright and careful people might be able to track & attack spammers, but for every one of those, there are a dozen or a hundred who will mis-type an IP, a URL, or be the unwitting dupe in a third-party DoS attack or in attacking a cracked system.
I'm not fan of spammers, but the collateral damage that can---and will---be caused by ANY automated 'strike back' system suggests that this is not the path to solving the spam problem.
>By the way, I have never had anyone tell me >his/her legitamate email was rejected by my >server. What false-positive rate do you consider >to be acceptable?
In our test, we had a bunch of vendors claim that because they were not aware of false positives that somehow there were vanishingly few. This is even true of systems that don't quarantine mail, although this is patently ridiculous---how are you even going to know about false positives if your mail disappears into a black hole? The answer is their marketing people pull numbers out of some occluded place normally covered by pants and claim those as false positive rates.
This, of course, is complete crap. False positives happen and mail has become so 'lossy' that we never even hear about some of them. For example, during our testing, I discovered a false positive in one of the RBLs that I was looking at. I found out because someone pointed it out (they got a 5xx or 4xx and had the presence of mind to send it in). Once I KNEW about the problem, I could go back in the logs and see a half-dozen other people who were affected by the same RBL error---but never bothered to complain or say anything.
This problem gets worse when you consider content-based systems, because they tend to false on the mail that you care the least about---bulk mail, mailing lists, etc. If you subscribe to a mailing list and it receives 50 messages a day, will you miss one? Almost certainly not... but that's likely where the false positives are happening.
InfoWorld, for all their faults, had a good idea to somehow separate out false positives from bulk mail as opposed to person-to-person mail. It's an interesting concept and I considered putting it into our test as well, but eventually dropped it. The reason is that some mailing lists are, to me, MORE valuable than person-to-person mail. For example, I'm on some software development project mailing lists, and a message there is likely to be more important to me than a message from a clerk saying "there's a package for you at the front desk from Amazon."
With all of this being said, I think that rejecting mail at SMTP time is really the way to do it. Whether you do it RBL or content or both, the fact that you can refuse to accept responsibility for the mail wins in so many ways. I depend on email heavily, and I fall into the "every sperm is sacred" school of thought with email. Quarantines are OK, but if I could know that either I got the mail or it got 5xx-rejected to the upstream, I'd prefer that solution by far. (Actually, the REAL preference is to let the network manager decide whether they want to accept/tag-and-deliver-quarantine/reject the mail)
I'm not sure that I agree that putting AV in front of AS is going to change much. What we do with AV is what people should be doing with AV today: delete that crap. So what that means is that the AS products didn't see viruses, or cleaned messages. Back before mass-mailing worms, we used to see a few viruses a day that could be cleaned and such, but now we see a constant flow of about 1000/day (about 10% of our flow) where they are ALL worm traffic. I can't find a non-worm message in this month's log, in fact.
So what would the effect on AS be? Well, some AS products DO detect mass-mailing worms (sure, why not); those guys didn't see them. Some AS products do NOT detect worms; those guys didn't see them. So I am not sure that we're going to see much of a change in the behavior, given that whether a worm is spam or not is something that varies from vendor-to-vendor. (It tends to be spam if the vendor only makes AS tools; it tends to be a virus if the vendor makes both and packages them in a single device).
In general, our detaching the product from first-hop-SMTP had to have some effects, as you & I have both just noted. Intuitively, what I think you would see is a higher spam catch rate (i.e., lower FN) but no change in the FP rate. For example, a common technique we are now seeing is that spammers will hit an open HTTP proxy, and do a POST to smtp-server:25 with a bunch of crap that happens to also contain a valid SMTP transaction. Many SMTP servers will accept what is effectively pipelined SMTP (even if it is not negotiated). So, if you were in the SMTP engine, you could look for "POST / HTTP" before the HELO/EHLO and say "this guy is a spammer, screw 'em, reject."
That would increase your spam catch rate if the spam would not otherwise have gotten caught. It should not increase your FP rate.
One thing I thought would be true is that people would have lots of subtle tuning that they wanted to do. So we opened up this review to let them do that---everyone could SSH or RDC into their boxes and try and tune things up. What we observed is that only a tiny number wanted to do that. My thinking is that the vendors did not have very subtle dials to turn regarding spam catch/false positives that might have been affected by this insulation we were artificially creating. So, I am inferring that FN might have gone up by some small number but FP not at all.
I regard that as A Good Thing. I think that 1st generation (whatever that means) anti-spam was VERY 'person' intensive in terms of tuning, management, etc. Actually, I KNOW that they were intensive; I wrote an anti-spam product back in 1995 that effectively implemented what we call "Graylisting" now, as well as put in rate-based controls. That was great for its day, but you had to watch it all the time to make sure that big senders didn't get blocked.
However, I believe that most email admins are sick and tired of playing with their anti-spam software. I talked to a bunch of them when I was constructing this test because I thought that the antispam market must basically be "done." Don't we already have nearly 100% adoption of anti-spam products? The answer is "yes, but..." Most of the people I talked to were looking for a better solution. They were early adopters and were dissatisfied with the amount of effort that went into maintaining their anti-spam solution. (Or they were looking for features, such as quarantine, that might not have been in the original product they bought.) They want it to work, and they want it to work well without a lot of screwing around. This is not an unreasonable thing to ask. We used to screw with O/S tuning all the time; then the O/S guys figured out that they could do it better than we, and now we rarely have to do that. Same thing for databases, etc. Now, the anti-spam guys are raising the state of the art as well.
This is why a lot of new products (and services) are coming out which are essentially untunable or are only tunable at a very coarse level. The
Yes, that's the methodology. Exactly. We replay them "immediately" for some value of "immediately" which can be translated as "if they're up, within a second or two, otherwise we queue." Not "real time" in the strict definition of the word, of course, but fast enough that we can claim that all products saw all messages at the same time as they showed up on our Internet pipes. I wouldn't say that it makes it repeatable, but it makes it supportable across 40 products.
And the issue of spamtool/zombie identification is a real one; thanks for bringing that up. As is the much more important issue of the 'sending' IP address. The 'sending' IP address problem in this methodology is one that can be dealt with by a good product (and several products can and do deal with it just fine); some others are so restricted that they cannot work effectively except as first hop. To me, that's a bug. I had one VP of Marketing scream at me "no one ever puts our product anywhere except as first hop," and I barely held back from saying "yeah, that's because your product is such a piece of crap it can't go anywhere except for first hop."
However, the "looking at the SMTP conversation" part is impossible to really deal with. But, I'll note that the majority of the products sit on top of an MTA. Either they install for themselves or they use tools like sendmail, qmail, or (for Windows boxes) the MS SMTP MTA. So those products don't actually know anything about the SMTP stream. There are a few products that specifically brought up this issue because they DO look at the SMTP stream, and they probably did not do as well. How "not well?" Maybe a couple of percentage points in spam catch rate. Maybe less.
In the review, I wrote a short side-bar where I admitted this up front: (http://www.nwfusion.com/reviews/2004/122004spamsi de5.html) "You may notice our numbers are not as optimistic as the marketing literature from vendors' products. There are four reasons for this:
1. Side effects from our test bed probably shaved a few points off of each product's ability to identify spam...."
I also brought that issue out when I wrote : "The false-positive and false-negative rates we found are useful for comparing products but a real installation will likely have a lower false-positive rate and higher spam-catch rate." (and mention things like the SMTP catch rate) in
http://www.nwfusion.com/reviews/2004/122004spamsid e.html
OK, so now that I'm done defending myself, what's the point? Well, one of the vendors told me "you know, all these products basically have no false positives and catch all spam." But that's completely wrong. We discovered a bunch of products that are still dark-age when it comes to catching spam, ones which have enormous false positive rates (in particular).
However, if you look at the top 10 or 12 products, you can see that while there are differences, they are not showing a huge variation in behavior.
What this means is that you can take a test like mine and use the spam catch rate/false positive rate as a "first cut." Because I believe that where you want to make your buying (or implemention, in the case of open source) decision is based on things besides just spam catch rate/false positive rate.
We have to do the FN/FP rate tests just to say "you must be this high to attack this problem." But from then, there are huge differences in the products, and that is what is important. I don't want to seem like I'm lashing out at the people who say "oh, I use (insert product here) and it never falses and never misses," but those folks just don't get it. It's not the spam catch rate that differentiates the products; it's everything else.
An easy example is CloudMark. Talk about Zen. This product doesn't do
Yes, it does help. You know all those RBLs that everyone seems to really love to much? It's complaints that get them populated (among other things).
So it's kind of altruistic if you ONLY complain and never use an anti-spam product, but there are a bunch of tools like DCC (distributed checksum clearinghouse), Vipul's Razor, and almost all of the RBLs that depend heavily on folks making sure that the bad guys are exposed...
"Every once in awhile I get a false positive, but the percentage is very trivial, and the problem gets immediately solved because, unlike content-based systems, the sender immediately knows his mail didn't go through and can bypass it manually."
You're mixing things up here incorrectly.
If I use an RBL OR if I use a content-based filter, I can either quietly drop or noisily bounce mail. I can do it at SMTP time, or I can do it after I've accepted the mail. Your choice of technology has nothing to do with what you choose to do with the mail.
Now, of course, there are products that don't give you that kind of flexibility, but that's a defect of the product and has nothing to do with whether the technology does or does not meet your needs to reject RBL-ish mail at SMTP-time.
Absolutely. SO there's a buyer's guide so you can learn about ALL the products. AND, then there's this mammoth test that is an independent, objective, scientific look at products that are a subset of the Buyer's Guide who wanted to engage in head-to-head competition and comparison.
In our test, it was very clear that we wanted to look at head-to-head products. You can't match TDMA against Brightmail---they do VERY different things for VERY different people.
If you wanted to write a feature, you could talk about Graylisting, TDMA, Content Filtering, etc., all at the same time and discuss the pros and cons of the technology. But in a test, you want head-to-head.
Magazines consider features and tests to be very different things.
So, in this article, we ONLY looked at SMTP-in/SMTP-out products which were directly comparable. We didn't look at TDMA, graylisting, client-side tools, commercial services (like gmail or yahoo or hotmail), or mail server add-ons that don't talk SMTP. And there were still too many products...
FrontBridge and MessageLabs pulled out of the review at the last minute, after their products had already been in place and in operation. You can infer any reason you want to that.
A full conflict-of-interest discussion appears in the review at http://www.nwfusion.com/reviews/2004/122004spamsid e2.html
Several of the vendors said something to the effect of "how can you do a fair review when you use one of our competitors?" (Obviously, the competitor would be 'disconnected' during the review, something that I think FrontBridge and MessageLabs didn't quite understand). My only answer is that anyone who thinks that they can review anti-spam products but is not yet using them is clearly not qualified to discuss the matter.
It'd be like asking my Mom to review Linux vs. Windows vs. Mac. Hey, she doesn't use any of those, so she's unbiased!
Actually, the reason that trade magazines don't often review open source materials is that they don't fit well within the methodology of how we review products.
Magazines cannot look at all products. So they have to look at the ones that have significant market share and are of interest to significant numbers of their readers. It is rare when an open source product rises to the level of prominence of a commercial product---for every Apache, BIND, Linux, and Spam Assassin there are a thousand other perfectly good programs that aren't really applicable.
But when the open source stuff does get sufficient market share, it's still very difficult to review. Products that enterprises use generally have things like technical support, documentation, release cycles, and such. Open source products tend to be underdocumented, undersupported, and come out "whenever the author thinks there should be a new revision." It's hard to put these things up against each other in a head-to-head review.
Much more importantly is the use of open source as a 'toolkit' for people. Enterprises use LOTS of open source things, but they generally do so by picking-and-choosing pieces and assembling their own best solution. This is often because the commercial products can't meet their requirements! But you can't review a "tool" in comparison with a fully blown commercial product.
Look at Spam Assassin, for example. It's one piece in a much bigger picture of how to manage spam. But it's just one piece. Without a mailer, quarantine support, a GUI, complete documentation, and technical support---it's not appropriate to put Spam Assassin up against a commercial product.
Does that mean that it cannot be a VERY successful tool that you build into your own customized spam solution? Absolutely not! But it does mean that if we download an RPM of some product and it doesn't measure up to a fully supported, fully documented, fully configurable, fully featured commercial product, it really doesn't seem like a fair review.
It's HARD to review open source products when there is not someone willing to say "I will represent that product." In the case of this review, we actually had a bunch of people who had incorporated SA into their commercial products---and so SA got a fair review. It just didn't get called "Spam Assassin."
What is VERY true is that commercial concerns have nothing to do with it. The editors want to serve the readers, and the readers have told us that they want to hear about open source. So the fact that SA will never buy an ad is irrelevant, at least for the reputable publications.
The only thing I can say about RBLs is that you need one that is an amalgam of others. This is the same theory that drives SpamAssassin: you may be able to fool one, but you can't fool them all.
I am doing testing with SenderBase and it gives any IP address a -10 to +10 score. Pick your own false positive/false negative threshold and you can slice out a big chunk of garbage. But SenderBase is not generally available except through a web interface. It's gone through a couple million messages of ours with one false positive.
I know that Symantec/Brightmail and Postini both have their own 'reputation-based' services as well that seem to work.
What I don't know of is any RBL that is itself an amalgam of other RBLs, returning a score (as opposed to a "go"/"no-go" answer). My own luck with RBLs before SenderBase was so poor that I basically discounted them as either (a) not helping enough to be worth the effort or (b) too many false positives.
A number of the products that I looked at had "RBL voting:" they lookup things in more than one RBL, and if they meet a threshold you set ("must appear in 2 RBLs..."), then the message is marked as spam. Others consider the RBL as a component---if it's in an RBL AND has "Viagra" and a URL in it, then it's probably spam.
I think that either a combo-RBL or RBL-voting has to be the way to go.
They seem to have gotten a lot better in the past couple of years.
Right. The table is poorly designed. This is what happens when you take a spreadsheet and feed it into an art department. The spreadsheet actually has 10 different columns.
But we don't want to add percentages. That would not be fair. You have two columns, each sorted from "best" to "worst" and you can read down each of the columns. What they should have done is broken them into two separate tables rather than one.
SO you have to actually say whether you care more about false positives (read left table) or false negatives (read right table) or make your own conclusions based on the combination of the two.
In general, people love scorecards and pure rankings because it means they don't have to actually come to their own conclusions. (I'm not accusing you of this; just explaining why articles often have them) In this case, the real answer is that you have to decide what's important to YOU and then you can rank them youself. For example, if you don't want to quarantine and you have zero tolerance for FP, a few FN won't bother you. On the other hand, if people have good quarantine or you REALLY need to see the FPs, but you want the FN rate to be low, that's a different set of criteria.
I'm opposed to compromises and mixing statistics together that don't belong together; see the rant about statistics in the review itself. (http://www.nwfusion.com/reviews/2004/122004spamsi de3.html)
It's hard to compress six months of research into an article, even one as long as this one...
For the spam test, approx. 75% was spam. The mail flow was about 30K messages. In Feb of 2003, the number was almost exactly 50%.
One of our consulting customers with a well-known domain name reports 95% spam (1 in 20 is not spam) over 20+million a day. Another one with a VERY well known email address reports 99%+ spam to that address---but around 50 to 75% for other, less well-known addresses.
Thanks for the compliment... because, you see, I first used the methodology in 2003, in the original Network World test (see http://www.nwfusion.com/reviews/2003/0915spam.html ).
Or, you could go back to February, 2003, and see the same methodology being prototyped at the Demo conference (http://www.nwfusion.com/reviews/2003/0224antispam demo.html)
Let's see: Feb 2003: 2 products. Sept 2003: 16 products, with 4 top overall performers. Dec 2004, 36 products, with 12 top overall performers.
And Network Computing? 23 products with 10 finalists, in between my two reviews for Network World.
Yeah, I'm feeling like what Network Computing does in between my reviews makes me a copycat...not.
What are you, a NWC ad salesman? Or just a bit clueless yourself?
Yes, you're right; it's an error. My notes show that you can turn on SSL for management, but what got written in the article is wrong. It'll get fixed online immediately. That crept in as part of the editing process.
On the other hand, I don't understand why ANYONE ships ANYTHING that talks on port 80 anymore. It's not like OpenSSL hasn't been proven through-and-through (or you can write your own). Port 80 might be fine for pictures of your vacation, but the management interface on a corporate mail server should be encrypted and authenticated.
However, if you want to discount a 10,000 word article for a single error, then you're going to have a hard time believing anything you ever read anywhere ever.
You would have difficulty finding stats that support the 95% assertion. Folks like Brightmail & Postini and SenderBase aim closer to 50%, but it's a different statistic: that's blocking 50% of the incoming TCP connects, not 50% of the spam. In our own testing before the spam review started, I got numbers similar to those using SenderBase as the reputation-based scoring ahead of our mail servers.
I would agree that a well-designed reputation-based DNS blacklist can immensely increase the spam catch rate AND block a bunch of mail before it hits the servers. However, if you did the intersection of all the random RBLs out there, you'd end up with an enormous false positive rate.
You can also take DNS BL information and mix it into your cocktail. I discussed that topic specifically in the article in greater depth.
It's also a question of environment. I have friends who have little 2-or-3-person mail servers that basically intersect ALL the blacklists they can find and are perfectly happy---because they don't correspond with more than a couple hundred different people. But talk about a big corporation with thousands of users, and the DNS BL strategy doesn't work quite as well because of the false positive issue.
As with everything, different strokes for different folks...
GFI got a horrible review last year. The product they submitted was a pure 'word checker' (i.e., if you've got Viagra anywhere, you're spam) and so their false positive rate went through the roof. They also had some horrible heuristics, such as "if you're not on the "to:" line, it must be spam." My experience is that it was architected for a small office where you can tune it out the wazoo. They have since (I have heard) fixed their product, but they were so heavily burned by last year that they didn't want to come and play this year. I can't really blame them; once burned, twice shy. But we'll never really know, will we?
MX Logic participated last year, but didn't get into the "final fab five" or whatever it was. I am not sure why they didn't participate this year. You'd have to ask them.
The Buyer's Guide is free and there is no fee to be included. Any spam product can be in it, and all were invited.
We didn't have a #1 choice, but NO ONE helped us with the review process. Where do you see that we had a choice, and where do we say that they helped us with the article?
Do you actually read the article, or do you just post?
The buyer's guide definitely is just pure marketing numbers. The article gives more realistic performance numbers, and that exposes some of what you're bringing up in the text. I found exactly what you're reporting (and mention it as an issue): vendors advertise based on 'oh, yeah, we throw out 50% of the mail using RBL-type technology...' kinds of things. It's broadly dishonest, which is why the performance numbers in the article are so very important to revealing 'worst case' scenarios.
> From deep within the article: >"Although these tests were conducted with the > assistance of Borderware, we where careful to > ensure results where fair and objective."
So deep that... they must be in some other article. I don't know where you cut-and-pasted that out of, but it sure wasn't the article referenced in this post.
Slashdot Mirror
I'm sure that if they're serious about actually showing that the statistics are useful then we can find 10 random sites who are willing to be 'ethically hacked.'
... well, they could read all those posts that are on the web site. Except they wouldn't be nicely formatted, but real men write HTML with vi anyway. Maybe they could store or corrupt data with the injection, and maybe they couldn't. Maybe (and this is most likely) they could cause the script to blow up. Is that "hacking" a web site? Hell, I get script explosion errors from web sites WITHOUT hacking them.
The astonishing thing is that most people who will read this press release just don't get it, and the depths of their not getting it are even more astonishing...
I am challenging the conclusion, not the data. I believe that they think that they have found vulnerabilities. I suspect they have found a lot of lousy code. No surprise here. 70%, sure. I'll bite off on that number. I'm not arguing with that.
But there is a huge difference between turning a vulnerability into a breach. Let me give you an example. A lot of Cross-Site Scripting attacks let you steal cookies. So they probably found those. But the question is: when you have a cookie, what can you do with it? Can you steal important data? Can you turn that cookie into a breach? Good web sites that use them also tie cookies to your IP address, which means that if you steal my cookie, you got nothing but crumbs. So the point is not that there are these vulnerabilities, but that they have done nothing to show whether these vulnerabilities are truly breachable and able to get an attacker real useful data.
Same for things like directory listing. You can do that to my web site. Is that a security problem? No, in fact, I turned it on specifically. If I didn't want people to read it, I wouldn't have put it on the friggin' web server.
Is a web site that's susceptible to an SQL injection attack hackable? Depends on where you get to inject the code. I'm sure that someone who put their mind to it could take a web site like, say, slashdot, and inject some SQL. Then they might be able to
Is being able to view a script a security vulnerability? it depends. It depends on the web site. The script. The webmaster's intentions.
What percentage web sites actually have data that's worth anything?
So the point is not that they've found a lot of theoretical issues, but whether they've actually found security issues. And the only way, in my mind, to see whether they have is to see if the issues can be exploited. If they can, I'll pay up. If they can't be exploited, then all they've done is made long lists of things that don't matter from a security point of view.
Very long lists.
Reacting to spam as an electronic vigilante seems to cause much more collateral damage than actually solve any problems. Most of us have been victims of unintentional Joe-job attacks, and that's bad enough... how's a couple million misdirected emails in a week to help your to-do list?
The same thing is true of people who go after spamvertised web sites and the ilk.
We used to run a public-access traceroute server specifically to help people diagnose network problems. It got picked up by an anti-spam site and started doing 10s of thousands of traceroutes a day.
Then, the "error rate" began to get to me. Calls at 3AM from morons who couldn't figure out what the hell they were doing, but had somehow had my name show up in some traceroute they were doing. Threatening letters (from more idiots), and even the occasional DoS attack.
Just got too much and I basically had to cripple the resource because of the false positives.
Extremely bright and careful people might be able to track & attack spammers, but for every one of those, there are a dozen or a hundred who will mis-type an IP, a URL, or be the unwitting dupe in a third-party DoS attack or in attacking a cracked system.
I'm not fan of spammers, but the collateral damage that can---and will---be caused by ANY automated 'strike back' system suggests that this is not the path to solving the spam problem.
>By the way, I have never had anyone tell me
>his/her legitamate email was rejected by my
>server. What false-positive rate do you consider
>to be acceptable?
In our test, we had a bunch of vendors claim that because they were not aware of false positives that somehow there were vanishingly few. This is even true of systems that don't quarantine mail, although this is patently ridiculous---how are you even going to know about false positives if your mail disappears into a black hole? The answer is their marketing people pull numbers out of some occluded place normally covered by pants and claim those as false positive rates.
This, of course, is complete crap. False positives happen and mail has become so 'lossy' that we never even hear about some of them. For example, during our testing, I discovered a false positive in one of the RBLs that I was looking at. I found out because someone pointed it out (they got a 5xx or 4xx and had the presence of mind to send it in). Once I KNEW about the problem, I could go back in the logs and see a half-dozen other people who were affected by the same RBL error---but never bothered to complain or say anything.
This problem gets worse when you consider content-based systems, because they tend to false on the mail that you care the least about---bulk mail, mailing lists, etc. If you subscribe to a mailing list and it receives 50 messages a day, will you miss one? Almost certainly not... but that's likely where the false positives are happening.
InfoWorld, for all their faults, had a good idea to somehow separate out false positives from bulk mail as opposed to person-to-person mail. It's an interesting concept and I considered putting it into our test as well, but eventually dropped it. The reason is that some mailing lists are, to me, MORE valuable than person-to-person mail. For example, I'm on some software development project mailing lists, and a message there is likely to be more important to me than a message from a clerk saying "there's a package for you at the front desk from Amazon."
With all of this being said, I think that rejecting mail at SMTP time is really the way to do it. Whether you do it RBL or content or both, the fact that you can refuse to accept responsibility for the mail wins in so many ways. I depend on email heavily, and I fall into the "every sperm is sacred" school of thought with email. Quarantines are OK, but if I could know that either I got the mail or it got 5xx-rejected to the upstream, I'd prefer that solution by far. (Actually, the REAL preference is to let the network manager decide whether they want to accept/tag-and-deliver-quarantine/reject the mail)
I'm not sure that I agree that putting AV in front of AS is going to change much. What we do with AV is what people should be doing with AV today: delete that crap. So what that means is that the AS products didn't see viruses, or cleaned messages. Back before mass-mailing worms, we used to see a few viruses a day that could be cleaned and such, but now we see a constant flow of about 1000/day (about 10% of our flow) where they are ALL worm traffic. I can't find a non-worm message in this month's log, in fact.
So what would the effect on AS be? Well, some AS products DO detect mass-mailing worms (sure, why not); those guys didn't see them. Some AS products do NOT detect worms; those guys didn't see them. So I am not sure that we're going to see much of a change in the behavior, given that whether a worm is spam or not is something that varies from vendor-to-vendor. (It tends to be spam if the vendor only makes AS tools; it tends to be a virus if the vendor makes both and packages them in a single device).
In general, our detaching the product from first-hop-SMTP had to have some effects, as you & I have both just noted. Intuitively, what I think you would see is a higher spam catch rate (i.e., lower FN) but no change in the FP rate. For example, a common technique we are now seeing is that spammers will hit an open HTTP proxy, and do a POST to smtp-server:25 with a bunch of crap that happens to also contain a valid SMTP transaction. Many SMTP servers will accept what is effectively pipelined SMTP (even if it is not negotiated). So, if you were in the SMTP engine, you could look for "POST / HTTP" before the HELO/EHLO and say "this guy is a spammer, screw 'em, reject."
That would increase your spam catch rate if the spam would not otherwise have gotten caught. It should not increase your FP rate.
One thing I thought would be true is that people would have lots of subtle tuning that they wanted to do. So we opened up this review to let them do that---everyone could SSH or RDC into their boxes and try and tune things up. What we observed is that only a tiny number wanted to do that. My thinking is that the vendors did not have very subtle dials to turn regarding spam catch/false positives that might have been affected by this insulation we were artificially creating. So, I am inferring that FN might have gone up by some small number but FP not at all.
I regard that as A Good Thing. I think that 1st generation (whatever that means) anti-spam was VERY 'person' intensive in terms of tuning, management, etc. Actually, I KNOW that they were intensive; I wrote an anti-spam product back in 1995 that effectively implemented what we call "Graylisting" now, as well as put in rate-based controls. That was great for its day, but you had to watch it all the time to make sure that big senders didn't get blocked.
However, I believe that most email admins are sick and tired of playing with their anti-spam software. I talked to a bunch of them when I was constructing this test because I thought that the antispam market must basically be "done." Don't we already have nearly 100% adoption of anti-spam products? The answer is "yes, but..." Most of the people I talked to were looking for a better solution. They were early adopters and were dissatisfied with the amount of effort that went into maintaining their anti-spam solution. (Or they were looking for features, such as quarantine, that might not have been in the original product they bought.) They want it to work, and they want it to work well without a lot of screwing around. This is not an unreasonable thing to ask. We used to screw with O/S tuning all the time; then the O/S guys figured out that they could do it better than we, and now we rarely have to do that. Same thing for databases, etc. Now, the anti-spam guys are raising the state of the art as well.
This is why a lot of new products (and services) are coming out which are essentially untunable or are only tunable at a very coarse level. The
And the issue of spamtool/zombie identification is a real one; thanks for bringing that up. As is the much more important issue of the 'sending' IP address. The 'sending' IP address problem in this methodology is one that can be dealt with by a good product (and several products can and do deal with it just fine); some others are so restricted that they cannot work effectively except as first hop. To me, that's a bug. I had one VP of Marketing scream at me "no one ever puts our product anywhere except as first hop," and I barely held back from saying "yeah, that's because your product is such a piece of crap it can't go anywhere except for first hop."
However, the "looking at the SMTP conversation" part is impossible to really deal with. But, I'll note that the majority of the products sit on top of an MTA. Either they install for themselves or they use tools like sendmail, qmail, or (for Windows boxes) the MS SMTP MTA. So those products don't actually know anything about the SMTP stream. There are a few products that specifically brought up this issue because they DO look at the SMTP stream, and they probably did not do as well. How "not well?" Maybe a couple of percentage points in spam catch rate. Maybe less.
In the review, I wrote a short side-bar where I admitted this up front: (http://www.nwfusion.com/reviews/2004/122004spamsi de5.html) "You may notice our numbers are not as optimistic as the marketing literature from vendors' products. There are four reasons for this:
1. Side effects from our test bed probably shaved a few points off of each product's ability to identify spam. ..."
I also brought that issue out when I wrote : "The false-positive and false-negative rates we found are useful for comparing products but a real installation will likely have a lower false-positive rate and higher spam-catch rate." (and mention things like the SMTP catch rate) in http://www.nwfusion.com/reviews/2004/122004spamsid e.html
OK, so now that I'm done defending myself, what's the point? Well, one of the vendors told me "you know, all these products basically have no false positives and catch all spam." But that's completely wrong. We discovered a bunch of products that are still dark-age when it comes to catching spam, ones which have enormous false positive rates (in particular).
However, if you look at the top 10 or 12 products, you can see that while there are differences, they are not showing a huge variation in behavior.
What this means is that you can take a test like mine and use the spam catch rate/false positive rate as a "first cut." Because I believe that where you want to make your buying (or implemention, in the case of open source) decision is based on things besides just spam catch rate/false positive rate.
We have to do the FN/FP rate tests just to say "you must be this high to attack this problem." But from then, there are huge differences in the products, and that is what is important. I don't want to seem like I'm lashing out at the people who say "oh, I use (insert product here) and it never falses and never misses," but those folks just don't get it. It's not the spam catch rate that differentiates the products; it's everything else.
An easy example is CloudMark. Talk about Zen. This product doesn't do
Yes, it does help. You know all those RBLs that everyone seems to really love to much? It's complaints that get them populated (among other things).
So it's kind of altruistic if you ONLY complain and never use an anti-spam product, but there are a bunch of tools like DCC (distributed checksum clearinghouse), Vipul's Razor, and almost all of the RBLs that depend heavily on folks making sure that the bad guys are exposed...
You wrote:
"Every once in awhile I get a false positive, but the percentage is very trivial, and the problem gets immediately solved because, unlike content-based systems, the sender immediately knows his mail didn't go through and can bypass it manually."
You're mixing things up here incorrectly.
If I use an RBL OR if I use a content-based filter, I can either quietly drop or noisily bounce mail. I can do it at SMTP time, or I can do it after I've accepted the mail. Your choice of technology has nothing to do with what you choose to do with the mail.
Now, of course, there are products that don't give you that kind of flexibility, but that's a defect of the product and has nothing to do with whether the technology does or does not meet your needs to reject RBL-ish mail at SMTP-time.
Absolutely. SO there's a buyer's guide so you can learn about ALL the products. AND, then there's this mammoth test that is an independent, objective, scientific look at products that are a subset of the Buyer's Guide who wanted to engage in head-to-head competition and comparison.
We're aiming for the best of both worlds.
In our test, it was very clear that we wanted to look at head-to-head products. You can't match TDMA against Brightmail---they do VERY different things for VERY different people.
If you wanted to write a feature, you could talk about Graylisting, TDMA, Content Filtering, etc., all at the same time and discuss the pros and cons of the technology. But in a test, you want head-to-head.
Magazines consider features and tests to be very different things.
So, in this article, we ONLY looked at SMTP-in/SMTP-out products which were directly comparable. We didn't look at TDMA, graylisting, client-side tools, commercial services (like gmail or yahoo or hotmail), or mail server add-ons that don't talk SMTP. And there were still too many products...
FrontBridge and MessageLabs pulled out of the review at the last minute, after their products had already been in place and in operation. You can infer any reason you want to that.
d e2.html
A full conflict-of-interest discussion appears in the review at http://www.nwfusion.com/reviews/2004/122004spamsi
Several of the vendors said something to the effect of "how can you do a fair review when you use one of our competitors?" (Obviously, the competitor would be 'disconnected' during the review, something that I think FrontBridge and MessageLabs didn't quite understand). My only answer is that anyone who thinks that they can review anti-spam products but is not yet using them is clearly not qualified to discuss the matter.
It'd be like asking my Mom to review Linux vs. Windows vs. Mac. Hey, she doesn't use any of those, so she's unbiased!
Actually, the reason that trade magazines don't often review open source materials is that they don't fit well within the methodology of how we review products.
Magazines cannot look at all products. So they have to look at the ones that have significant market share and are of interest to significant numbers of their readers. It is rare when an open source product rises to the level of prominence of a commercial product---for every Apache, BIND, Linux, and Spam Assassin there are a thousand other perfectly good programs that aren't really applicable.
But when the open source stuff does get sufficient market share, it's still very difficult to review. Products that enterprises use generally have things like technical support, documentation, release cycles, and such. Open source products tend to be underdocumented, undersupported, and come out "whenever the author thinks there should be a new revision." It's hard to put these things up against each other in a head-to-head review.
Much more importantly is the use of open source as a 'toolkit' for people. Enterprises use LOTS of open source things, but they generally do so by picking-and-choosing pieces and assembling their own best solution. This is often because the commercial products can't meet their requirements! But you can't review a "tool" in comparison with a fully blown commercial product.
Look at Spam Assassin, for example. It's one piece in a much bigger picture of how to manage spam. But it's just one piece. Without a mailer, quarantine support, a GUI, complete documentation, and technical support---it's not appropriate to put Spam Assassin up against a commercial product.
Does that mean that it cannot be a VERY successful tool that you build into your own customized spam solution? Absolutely not! But it does mean that if we download an RPM of some product and it doesn't measure up to a fully supported, fully documented, fully configurable, fully featured commercial product, it really doesn't seem like a fair review.
It's HARD to review open source products when there is not someone willing to say "I will represent that product." In the case of this review, we actually had a bunch of people who had incorporated SA into their commercial products---and so SA got a fair review. It just didn't get called "Spam Assassin."
What is VERY true is that commercial concerns have nothing to do with it. The editors want to serve the readers, and the readers have told us that they want to hear about open source. So the fact that SA will never buy an ad is irrelevant, at least for the reputable publications.
The only thing I can say about RBLs is that you need one that is an amalgam of others. This is the same theory that drives SpamAssassin: you may be able to fool one, but you can't fool them all.
I am doing testing with SenderBase and it gives any IP address a -10 to +10 score. Pick your own false positive/false negative threshold and you can slice out a big chunk of garbage. But SenderBase is not generally available except through a web interface. It's gone through a couple million messages of ours with one false positive.
I know that Symantec/Brightmail and Postini both have their own 'reputation-based' services as well that seem to work.
What I don't know of is any RBL that is itself an amalgam of other RBLs, returning a score (as opposed to a "go"/"no-go" answer). My own luck with RBLs before SenderBase was so poor that I basically discounted them as either (a) not helping enough to be worth the effort or (b) too many false positives.
A number of the products that I looked at had "RBL voting:" they lookup things in more than one RBL, and if they meet a threshold you set ("must appear in 2 RBLs..."), then the message is marked as spam. Others consider the RBL as a component---if it's in an RBL AND has "Viagra" and a URL in it, then it's probably spam.
I think that either a combo-RBL or RBL-voting has to be the way to go.
They seem to have gotten a lot better in the past couple of years.
Right. The table is poorly designed. This is what happens when you take a spreadsheet and feed it into an art department. The spreadsheet actually has 10 different columns.
i de3.html)
But we don't want to add percentages. That would not be fair. You have two columns, each sorted from "best" to "worst" and you can read down each of the columns. What they should have done is broken them into two separate tables rather than one.
SO you have to actually say whether you care more about false positives (read left table) or false negatives (read right table) or make your own conclusions based on the combination of the two.
In general, people love scorecards and pure rankings because it means they don't have to actually come to their own conclusions. (I'm not accusing you of this; just explaining why articles often have them) In this case, the real answer is that you have to decide what's important to YOU and then you can rank them youself. For example, if you don't want to quarantine and you have zero tolerance for FP, a few FN won't bother you. On the other hand, if people have good quarantine or you REALLY need to see the FPs, but you want the FN rate to be low, that's a different set of criteria.
I'm opposed to compromises and mixing statistics together that don't belong together; see the rant about statistics in the review itself. (http://www.nwfusion.com/reviews/2004/122004spams
It's hard to compress six months of research into an article, even one as long as this one...
For the spam test, approx. 75% was spam. The mail flow was about 30K messages. In Feb of 2003, the number was almost exactly 50%.
One of our consulting customers with a well-known domain name reports 95% spam (1 in 20 is not spam) over 20+million a day. Another one with a VERY well known email address reports 99%+ spam to that address---but around 50 to 75% for other, less well-known addresses.
Thanks for the compliment... because, you see, I first used the methodology in 2003, in the original Network World test (see http://www.nwfusion.com/reviews/2003/0915spam.html ).
m demo.html)
Or, you could go back to February, 2003, and see the same methodology being prototyped at the Demo conference (http://www.nwfusion.com/reviews/2003/0224antispa
Let's see: Feb 2003: 2 products.
Sept 2003: 16 products, with 4 top overall performers.
Dec 2004, 36 products, with 12 top overall performers.
And Network Computing? 23 products with 10 finalists, in between my two reviews for Network World.
Yeah, I'm feeling like what Network Computing does in between my reviews makes me a copycat...not.
What are you, a NWC ad salesman? Or just a bit clueless yourself?
RTFA. Postini was in it, both in the big table and in the Dirty Dozen finalists.
Yes, you're right; it's an error. My notes show that you can turn on SSL for management, but what got written in the article is wrong. It'll get fixed online immediately. That crept in as part of the editing process.
On the other hand, I don't understand why ANYONE ships ANYTHING that talks on port 80 anymore. It's not like OpenSSL hasn't been proven through-and-through (or you can write your own). Port 80 might be fine for pictures of your vacation, but the management interface on a corporate mail server should be encrypted and authenticated.
However, if you want to discount a 10,000 word article for a single error, then you're going to have a hard time believing anything you ever read anywhere ever.
My apologies:
t ml
http://www.nwfusion.com/reviews/2003/0915spam.h
is last year's review.
You would have difficulty finding stats that support the 95% assertion. Folks like Brightmail & Postini and SenderBase aim closer to 50%, but it's a different statistic: that's blocking 50% of the incoming TCP connects, not 50% of the spam. In our own testing before the spam review started, I got numbers similar to those using SenderBase as the reputation-based scoring ahead of our mail servers.
I would agree that a well-designed reputation-based DNS blacklist can immensely increase the spam catch rate AND block a bunch of mail before it hits the servers. However, if you did the intersection of all the random RBLs out there, you'd end up with an enormous false positive rate.
You can also take DNS BL information and mix it into your cocktail. I discussed that topic specifically in the article in greater depth.
It's also a question of environment. I have friends who have little 2-or-3-person mail servers that basically intersect ALL the blacklists they can find and are perfectly happy---because they don't correspond with more than a couple hundred different people. But talk about a big corporation with thousands of users, and the DNS BL strategy doesn't work quite as well because of the false positive issue.
As with everything, different strokes for different folks...
GFI got a horrible review last year. The product they submitted was a pure 'word checker' (i.e., if you've got Viagra anywhere, you're spam) and so their false positive rate went through the roof. They also had some horrible heuristics, such as "if you're not on the "to:" line, it must be spam." My experience is that it was architected for a small office where you can tune it out the wazoo. They have since (I have heard) fixed their product, but they were so heavily burned by last year that they didn't want to come and play this year. I can't really blame them; once burned, twice shy. But we'll never really know, will we?
MX Logic participated last year, but didn't get into the "final fab five" or whatever it was. I am not sure why they didn't participate this year. You'd have to ask them.
Thanks for reading the article so carefully.
The Buyer's Guide is free and there is no fee to be included. Any spam product can be in it, and all were invited.
We didn't have a #1 choice, but NO ONE helped us with the review process. Where do you see that we had a choice, and where do we say that they helped us with the article?
Do you actually read the article, or do you just post?
The buyer's guide definitely is just pure marketing numbers. The article gives more realistic performance numbers, and that exposes some of what you're bringing up in the text. I found exactly what you're reporting (and mention it as an issue): vendors advertise based on 'oh, yeah, we throw out 50% of the mail using RBL-type technology...' kinds of things. It's broadly dishonest, which is why the performance numbers in the article are so very important to revealing 'worst case' scenarios.
> From deep within the article:
>"Although these tests were conducted with the
> assistance of Borderware, we where careful to
> ensure results where fair and objective."
So deep that... they must be in some other article. I don't know where you cut-and-pasted that out of, but it sure wasn't the article referenced in this post.