Slashdot Mirror

Disclosure: my name is Bruno Bowden and I managed the engineering team on Enterprise Gmail many years ago at Google before leaving to work in venture capital. My profile is www.linkedin.com/in/brunobowden. Though I didn't work on spam fighting directly, I interacted a great deal with the spam team while I worked there.

One of the main architects of the spam fighting system - Brad Taylor - published a scientific paper on "Sender Reputation in a Large Webmail Service" - http://www.ceas.cc/2006/19.pdf. This has a lot of detail about the system. We keep much of the internals secret as it reduces the chance that a spammer can reverse engineer and work around the system. If you'll allow me to be vague, the number of signals it uses was stunning to me. There's a mixture of hard wired tests (e.g. is the sender in someone's address book), reputation (domain and content), machine learning and anything else we can make work.

One of the principle improvements came when we switched to user classification through the "Report Spam" button. People have different opinions on what constitutes spam, so individual filtering is far more effective. It also avoids the politics of certain lists of domains and IPs from third parties which can be controversial. Even then it has challenges, as sometimes users will mistakenly pick out a phishing email and mark it "Report Not Spam". Because of that, Gmail now adds a red warning banner to indicate more strongly what is a likely a phishing attempt. In general, Google has tried to be very supportive of encryption, e.g. DKIM for authentication (and SPF) to STARTTLS for privacy. I would also like to mention the abuse team that works hard to prevent gmail being used as a source of spam, shutting down accounts as soon as possible after suspicious email is sent, then helping affected users to recover their account.

In general, the Gmail has received a lot of compliments on the spam filtering, I'm sure the team will be grateful for the positive comments here on Slashdot. There are still things that can confuse the system, e.g. receiving forwarded email (which might be missing source IPs) or genuine email that is sent to the wrong address. Though the system isn't perfect, I know the team will continue to work hard on it.

Someone at CEAS2006 did exactly this, with great results. The Spamalot system was great fun, and I think they had something like a 19 email chain going on before the scammers gave up. http://www.ceas.cc/2006/5.pdf (pdf notes only)

Many of the questions asked here are answered in the Challenge Call for Participation

Or the overview talk that Rich Segal gave at the MIT Spam Conference.

The guidelines are scheduled to be finalized May 1.

A second issue is that you want current spam; the global characteristics of spam change from week to week. So what's the use of an ancient archive?

And perhaps the biggest problem is that SpamArchive is a hodge-podge of mail from different sources, vetted only by the people who send it in. It isn't a sample of spam in any statistical sense.

Finally, there is no scarcity of spam. Ham is what people don't want to share.

So a collection of spam, particularly an old one sent in by self-selected volunteers, is of little practical use. The hard thing to get is a collection of spam and ham from a common place.

The TREC tests use private corpora that have legitimate mixes of ham and spam. They also use public corpora in which the spam has been carefully spoofed to make it appear to have been sent to the same recipients as the ham. Collecting the spam for the corpus was easy; spoofing was not.

This was a technique described at CEAS 2006 (papers and slides should be on the website). It worked well for the ISP in the States that piloted it, although they were less invasive at first - hosts that had high outgoing email activity got a banner applied over the top of their web pages (or a click through). The idea was the banner got them to ring in and get help to clear their machine or get them to explain what they were doing. There were some other ideas presented too, such as an automated system for replying to 419 scams - that was pretty cool. I think they managed to get a chain of 19 emails to/from this bot before the scammer gave up. Consume their resources if they try and consume yours! :)

Fidelis Assis (who has now gone solo after having participated in the CRM114 project) shows great results for his recent solo effort: OSBF-lua Bratko's PPM spam filter -- the one that did great at TREC -- is not yet packaged as a drop-in filter. Same for my DMC spam filter.

The actual TREC 2005 tests referred to in TFA are here.

TREC 2006 evaluations are now underway.

While it is reasonable to conjecture that spam has changed so as to defeat spam filtering techniques, or will change so as to defeat the PPM technique that did well at TREC, the historical evidence does not support this conjecture. In particular:

• The spam filters tested in 2004 give pretty well exactly the same performance on 2005 and 2006 data.
• New versions of the filters are a little bit better, but not by leaps and bounds, and also get about the same results over the last 2.5 years of data.
• There is no evidence that "Bayesian poisining" is a viable technique for defeating statistical spam filters in anything but a very artifical laboratory environment where the poisoner has access to the recipient's inbox

Andrej Bratko used PPM -- a well-known data compression technique to compress ham and spam separately. Well actually he didn't compress them but just build the statistical model necessary to compress them. Then he simply (tentatively) added the unknown message to each model and chose the one that compressed it best. The general technique of using compression has been mentioned here and elsewhere but Bratko used a much stronger compression scheme and was somewhat clever about it.

I later reproduced Bratko's results using DMC -- a compression schem that I invented 20 years ago -- and got some interesting results. We have a journal article in press describing it and also an evaluation paper at CEAS 2006.

Bratko A., Cormack G. V., Filipic B., Lynam T. R. and Zupan B., Spam Filtering Using Statistical Data Compression Models

Unsolicited, unwanted email that was sent indiscriminately, directly or indirectly, by a sender having no current relationship with the re- cipient.

The TREC tests involved tests on 350,000 email messages. A 92,000 message public corpus from this effort is available for free download.

John Graham-Cumming (no relation to TREC) has created SpamOrHam -- a community-based effort to adjudicate the judgements in the TREC corpus. This'll let us test in a big way Yerazunis' contention that spam filters are better than humans.

Any filter writer can participtate in TREC 2006 by submitting a letter of intent now and a filter in due course.

There's also an upcoming scientific spam conference this summer - CEAS.

If a user has to go into the SPAM box and double check that no mistakes have been made then the system is worse than not having any SPAM checking at all.

I've found greylisting to be the best solution so far

Incidentally, I've been working with Gordon Cormack to try and figure out what the heck went wrong with his first set of dspam tests. So far, we've made progress and ran a successful test with an overall accuracy of 99.23% (not bad for a simulation).

Lynam and I said that DSPAM 2.8.3, in its default configuration, achieved 98.15% accuracy on the same corpus to which Zdziarski refers above. The report also argued that accuracy was a very poor measure of filter performance and that a false positive rate such as the 1.28% demonstrated by DSPAM would likely be unacceptable to an email user.

In recent correspondence, Zdziarski suggested three configurations of DSPAM (available here on the web) that achieved the following results:

dspam(tum) fpr 1.81% fnr 0.80% accuracy 99.20%
dspam(toe) fpr 1.94% fnr 0.59% accuracy 99.16%
dspam(teft) frp 1.85% fnr 0.53% accuracy 99.32%

More detailed results and comparisons will be made available when our current study is complete. Don't take my word (or Jonathan's) for anything; run this filter and others on your own email. But please take great care in constructing your gold standard.

Gordon Cormack

Sheesh! Slashdot has gotten really lame.

"Other anti-spam workers" is none other than John Levine, Ph.D, co-author of the BEST SELLING INTERNET BOOK OF ALL TIME (I kid you not) "The Internet for Dummies" (Now in its ninth edition). Some of you cretins need to read it.

In Commonwealth of Virginia v. Jeremy Jaynes Dr. Levine served as an expert witness for the prosecution. His testimony helped send Jaynes to prison for nine years.

At the second annual Conference on Email and Spam Levine presented a technical paper on his experiences with greylisting.

Dr. Levine is the chair of the IRTF Anti-Spam Research Group. He's a founding member of the Coalition Against Unsolicited Commercial Email. He runs the Network Abuse Clearinghouse.

"Other Anti-Spam Worker" indeed.

Take a good look at Blue Security's product. I think you'll see that it's little more than an HTTP DDoS tool. BlueSecurity claims that it's okay to DDoS spammers, and that they make very sure that only spammers are DDoS'd (although their careful not to call what they do a DDoS).

I'm given to understand that they moved their hosting to Israel when Verio terminated their service for violations of Verio's acceptable use policy. Verio doesn't allow folks to host denial of service tools on their network (nor will any normal ISP do so).

Someone should ask BlueSecurity about their legal threats against Everyone's Internet for attempting to do the same.

These are not nice people. The only difference between them and the normal crop of script-kiddie miscreants, is that they have found venture capital.

Sheesh! Slashdot has gotten really lame.

"Other anti-spam workers" is none other than John Levine, Ph.D, co-author of the BEST SELLING INTERNET BOOK OF ALL TIME (I kid you not) "The Internet for Dummies" (Now in its ninth edition). Some of you cretins need to read it.

In Commonwealth of Virginia v. Jeremy Jaynes Dr. Levine served as an expert witness for the prosecution. His testimony helped send Jaynes to prison for nine years.

At the second annual Conference on Email and Spam Levine presented a technical paper on his experiences with greylisting.

Dr. Levine is the chair of the IRTF Anti-Spam Research Group. He's a founding member of the Coalition Against Unsolicited Commercial Email. He runs the Network Abuse Clearinghouse.

"Other Anti-Spam Worker" indeed.

Take a good look at Blue Security's product. I think you'll see that it's little more than an HTTP DDoS tool. BlueSecurity claims that it's okay to DDoS spammers, and that they make very sure that only spammers are DDoS'd (although their careful not to call what they do a DDoS).

I'm given to understand that they moved their hosting to Israel when Verio terminated their service for violations of Verio's acceptable use policy. Verio doesn't allow folks to host denial of service tools on their network (nor will any normal ISP do so).

Someone should ask BlueSecurity about their legal threats against Everyone's Internet for attempting to do the same.

These are not nice people. The only difference between them and the normal crop of script-kiddie miscreants, is that they have found venture capital.

Here's my solution to the greater unwanted communication Anti-spam paper submitted to Conference on Email and Anti-Spam

Lawrence Lessig has been pushing a bounty system for spammers for a long time. See this Interesting People post, for example. He was still pushing the same concept recently at his talk at the Conference on Email and Anti-Spam (July 2004). I'm surprised that he isn't mentioned in TFA.

I have analytically and experimentally proven that over time, those random words will break your spam filter. I hope to publish a paper on the subject this summer at the First Conference on Email and Anti-Spam (CEAS). If you are interested, contact me by e-mail and I can send you a pre-print once the paper is finished and submitted.

A shaky technique coming from a presentation without a paper at a conference that did not referee submissions or publish a proceedings?

*gasps*

It doesn't require much background reading to understand why naive Bayes works for text classification and just how easy it is to trick it if you want to.

If you're interested in anti-spam research that goes beyond the hand waving and mutual back patting that happens at The Spam Conference, check out The First Annual Conference on E-Mail and Spam.