Distributed Spam Detection

A reader writes "There's an interesting project at SourceForge, called, "Vipul's Razor", that uses a gnutella like system to let users exchange spam "signatures" to filter spam. I work at an ISP in Ottawa, we have been using it for last two weeks to stop bulk of spam coming to our POP3 accounts. More impressively, it hasn't tagged any valid mail as spam yet. Here's the scoop from its webpage: "Vipul's Razor is a distributed, collaborative, spam detection and filtering network. Razor establishes a distributed and constantly updating catalogue of spam in propagation. This catalogue is used by clients to filter out known spam. On receiving a spam, a Razor Reporting Agent (run by an end-user or a troll box) calculates and submits a 20-character unique identification of the spam (a SHA Digest) to its closest Razor Catalogue Server. The Catalogue Server echos this signature to other trusted servers after storing it in its database. Prior to manual processing or transport-level reception, Razor Filtering Agents (end-users and MTAs) check their incoming mail against a Catalogue Server and filter out or deny transport in case of a signature match."" Cool idea. I'm up around 80% spam a day on my main mail account. Might be worth a try.

Comments

So...

[DagSverre]

...what stops this from being abused? Say I set up a box that automatically reports all mails on the most popular mailing lists as spam, effictively making the ISPs around the world start to filter out the mailing lists...

It's a great initiative, I really hope no troll out there takes my word on this and actually do this.

Re:Great use of p2p -- Wont work.

[DLG]

>> This wont work. All that will happen is that the spammers will just modify their spam programs to slightly modify each message they send out.

It will however require them to send each specific message separately rather than sending large cc's or using some sort of relay. That alone is a big step since right now most spammers can get away with sending a single email message and relying on an open relay to retransmit to a larger group.

Furthermore I have doubts that for the time being this project will concern spammers. Infact I am pretty sure spammers are not really interested in wasting their own time trying to spam people who consider spam a violation. It is more convenient to ignore those people (which is why they don't bother to check if you want spam or not before they send it to you).

DLG

Virus Detection

[doorbot.com]

This seems like it would be a great method for virus detection on a non-Windows machine. For those of you who run *nix mail servers which eventually filters down to Windows clients, having a mail tagged as viral would be nice to have it be immediately denied at the server. So I'm assuming all it would take is a smart admin to tag the email as spam, and then it will propagate around to the other servers (less than 1k would transfer!).

Re:Great use of p2p -- Wont work.

[kevinank]

Interesting work, but I notice that you are only examining trigrams, and you are using an even weight factor. To improve selection you probably at least need to use variable weights (a fuzzy logic neural network rather than binary logic) and train the network with more sample spam.

I've been working on a similar project but using additional factors that help identify spam such as violations of the mail RFC's, and other header indicators, in addition to NLP. I have a prototype that I'm using to score all of my inbox e-mail and am using that to tune the weight factors and add in new factors as I encounter them. It would be interesting to combine your approach with mine I think, since I hadn't thought of analyzing trigrams.

Anyway, if you are interested send me an e-mail and I'll give you my current perl code.