Slashdot Mirror

← Back to Stories (view on slashdot.org)

Distributed Spam Detection

Posted by CmdrTaco on from the interesting-ideas dept.

A reader writes "There's an interesting project at SourceForge, called, "Vipul's Razor", that uses a gnutella like system to let users exchange spam "signatures" to filter spam. I work at an ISP in Ottawa, we have been using it for last two weeks to stop bulk of spam coming to our POP3 accounts. More impressively, it hasn't tagged any valid mail as spam yet. Here's the scoop from its webpage: "Vipul's Razor is a distributed, collaborative, spam detection and filtering network. Razor establishes a distributed and constantly updating catalogue of spam in propagation. This catalogue is used by clients to filter out known spam. On receiving a spam, a Razor Reporting Agent (run by an end-user or a troll box) calculates and submits a 20-character unique identification of the spam (a SHA Digest) to its closest Razor Catalogue Server. The Catalogue Server echos this signature to other trusted servers after storing it in its database. Prior to manual processing or transport-level reception, Razor Filtering Agents (end-users and MTAs) check their incoming mail against a Catalogue Server and filter out or deny transport in case of a signature match."" Cool idea. I'm up around 80% spam a day on my main mail account. Might be worth a try.

23 of 304 comments (clear)

  1. SpamBouncer by joib · · Score: 5, Informative

    I'm personally using SpamBouncer, a procmail-based spam filter. Works fine for me.

  2. Great use of p2p by astrashe · · Score: 5, Insightful

    This is a great use of p2p -- something that doesn't involve piracy. I wish I had heard of it before.

    Are there any other innovative non-piracy p2p apps out there that we should know about?

    1. Re:Great use of p2p by Sarcasmooo! · · Score: 5, Informative

      Just because most people on a P2P network use it for piracy, it doesn't become a pirate-app. I can, and have, used programs that are under attack by the RIAA do download speeches, text documents, etc. At the early point of the 2000 Nader campaign, when he couldn't get 30 seconds of time on M$NBC (much less a place in the debates later on), I used Napster and Scour to find speeches he's given. And when the Department of Commerce kicked of it's 'Safe Harbor' privacy program by failing to put the confidential information provided by the companies involved on a secure site, I downloaded the pages in a zip file despite the site being closed for a fix. Using programs like Scour, I found reading material on scientology, COINTELPRO, and more, all the way up until the day that lawsuits shut them down.

  3. So... by DagSverre · · Score: 5, Interesting

    ...what stops this from being abused? Say I set up a box that automatically reports all mails on the most popular mailing lists as spam, effictively making the ISPs around the world start to filter out the mailing lists...

    It's a great initiative, I really hope no troll out there takes my word on this and actually do this.

    1. Re:So... by Greyfox · · Score: 4, Insightful

      Spammers themselves are generally interested in ways to disrupt those lines of defense. If this project grows in popularity and shows itself to effectively block spam, they'll start gunning for it. Considering potential holes in the system before that starts happening really isn't a bad idea.

      --

      I'm trying to teach myself to set people on fire with my mind... Is it hot in here?

    2. Re:So... by dev0n · · Score: 4, Insightful

      Seems like everyone hates spam with a passion, except maybe the spammers themselves

      well, i would have to disagree with you on this point.. i work at a web hosting company as the technical support manager, and handling abuse complaints falls into my realm of responsibility... and i have found that a significant number of first time spammers do not KNOW that spam is "wrong", and get quite upset that they were "taken" by companies that send bulk messages on their behalf. i had one gentleman send me an apology letter that actually made me feel sorry for him. he, and many other people on our network, have never been repeat spammers.

      i know that there are many people out there who don't care, but we can't automatically assume that all spammers are evil. some of them are just ignorant.

  4. Authentication with servers? by GlassUser · · Score: 5, Insightful

    I read some of the documentation, but I can't find details on a couple of questions. Do the servers authenticate with each other? It was implied, but how deep is it? Are the SHA signatures signed to the originating server (or client/trollbox) too? I think this kind of model is great, but if you don't have some nifty authentication/accountability, it can be wide open for abuse. I'm sure anyone reading slashdot can imagine a vengeful spammer flooding the network with bogus or malicious hashes.

    --
    funny munging
  5. Fighting spam by Brian+Kendig · · Score: 5, Informative

    I'll post my usual public service announcements here:

    SpamCop is a great service for reporting spam; just paste the spam message into the web form, and it'll automatically figure out where the smap came from and send complaints off to the appropriate people.

    The Spam Bouncer is a procmail-based personal spam screening tool. It's got some interesting features, but I haven't used it in a long while.

    The way I avoid spam is to have my mail client screen out any email which contains any of these phrases:

    to be removed
    to be permanently removed
    to get removed
    to get off the list
    to get off this list
    to be taken off
    to remove yourself
    removal instructions
    remove in subject line
    "remove" in subject line
    remove in the subject
    "remove" in the subject
    'remove' in the subject
    S.1618
    S. 1618

    This list by itself catches about 80% of the spam I get.

  6. idea won't work if reaches critical mass by intuition · · Score: 4, Insightful

    Razor catalogs spam by hashing the entire text of the message. Later potential spam is "detected" by hashing entire texts of messages to see if the hash matches any of the existing hashes in the spam catalog.

    To get around this all a spammer has to do is change/add at least one charachter to each spam. This would make all the hashes unique and no spams would be detected.

  7. How do you compute a signature? by cperciva · · Score: 5, Informative

    As far as I can tell from a quick glance at this, it looks like the entire message body is being used to compute the signature. This isn't going to work very well -- over half of the spam I receive is "personalized", and that fraction is growing every day.

    This could work very well, but we need some way of computing signatures which will be invariant across different copies of personalized spam for this to be effective.

    --
    Tarsnap: Online backups for the truly paranoid
  8. SpamAssassin uses Razor by wideangle · · Score: 5, Informative
    From http://spamassassin.taint.org/:

    SpamAssassin is a mail filter to identify spam.

    Using its rule base, it uses a wide range of heuristic tests on mail headers and body text to identify "spam", also known as unsolicited commercial email.

    The spam-identification tactics used include:

    • header analysis: spammers use a number of tricks to mask their identities, fool you into thinking they've sent a valid mail, or fool you into thinking you must have subscribed at some stage. SpamAssassin tries to spot these.
    • text analysis: again, spam mails often have a characteristic style (to put it politely), and some characteristic disclaimers and CYA text. SpamAssassin can spot these, too.
    • blacklists: SpamAssassin supports many useful existing blacklists, such as mail-abuse.org, ordb.org or others.
    • Razor: Vipul's Razor is a collaborative spam-tracking database, which works by taking a signature of spam messages. Since spam typically operates by sending an identical message to hundreds of people, Razor short-circuits this by allowing the first person to receive a spam to add it to the database -- at which point everyone else will automatically block it.

    Once identified, the mail can then be optionally tagged as spam for later filtering using the user's own mail user-agent application.

    SpamAssassin requires very little configuration; you do not need to continually update it with details of your mail accounts, mailing list memberships, etc. It accomplishes filtering without this knowledge, as much as possible.

    Call your ISP and ask if they use it.
  9. This is just a temporary solution. by mrsam · · Score: 5, Informative
    Spam generators have been trying to hash-bust these kinds of filters for years now. A four year spam generator automatically appends random junk at the end of the Subject header or at the tail end of the message, in order to defeat the early hash-based spam filters.


    This is probably a 'fuzzy' hash function that should ignore minute variations. However, it goes without saying that if this hash-based spam filter becomes widespread, then the spammers will simply figure out how to hash-bust their way past it.


    To have any hope of working over the long term, this kind of an approach must include the ability to distribute not just the hashes themselves, but the hash function as well, so that the hash function itself can be adjusted, when needed.

  10. One way around potential abuse. by chris_7d0h · · Score: 5, Insightful

    To eliminate the situation where one person posts a lot of "incorrect" signatures, a ranking system could be applied.
    The thought goes like this.
    A person submits a signature of "identified" spam mail to a "supernode" for ex. and the submission gets a ranking of 1. Each additional submission (by other users) increases the score by a number.

    This way, there are several classifications which could be used to filter incoming mail. For the mail providers, they could opt for only removing mail matching signatures with a very high score (thus very likely these will be actual spam) or they could filter anything reported.

    The purpose of allowing the use of classifications is that it will take longer time to get higher scores, since more people have to report the specific spam mail. Some people whish to eliminate things the least bit suspected, but mileage may vary.

    Do you see a resemblance with the ./ moderation?

    --
    In a society that believes in nothing, fear becomes the only agenda ~ Bill Durodié
  11. Bogus hashes won't tag valid mail by morzel · · Score: 4, Informative
    The beauty of a cryptographic hash function is that it's purely one-way: it is very easy to check if two messages are the same (they calculate to the same hash), but it is nearly impossible (or at least very very very hard) to calculate the message for any given hash.

    Injecting random hashes into the network won't result in valid emails being tagged, but can flood/DOS the catalogue machines.

    It would be possible to create hashes for a number of "probable" emails, but diversity in messages is so big, the chances are quite slim to actually stop a legitimate mail.

    --
    Okay... I'll do the stupid things first, then you shy people follow.
    [Zappa]
  12. X-YahooFilteredBulk by Malc · · Score: 4, Informative

    I noticed that a lot of spam coming through my Yahoo account had been tagged with the header "X-YahooFilteredBulk". I added this to my Exim system filter and I've gone from 20+ spams a day in my inbox to 2 in a week. Thank you Yahoo!

    Unfortunately, a lot anti-spam measures (including Exim 3's system filters) only take place after a message has been accepted for delivery. For me, this results in a lot of bounce messages frozen in the queue as they cannot be returned (Hotmail mailbox full, etc). I've switched on features like verifying the sender and the headers, but this doesn't catch them all, and in some cases might even stop some legitimate spam (one of my mailing lists uses incorrect syntax for the "RCPT TO:").

    More effective anti-spam systems need to filter before the message has been accepted. If you wait until then, it is already too late and it is on your system. No, refusing accept delivery is much effective IMHO, and forces the MTA's further up the chain to deal with it. They shouldn't have accepted it in the first place! When you get spam, return 550 (or whatever the code is) and let the SMTP client deal with it. In an ideal world, ever provider (ISP, or free service like Yahoo) will implement stricter MTA's. If the spam rejection can be pushed far enough up the chain, life for everyone will easier.

    BTW, according to Philip Hazel (a message I recieved to a question I posed on the Exim mailing list), Exim 4 will offer much more functionality along these lines, including the invocation of C funtions after the DATA phase of the SMTP input. I guess this would be the spot to plug in Vipul's Razor, although I don't know what kind performance hit that would lead to. Mr. Hazel also pointed out that some stupid clients are in contravention of the RFC and will continue to try and delivery a message if they recieved 5xx after the DATA phase... oh well: they'll be using my bandwidth but they won't be putting any crap on my server.

  13. Re:Great use of p2p -- Wont work. by DLG · · Score: 5, Interesting

    >> This wont work. All that will happen is that the spammers will just modify their spam programs to slightly modify each message they send out.

    It will however require them to send each specific message separately rather than sending large cc's or using some sort of relay. That alone is a big step since right now most spammers can get away with sending a single email message and relying on an open relay to retransmit to a larger group.

    Furthermore I have doubts that for the time being this project will concern spammers. Infact I am pretty sure spammers are not really interested in wasting their own time trying to spam people who consider spam a violation. It is more convenient to ignore those people (which is why they don't bother to check if you want spam or not before they send it to you).

    DLG

  14. Virus Detection by doorbot.com · · Score: 5, Interesting

    This seems like it would be a great method for virus detection on a non-Windows machine. For those of you who run *nix mail servers which eventually filters down to Windows clients, having a mail tagged as viral would be nice to have it be immediately denied at the server. So I'm assuming all it would take is a smart admin to tag the email as spam, and then it will propagate around to the other servers (less than 1k would transfer!).

  15. One flaw, depending on your perspective... by wirefarm · · Score: 4, Interesting

    I spent the last few days hacking together a bulk mailer in perl. I did so with a lot of sensitivity and a bit of trepidation and a lot of social engineering to my employer who wanted to put together a way to send invitations to a party via email, rather than the very expensive snail mail method that we had been using.

    This was emailed to our real customers - our 'A list'. These are the people who get invited to these parties each time - people who come and enjoy the food and drinks, no strings attached.

    But, yet, technically, it *is* bulk email and this first time, unsolicited. A very large percentage of the people responded enthusiasticly that they want to remain on the list for this, but a few (8 out of 3500) asked to be removed from the list. One guy seemed annoyed and I typed him a personal apology. (In fact, I doubt that this guy read the email before sending off his remove request.)
    What if that guy had submitted the email as spam to this system?
    In that case, the rest would miss out on coming to a good party.

    I hate spam as much as anyone on slashdot. I was asked to set up a bulk email and found that it could be done in a way that was not offensive in this case. Had it conflicted with my conscience, I would have refused.

    Maybe the system needs some sort of moderation as a filter, too. At least that would allow valid bulk email to survive one trigger-happy end-user.

    Ok, go ahead and tell me that I'm wrong in this...
    Cheers,
    Jim in Tokyo

    --
    -- My Weblog.
  16. List of server-based spam filter systems by tgeller · · Score: 5, Funny
    A canonical list of server-based spam filtering systems is on the SpamCon Foundation site, along with other sysadmin resources.

    --
    Tom Geller
  17. Foreign spam removal by wideangle · · Score: 5, Informative

    For the many /.ers who:

    a. Use Outlook secretly
    b. Receive loads of foreign spam
    c. Don't know any foreign languages
    d. Don't have any foreign friends
    e. Don't have any friends

    This Outlook rule is for you!

    Apply this rule after the message arrives
    with
    Ô or ¾ or Ç or or É or ½ or Í or ò or Ë or ® or Ä or ã or Ï or Ö or Ô in the subject or body
    delete it
    and stop processing more rules.

    This blocks 99% of foreign spam. Sue Mosher wrote about other effective methods for killing spam in Outlook. Finally, before you reply saying "You dummy, that filter works in any client!" -- You're right.

  18. Re:Great use of p2p -- Wont work. by friscolr · · Score: 4, Interesting
    Maybe some kind of AI algrorithm

    everytime spam gets mentioned on slashdot, someone says this, and everytime i respond with the work i've been doing-
    pattern matching spam
    uses word counts and phrase counts from known spam and known good mail to match against incoming mail. requires a certain amount of known spam/not spam, but otherwise it has a good rate of matching spam/not spam and doesn't require the incoming mail to at all known beforehand.

    --

    -f
    www.blackant.net

  19. Re:Great use of p2p -- Wont work. by kevinank · · Score: 5, Interesting
    Interesting work, but I notice that you are only examining trigrams, and you are using an even weight factor. To improve selection you probably at least need to use variable weights (a fuzzy logic neural network rather than binary logic) and train the network with more sample spam.

    I've been working on a similar project but using additional factors that help identify spam such as violations of the mail RFC's, and other header indicators, in addition to NLP. I have a prototype that I'm using to score all of my inbox e-mail and am using that to tune the weight factors and add in new factors as I encounter them. It would be interesting to combine your approach with mine I think, since I hadn't thought of analyzing trigrams.

    Anyway, if you are interested send me an e-mail and I'll give you my current perl code.

    --
    LibBT: BitTorrent for C - small - fast - clean (Now Versio
  20. Answers to some questions raised on slashdot. by vipul_ved_prakash · · Score: 5, Informative
    Hi,

    Some of you point out that Razor's use of SHA-1 signatures can be defeated by introducing randomness in the message. This is true; SHA-1 will eventually be phased out and replaced by a fuzzy hashing mechanism like nilsimsa in future. [http://lexx.shinn.net/cmeclax/nilsimsa.html] [http://www.geocrawler.com/archives/3/2539/2001/7/ 0/6173567/] The protocol is structured to aid change of hashing algorithms seamlessly, without breaking the existing system. Regarding the possibility of poisoning the database, we are working on a reputation system that will assign credit to honest reporters. Once we have a critical mass of users, it would be hard for dishonest reporters to even join the reporting network, much less be able to mount a DOS attack. Some of these issues have been discussed on the razor-users mailing list. The list archives are located at [http://www.geocrawler.com/archives/3/2539/2001/] best, vipul.