Slashdot Mirror

← Back to Stories (view on slashdot.org)

How Apple's Mail.app Junk Filter Works

Posted by pudge on from the would-you-like-to-buy-a-monkey? dept.

fmorgan writes "O'Reilly has now posted the second part on an article about Mac OS X Mail.app spam filtering with more details on what this technology is (and isn't): 'Many myths have emerged about Mail's junk mail filter. No, it's not an extremely complex set of rules, no it doesn't look for keywords, and no, it doesn't use white magic ... Interestingly enough, the technology that underlies the Junk Mail filter began its life as an information retrieval system.'"

3 of 273 comments (clear)

  1. Maybe... by ErichTheWebGuy · · Score: 5, Interesting

    Microsoft can learn a lesson here? Especially in the light of this hole, from which a spammer can clearly see that you have opened their messages and validate your address...

    --
    bash: rtfm: command not found
  2. how does it compare to Bayesian? by the+quick+brown+fox · · Score: 5, Interesting
    Is there any hard data out there that shows the cluster analysis actually improves on the better Bayesian algos out there? After all, most of the good ones also achieve the 98%+ that this article cites.

    According to the FAQ of SpamBayes (I think), they're always getting suggestions of ways to tweak their algos that would "obviously" improve the result, but in almost every case it either makes no difference or hurts accuracy, when actually tested on real data.

  3. Sounds sufficiently different to me by Anonymous Coward · · Score: 5, Interesting

    Actually from my understanding of it, its fairly different.

    I thought mozilla used bayesian (which you've mentioned) where words in the email get assigned a probably factor of being spam. These factors are totaled at the end; if the total factor is greater than some predefined value the message is flagged as spam.

    What this does (in my understanding) is count the number of occurances of each word in every email, and store that in a huge table. Then it relates messages together based on these word counts. So its like you get email clusters in N dimensional space, where each axis is a word, and an emails position on the axis is the number of times that emails uses that word. Then the clusters that have a lot of spam mail in in them are marked as spam clusters. All the emails in that cluster are then assumed to be spam

    The advantage to this method I would suppose is to fold:

    A) When you reduce the the N dimensional space, you would start by eliminating noise words (ie words that only occur in a single email). Spam emails that put fake words in to lower their spam probability in the bayesian method would not benefit with this method.

    B) Messages are grouped by content, so its possible that the client could group email by a common subject, kind of like automatic intelligent sorting. They do mention that this technology can be used to generate email summaries. So (in theory) not only could spam be sorted out, but so could any other key topics, like work, relatives, viagra purchases...

    At least thats my understanding of it.