Slashdot Mirror
Using Email Networks as P2P Spam Filters
Oscar Boykin writes "New Scientist is running a story on using the social network in email as a P2P network.
The idea is that email networks have structure that is conducive to a type of search called percolation search . This means email clients could query the social network of email users to filter spam.
This story is based on a preprint available."
The authors propose that their system have access to inbound and outbound contacts. For trusted email accounts, that might work. But what about email accounts that people may want to creat to sheild their identity (political dissidents, whistleblowers). They would have to live outside of the spam protection network and would, I assume, be seed accounts for spammers.
Am I missing something in this analysis?
"Rocky Rococo, at your cervix!"
Since switching to Thunderbird, I get nearly no spam...maybe one or two per day. I like fancy stuff, but when simple works, go with it!
I'm not a troll, but I play one on Slashdot.
...Now the RIAA's going to sue me for getting spam.
Pulp Audio Weekly - Geek News and Reviews
Imagine the potential for harm if I infiltrated a social network and then identified my enemies as spammers, either deliberately or because I or the software agent I use was somehow tricked into doing so.
Social network-based spam-detection is a part of, not a total, solution, and its limits need to be recognized.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
We do a mailing once to twice a week and there are hundreds of wrong/fake mailaddresses on it. We do not filter this, the mailserver dumps the returns. However now we are already marked sometimes as spam, while we craft the messages in such a way that at least spamassassin does not mark it as spam. It is a user subscription list, certainly not spam. This kind of solution will make this situation worse.
My wife's sketchblog Blob[p]: Gastrono-me
Granted, I just skimmed the article, but isn't this exactly how Razor works? (simplified) Communities of people flag messages, senders, etc. as spam, and the mail server (or in my case, spamassassin) compares the messages to the community spam archive for matches before delivery.
Do you really need reason for beer? Wingman Brewers
..but most of the large spam companies do exactly that, in fact their selling point to company A is that they have thousands of other customers, so if customer B gets a spam or virus, customer A is protected even though they never received it.
many many of the hosted and appliance based commercial spam filters do exactly that, they report up to the home base.
You click a multi-user message as marked as spam, then it turns into spam for everyone else too.
God spoke to me.
So it can't deal with spam that includes a unique random ID and would tag emails from a mailing list as spam. Once more: nice try, but it won't work in the real world.
In Korea, only old people get P2P spam.
Actually, I think we should find a way to attach the same stigma to spam customers that we do to the spammers. Why do spam customers not have to go to jail? They're as much the problem as the spammers.
I can see something like having all the spam customers' names published online, so you google for "spam" and "lheal" and up pops my list of purchases. The other spammers then get a very clean list of people to spam. Over time, the net would be segregated into those who like spam and those who don't.
Yeah, unworkable idea, but so are all the others.
Raise your children as if you were teaching them to raise your grandchildren, because you are.
When you thought it was safe to use email again...
Man, and I thought a Percolation Search was what I did last weekend in backwoods North Dakota when I couldn't find a local Starbucks.
He who knows best knows how little he knows. - Thomas Jefferson
What strikes me is that the idea of "pooling information" isn't really new. When one yahoo-mail/HotMail/Gmail user marks a particular mailing as spam, it affects the likelihood that the same email would be marked as spam for other yahoo users. So, the idea of "pooling information about spam" (from article) is already in use! However, it would be nice to create explicit protocols to allow such data (what mailings I have marked as spam) to be made public so that people using other email providers or their own mail servers can share in this pool of knowledge. Of course, the big three email providers (yahoo mail, hotmail, and gmail) will be foolish to make this information public: the spam filtering is one thing that makes a yahoo/gmail account more attractive to potential users! Good idea in theory, but bad business prospects. To add insult to injury, there is no way for the researchers to profit from the arrangement.
This just in, Spammers now use virtual social engineering to clog new P2P network spam filtering, Inboxes bulging with "extra inches"
If I were a spammer:
I'd change an email client to respond with any message from certain folks I don't like to report all of their messages as spam to poison the social network. a couple of clients out there saying "yup, I've already got a message like that here, and my user marked it as spam".
think globally, act locally, right?
This isn't a new idea... except that they propose to integrate it into the mail client and have everybody you've ever sent mail to or received mail from be a potential contact, weighted by frequency that you email them. That's a bit new, but not as effective as it seems.
For one thing, it would block mailing list messages, which are messages that you probably do share with your contacts.
For another, it does not consider that most spam has random keywords seeding into every copy sent, so those would have to be ignored somehow, which introduces a fuzzy match algorithim, which means the possibility of false matches exists, and since you're asking others (probably all using the same algorithim against their databases) you have increased the chances of a false match being found.
In any case, collaborative networks already exist in a better form. Users mark messages as spam when they get them, a flag is created and sent to some central place that all users check against for matches. The algorithim for fuzzy matching resides in one place and is only used as an indicator in spam assassin in any case, not as the sole indicator..
Large scale systems like Google's GMail can use people flagging messages as spam to filter similar enough messages from other users, sort of thing. I'm pretty sure they do something like this, in fact, as my GMail account has *never* made a mistake in it's spam detection.
And so forth. There's better ways than relying on a random query of your contacts to see what they think.
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.
Sorry, I can't read the article. There were too many buzzwords in the post.
I've already had this idea for years, seriously, I almost made it my graduation project.
Create a distributed spam filter that fingerprints incoming mail based on a number of criteria, have the user mark spam with a certain 'undesiredness factor', blacklist email fingerprinted as spam and propagate this information to other people using the same system... This way it should be possible to create some kind of 'network' that classifies email much more reliable than a simple content filter or address blacklist.
I even thought about how to integrate something like this in existing mail transport agents like exim and into email clients to have them forward/download the fingerprints and scores of incoming email before the email itself is sent on, and how you could create some kind of hierarchy of 'filter authorities' that collect all fingerprints and their scores from lower levels, possibly going up from the user to the ISP to the backbone provider... The user could have a special inbox that shows the fingerprints/subjects of messages on the mail server that are marked as 'undesired' by other users of the network, with the option to re-rate them or to download them anyway...
Anyways, the way it is put in this article it will not be of much help to the internet in general, because the spam messages are only scored by the 'social network', but they will still be delivered to their destination, and thus wasting bandwidth...
You could collect email adresses in a hashed form, just like passwords are stored on a server. You would be able to check if the sender is in the list, but not be able to "un-hash" the list back into real adresses. The way to get around it would be for spammers to attach their sender adresses to these funny mails people do like to forward to their friends.
10 ?"Hello World" life was simple then
Skipping past the security issues. One of the goals of spam filters should be reducing network load not increasing it. If we have to send our spam to several differnt peers to be scored this would compound the network load problems. Mostly this is a bad idea(tm) from the get go. I think the only thing that will really stop spam is to force something like pgp(gpg) signatures on all mail. Here's hoping the new national ID cards will have public certs encoded on them. It would be cool if someone would step in and get PKI working for the rest of us. Also we should drag the boddies of spammers through major cities behind a horse, while allowing victums to beat the spammer with large sticks like golf clubs.
I have 2 email accounts;
;-)
bogus@cox.net and real@cox.net
bogus is NEVER used for real communication but freely accessible on different web-sites and sent to a couple of "relevant" news servers
bogus emails are always fetched first and kept 1 week.
real@cox.net has a filter comparing properties from incoming emails to emails in bogus - if there's a match:
emailmessage >> trash!!
That's iT!!!!
Spammers can ranodomly generate content for their spam to bypass this. Have the actual spam message text as a JPEG image followed by random, gibberish text in both the same background and foreground colour so it is invisible. If the system looks for messages that are identical by comparing the text, then spam messages would appear different from the computer's point of view.
This will just be an ongoing thing. Spammers will figure a way around this, then people will get on to what the spammers are doing and fix things so they block the spam again (like filter out text with the same foreground and background, compare JPEGs), but then the spammers come up with new methods... Besides, it seems like in order to compare emails, it will have to distribute more, so this method will cause Spam to take up even more network traffic.
I think a real simple solution would be to have an email client allow only emails from addresses in the address book or from websites that have been bookmarked. Other emails should be kept separate and dealt with as potential spam.
Cloudmark.
I signed up for the free beta and was told that it would be free forever (they were going to charge businesses, IIRC). Then they chagned their mind but said that early adpoters/beta users would get it free for life. Then it left beta and they offered me a $5 discout (one time) for their subscription service (or some other pointless trinket offer like that). As far as I'm concerned they ripped me off.
That set me off trying other things, and I eventually found POPFile, which I use to this day (great software). I've posted this to Slashdot before (a long time ago). Some nice guy from a anti-spam company gave me a code for a free version of their product to be nice (I never used it, I had found something by then and didn't feel like switching again).
The point of all this is that it is a nice method that really works. If there was an open source project that did the same thing, I would use it. Untill then, I've got a solution that works fine.
But this isn't new (if I'm right about what it is, the article is down).
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
This sounds a lot like Vipul's Razor.
Nah... Congress can't make stupidity illegal; they'd lose too many votes. The universe, not being elected, can... but tends to be in favor of capital punishment as a way of preventing repeated behavior.
An utterly illegal and unethical solution would be to start up a V1AGRA spam outfit, and taint the supply so that one pill in twenty was actually a disguised lethal dose of cyanide. This would cut into demand sharply, and possibly decrease average human gulibility. Of course, when you got caught, the electric chair might be the only thing that saved you from the lynch mobs.
A marginally less illegal and unethical approach would be starting an urban legend that in fact evil fanatic group was doing exactly that, and that some number of people are confirmed dead from it, with minor male Hollywood sleaze in the hospital having barely survived due to semi-plausible escape. I hypothesize there is a large overlap between those susceptible to spam purchasing and those who believe urban legends without checking to provide effective innoculation.
Lessee, how about "Al Qaeda, 6, Cliff Robertson, alcohol induced vomiting preventing the full dose from being absorbed" for the first round?
//Information does not want to be free; it wants to breed.
Your post advocates a
(X) technical ( ) legislative ( ) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)
( ) Spammers can easily use it to harvest email addresses
(X) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
(X) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
(X) Anyone could anonymously destroy anyone else's career or business
Specifically, your plan fails to account for
( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
(X) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
(X) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
(X) Bandwidth costs that are unaffected by client filtering
( ) Outlook
and the following philosophical objections may also apply:
( ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
(X) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough
Furthermore, this is what I think about you:
(X) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!
Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.
What is one person's spam is another person's desired mail. I'm not talking about advertising, either. For example, I know for a fact that there are a lot of people out there that "knee-jerk" react to service messages from their bank, credit card, whatever... stuff they even signed up for that they mark as spam. Since I want to get my "your payment has posted" email, do I want to rely on the network of people around me that signed up for the same thing with the same company and report it as spam because they're too lazy to just unsubscribe?
Diplomacy is the art of saying, "Nice doggie!" until you can find a rock.
I have been using Spamihilator for a while now, with the DCC plug-in activated, that checks a fuzzy check-sum of the message with servers that hold a list of other users who use this filter. I have found that it does block a number of newsletters that large numbers of people receive, however a simple list of newsletter definitions do a good job of preventing this problem. I just put this filter with a DNSBL filter that checks Spamcop and other blacklists, and a learning filter, with no spam reaching my inbox, and all real messages getting in fine. More Information: http://www.rhyolite.com/anti-spam/dcc/ http://www.spamihilator.com/
..you insensitive clod!
Harald
Because they won't be part of your trusted network.
Especially hot babes or hunks you meet at parties.
Next!
-- Tigger warning: This post may contain tiggers! --
In the last few months, as gmail's customer base has grown, their spam capturing capabilities have reached about 99.5% with a 0% false positive rate. And I get about 100 spams per day. It has been weeks since Gmail last falsely identified an incoming spam for me.
This type of searching (i.e efficiently searching through a long-tailed distribution) my contacts and archived mail is probbaly just one part of the equation - only about 25% of my email is from other gmail users. But nearly all of my legit email is from people I have emailed to or from before. I am a perfect candidate for this kind of protection.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
People forget that the true detriment of spam these days is the traffic it causes, not cluttering your inbox (if you're smart)
You've got to be kidding. Spam is text, or very nearly so (HTML). Unless you are using floppies and a 2400bps modem, the bandwidth/storage costs are irrelevent.
What is relevent is that it forces people to either use spam filters that randomly throw good messages away or they miss good messages becuase they can't be seen among all that spam. In either case, the loss rate goes from negligable to very noticeable and that makes email a whole lot less useful than it should be.
*Worms*, on the other hand are a storage problem, since they include largish binaries. But the methods for dealing with worms is rather different than for spam.
Thunderbird spam filter chokes once the number of email goes over few thousands. It's not grinding to the halt, it just hits the wall per se - no spam is getting marked at all ...
For people that have to pay the costs associated with building out network capacity and maintaining it, spam's bulk is indeed the main problem.
No, actually it isn't. I run my own mail server. I keep all mail, including spam. I get something like 200 spams/day. All spam for 2004 amounts to a bit more than 100MB. At the somewhat inflated price of $0.50/GB, that is about 5 cents to store all the spam for 2004. You may quibble over the exact number but you would be hard pressed to come up with storage costs over $1.
My ISP charges $5/GB for excess bandwith when web hosting. That means a bandwidth charge for all of 2004 spam of about $0.50
That means the total is still under a buck.
As you point out, the big three provide a massive surface area to do mass mailing tests on but have no commercial interest in sharing this information.
A company called Cloudmark has, for several years now, been running a similar system for cross-provider spam pooling. It's an outlook plugin that weights user opinions against email voted as desirable or otherwise; pretty much exactly as the article describes only on a more centralised (commercial) basis. A friend who ran it a few years back reliably informed me it was very effective.
http://www.cloudmark.com/products/safetybar/howitHere is something I thought you spamassassin users would find useful. Teaching the bayesian filter is difficult when just starting a new mail server, so here is a couple years of archived spam mail for you. Run these through 'sa-learn' and you'll have reduced your spam quite a bit: Spam Archive
[%] Cingular Ringtones
Social filtering of email is already widely implemented. It's not implemented at the level of the end user email client, but at the level of email servers, which compare messages to users and blackhole addresses.
A P2P approach and querying of other people's address books has huge privacy and compatibility problems without any obvious advantages.
This is an on-topic, one line ad for a software program I wrote. If you hate ads then READ NO FURTHER!
My mailserver does everything it can to prevent spammers from using the SMTP DATA command to send their spam.
we must assume you value convenience above privacy.
I was thinking spam could be solved by having any new senders have to click a confirm link to get the mail in the inbox. once added to trusted lists or 2nd email onwards it could skip that step.
/. people. if yer not smart coming in here you are by the time you read all these +5 comments.
this would waste the spammers time and anyone with anything important to say WILL click the confirm link. or those letters you gotta type in for getting a free account on sites.
hope this idea is useful to one of you amazing
bluetigerbc at gmail
I always assume this is what Yahoo!, Gmail, and Hotmail were doing anyways. Why wouldn't they? They have all your emails.....
Yet another fake eBay site.... (>_<);
I Co mmand=SignInFPP
i gn In&UsingSSL=1&pUserId=&ru=http%3A%2F%2Fservlet%2Ee bay%2Ecom%2FForumController%3Fdest%3Dhttp%3A%2F%2F projekt%2Eig%2Dimmobilien%2Ecom%2Fsignin%2Ehtml
:p
:)
I entered a bogus but properly formatted CC# but it appeared to reject it. Oh well. Enjoy the relevant information and use it to avoid being duped....
The phish was sent from 80.247.227.76 in France through a redirect page at href=http://projekt.ig-immobilien.com/signin.html in Germany to the phish site itself at:
href=http://61.190.66.139/ws/index2.php?MfcISAP
in China via the phish email link:
href=https://signin.ebay.com/ws/eBayISAPI.dll?S
Amazing.... Credit card fraud 'courtesy' of miscreants using online resources in three countries using USA e-commerce giant eBay to do their dirty work. Truly an international effort if I ever saw one....
P.S. It looks like eBay closed up this security hole. Good for them.