Slashdot Mirror
Plan for Spam, Version 2
bugbear writes "I just posted a new version of the Plan for
Spam Bayesian filtering algorithm. The big change is to mark tokens by context. The new version decreases spams missed by 50%, to 2.5 per 1000, even though spam has gotten harder to filter since the summer. I also talk about how spam will evolve, and what to do about it."
The latest development Spamassassin has an interesting application of Bayesian filtering. Basically, it takes all of SA's existing heuristics, uses that to develop a sense of what is and is not spam, and then pumps the results through a Bayesian filter that learns from these messages.
As with any other SA test, no single element of the chain is trusted enough to definitively call something spam, but if a message would have squeeked through before, this new filter can put the final nail in its coffin through word analysis against previous spam.
So, why did I use a subject about "ENDING spam"? Because one of the tools that spammers have is SA itself. They can use it to score their messages and determine how "spamish" it is. The problem now is that each SA installation will have subtly different scoring, and the message may be "ok" according to the spammer's version, but my version has a better sense of the mail that *I* get.
SpamAssassin is definitely a tool worth checking out if you have not already. Install it in daemon mode (spamd) and then use "spamc -f" in your procmailrc or the equiv for your MTA.
Very nice tool, and a real time-saver for me.
I'm really excited about all of the neat stuff happening with Bayesian filtering and related technologies, but I just wanted to put in a plug for TMDA, Tagged Message Delivery Agent, which uses a whitelist-centric strategy. Since I began using it, the amount of spam I have to look at is virtually at zero. If you haven't read about it yet, check it out.
Just curious - did you follow up w/ him to see *why* he thought you signed up to receive the spam? Is it possible that you inadvertantly allowed them to send you spam the last time you renewed your driver's license? I ask because most of the spams I get say "you signed up with one our partner sites" and i've always wanted to (but have been too lazy to) go back and see how far up the chain I could get.
The article mentions compiling a vast collection of spam. Such a project is already underway at SpamArchive.
Content-Type: text/html (or text/plain)
Content-Transfer-Encoding: base64
Because a lot of filters don't know how to decipher this. For me, this makes it a lot easier to filter, though. I get no legitimate e-mail encoded this way, so I just have procmail dump any e-mail encoded this way. Problem solved, and without the CPU burden of decoding or running expensive spam filters.
The url for the project is popfile.sourceforge.net
I didn't try it yet, but it I will try it really soon now!
OK, signal and noise. What if the signal was all in one frequency band and the noise all in another. Problem separating them? No.
What if, in effect, a similar distinction held for spam in the transmission channel - that spam by itself selected a pathway to the recipient that was never used by the signal? Block that pathway and the spam never gets through.
Spam doesn't select a pathway but spammers do. If you could block relay spam at the open relays it would be dead. You can't, of course - the open relays are controlled by people who don't know the need to block spam. You know that, I know that. If you can't change the people then change the open relays (from the spammers' points of view.) Set up a system that looks like an open relay and stop the spam. An open relay honeypot.
I asked an operator of such a honeypot how he did last year:
> How did 2002 end?
From March 7 to December 26 2002, the total was:
235,624,232
Using one Pentium 90 he stopped spam to 235 million recipients. Think about that number when you see filter people reporting what they stop just for their own domains. This was spam to recipients all over, not simply to the honeypot operators domain: he operates at the relay level. He stopped 100% of the spam, no deception deceived him, no tuning was needed, no valid email was caught - it is perfect filtering. Perfect filtering - who else has that?
And you can do it at home on your DSL or cable connection (the guy above uses sendmail -bd, but Windows users have a program they can use):
http://jackpot.uk.net/
Yeah, I know, spammers are switching to open proxies. So, write an open proxy honeypot. That, too, will be 100% efficient. In addition you now are giving spammers reason to fear every open relay and every open proxy they detect. FEAR. The SPAMMERS have to scramble. They have to scramble and they have to show everything they do to overcome the technique - there is no stealth way to look for open relays and open proxies.
The problem is solved, it is a matter of implementation and of getting active systems everywhere in the net space (so there's no safe IP space for the spammers anywhere.)
Remember: A single Pentium 90, 235 million spam messages stopped in 10 months.
It doesn't catch all the spam, and it occasionally has a false positive. This will be true of any spam filter we implement, because spam continues to change. SpamAssassin runs on some of the mailservers I connect to, but it tends to perform worse than Mail.app. So until we can get each user's spam filter customized at the server, spam identification is going to have to stay client-based. It sounds like Paul Graham's tools are getting a little more efficient, but does any of this make a big difference for the end user?
Can't you see that everyone is buying station wagons?
...until the email server at work got hacked and someone stole the entire address list. Since then, all of us have been getting spam by the bucketloads. And since I depend on people being able to get my current work address, I can't change it. Thank God for SpamAssassin!
-Looking for a job as a materials chemist or multivariat
It's all fine and dandy to have a spamtrap account if you never plan to read it, but what if you want to get online bank statement notifications or other important notices? I just noticed my friendly credit card company (Capital One) took it upon themselves to introduce my previously spam-free e-mail account to their business partners so they could introduce me to the wonderful world of buying fucking flowers for valentines day. Thanks alot assholes. And no, they have NO option to opt out of this fucking crap. The spam is posted from the same address as the statement notifications with a friendly disclaimer saying they're not in any way affiliated. Nice.
I host several domains as a hobby for my family. Recently my ip address made it into a listing on spews.org. Am I a spammer? By no means. Am I screwed? Absolutely. After reading spamming newsgroups I found that I am not alone. At first I was just getting blocked because I was sending mail ( my own smtp server ) from a "known" spamming source when in fact I'm not a source of spam. My IP happens to fall into a larger block of ip's that my ISP owns, some of which are sources of spam.
This was a minor setback, but now other services are starting to use bulk email sources as deny lists for their offerings. My free dns provider, zoneedit now prohibits me from adding / modifying any of my zones. This is simply not acceptible to me. The way spews is set up, it is not easy for my ip to get off the list. My ISP cannot just call them up and take me off. There has to be a way to avoid this, and eliminating spam at a higher level would be a good start.
sneakemail.com is my new way of eliminating spam.
It sounds like you're using TMDA. Or, if you're not, you should be. :) Check out my related post on this story.
I went through over 500 spam a day down to about 3 or so and I figured out that those last 3 are due to the fact that they are bypassing the filter (I have a bunch of different urls and the server that it is all hosted on also has its own name - so mail sent to that username at that host doesn't get sent through any filters and the way that the filters are setup there - pair.com - I can't trap that particular servername).
I have been very impressed with SA and am writing scripts to track the stats even better (I love seeing what it has pulled out everyday).
So far I have had zero false positives out of about 1-2megs of mail being filtered everyday for nearly a month now.
SA has multiple different ways of searching the mail - any one of them can be easily bypassed by any given e-mail - but all of them together are really damn good at getting rid of spam.
I'm very impressed with it and how well it learns (although straight "out of the box" - or perhaps I should say "straight out of the tar.gz" it brought me down from 500+ spam to 5-10 a day and then I tweaked how my accounts were filtering into SA and that fixed the rest.
There are some odd things afoot now, in the Villa Straylight.
I think you misunderstand how easy bogofilter is.
I initially trained on about 200 emails. At first, I got 1 spam per day, or so. There have not yet been any false positives (good mail classified as spam).
A week later, I get 1 spam in my inbox every 3-4 days, and no good mail has been classified as spam. All I need to do it take the false identifications and re-classify them. That means, every 3-4 days I take the spam in my inbox and re-scan it through bogofilter (cat SPAM | bogofilter -S). That is all. It is not any effort, really, after the initial training. Then, the filter does all the work, and you don't need to worry about blacklisting or whitelisting or anything.
The really important thing is that the filter statistically optimizes YOUR manual email classification. The best source of email classifying is YOU looking at an email, and Bayesian filtering is the only method that is optimized to do that.
Wouldn't work. The algorithm only cares about the most statistically significant 15 words. TEENS easily beat yams.
Another program is SpamPal, which also acts as a pop proxy. It also has a plugin structure, and one of the plugins is a Bayesian filter. This is in addition to included support for using available spam blacklist stuff like SPEWS, ORDB, SpamCop and a whole bunch of other DNSBL lists (even the ability to block entire domains like .kr, .ch and so on). It's a rather cool piece of software.
-'fester
Without using filtering software.
1. Change your e-mail address and drop the old one. (This way you are starting off with a clean slate and not on any mailing lists.)
2. Make sure your ISP dosent post or sell your e-mail address.
3. Make your email address simple for people to rember but hard for a computer to crack example m1nam3@isp.com. Use simular methods as you would in making a password. That prevents common name email address.
4. On your webpage make a CGI/PHP/ASP whatever form to send you an e-mail. When you want people to e-mail you give them the link to that page. Make sure that there are no prameters that can make your program e-mail others, and also that your e-mail address is not listed in any of the source that is visable to the web user.
5. Only give your e-mail to people you can relitvly trust. If you cant trust them then give them a link to you weppage.
6. When filling out forms on the network asking for your e-mail ether use an alternate e-mail or read the companies privicy clames and make sure that you do not check or uncheck something stating that they will send you e-mail or adds.
7. Use spamassasan or other email filtering on your system.
8. Forward all spam to ucs@ftc.gov with all the headers.
9. See if your email client has a automatic bounce back. If so bounce the message back to sender.
10. if you want to post your e-mail address then I would make a graphical jpg, png as your e-mail. That way it slows down most computers from reading it.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
One of my ISPs's implementation of SpamAssassin seems to be using it as part of their rating heuristic.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Praed argued, very eloquoently & persuasively (hey, he's a lawyer :) that there are laws on the books banning spam in nearly every state. All you have to do is find a way to bring those laws to your assistance. In particular, note that:
As a lawyer that has successfully prosecuted a number of spammers, Praed was able to talk about all of this with some authority. He cautioned everyone though that laws will never eradicate spam -- as he put it, "people still rob banks since that's where the money is". But legislation & prosecution can still be a very valuable tool in fighting spam, and an important supplement to things like better mail filters. This is a big problem, and is going to need a variety of tiered solutions to control it.
DO NOT LEAVE IT IS NOT REAL
Hi, that was me . Unfortunately this only works for Outlook (not even Outlook Express), but it's been working great for me.
As others have pointed out, Vipul's Razor is a great open-source solution.
Checking SourceForge , I found the following additional packages:
BogoFilter
SpamAssassin
JoeEmail
Bayesian anti-spam classifier
Anti-Spam SMTP Proxy Server
Bayesian Mail Filter
JunkFilter
SpamProbe - fast bayesian spam filter
Mailfilter
IMAPAssassin
That's just from the first page of search results. If you'd like to see all the results (I did a search for "spam" from their search box), click here .
I feel fantastic, and I'm still alive.
It's not just about the US, in many European countries spam is illegal already now (clear cases are Norway and Austria), and the European Union as a whole has decided to outlaw spam, it should be implemented this year. I don't know exactly about the situation in East Asia, but I don't think the Chinese and Koreans like it too much that their resources are misused for sending spam all over the world, so they could follow soon. Yes, there certainly will be some smaller countries where spam is still legal, but once spam is illegal in the European Union, the United States, China and many other big countries no one who has sent thousands of spam mails to harvested addresses can reasonably claim that he or she believed that all the addresses were only of people in a few offshore countries.
Furthermore, the US American conception of law has, as far as I know, the principle of being applicable exterritorially, which is in general quite controversial, but could be useful here - it would probably be possible to forbid any companies that do business in the US to send spam, even if the spam is only sent from other countries and only to people living outside the United States.