The Spam Problem: Moving Beyond RBLs

whirlycott writes "I just published a paper called The Spam Problem: Moving Beyond RBLs on my site. I comprehensively describe RBLs and list eight specific problems with them. I also get into ideas that next generation antispam system creators should read. I hope that this will be useful to anybody who is attending the Spam Conference at MIT on Jan 17th."

Comments

You know, that would suck.

[aetherspoon]

(refering to the intro in the article)
I mean, you can compare it to having your entire town roped off because one person was a fraud... completely destroying said town, but you still live in it.

Wasting an entire netblock by blacklisting it is not good....

Not true

[Anonymous Coward]

There was this woman who spammed and made a living out of selling anti-spam services. A bit like the mob, really.

She doesn't really on people clicking through - all she needs is to drive up the irritation factor.

EFF said it better

[Lumpish Scholar]

whirlycott's article points to the Electronic Freedom Foundation's Public Interest Position on Junk Email (Google cache), which begins:

Executive Summary: Any measure for stopping spam must ensure that all non-spam messages reach their intended recipients.

For the past several years, the Electronic Frontier Foundation (EFF) has watched with great interest the debate regarding what to do about unsolicited bulk email from strangers, or spam. We have been asked to lend our support to bills that have been introduced in Congress, and we have been approached in various other ways to help lead the fight against this annoying intrusion into people's email mailboxes.

While members of the EFF staff and board find this unsolicited email to be as annoying as everyone else, we believe that the two most popular strategies for combatting it so far--legislation and anti-spam blacklists--have failed in their fundamental design. Anti-spam bills have been badly written, are unconstitutionally overbroad, and frequently wander into areas where legislators have no expertise, such as the establishment of Internet standards. And anti-spam blacklists, such as the MAPS RBL (Mail Abuse Prevention System Realtime Blackhole List, the most popular), result in a large number of Internet service providers (ISPs) surreptitiously blocking large amounts of non-spam from innocent people. This is because they block all email from entire IP address blocks--even from entire nations. This is done with no notice to the users, who do not even know that their mail is not being delivered.

The focus of efforts to stop spam should include protecting end users and should not only consider stopping spammers at all costs. Specifically, any measure for stopping spam must ensure that all non-spam messages reach their intended recipients. Proposed solutions that do not fulfill these minimal goals are themselves a form of Internet abuse and are a direct assault on the health, growth, openness and liberty of the Internet.

Email is protected speech. There is a fundamental free speech right to be able to send and receive messages, regardless of medium. Unless that right is being abused by a particular individual, that individual must not be restricted. It is unacceptable, then, for anti-spam policies to limit legitimate rights to send or receive email. To the extent that an anti-spam proposal, whether legal or technical, results in such casualties, that proposal is unacceptable.

Re:EFF said it better

[Zeinfeld]

Executive Summary: Any measure for stopping spam must ensure that all non-spam messages reach their intended recipients.

The problem with the vast majority of psuedo-solutions to spam is that the promoters simply will not listen to any ideas other than the one they first thought of and they simply won't listen to people who point out that blocking good mail is a serious problem.

The 'cry me a river' response is as idiotic as it is arrogant. SPAM is a problem, failure to deliver email is a bigger problem.

That does not mean that we don't address the problem of SPAM, it just means that we have to approach the problem from both ends, identifying the good signal as well as eliminating the bad.

The MIT conference is likely to be a failure because the organizers are only presenting the tried and failed filtering approaches of the past. Those approaches are now well understood, they can mitigate the problem but can never do more than that. Filters suffer from reverse network effects, the more widely used they are the greater the incentive to program arround them.

Blacklists fail for many reasons, not least complete lack of accountability. As the paper reports the operator of one blacklist that claimed to only list open relays actually listed sites for other reasons. Ultimately a blacklist that does not have some robust accountability structure is simply a vigilante operation. Vigilantes are frequently popular with people who think they are victims of crime regardless of whether they create more problems than they solve.

The tools we need to start applying are digital signatures and email authentication in combination with whitelists. This follows sound business process, if you want to talk to someone well known their secretary will use a two step process, first ask who you are and check to see if you match the access criteria (e.g. to set up a cold call meeting with a Fortune 100 CEO you had better be a Fortune 500 CEO), then check to see if you really are who you claim to be.

Authentication and Authorization requires no heuristics and there is no feasible counter-strategy for the spammers.

I believe that the way to stop spam in the long term is to deploy signed email ubiquitously. Self signed certificates are sufficient for this purpose if we can provide a lightweight authentication via a DNS-linked PKI.

For example consider the problem of stopping spam to email lists. These are a prime target for spammers as the email server does most of the work. As a result most email lists are now filtered so that only subscribed readers can post. This has in turn been gamed by the spammers who use automated tools to scan the archives of an email list and send emails with forged headers purporting to come from another subscriber. Authentication and authorization prevents this mode of attack.

The counter-argument to using authentication is that the spammers can get their own credentials. If you spend some time analysing SPAM however you will find out that this is unlikely. Almost every spam has forged or obscured headers. While this does not prove that this is a requirement it is certainly indicative of the fact that the spamers do not want this type of visibility.

Even if a spammer can get a credential they are most unlikely to get a credential that would match my personal whitelist which would consist of the signing keys of the email lists I subscribe to and the domain names of the member companies of W3C and OASIS.

One possible solution ...

[JSkills]

Ok this one's not for everyone. What we did at goofball.com is to set up a user configurable spam filtering system based on a combination of rules and use of the RBL.

There is a simple web based front-end that allows users to add and modify rules for accepting or rejecting mail based on a variety of factors - all saved in the datbase. Things like checking the subject, to, from, or the body of an incoming email for the presense (or lack) certain strings is a simple example.

All of this is done is Perl using Mail::Audit of course. I know there's Spam Assassin, but this was a little more fun (and customizable) for us.

The final check is the Realtime Blackhole List. When we first implemented this solution, we noticed in the logs that almost everything was on the RBL (even mail from yahoo.com). In fact, our own server was on the RBL. We'd never sent spam before, but I'm sure our relay was open at one time or another.

Since the system is configured to look for "accept mail" rules first, the solution came down to adding "accept" rules for pretty much everyone we knew, so that mail from known parties would be accepted even if on the RBL.

So now I get no spam at all - ever. I get very little mail at all in fact. It's really analogous to having an unlisted phone number. It's not the perfect solution by any means, but I'll take it any day over slogging through literally hundreds of spam mails every day ...

Re:RBLs in Spamassassin

[spacefight]

I don't really see why anyone would use RBLs just by themselves.

That is easy. While spamassasin does the work pretty good - you still have to download the whole crapload. RBL enabled MTAs won't accept any email as soon as a blacklisted IP wants to connect. This saves bandwith, disk space, client side filtering (read: cpu time) and so on.

Re:Bollocks!

[Doc Hopper]

You've voiced an opinion on Slashdot which will be both unpopular and likely to quickly be moderated down. What we have here is two people arguing about an opinion:

• Spammers are not thieves. Under your logic, the weekly coupons your supermarket sends out snail mail would make your supermarket thieves.

Contrary to what many anti-spam activists would have people believe, you are correct. Spammers are not thieves. They are, however, engaged in an unethical form of guerrilla marketing which has the net effect of shifting costs of advertisement to the consumer, rather than to the advertiser. Much like the RIAA labels people that infringe copyright as "pirates" and "thieves", to little effect, calling spammers "thieves" is probably over the top and unlikely to bring positive change.

• Just be thankful that with computers you can filter the through all the static. You can't filter out billboards, newspaper ads, loudspeakers, etc. etc. etc.

The key difference you've missed is where the costs are borne. My company pays $650 a month for our T-1. By 9 AM this morning, we had received over 11,000 attempted emails. We have 300 employees. Of those 11,000, roughly 200 were legitimate mail. The rest were spam, double bounces, or roughly 4,000 attempts in nine hours to send mail to addresses which do not exist in our domain. We used to accept these and send bounces directly from our Groupwise server; I put Groupwise behind a firewall and Postfix mail relay shortly after I was hired here, and noticed that in two weeks we had over two gigabytes of double-bounces sitting in our queue. Yes, I use RBL, Anomy, and SpamAssassin. Nevertheless, the amount of time that I have had to spend to limit the problem to manageable levels that don't drive us out of disk space and bandwidth has cost my company dearly from payroll. I can drive past billboards and ignore them. I can choose not to listen to the radio or watch television. I don't miss much by not watching TV or listening to the radio, and thankfully any important global news I get relayed through my co-workers. In today's world, however, it would be extraordinarily difficult to decide to not use the telephone, or, if you conduct a great deal of business using electronic mail, to suddenly decide to stop using the medium entirely.

I'm not saying you are wrong, but it seems to me that both the "spammers are thieves" and "spammers are not thieves" arguments are not quite hitting the mark. Whether unintentionally or intentionally, a single unsolicited commercial email can end up collectively costing the world a great deal of money. It seems that the best analogy I could use is that spammers are like cigarette smokers. A smoker's behavior hurts only themselves, but the secondary aspects of their behavior (second-hand smoke) impact the health of others, and so that portion is regulated. It's difficult to find a public building in the U.S. these days that allows smoking due to the known, unhealthy side effects. But it took the human race hundreds of years of dealing with "annoyance levels" of the problem before coming face-to-face with the predictable health consequences of the smokers' actions.

No offense meant to smokers! I know it's a hassle to have to go outside when it's sleeting and windy in order to find a place to smoke. Spammers, also, are exercising their right to free enterprise and free speech, but, ultimately, I think spammers will find themselves in a similar regulatory position, that they must practice their craft only in designated, acceptable areas, and that spamming outside of those lines will have significant legal repercussions. Digital signatures are part of the solution, as are whitelisting and blacklisting. If spammers can be forced to operate legitimately, using only legitimate information so that they can be contacted and held liable for their actions, it would be a truly enormous step in the right direction. I don't think that part will happen through legislation, but through very large installed bases of users beginning to use mail platforms which transparently implement this kind of functionality...

Passive denial doesn't work

[The Spoonman]

It only blocks LEGITIMATE e-mail from servers that may, at some time in the future possibly, be used by spammers as a relay. It does block from machines that have sent spam, but also those that have never done it, just the potential is there. It does not, however, block spam! At least, not effectively.

And, that's where the problems lie. Administrators are putting these things in, assuming they'll stop spam, and then getting pissy when you tell them legitimate mail isn't getting through.

I used to be the e-mail admin for my company. We somehow ended up on the worst of these lists, osirusoft. This, despite the fact that we used SMTP AUTH; YOU COULDN'T SEND MAIL WITHOUT A PASSWORD! And, once you get on one of the lists, you're on them all.

So, I spent the better part of a couple of days going through them all and having to prove I wasn't an open relay. They all but one removed us within a week, but that was a week we couldn't send mail to a few customers.

And, the one that didn't remove us in a week...osirusoft...they took over a month. Every day I went to their site and ran the "autotest". Every day I watched it say, "Relaying Denied, deleting from list". Every day, I watched another "proof" of our spamminess posted onto their list.

And, the idiot admins of the ISPs? "Well, you're obviously an open relay. I see dozens of spams being sent from your site on the osirusoft list!"

BTW, the osirusoft rbl is run by some loser in his basement. Great plan, basing your company's e-mail on some unemployed idiot with a chip on his shoulder.

Look at your spam, where does the majority come from? That's right, AOL & Hotmail. But, your company would NEVER allow you to block from them, they'd lose too many customers. Install an active filter, you'll see better results and less spam.

Re:Time to ditch SMTP

[mstefan]

The contents of the header fields in a message have nothing to do with SMTP protocol itself; the sender is identified with MAIL FROM at the beginning of the transaction and could choose to validate the address if they wish (either returning a 550 or 553 result code if they don't like it for some reason). Requiring that the From: and/or Reply-To: header fields match the return path means that you're effectively eliminating relaying, "smart hosts" and some gateways that forward messages from different mail systems.

And, bottom line, it wouldn't do anything to stop spammers, or even slow them down. So what if they have to use a "traceable user account" with some service provider? Spammers move from provider to provider (often providing false information), or they use spam-friendly/neutral providers who don't give a damn as long as they get their monthly fee.

ISP Volume Reduction and Defense In Depth

[billstewart]

If you're an individual user, a computation-intensive spamassassin approach can do a really good job of blocking most spam and blocking very little non-spam. But if you're an ISP or Mail Service Provider, having a conservative RBL can save you a lot of resources, including bandwidth and computation, by throwing away the high-volume relay-abuse spams with as little work as possible, saving the more complex work for mail that's less likely to be spam. (By conservative, I mean "trying to only block actual relays and other known spammer systems", as opposed to "broad-spectrum insecticides and lists that do collateral damage to pressure ISPs or harass their competition.") That might be a 25-50% reduction in total email that the ISP needs to handle, but from an instantaneous-resources standpoint, it's probably higher than that, because spam tends to come in high-volume blasts, while real email is mostly Poisson arrivals. And if an ISP's failure responses are the "Temporarily inaccessible, try again later" type as opposed to permanent rejections, real email systems are much more likely to try again later than spammers are (though of course open relays may still try again later, because they're just mal-administered, not necessarily broken.)