Slashdot Mirror
The Spam Problem: Moving Beyond RBLs
whirlycott writes "I just published a paper called The Spam Problem: Moving Beyond RBLs on my site. I comprehensively describe RBLs and list eight specific problems with them. I also get into ideas that next generation antispam system creators should read. I hope that this will be useful to anybody who is attending the Spam Conference at MIT on Jan 17th."
Tell EVERYONE you know never to click on any spam links, or buy spamvertised products. People spam because it WORKS. The only real way to stop it is to STOP BUYING SPAMMED PRODUCTS.
You'll notice that he listed and then did not address the "Common Arguments and Justifications" for running and/or using a RBL. Just couldn't come up with a reason why privately owned servers have to accept mail from any particular person or group if they don't want to.
1. Don't let a spammer verify your email address
2. Don't post your email address on the internet
3. Secure your email client
4. Avoid common email traps
5. Fight back
Let me know if these can be improved.
Read my sig if you like, but I'll never see yours, thanks to Discussions, Viewing, Disable sigs...
My spamassassin-tagged mail usually scores between 1 and 1.5 ( a 5 is needed for a **SPAM** tag) - which in the grand scheme of things seems to be enough of a weigh for the value of an RBL. Don't absolutely trust it's value, but don't ignore it completely either.
I don't really see why anyone would use RBLs just by themselves. Personally, I have spamassassin catching the "big spams", you know the ones with webbugs, html-only, forged headers, etc. etc. I occasionally tag those as junk in my Mozilla Mail, while tagging my normal mail as not-junk. The Bayesian filter takes care of the occasionally sneaky spam. Once trained it's an awesome combination.
My company was collateral damage on SPEWS last month and I kicked the *^&^#$* out of our ISP for hosting Global Travel on our netblock. They got booted and we got cleaned off the list. Bada-bing bada boom.
5 2%24Db4.726975%40twister.tampabay.rr.com
RBL's are like a fever. They tell you when something it wrong and only a dork blames the fever when the problem is the disease. Get your ISP to whack the spammer or change ISP's.
http://groups.google.com/groups?threadm=Fc6K9.262
My God! It's full of Voids!
Stupid job ads, weird spam, occasional insight at
There is a simple web based front-end that allows users to add and modify rules for accepting or rejecting mail based on a variety of factors - all saved in the datbase. Things like checking the subject, to, from, or the body of an incoming email for the presense (or lack) certain strings is a simple example.
All of this is done is Perl using Mail::Audit of course. I know there's Spam Assassin, but this was a little more fun (and customizable) for us.
The final check is the Realtime Blackhole List. When we first implemented this solution, we noticed in the logs that almost everything was on the RBL (even mail from yahoo.com). In fact, our own server was on the RBL. We'd never sent spam before, but I'm sure our relay was open at one time or another.
Since the system is configured to look for "accept mail" rules first, the solution came down to adding "accept" rules for pretty much everyone we knew, so that mail from known parties would be accepted even if on the RBL.
So now I get no spam at all - ever. I get very little mail at all in fact. It's really analogous to having an unlisted phone number. It's not the perfect solution by any means, but I'll take it any day over slogging through literally hundreds of spam mails every day ...
Having briefly looked at the paper, it seems like the usual complaining about RBLs as being too broad you see all the time in NANAE (news:news.admin.net-abuse.email).
Summary: someone tries to send email and finds that they're listed on SPEWS. They complain because "we're not an open relay", without figuring out just why they're on that list. Almost invariably, they're on the list because their ISP persistently ignores spam complaints and prefers spammer money to honest customer money. I think there's been about two or three actual mistakes in the SPEWS listings in the year or so I've been following NANAE. Otherwise, it's all been a legitimate extension of the block because the ISP knowingly ignores complaints and supports spammers.
Spam is theft. Theft of Bandwidth, theft of service and theft of time. It's that simple. Spammers are thieves. ISPs which support spammers are thieves. Soon, they'll be blocked from the public internet for anti-social behaviour. After all, if your local bargain supermarket ignored the thieves stealing 20% from every transaction you make with them, will you go back?
Many South American and Asian ISPs are blacklisted because they were quite happy to spam everyone when they could steal bandwidth and service from other ISPs. Now that they're blacklisted, they're whinging and moaning about 'freadom of speach', interference with interstate commerce, and other such bullshit.
It's about none of these things. Blacklists are about protecting your network from a Denial of Service attack by spammers.
People who complaing about RBLs (OR DNSBLs, to be more accurate) are missing the point. They should be complaining about spammers who think it's acceptable to steal my bandwidth and your bandwidth to advertise their product..
dave "the only good spammer is a rotting corpse, dangling from the noose"
and
Scalable (resources)
Aren't mutually exclusive?
(1) You (and I) get too much spam.
(2) Your e-mail system administrator (and mine) need to keep beefing up the servers because the sheer volume of e-mail is growing so quickly.
To a first approximations, filters solve (1) but not (2), and black hole lists solve (2).
whirlycott summarizes the problem with (2) in two words: "collateral damage." How much of the e-mail network do we need to destroy in order to save it?
We need to move past first approximations. We need systems that work at the server level, but that somehow address the problems of collateral damage and false positives.
This is only the tip of the iceberg. Any network messaging medium is vulnerable to abuse by spammers. The problem started with Netnews, it continued with e-mail, it's happening now with instant messaging. We need at least high level solution that helps solve the problem regardless of prototcol.
I wish I had one.
Stupid job ads, weird spam, occasional insight at
The problem, as I've said here before, is SMTP itself.
The RFC pretty much states that to be compliant, you have to accept the mail as it is presented. Can't achieve accurate or trusted reverse name lookup information on the sending system? Well, that's tough, take the mail (read this for yourself).
This problem stems from when systems on the Internet were inherrently trusted. That's not the case any longer, and it's time for a new mail transmission standard.
For starters, it should allow system administrators the ability to give priority to systems that can present some form of credentials. SSL or keyed encryption, whatever the standard is, it will permit systems to give totally trusted access to systems that meet the specific security and trust guidelines of the receiving system, not the RFC (times have changed, tough).
Those systems that do not meet minimum trust levels will either have to clean up their act or take the time to contact the remote system to figure out the issue.
It won't stop spam, but it will go a long way to slowing it down and possibly providing some secure method of mail transport in the process.
It's important to realize the point of RBL blocking. It isn't to make end-users happy, it's designed to lower traffic on the mail servers. So a proposed solution needs to be something that the ISP can execute without having to analyze the email. RBLs monitor a single variable, IP, to determine whether it should be accepted or not. If someone could come up with an idea that processed emails based on another single variable, then we'd have ourselves a good spam filter.
One proviso: if anyone complains, I will look at it.
RFCs require that one accepts mail for postmaster@domain.com and from the empty envelope sender. Since I do this, I believe I am fully RFC compliant.
So stop whining about DNSBL. The problem is wider than that, and will not be solved by getting rid of DNSBL. The system isn't perfect, but that is not the issue.
Conversion Rate Optimisation French / English consultant
I have been a very loud protestor about collateral damage in news.admin.net-abuse.email. I well understand the problem but I think you over-estimate it. SPEWS deliberately lists non-spam-source IPS - that's collateral damage, that's wrong and avoidable. Take that away and the remaining collateral damage is unfortunate but not severe.
Many have changed how they use RBLs - instead of simply rejecting they send a reply asking for confirmation the sender is a real human. If that confirmation is made the original message is delivered. That seems to be simple, straightforward, and capable of reducing collateral damage to a very low level. It even has intelligence behind it.
I advocate relay spam honeypots (and open proxy honeypots - move with the times, keep up with the spammers). The white paper doesn't even mention these. The WP has the section asking if open relays are necessary. Well, no, they probably aren't. Is there a point? For how many years has there been an effort to secure open relays? Has it succeeded? The fact is that they are there - asking if they are necessary may inform you but it doens't change the situation in any useful way.
For all these years the spammers have been given free access to the relay level - there's a self-satisfying division into the secure systems run by the wise and the open relays run by inept administrators. that division allows the operator of a secure system to condemn the operator of an open relay with confidence - he can strut. Yipee. As a spam-fighting tool it's a close to a complete bust. Well, yeah, lots of open relays have been secured. BFD - there's still enough for the spammers, and RFC 2505 said it would be this way. Yo: RTFM (in this case RTFRFC.)
You want to hurt the spammers? OK, hurt them. It's not like you have to go out of your way - accept and deliver one of their relay tests and the chances are excellent they'll send you spam that you can discard. That's still a secure system, but it has teeth instead of gums.
There's all these people falling over themselves devising elaborate filters. If you simply open up a relay enough to accept the spam but not deliver it there's no filter needed - a non-mail-server system that receives relay email receives close to pure spam - you will never get a filter as selective as that. Accept and deliver the relay tests and you have screwed the spammer. I won't even enumerate all the ways he is or can be screwed but there's a bunch.
If 5% of the Windows systems with network connections ran Jackpot then spam would be dealt a mortal blow:
http://jackpot.uk.net/
It isn't hard, and it does tremendous good. Check it out.
I'd have to say, yes.
Personally I use Spamcop's RBL and reporting service. I check the held mail page a couple of times a day. I have yet to see a legitimate mail be blocked and it's reduced the number of spams a day I get from hundreds to 2 or 3.
Maybe some RBLs still work the way the author decribes but from what I'm hearing that's not the way many work now. Now it's more like a reporting user recieves a spam (hopefully very near the start of the spamming run) and reports it. The reporting system works out the most probable source and lists it (due to the fact that spoammers often move within a netblock the netblock rather than the individual IP address has to be blocked for the RBL to be effective), the system also mails the admin address for the appropriate domain (and any listed interested third parties) with the information required to identify the spammer and asks them to deal with them. That IP address is also monitored by the RBL. When the spammer stops sending spam or the administrator informs the RBL operator that they've dealt with the problem the netblock is taken off the RBL.
If the mail system administrator are on the ball and not asleep at the switch there's no reason why the total time from a netblock being entered into an RBL to being removed need be more than a couple of hours. If they're crap at their job or beligerant then they don't deserve honest customers.
The complaints made by the author of this paper are very reminisent of some of those I've seen on antispam/pro-RBL mailing lists from spammers who've had their spams stopped by RBLs. Draw your own conclusions, but I'm inclined to go with "If it looks liek a duck, it quacks like a duck nd tastes great with plum sauce...".
Stephen
"Don't write down to your readers, the only people less intelligent than you can't read" - Sign on Newspaper Office Wall
People spam because it's dirt-cheap. If spammers had to pay 10 an email, you'd better believe they'd be a heck of a lot more cautious about who they send to.
And a "Stop Buying Spam Products" is doomed to fail, anyway, because it's a numbers game. If 1 person out of every 100 people spammed buys something, then it's probably an outrageously successful campaign.
The fact is, you may be throwing out 50 spam emails a day, but if you see a subject line that speaks to an immediate need, you're probably going to stop, read it, and consider a purchase.
Isn't this how a blacklist is supposed to work? I thought the idea was precisely to annoy the honest users, such that they complain to the ISP. If the users know that they are blacklisted because of a spammer, they are likely to either leave the ISP or pressure it to turn the spammer off. It's not nice, but the intent is to get results.
I assert ownership of all trademarks and copyrights on this page.
A huge amount of spam is being sent through unsecured relays in Asia and South America. Consequently, an overwhelmingly large percentage of the hosts listed on RBLs are in fact based in these countries (see Wired article: Not All Asian E-Mail Is Spam). This amounts to nothing less than discrimination and isolationism that is being used to slowly cut off countries that have a critical importance in global matters
Obviously, if a huge amount of spam is coming from a huge amount of servers in a country, a huge amount of servers in that country are going to get blocked.
How about we drop the sensationalism here?
It's not some conspiracy to block all mail from Asia.
Look, maybe some people need to get mail from Asia, but I don't have any reason to. I'm not obligated to let anyone on the internet contact me at will. I can pick and choose who to block/accept at will. If people in don't want their servers to get blocked, maybe they should deal with their spam problem. I don't have time to fix it for them.
Look at it this way:
The internet is this huge shared network. It has a finite amount of bandwidth and it works because everyone carries data to its destination.
The question here should not be if any nodes should ever get blocked. The question should be: How much junk traffic should a single node on the network have to generate before it happens?
At some point you have to start blocking people. If I start DOSing an email server (almost what spam is), I can expect to have my traffic blocked at some point. Maybe I have to send a million junk messages, maybe a billion, but at some point it's costing too much to carry and process my traffic. Yes, bandwidth costs money. That's just the way a system like the internet has to work. There have to be mechanisms in block to handle the case were a node starts misbehaving. One of those mechanisms has to be dropping traffic from that node.
Carrying junk traffic costs money. Filtering costs money. At some amount of traffic, the cost becomes too high, and you have to block the traffic. Think of it as a signal to noise ratio. There always needs to be some number, at which you pull the plug, because the data isn't worth dealing with anymore.(And filtering it is too expensive)
Any time you share something you're going to need the ability to do this. If I start driving in the middle of a two lane highway, I can expectect to get pulled over and have my license revoked (eventually). It should be. I'm messing up things for everone else and the sensible way to fix it is to remove me.
Life is too short to proofread.
The problem is that the relevant "people" are not necessarily the ones stupid enough to respond to spammed come-ons. Even in the (unattainable) case in which nobody ever responds to spamvertising, spammers will still make money.
Large-scale spammers don't sell their own crap; they sell the "service" of spamming advertisements for other people's crap. Even if nobody responds to the spam, the spammer still has the money. Eventually, some of the clients get tired of flushing their money down the toilet, but there will always be customers for the spammer's snake-oil pitch.
/. If the government wants us to respect the law, it should set a better example.
It only blocks LEGITIMATE e-mail from servers that may, at some time in the future possibly, be used by spammers as a relay. It does block from machines that have sent spam, but also those that have never done it, just the potential is there. It does not, however, block spam! At least, not effectively.
And, that's where the problems lie. Administrators are putting these things in, assuming they'll stop spam, and then getting pissy when you tell them legitimate mail isn't getting through.
I used to be the e-mail admin for my company. We somehow ended up on the worst of these lists, osirusoft. This, despite the fact that we used SMTP AUTH; YOU COULDN'T SEND MAIL WITHOUT A PASSWORD! And, once you get on one of the lists, you're on them all.
So, I spent the better part of a couple of days going through them all and having to prove I wasn't an open relay. They all but one removed us within a week, but that was a week we couldn't send mail to a few customers.
And, the one that didn't remove us in a week...osirusoft...they took over a month. Every day I went to their site and ran the "autotest". Every day I watched it say, "Relaying Denied, deleting from list". Every day, I watched another "proof" of our spamminess posted onto their list.
And, the idiot admins of the ISPs? "Well, you're obviously an open relay. I see dozens of spams being sent from your site on the osirusoft list!"
BTW, the osirusoft rbl is run by some loser in his basement. Great plan, basing your company's e-mail on some unemployed idiot with a chip on his shoulder.
Look at your spam, where does the majority come from? That's right, AOL & Hotmail. But, your company would NEVER allow you to block from them, they'd lose too many customers. Install an active filter, you'll see better results and less spam.
Which is more painful? Going to work or gouging your eye out with a spoon? Find out!
http://www.workorspoon.com
The author of the article is yet another person who misunderstands the problem. The problem is not how to prevent the delivery of spam; that has already been solved. The problem is how to get the ISPs hosting the spammers that continue to eat up our bandwidth to disconnect them from the network. Decent ISPs will just do that upon the discovery they have spammers. And it is acceptable to slap their hand once or even twice, but three spams and you're out. The problem is many ISPs are not decent at all, and will only act upon a financial incentive. Blocking the whole ISP is what is required. DNSBLs such as SPEWS are doing that incrementally with the intent to minimize the number of others affected for long enough to show to the ISP that they had better get rid of the spammers. At this point most ISPs will realize they will lose customers in the future, and will get rid of the spammers. A few will be stubborn, and will eventually have their entire address space listed. Not only do we not want mail from spammers, we don't want mail from anyone who supports spammers. And if you are paying money to an ISP who runs in turn is providing services to a spammer, then you are indirectly supporting spammers through financial benefits, such as the ISP offering the spammers lower rates through economy of scale. And do not forget that if you are doing this, that you and your ISP are benefitting off the costs incurred by others. All this article is, is a reflection of frustration by an individual who just doesn't get it, that he needs to either turn his ISP around to be a decent member of the internet community, or he needs to switch to another ISP. It looks like a lot of work went into it, but the premise being all wrong, the article is worthless and offers no solutions.
now we need to go OSS in diesel cars
"How about a pizza company refusing to accept orders from a paticular motel because often noone will admit to ordering there? Stay at a different motel."
Um, exactly how much research are you expecting people to do on motels? Call them up and say "Can I order pizza there?"
"If you are using an ISP that does not enforce acceptable use policies restricting unsolicited email, you are supporting spaming activity."
As opposed to what? Exactly how is one supposed to go about finding out about how effective an ISP's attempts to filter spam are? The biggest problem with your argument is that spammers always change how they operate.
Sorry, but your answers struck me as oversimplified and unhelpful. How that was modded up as 'insightful' I'll never know.
Some experienced sysadmins do not endorse SPEWS' wholesale blacklisting of entire netblock neighborhoods. Those admins choose not to use SPEWS RBL, but may choose to use RBLs that cause less collateral damage. Some experienced sysadmins use SPEWS RBL because they do endorse SPEWS' clearly documented process which bears many similarities to economic extortion.
Many inexperienced sysadmins use osirusoft (e.g via SpamAssassin) without knowing the difference between SPEWS and other RBLs aggregated by osirusoft. Without knowing that difference, these inexperienced sysadmins unknowingly endorse SPEWS' clearly documented process which bears many similarities to economic extortion.
One answer is a SPEWS whitelist + reciprocal blacklisting. Create a whitelist of SPEWS-blacklisted-but-collateral-damage IPs which have *never* been accused by SPEWS (or other RBL) of spamming. When an ISP causes collateral damage by enforcing the SPEWS RBL against a presumed-guilty-but-never-accused IP that exists in the SPEWS whitelist, ask the individual sysadmin to use the SPEWS-collateral-damage whitelist.
If an individual sysadmin uses the SPEWS RBL but chooses not to use the SPEWS-collateral-damage whitelist, they would be endorsing SPEWS clearly documented process which bears many similarities to economic extortion. Such explicit endorsement will earn such individual sysadmins membership in an IP blacklist of "sysadmins who support SPEWS' clearly documented process which bears many similarities to economic extortion". This blacklist would then be enforced by sysadmins whose IPs are SPEWS-blacklisted-without-spam-accusation .
This unbundling mechanism provides a technical means for individual sysadmins to endorse SPEWS valuable spam-fighting contributions without endorsing SPEWS' clearly documented process which bears many similarities to economic extortion.
Long-term, the solution is pseudonymnous, non-profit TLS certificates for SMTP servers with social (not economic or calendar) seniority (c.f. Apache Incubator). The economic variety exists at bondedsender.org, along with whitelist patches for popular open-source MTAs.
If you're an individual user, a computation-intensive spamassassin approach can do a really good job of blocking most spam and blocking very little non-spam. But if you're an ISP or Mail Service Provider, having a conservative RBL can save you a lot of resources, including bandwidth and computation, by throwing away the high-volume relay-abuse spams with as little work as possible, saving the more complex work for mail that's less likely to be spam. (By conservative, I mean "trying to only block actual relays and other known spammer systems", as opposed to "broad-spectrum insecticides and lists that do collateral damage to pressure ISPs or harass their competition.") That might be a 25-50% reduction in total email that the ISP needs to handle, but from an instantaneous-resources standpoint, it's probably higher than that, because spam tends to come in high-volume blasts, while real email is mostly Poisson arrivals. And if an ISP's failure responses are the "Temporarily inaccessible, try again later" type as opposed to permanent rejections, real email systems are much more likely to try again later than spammers are (though of course open relays may still try again later, because they're just mal-administered, not necessarily broken.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Read the above post more carefully. The spammer was successful in spoofing the IP address of a TCP session, because he controlled both the dialup account and the high-speed account.
SYN from the dialup account.
SYN+ACK from the helpless email server back to the dialup account. Dialup account now has observed both sequence numbers.
ACK from the dialup account, and the SMTP transaction begins.
As sending mail consists mostly of uploading, upload packets to the server are forged from the high-speed account to the server. The dialup account only needs to receive the ACK for the sent data, and the SMTP responses from the server. The spammer uses both the dialup and the high-speed accounts in tandem to keep the connection alive, in effect intentionally hijacking his own TCP connection.
Very clever! The spammer must have had some help in setting up a scheme like this. I don't think he'd be smart enough to write the software on his own.
Dr. Demento On The 'Net!