Slashdot Mirror
The Spam Problem: Moving Beyond RBLs
whirlycott writes "I just published a paper called The Spam Problem: Moving Beyond RBLs on my site. I comprehensively describe RBLs and list eight specific problems with them. I also get into ideas that next generation antispam system creators should read. I hope that this will be useful to anybody who is attending the Spam Conference at MIT on Jan 17th."
Tell EVERYONE you know never to click on any spam links, or buy spamvertised products. People spam because it WORKS. The only real way to stop it is to STOP BUYING SPAMMED PRODUCTS.
You'll notice that he listed and then did not address the "Common Arguments and Justifications" for running and/or using a RBL. Just couldn't come up with a reason why privately owned servers have to accept mail from any particular person or group if they don't want to.
1. Don't let a spammer verify your email address
2. Don't post your email address on the internet
3. Secure your email client
4. Avoid common email traps
5. Fight back
Let me know if these can be improved.
Read my sig if you like, but I'll never see yours, thanks to Discussions, Viewing, Disable sigs...
Quite a bit, actually. This reads like a topical treatment by someone who really doesn't know the subject. For example he mentions whitelisting, but in the solutions section, completely ignoring the fact that there are already solutions, both commercial and open source, that use whitelisting, blacklisting, and greylisting. In fact, I wrote one about 6 months ago for a client, and they are quite happy about it, it affords them complete spam protection.
-- Ed Carp, N7EKG erc@pobox.com PGP KeyID: 0x0BD32C9B What I'm up to: http://intuitives.mine.nu
Having briefly looked at the paper, it seems like the usual complaining about RBLs as being too broad you see all the time in NANAE (news:news.admin.net-abuse.email).
Summary: someone tries to send email and finds that they're listed on SPEWS. They complain because "we're not an open relay", without figuring out just why they're on that list. Almost invariably, they're on the list because their ISP persistently ignores spam complaints and prefers spammer money to honest customer money. I think there's been about two or three actual mistakes in the SPEWS listings in the year or so I've been following NANAE. Otherwise, it's all been a legitimate extension of the block because the ISP knowingly ignores complaints and supports spammers.
Spam is theft. Theft of Bandwidth, theft of service and theft of time. It's that simple. Spammers are thieves. ISPs which support spammers are thieves. Soon, they'll be blocked from the public internet for anti-social behaviour. After all, if your local bargain supermarket ignored the thieves stealing 20% from every transaction you make with them, will you go back?
Many South American and Asian ISPs are blacklisted because they were quite happy to spam everyone when they could steal bandwidth and service from other ISPs. Now that they're blacklisted, they're whinging and moaning about 'freadom of speach', interference with interstate commerce, and other such bullshit.
It's about none of these things. Blacklists are about protecting your network from a Denial of Service attack by spammers.
People who complaing about RBLs (OR DNSBLs, to be more accurate) are missing the point. They should be complaining about spammers who think it's acceptable to steal my bandwidth and your bandwidth to advertise their product..
dave "the only good spammer is a rotting corpse, dangling from the noose"
and
Scalable (resources)
Aren't mutually exclusive?
(1) You (and I) get too much spam.
(2) Your e-mail system administrator (and mine) need to keep beefing up the servers because the sheer volume of e-mail is growing so quickly.
To a first approximations, filters solve (1) but not (2), and black hole lists solve (2).
whirlycott summarizes the problem with (2) in two words: "collateral damage." How much of the e-mail network do we need to destroy in order to save it?
We need to move past first approximations. We need systems that work at the server level, but that somehow address the problems of collateral damage and false positives.
This is only the tip of the iceberg. Any network messaging medium is vulnerable to abuse by spammers. The problem started with Netnews, it continued with e-mail, it's happening now with instant messaging. We need at least high level solution that helps solve the problem regardless of prototcol.
I wish I had one.
Stupid job ads, weird spam, occasional insight at
The problem, as I've said here before, is SMTP itself.
The RFC pretty much states that to be compliant, you have to accept the mail as it is presented. Can't achieve accurate or trusted reverse name lookup information on the sending system? Well, that's tough, take the mail (read this for yourself).
This problem stems from when systems on the Internet were inherrently trusted. That's not the case any longer, and it's time for a new mail transmission standard.
For starters, it should allow system administrators the ability to give priority to systems that can present some form of credentials. SSL or keyed encryption, whatever the standard is, it will permit systems to give totally trusted access to systems that meet the specific security and trust guidelines of the receiving system, not the RFC (times have changed, tough).
Those systems that do not meet minimum trust levels will either have to clean up their act or take the time to contact the remote system to figure out the issue.
It won't stop spam, but it will go a long way to slowing it down and possibly providing some secure method of mail transport in the process.
It's important to realize the point of RBL blocking. It isn't to make end-users happy, it's designed to lower traffic on the mail servers. So a proposed solution needs to be something that the ISP can execute without having to analyze the email. RBLs monitor a single variable, IP, to determine whether it should be accepted or not. If someone could come up with an idea that processed emails based on another single variable, then we'd have ourselves a good spam filter.
One proviso: if anyone complains, I will look at it.
RFCs require that one accepts mail for postmaster@domain.com and from the empty envelope sender. Since I do this, I believe I am fully RFC compliant.
So stop whining about DNSBL. The problem is wider than that, and will not be solved by getting rid of DNSBL. The system isn't perfect, but that is not the issue.
Conversion Rate Optimisation French / English consultant
I'd have to say, yes.
Personally I use Spamcop's RBL and reporting service. I check the held mail page a couple of times a day. I have yet to see a legitimate mail be blocked and it's reduced the number of spams a day I get from hundreds to 2 or 3.
Maybe some RBLs still work the way the author decribes but from what I'm hearing that's not the way many work now. Now it's more like a reporting user recieves a spam (hopefully very near the start of the spamming run) and reports it. The reporting system works out the most probable source and lists it (due to the fact that spoammers often move within a netblock the netblock rather than the individual IP address has to be blocked for the RBL to be effective), the system also mails the admin address for the appropriate domain (and any listed interested third parties) with the information required to identify the spammer and asks them to deal with them. That IP address is also monitored by the RBL. When the spammer stops sending spam or the administrator informs the RBL operator that they've dealt with the problem the netblock is taken off the RBL.
If the mail system administrator are on the ball and not asleep at the switch there's no reason why the total time from a netblock being entered into an RBL to being removed need be more than a couple of hours. If they're crap at their job or beligerant then they don't deserve honest customers.
The complaints made by the author of this paper are very reminisent of some of those I've seen on antispam/pro-RBL mailing lists from spammers who've had their spams stopped by RBLs. Draw your own conclusions, but I'm inclined to go with "If it looks liek a duck, it quacks like a duck nd tastes great with plum sauce...".
Stephen
"Don't write down to your readers, the only people less intelligent than you can't read" - Sign on Newspaper Office Wall
Isn't this how a blacklist is supposed to work? I thought the idea was precisely to annoy the honest users, such that they complain to the ISP. If the users know that they are blacklisted because of a spammer, they are likely to either leave the ISP or pressure it to turn the spammer off. It's not nice, but the intent is to get results.
I assert ownership of all trademarks and copyrights on this page.
A huge amount of spam is being sent through unsecured relays in Asia and South America. Consequently, an overwhelmingly large percentage of the hosts listed on RBLs are in fact based in these countries (see Wired article: Not All Asian E-Mail Is Spam). This amounts to nothing less than discrimination and isolationism that is being used to slowly cut off countries that have a critical importance in global matters
Obviously, if a huge amount of spam is coming from a huge amount of servers in a country, a huge amount of servers in that country are going to get blocked.
How about we drop the sensationalism here?
It's not some conspiracy to block all mail from Asia.
Look, maybe some people need to get mail from Asia, but I don't have any reason to. I'm not obligated to let anyone on the internet contact me at will. I can pick and choose who to block/accept at will. If people in don't want their servers to get blocked, maybe they should deal with their spam problem. I don't have time to fix it for them.
Look at it this way:
The internet is this huge shared network. It has a finite amount of bandwidth and it works because everyone carries data to its destination.
The question here should not be if any nodes should ever get blocked. The question should be: How much junk traffic should a single node on the network have to generate before it happens?
At some point you have to start blocking people. If I start DOSing an email server (almost what spam is), I can expect to have my traffic blocked at some point. Maybe I have to send a million junk messages, maybe a billion, but at some point it's costing too much to carry and process my traffic. Yes, bandwidth costs money. That's just the way a system like the internet has to work. There have to be mechanisms in block to handle the case were a node starts misbehaving. One of those mechanisms has to be dropping traffic from that node.
Carrying junk traffic costs money. Filtering costs money. At some amount of traffic, the cost becomes too high, and you have to block the traffic. Think of it as a signal to noise ratio. There always needs to be some number, at which you pull the plug, because the data isn't worth dealing with anymore.(And filtering it is too expensive)
Any time you share something you're going to need the ability to do this. If I start driving in the middle of a two lane highway, I can expectect to get pulled over and have my license revoked (eventually). It should be. I'm messing up things for everone else and the sensible way to fix it is to remove me.
Life is too short to proofread.
In your case it worked out. If you had simply been asked to persuade your ISP to boot the spammer would you have ignored the request? Are you actually so dense that it takes blocking your email to get you to act?
Note that I'm not trying to claim you are dense or prove it - my point is that you could have been reached in a way that led to the same result but that DID NOT block your valid email. Is there any reason why the brutal method should be the one chosen first? Uh, any good reason - surely there are thugs who enjoy using their power to abuse others.
Not to mention that there's been more than one case in NANAE where the collateral damage was suffered by someone related to an ISP that had long ago booted the spammer but had not removed all traces. No spam flowed because of the omission, the listing was long after the spammer was removed, no risk to anyone existed. Still, the IP of an innocent party was wrongly listed, wrongly blocked, much time and energy was spent discussing it in NANAE, a person and organization that could perhaps have become spam opponents were given reason to hate the guts of spam fighters. No win of any kind I can see in that.
And, of course, the brutal blocking actions haven't ended spam, other than the occasional anecdotal victory. I ran an open relay honeypot, I saw how modern bulk spammers operate. The DNSBLs are a weak tool to deal with that. Don't take my word for it: run your own open relay honeypot. You'll quickly learn a lot about how spammers operate. All the while you'll be stopping their spam, too. Open proxy honeypot? Bless you - you'll also do wonders.
(Any of you sendmail experts able to figure out my pseudonym?)
RBL's are like a fever. They tell you when something it wrong and only a dork blames the fever when the problem is the disease.
It's not like any fever I've come across. For the analogy to hold, when I'm ill my entire village would get a fever, and some of the population might die, in the hope that the sound of the ambulances and funerals might alert me to the fact that I have a problem.
I'm glad you are so happy about having your reputation threatened when you have done nothing wrong. Our business is hosting websites on our own machines in a server park. Server parks are always going to be a good place for spammers to rent cheap machines, and if our clients start getting their mails bounced, they don't write to the server park owners, they cancel their contracts with us. And, no, we can't just take our servers elsewhere at 3 minutes' notice, so the RBL puts zero economic pressure on our server park (which seems to act fairly promptly on abuse compaints anyway).
RBLs punish the innocent to get at the guilty. This is wrong. The next time my business is hit by SPEWS or any other such system, I'm going to start writing pithy articles for the general press, with the aim of scaring customers away from ISPs that use RBLs, eg "Do you want your ISP to tell you what email you can read?. And I shall certainly take legal advice on whether I can sue companies who bounce my mail with any rejection message containing the word 'spam' for libel or something similar.
Virtually serving coffee
The problem is that the relevant "people" are not necessarily the ones stupid enough to respond to spammed come-ons. Even in the (unattainable) case in which nobody ever responds to spamvertising, spammers will still make money.
Large-scale spammers don't sell their own crap; they sell the "service" of spamming advertisements for other people's crap. Even if nobody responds to the spam, the spammer still has the money. Eventually, some of the clients get tired of flushing their money down the toilet, but there will always be customers for the spammer's snake-oil pitch.
/. If the government wants us to respect the law, it should set a better example.
While the article was very interesting, it seems that this Property of a Real Solution is not fully cooked. Nations, States, and Unions are defined by their laws and by the territorial boundaries over which those laws are operative. Removing the jurisdictional considerations from the law removes the basic tenet that one must have notice that any particular activity is criminal.
There is the saying that "Ignorance of the law is no excuse." This proposed legal solution requires not just knowledge of the law of your State of residence, but of every other jurisdiction in the world as well. That is indeed an untenable proposition, as there is unlikely any person alive that knows the laws, rules, and regulations of every jurisdiction.
While UCE is annoying, it is nothing special from a legal perspective. A *solution* to UCE (or any other annoyance) is not worth the consequences of a legal theory that subjects every person on the planet to every law on the planet.
Ok hotshot, I've just added cyberporte.co.uk to our local RBL list and taken the liberty of posting a link (with a C&C warning) to your post on NANAE. Would you like the address of our attorney now....
This is great, you've just demonstrated that RBLs are not neutral, and are driven more by a desire to punish than to solve the problem. If I ever need to send an email from that domain, I'll use one of our other smtp servers, or that of one of my ISPs, or rent a clean one, or... the problem last time was that I didn't know how ineffective RBLs are. The one thing I'm not going to do is change my server park because someone on the other side of the world is on a quixotic crusade. It's not my battle, and I object to people trying to enlist me.
Why your netblock or address range has been rejected.
In our case, it is because one machine in our 16-bit IP range had been used for spam, so SPEWS blocked 65,000 machines, each of which is administered by a different person/company. How does jeopardising the existence of my company, whose smtp server is clean, help to fight against spam? Like I said, we can't just pick up a fairly full server and take it somewhere else, so there is no real economic pressure on the server park.
Joe Internet user is tired of spam
See n previous /. discussions about this, but the (statistically) average email address gets about 3 a day. Quite a lot of /.ers say they get very few spams, and many of those who do say that the annoyance value is pretty low. On the other hand, if you are trying to buy a skyscraper (real example) and you can't get emails from the estate agent, who happens to be in a different continent, that is extremely annoying, especially if there is absolutely no reason for blocking that particular server.
Any decent way to block spam
Err, if netblock is such a greeeeat system, how come spam is increasing? Am I missing something? If there is a consensus that spam is a major problem, legislate against it. I don't have a problem with that. I do have a problem with what mrneutron calls 'collateral damage', ie people damaging my reputation to get at someone else, especially when the system obviously isn't reducing the amount of spam sent globally.
Virtually serving coffee
I run a spam filtering service which uses DNSBLs along with other measures to reduce the spam that my customers receive. The customers who sign up for this service typically are completely swamped by unwanted email, in fact - one customer has a hit rate of over 60%. Yes, 60%. They had reached the point where their email was becoming useless, so they had to do something about it.
DNSBLs are a valuable tool when combined with other technologies and have a very low 'collateral damage' rate. For example, the customer mentioned above has never called to complain that valid email was blocked even though I remove over half of their mail before they get it.
As for someone's right to run an open relay, I guess they do have the right to run their server however they choose, but that right ends at my door. My server, my T1, my customers asking for help. I explain the risk of collateral damage to potential new customers, and explain they must trust me to make decisions on what is blocked and what is not. I try very hard not to be overzealous and it has served me well because no customer has ever left the service once they signed up.
I'm very sorry if the author of this article was inconvenienced by being blacklisted. But the needs of the many outweigh the needs of the few... or the one. (TM)
You seem to be upset that some groups have demanded that the smaller ISP's and less technological countries do the main work in solving the Spam problem. THEY ARE THE ONES RESPONSIBLE FOR IT IN THE FIRST PLACE. Yes, they may not personally be the people doing it, but they are part of a group that IS doing it. I think Blocking is TOTALLY appropriate Punishment to the Asian Countries for their failure to police their ISP's and fight the evil of Spam. Note, I personally have had my email to a friend blocked because of the RBLs. He gave me a new email address, (at another small ISP) and the problem was solved. If you have that problem, SOLVE it by moving AWAY from the SPAMMERS, instead of supporting them by your lazyness.
excitingthingstodo.blogspot.com
What? With most blocklists the blocks are aimed verified spam sources, exclusively. SPEWS alone escalates, and it appears you assume you know how they escalate. Who, other than SPEWS, operates in a manner even remotely resembling what you claim?
If you would limit yourself to dealing with facts then you'd find factual episodes in which SPEWS escalated a listing long after the spammer was removed, escalating apparently because some non-useful, non threatening vestige of the spam operation (like a DNS entry) remained. In such a case there is no spam threat, no need to list, no need for collateral damage. Your glib explanation doesn't apply: it's a screw-up, an over-zealous action taken carelessly. SPEWS apparently started to believe the extravagant claims being made for it. It's often dangerous to start believing your own PR. Apparently it's dangerous even when you don't originate the PR.
There have been episodes of egregious collateral damage. The total of these do not begin to approach a reason to stop using DNSBLs. Even one episode is reason enough to re-examine and revise a listing policy - the enemy is spam, make sure you hit spam and spam only. Fight the enemy. Making excuses for shooting the innocent is not fighting the enemy, nor is making incorrect claims about what is done.
How was SpamCop missed in the "research" ?
By the stated definition (Technology, 1) there is only the act of theft but no such a thing as a thief ?
For the writing to be taken seriously it somehow needs to add some value to an intelligent discussion. Just stating that RBLs are not perfect is like stating that operations and amputations have drawbacks.
In your case it worked out. If you had simply been asked to persuade your ISP to boot the spammer would you have ignored the request? Are you actually so dense that it takes blocking your email to get you to act?
Dense?
Why are you even mentioning the word dense?
He was a friggin customer! His email being blocked was the first indication he had that a spammer was hosted by his isp.
So what next? He asks his isp to boot the spammer. If they refuse, he doesn't want to have an acount with them anyways, so he'll go somewhere else. Seems fine to me.
It's hardly "brutal" anyways. The email bounces, it doesn't just disappear and leave him wondering why no one ever replies.
Finally, if the isp is only partially fixing a spam problem, after booting the spammer, then they're incompetent and you don't want to be working with them anyways. The ip you complain was "wrongly blacklisted" was actually rightly blacklisted. It just wasn't removed from the list, because someone wasn't doing their job.
If an isp gets a notification that an ip has been added to a blacklist, isn't it obvious that they should contact the maintainer of that blacklist when the problem is fixed? The fault in your example does not lie with the blacklist, but with the isp. If you choose a crappy isp, expect problems.
Life is too short to proofread.
Not to mention that there's been more than one case in NANAE where the collateral damage was suffered by someone related to an ISP that had long ago booted the spammer but had not removed all traces.
That's life in the big city. Most of the time this happens when an ISP thinks that it's good enough to just remove the web site but still host DNS or mail for the spammer. This is called "spam support services" and is a no-no. Even on the rare occasion when it's something like IP addresses still showing up as being allocated to the spammer, how is anyone outside the ISP supposed to know that the spammer is no longer a customer? So many ISPs come to NANAE begging to be delisted when they have done literally nothing about their blatant spam problems that why should the one out of ten that is simply incompetent be given special consideration?
And, of course, the brutal blocking actions haven't ended spam
Oh somebody call a waaaaam-bulance. Free clue: Nothing will end spam. Even if e-mail becomes metered you will still get spam -- it will just come from the people who send you paper junk mail instead of Alan Ralsky.
Some experienced sysadmins do not endorse SPEWS' wholesale blacklisting of entire netblock neighborhoods. Those admins choose not to use SPEWS RBL, but may choose to use RBLs that cause less collateral damage. Some experienced sysadmins use SPEWS RBL because they do endorse SPEWS' clearly documented process which bears many similarities to economic extortion.
Many inexperienced sysadmins use osirusoft (e.g via SpamAssassin) without knowing the difference between SPEWS and other RBLs aggregated by osirusoft. Without knowing that difference, these inexperienced sysadmins unknowingly endorse SPEWS' clearly documented process which bears many similarities to economic extortion.
One answer is a SPEWS whitelist + reciprocal blacklisting. Create a whitelist of SPEWS-blacklisted-but-collateral-damage IPs which have *never* been accused by SPEWS (or other RBL) of spamming. When an ISP causes collateral damage by enforcing the SPEWS RBL against a presumed-guilty-but-never-accused IP that exists in the SPEWS whitelist, ask the individual sysadmin to use the SPEWS-collateral-damage whitelist.
If an individual sysadmin uses the SPEWS RBL but chooses not to use the SPEWS-collateral-damage whitelist, they would be endorsing SPEWS clearly documented process which bears many similarities to economic extortion. Such explicit endorsement will earn such individual sysadmins membership in an IP blacklist of "sysadmins who support SPEWS' clearly documented process which bears many similarities to economic extortion". This blacklist would then be enforced by sysadmins whose IPs are SPEWS-blacklisted-without-spam-accusation .
This unbundling mechanism provides a technical means for individual sysadmins to endorse SPEWS valuable spam-fighting contributions without endorsing SPEWS' clearly documented process which bears many similarities to economic extortion.
Long-term, the solution is pseudonymnous, non-profit TLS certificates for SMTP servers with social (not economic or calendar) seniority (c.f. Apache Incubator). The economic variety exists at bondedsender.org, along with whitelist patches for popular open-source MTAs.
The problem is not you making a personal decision to create false positives for yourself. The problem is other people making decisions for you which block mail which is not spam without your knowledge.
The problem is some ISP between you and your friends/family/coworkers deciding that your friends'/family's/coworkers' mail is spam without you having any say in it.
The idea is that YOU should decide what false positives to deal with, not a government or an unaccountable entity like an ISP.
Read the above post more carefully. The spammer was successful in spoofing the IP address of a TCP session, because he controlled both the dialup account and the high-speed account.
SYN from the dialup account.
SYN+ACK from the helpless email server back to the dialup account. Dialup account now has observed both sequence numbers.
ACK from the dialup account, and the SMTP transaction begins.
As sending mail consists mostly of uploading, upload packets to the server are forged from the high-speed account to the server. The dialup account only needs to receive the ACK for the sent data, and the SMTP responses from the server. The spammer uses both the dialup and the high-speed accounts in tandem to keep the connection alive, in effect intentionally hijacking his own TCP connection.
Very clever! The spammer must have had some help in setting up a scheme like this. I don't think he'd be smart enough to write the software on his own.
Dr. Demento On The 'Net!