Paul Graham Describes Dangers of Spam Blacklists

CRoby writes "Paul Graham posted an essay describing the danger and corruption of the main spammer blacklists today. It discusses MAPS and the SBL, the blacklist created to try to alleviate the abuses of MAPS, and suggests (maybe) another blacklist's creation."

Comments

$article_title by $blowhard

[Neil Blender]

$idea will not help cut down on spam. In fact, it is detrimental. This has been know for $num_years years, but I feel I must prove that I am really smart by writing an article about it.

A few comments

[alanw]

From Paul Graham's original article http://paulgraham.com/spamhausblacklist.html

any filter relying on the SBL is now marking email with the url "paulgraham.com" as spam

The primary use of the SBL is to allow sysadmins to refuse e-mail coming from listed IP addresses. The mail should be rejected during the SMTP header conversation, and the senders of genuine (non-spam and non-virus) e-mails will receive a non-delivery report from their outgoing MTA.

I assume that what Paul Graham is complaining about must be SpamAssassin, or some other content filter, applying a score to articles containing URLs, which when looked up in DNS resolve to listed IP addresses. This is much less acceptable, since the sender has no way to know that their e-mail may have been classified as spam.

The details of the listing can be found at http://www.spamhaus.org/sbl/sbl.lasso?query=SBL279 45. This is a /32 - i.e. a single IP address. I don't know why Paul Graham's web site (which has that IP address) has been associated with textileshop.com, which has a completely different IP address.

The other Yahoo listing on the SBL is also a /32.

I also note in another of Paul Graham's articles http://paulgraham.com/sblbad.html he claims

The most notorious example is the MAPS RBL

As any fule kno, the most notorious spam blacklist is SPEWS. ~

Vigilante it ain't

[Rosco P. Coltrane]

The problem was, as vigilantes so often do, the guys at MAPS got carried away

For some reason, journalists keep calling blackmail lists "vigilantes". But there's something they don't understand: nobody forces email system administrators to use those lists.

These lists are provided by people for free. They decide to list bad email servers, but they may as well include any server they want. After all, who's to force them to provide quality of service?

The real problem, of course, is that blacklists are needed in the first place. If ISPs did their jobs a little better (aol, hotmail and the likes), the amount of spam would already decrease significantly. And don't speak to me about chinese ISPs, since most spam comes from the US.

Re:Vigilante it ain't

[Maestro4k]

For some reason, journalists keep calling blackmail lists "vigilantes". But there's something they don't understand: nobody forces email system administrators to use those lists. No, but the non-spamming sites that end up on it would certainly disagree with you, they didn't do anything to merit the block.

You seem to be confused about what a vigilante is, dictionary.com gives me this: "One who takes or advocates the taking of law enforcement into one's own hands." Note it doesn't say anything about them forcing others to agree with their views or take part in them. If you decide to take legal actions in your own hands, then you are, by definition, a vigilante. So it does apply here, just because they don't force anyone to use their lists doesn't change that.

These lists are provided by people for free. They decide to list bad email servers, but they may as well include any server they want. After all, who's to force them to provide quality of service? TFA's point was that these lists start out listing just IPs/hosts/sites they know are sending spam, then later the power corrupts ("power corrupts, absolute power corrupts absolutely") them and they start using the power they've gained by their blacklist being used by many people to start trying to force ISPs to comply with them by blocking bunches of innocents at the same ISP. That indeed has happened, although I'm really not sure if it's happened here or not. The risk of it occuring is pretty high, humans are, after all, only human and it's hard to resist that temptation, especially when you're a strong enough anti-spam advocate to run a blacklist. The real problem, of course, is that blacklists are needed in the first place. If ISPs did their jobs a little better (aol, hotmail and the likes), the amount of spam would already decrease significantly. And don't speak to me about chinese ISPs, since most spam comes from the US. The real problem is human nature in all of this. In spam existing in the first place (greed), in ISPs not blocking things they should (laziness, lack of knowledge or time), in people actually buying from spam (greed (getting something cheaper than legal means would allow), sexual desire (gotta have a longer penis!) or just simply a criminal desire to purchase illegal goods (prescription drugs for example)) as well as humans becoming corrupted by power when their blacklists get to be popular.

So basically if we can solve how to get people to stop being, well, people and giving in to baser instincts we can stop spam. Of course we'd also stop crimes of all sorts as well and we've not managed that in hundreds of years so I'm not holding my breath for it to happen.

Not like people get all radical about it...

[dmorin]

Actual quote I have heard on the subject of spam blacklists: "I don't care that you're not a spammer. Your ISP allows spammers in their midst and therefore you all go on the list. Get a new ISP."

Oh, ok. Nothing like over reacting a bit.

Re:Not like people get all radical about it...

[Uruk]

No, the principle is that if ISPs know that this kind of overreaction will occur, they will make quite sure that they don't have spammers in their midst. In essence, it's an attempt to incentivize ISPs to police themselves.

What's the alternative? Having some centralized, international spam cop whose job it is to clean up every ISP on the planet? If ISPs get a completely free pass on spam and don't have to care whether their subscribers are abusing other people or not, where is their incentive to prevent the abuse? The way you avoid the tragedy of the commons is by getting people to see their individual stake in the issue.

Certainly the quote that you're pointing out isn't the most diplomatic or effective way of putting it, and I doubt this kind of thinking is behind that quote - it probably is the knee-jerk reaction that you're identifying it for. Still, the idea might have some merit.

Pure and simple...

[jellisky]

I had the unfortunate "joy" of being blocked by some of these draconian blacklists. My sister requested some information from me for a trip that she has upcoming via my yahoo.com account. After it bounced from her ISP saying that I was sending it from a "spam-hosting" ISP, I sent it from my mac.com account. Same schtick. After a couple other choices, I finally got it sent from my .edu account.

Her ISP uses SpamBag for their blacklist. SpamBag? ScamBag is more like it.

No wonder my sister is disenchanted by email. Her yahoo account got spammed to no end, then she can't get emails from most of her friends since they get bounced back by her ISP's stupid blacklist.

Blacklists are fine and dandy in principle, but practice has shown them to be useless. IT managers, just drop them. They're more annoying than anything.

-Jellisky

Whiskey. Tango. Foxtrot. Over.

[Skye16]

So...it's okay if he goes to Federal Pound-Him-In-The-Ass penitentiary just because he rented a car from a place that also rented a car to a crack dealer?

Huh?

Sorry, but that's still bullshit. He states it clearly in his article: You can't screw over innocents just to make the guilty pay. Does the your government put a neighbor family through torture just because you got a parking ticket? No. It's YOUR fault and YOU should be punished. Not some innocent bystander.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Pay and you get removed

[tmk]

I have found an interesting offer: pay 50 bucks and you are removed immediately from the spam list. Have a look here.

Interesting: The company won't say who they are. They say this was approved by local authorities, but this is bullshit. Local authorities can not brake federal law in Germany.

There is a problem with blacklists

[WebHostingGuy]

We deal with this all the time. Leaving any IP on a blacklist for any period of time doesn't help. Most spammers nowdays spam and run. They unload from a hacked account through a broken formmail script or a zombie computer. After 36 hours they have dumped their million emails and moved on to another IP. Blacklists generally don't get this though. They just make a bigger and bigger list. The problem with this approach is that they already missed the spammer. One time we dealt with someone who was running a blacklist and when we asked why an IP was on the list they said because it spammed years ago. When we said we have controlled the IP for the past three years they said it doesn't matter. It's like give me a break...

The solution to blacklists is to use an AOL model in which dynamic IP blocking is used. When spam is noted from an IP that IP is automatically blocked for 24-36 hours after the last spam comes in. That way the innocents are not being blocked and the spammers email doesn't make it through. There are a couple blacklists which do this but more should.

Compare this to the opposite blacklists like BLARS which requires a thousand dollars for "him" to investigate whether an IP should be removed. I have never seen an IP which is not listed with BLARS.

Re:Definitely a bad idea...

[Vainglorious Coward]

I'd take all the SPAM anyday vs. not being able to send legitimate emails.

Except that blocklists don't stop you sending email, they merely allow others to decide whether to accept that mail. Or do you think other people should be forced to accept any and every email you send?

Re:Definitely a bad idea...

[Seumas]

John Reid of the SBL told me this wasn't true-- that the SBL was still clean, and that they only blacklisted hosting companies' mail servers when they were spam hosts who took on innocent users as camouflage:

He is right. That definitely is NOT how SBL actually operates. I have a site that is heavily trafficked (millions per month) and they blocked my email (from my own personal server) that has delivered mail for my site for seven years with absolutely no outgoing spam or relaying having ever occurred in its entire life.

However, a spammer with false credentials faked his way into a hosting account with my colo provider and as a result, SBL blocked multiple entire submnets, rendering my entire site and service useless for almost an entire month (we deal with auctions, meaning nobody was getting closed notices, won notices, outbid notices, addresses to send payment, registration emails, lost password emails - and when they complained, I couldn't respond to help them and explain it to them).

SBL couldn't have cared less. As far as they are concerned, if one IP is a source of spam, they all are. And they'll get to fixing it in their own damn sweet time.

But the defense of SBL fan-boys is typically "well it's VOLUNTARY!".

Yeah. Whatever. Fuck off.

Guideline, not a rule

[bitflip]

I use blacklists all the time. Rather than simply rejecting the mail, if the server is on a blacklist, the initial OK is delayed by five seconds.

If you're sending a ton of mail, i.e., spam, little of it gets through. If you're only sending one or two messages, ie, likely legit mail, it goes through just fine.

Combined with more specific stuff further back (bayes, et. al), it's been quite effective at reducing the amount of spam sent, and the amount of mail that gets scanned.

The problem isn't blacklists, its how people use them.

"Power-hungry weenies"

[slavemowgli]

Interestingly enough, the owner of the acme.com domain who was recently featured in a story due to his getting more than a million spam mails (well, attempts to send spam) a day, agrees:

DNS-RBLs - Domain Name System Realtime Black Lists. In theory the idea is fine. You have a set of sites that you blacklist, and you want to let other folks use the same list so you distribute it using DNS, which is a nice efficient de-centralized database. What's not to like?

Well, I don't know why, but in practice every single DNS-RBL eventually comes under the control of power-hungry weenies. They start listing sites unreliably, and if you complain you find yourself listed. And there's usually no way to get off the list.

A lot of people tell me I'm wrong about this. They say that certain DNS-RBLs are ok, with objective criteria for inclusion and simple procedures for getting off the list. The thing is, they give conflicting recommendations for which lists are good and which are bad. Some of these folks recommend lists which I know from personal experience are bad.

This problem is really inherent in the way DNS-RBLs are set up. You cede control of your mail system to a third party, with no real possibility of checking how they are doing. The people running the lists get overwhelmed with bogus feedback from spammers and/or idiots, to the point where they assume all their mail about the lists is from spammers and/or idiots.

If the lists you use have not yet descended into corruption and chaos, consider yourself temporarily lucky.

Do not use DNS-RBLs.

(from http://www.acme.com/mail_filtering/shame_frameset. html)

Re:Paul is just pissed because...

[deacon]

66.163.161.45 is filthy. Blocking mail that has URLs pointing there will stop a fair amount of spam. Not an approach I'd use myself, but certainly a lot more effective (in terms of spam caugh and false positives) than many, many other approaches in widespread use. Paul chose to host his website there, despite supposedly knowing a lot about the spam issue. That was probably not a good call.

Let me reword your justification of of this behaviour so others can see the flaw in it more clearly:

[66.163.161.45 is a filthy neighborhood. Lots of criminals live there. So, a group of vigilantes randomly started machine gunning people walking the street. Not something I'd do myself, I prefer to use a shotgun, but certainly more effective then using the court system. Paul chose to live there, and he should have known it's a bad area. If he gets shot at random, well, too fucking bad, he should have known better. Living there was probably not a good call.]

Some days it's hard choosing between deleting 400 spams a day and dealing with the exsistance of "spam blocking" groups. Then I read a comment from an "anti-spam" person and I think I'll be safer choosing to work that delete key.

So what

[Vainglorious Coward]

I reserve the right to block (or accept) any mail I choose on my own system. I also make that decision on behalf of my users, weighing the pros and cons, and especially the listing policies, of any RBLs. If I get it wrong, then yes, my users won't be happy. I'm all for doing what makes my users happy. Blocklists do make my users happy. They work. The fact that there's sqealing about the effect shows that they work. I reject utterly the contention that I should somehow be forced to accept anything I don't want to receive

Re:Abuse my hind end

[jamie]

Obviously you feel very strongly about spam. You feel that spam is so important that websites which offer to sell spam software should be blacklisted, along with many other innocent websites hosted at the same ISP.

What else do you feel strongly about?

There are websites, I am sure, that describe in detail how to commit murder and get away with it. Some readers may find those sites, and using that knowledge, go commit violent crimes -- just as some readers of spam sites may purchase email harvesting software and then go commit the crime of sending bulk email. I assume you would support blacklisting ISPs that host violent-crime advice, since surely everyone agrees that murder is worse than spamming.

There are ISPs that host neo-Nazi propaganda calling for the murder of all non-whites. Do you think that's better or worse than offering spam software for sale? Should those ISPs be blacklisted?

Escort services? Simulated rape porn? "The Anarchist's Cookbook"? A list of abortion providers' addresses? Al Qaeda recruitment and propaganda? I want to know which of these you think is equally as bad as, or worse than, hawking a CD with a million email addresses on it. How many things do you think merit blocking all of an ISP's innocent websites?

You have your list. Others have their own lists -- and, frankly, there are a billion people who think porn is vitally important and your fixation on spam is stupid. Do you really want the internet segmented? Do you think advancing your pet cause is worth walling off the internet into warring quarters? Do you really want to wield a censor's black pen?

Re:Definitely a bad idea...

[Seumas]

That's the point - it doesn't matter how fast you respond to a spammer. If you ditch the spammer instantly, you're still going to end up on the list indefinately. In the case I cited, the spammer was kicked off within hours. I'm sure he was off to some other unwitting place to spam from while the rest of us went weeks without being able to send from our servers.

How is it an incentive for admins to be "responsive" when dealing with spammers if you're going to punish everyone within a certain radius for days or weeks even if the problem was terminated within hours?

What exactly is so wrong with blocking an IP at a time? You do away with the innocent bystanders while still nailing the spammers. Anyway, the reason they block the entire subnet has NOTHING TO DO WITH PREVENTING SPAM. It's merely a way of pissing off enough legitimate people to force the bad person to be dealt with (even if they've already been dealt with or it was an honestly unavoidable situation or what have you).

If you've identified chronically spam-friendly hosts and want to widen your net for them, that's great. But don't take out the entire neighborhood because of one bad neighbor.

Load of FUD by Paul Graham, competitor to Spamhaus

[Steve Linford, Spamh]

Gentlemen,

You do realize that Paul Graham is in the business of pushing Bayesian anti-spam filtering, which he claims as 'the best' solution to spam. For a long time Graham has been spreading FUD about other anti-spam solutions, in particular blocklists. We're well used to hearing utter bollocks about blocklists spread by him.

Yesterday we listed on the SBL an IP of a spammer which as luck would have it is being shared by Paul Graham. We of course can not simply give the spammer carte blanche to spam our users because Paul Graham is also using the same IP. Graham has no concern for the fact he's sharing his IP with a spammer, and rather than contact his ISP to ask what a spammer is doing sharing his IP he simply sees a PR oppurtunity to bolster his "blocklists are evil, bayesian is good" campaign. I'm only surprized this actually made Slashdot.

Steve Linford, CEO, Spamhaus