Paul Graham Describes Dangers of Spam Blacklists

CRoby writes "Paul Graham posted an essay describing the danger and corruption of the main spammer blacklists today. It discusses MAPS and the SBL, the blacklist created to try to alleviate the abuses of MAPS, and suggests (maybe) another blacklist's creation."

Comments

$article_title by $blowhard

[Neil Blender]

$idea will not help cut down on spam. In fact, it is detrimental. This has been know for $num_years years, but I feel I must prove that I am really smart by writing an article about it.

Definitely a bad idea...

[nev4]

We've been blacklisted before and the sysadmins who run these things often WILL NOT remove you, no matter what. I'd take all the SPAM anyday vs. not being able to send legitimate emails.

Re:Definitely a bad idea...

[Anonymous Coward]

You really don't get it.

The point isn't *me* using MAPS/SBL. The point is that others use it, thinking it makes a difference. Your netblock (that is, your ISPs netblock, or your ISPs ISPs netblock, etc) gets included in that list and *bang* you're a casualty of war.

Get it yet?

Re:Definitely a bad idea...

[Vainglorious Coward]

I'd take all the SPAM anyday vs. not being able to send legitimate emails.

Except that blocklists don't stop you sending email, they merely allow others to decide whether to accept that mail. Or do you think other people should be forced to accept any and every email you send?

Re:Definitely a bad idea...

[Seumas]

John Reid of the SBL told me this wasn't true-- that the SBL was still clean, and that they only blacklisted hosting companies' mail servers when they were spam hosts who took on innocent users as camouflage:

He is right. That definitely is NOT how SBL actually operates. I have a site that is heavily trafficked (millions per month) and they blocked my email (from my own personal server) that has delivered mail for my site for seven years with absolutely no outgoing spam or relaying having ever occurred in its entire life.

However, a spammer with false credentials faked his way into a hosting account with my colo provider and as a result, SBL blocked multiple entire submnets, rendering my entire site and service useless for almost an entire month (we deal with auctions, meaning nobody was getting closed notices, won notices, outbid notices, addresses to send payment, registration emails, lost password emails - and when they complained, I couldn't respond to help them and explain it to them).

SBL couldn't have cared less. As far as they are concerned, if one IP is a source of spam, they all are. And they'll get to fixing it in their own damn sweet time.

But the defense of SBL fan-boys is typically "well it's VOLUNTARY!".

Yeah. Whatever. Fuck off.

Re:Definitely a bad idea...

[Seumas]

Oh, NEAT. So you can afford the downtime of a service/site that must be available 99.999% of the time to find and move to another colo provider and deal with weeks of inavailability inbetween (due to the SBL block) every time SBL decides to block a slew of subnets around you just because some jerkoff decided to spam from it?

I'm glad you're so flexible. In the real world, most of us aren't.

Re:Definitely a bad idea...

[hawkbug]

Right on - a company can't simply get out of an ISP contract for a lot of reasons. Technical reasons aside, imagine getting out of a 3 yr contract after 2 months. It's not going to happen.

Re:Definitely a bad idea...

[henrywood]

It's a very difficult problem. Being charged with implementing Spam filtering measures for my company I know how difficult a line it is to walk. When you're handling mail for 600+ users you do get a different perpective on the problem.

We ended up by deciding to temporarily block mail from servers on certain blacklists (Spamhaus and Spamhaus XBL), sending a message back to the sender which allows them to release the mail. We also use SpamCop, but in a looser way; only if the mail comes from a SpamCop listed server and fails certain other tests do we, again temporarily, quarantine it. Otherwise we mark it as Spam, pass it through, and ask the recipient to tell us if it was Spam so that we can block it next time.

In either case the original sender, presuming it's a real person, has the ability to release the mail. (Of course we check all released mail, and if it's Spam the sender goes on our own permanent blacklist!).

I'm all too aware that this has the potential to add more useless mail to the system, but in practice most of these relase messages never even leave our server because the original came from a non-valid address. And it does work pretty well.

These, and other, rules allow us to block most of the Spam, which amounts to about 2/3 of all the mail we receive. And I've had a lot of compliments from the end users, so they appreciate what we're doing.

The moral is you can't trust the blacklists absolutely, but they have a very useful advisory role to play.

Re:Definitely a bad idea...

[Seumas]

The best solution is to not let your blacklist be the final word. I use SBL on my server (though I dislike them due to personal experiences when a network I was on had a spammer on it for a day and it took three weeks for my own mail from my own email server on my own rackmount to flow again) - but I don't block mail just because it's on the list. I count it in the final spamassassin score. So if you are on the list, but little or nothing about the content seems to be spam - no problem.

If you are from a blacklist and your message has lots of chick-scratch in it or other spammer tricks and it generally looks like a piece of spam, it's more likely to be caught and blocked.

But using the SBL alone and giving it the final decision over accepting mail is just giving it way too much power.

Re:Definitely a bad idea...

[Seumas]

Providers don't have a choice very often. It's incredibly easy for someone to use any number of credit cards (even stolen ones that haven't been reported) and various false identities to purchase hosting accounts. If a provider doesn't respond and just keeps letting the spammer have at it, that's fine. But if someone is cut off quickly, then restore their SBL credibility immediately. Duh.

Anyway, they shouldn't be blocking entire blocks of IPs. That doesn't even make sense. What does one guy on one IP out of hundreds or thousands who spammed for most of a day before he got caught have to do with my server which has run clean and reliable and secure and in good faith (including SPF and everything else) for the better part of a decade?

As Paul Graham already stated, this is just a strongarm tactic to harass as many innocent parties as possible. There's no other explanation for it. Are two spammers really worth denying tens of thousands of (in the case of Paul Graham) Yahoo customers?

There are bad-actors; rogue hosts. It's pretty clear when you're dealing with one who isn't. And if you were quick to put people on the SBL list, then take them down just as quickly. It is unacceptable that it took three weeks after the incident for them to finally remove them from the list.

Re:Definitely a bad idea...

[Seumas]

That's the point - it doesn't matter how fast you respond to a spammer. If you ditch the spammer instantly, you're still going to end up on the list indefinately. In the case I cited, the spammer was kicked off within hours. I'm sure he was off to some other unwitting place to spam from while the rest of us went weeks without being able to send from our servers.

How is it an incentive for admins to be "responsive" when dealing with spammers if you're going to punish everyone within a certain radius for days or weeks even if the problem was terminated within hours?

What exactly is so wrong with blocking an IP at a time? You do away with the innocent bystanders while still nailing the spammers. Anyway, the reason they block the entire subnet has NOTHING TO DO WITH PREVENTING SPAM. It's merely a way of pissing off enough legitimate people to force the bad person to be dealt with (even if they've already been dealt with or it was an honestly unavoidable situation or what have you).

If you've identified chronically spam-friendly hosts and want to widen your net for them, that's great. But don't take out the entire neighborhood because of one bad neighbor.

Re:Definitely a bad idea...

[Vainglorious Coward]

Except that I have been listed. And I had to go through contortions to fix that situation, which did not occur because of anything I did. What were you saying about acting like a dick?

As I already said, yes, I do assume the role of telling people to fuck off on behalf of my users. And I'm accountable for that. If I choose lists with inappropriate policies, or continue to use a list after its policy has changed for the worst, then I deserve to have my users demand change or my removal. No-one is pretending that RBLs are a magic bullet, or even that that they're a "configure & forget" solution. Of course there will be false-positive listings, malicious smear attacks (which is what this case appears to have been) and so on. My experience is that the damage arising from such cases is minimal when compared to the benefit of using RBLs. Simply put, RBLs work more effectively than just about any other technique (for today, at least).

And frankly, on a practical level, what are you going to do about it? Do you think you can stop groups of people organising themselves and exchanging opinions on the activities of others?

Re:Definitely a bad idea...

[capilot]

We've been blacklisted before ...

Was it for -- wait, let me guess -- was it maybe for spamming? Maybe next time you won't spam or let your users spam. Just a thought.

the sysadmins who run these things often WILL NOT remove you

Which sysadmins are those? Certainly that's true for my system. Once I drop a spammer into the system blacklist they're there for life. I don't have the time or energy to audit my block list, and what would be my motivation anyway?

The major RBL's on the other hand, will remove you if -- and this is the important part -- if you stop spamming. In this sense, the RBLs are doing you a great service. If the RBLs list you before I get mad enough to block you myself, then you have a chance to eventually get unblocked. Would you care to name a major RBL that continued to list you even after you cleaned up your act?

I'd take all the SPAM anyday vs. not being able to send legitimate emails

Ahh, but you weren't really listed for sending legitimate emails were you? If you're willing to accept spam in exchange for the ability to send it, then that seems perfectly fine to me. All the sites that want to send spam, and are willing to receive it in return need merely not subscribe to the RBLs. Voilla! The system works.

I, on the other hand, am perfectly willing to not receive spam in exchange for your inability to send it to me. The system works again!

Re:Definitely a bad idea...

[Gorm the DBA]

Except for one not so minor thing...

Credit bureaus are *heavily* regulated. If they have a file on you you can get a copy of it every few months. If there is an error, there is a defined process to follow to clear it up, and they are forced by law to resend new reports to anyone who accessed your report during the time the error was present.

"Blacklists" are not regulated at all. There is no accountability, no way to protest a listing if you believe it is incorrect. No recourse.

If you can't see a difference...then I pity you and whatever school system you went to.

Re:Definitely a bad idea...

[syukton]

Actually, I'm with singletoned, and I think it's you that has a problem with understanding. Understanding something involves realizing implications which are not immediately obvious. Understanding is something that few people ever really do. Reading the facts isn't enough, you need to be able to manipulate those facts and draw provable conclusions from them. THAT is understanding.

For example, in order to get revenge on people they believed were spamming, MAPS would blacklist the mail server of the company hosting their site.

The problem with blacklists is that they're human controlled and extremely susceptible to egotistical vigilante-ism. If I'm getting spam from a server, I don't have to block just that server. I could block every server in the headers, for example. What I choose to add to my blocklist can be totally arbitrary, and that's the problem with blocklists controlled by individuals that can block huge IP blocks.

And, in terms of preventing the "sending" of mail, you could consider a blacklist to be a postman who would, whenever he saw a letter from a given return address, he'd destroy it. Any time you got a New Scientist magazine? destroyed, at their discretion. How many companies use a blacklist without saying what's on the blacklist, or making the blacklist easily searchable and editable? Does a user ever get a message on a regular basis "Hello so and so, you've received 274 emails this week from addresses in our blocked address list (which contains mostly spammers; click here to make a change." ? No, they don't provide that helpful information with links to the relevant information.

The mail is just blocked, it disappears into a void. By intercepting it before it reaches its intended recipient you are effectively preventing it from being sent. Because it's not the addressed recipient that decides whether or not to accept the mail according to the blacklist, it's an unnamed middle-man or middle-men. A blacklist allows any server in-between the sender and the recipient to say "no, sorry, your ass is blocked."

I do think people should be forced to accept every email that I send. They shouldn't be forced to READ them all, but they should be forced to accept them. As email becomes more and more prevalent as a form of legally recognized communication (emails are used in court as evidence) it's important to recognize the implications of interfering with that communication without disclosing such interference. Would you like it if I were your postman and every time I saw your electric bill, I took it and destroyed it because I didn't like the electric company and I didn't think anybody should be subjected to their tortures? Would you like me totally interfering with your legal communication and then not telling you, not even sending you a friendly "the electric company is evil, go solar!" letter? Would you like the way that could impact your finances, your credit, your reputation? What happens when somebody adds an obscure credit union to a blacklist and people don't get fraud alert emails from the CU, just because one server in their datacenter was compromised and used to send 10,000 spams? Do you REALLY understand, now? I still don't think you do.

The blacklist themselves aren't really responsible for breaking any rules, which they believe absolves them of acting responsibly. The fact of the matter is that blacklists are often implemented in the most infuckingcredibly ignorant ways possible, unfortunately. No e-mails as per my suggestion above, no way for the sysadmins that use the blacklist to audit/edit it, etc.

We need a wiki-style collaborative blacklist that has a membership of thousands who all collaborate on this issue. It's just one more example of how giving one person too much power before they're ready to use it responsibly with proper discretion results in a disaster. A blacklist affects too many people to be implemented so willy-nilly at only a few peoples' (poor) discretion. We need a collaboration, a large committee who will not become corrupted by power (as none of the members will individually have any power) but will be a gathering of individuals who maintain their individual opinions and ensure that the system remains fair and balanced.

Re:Definitely a bad idea...

[keraneuology]

this is just a strongarm tactic to harass as many innocent parties as possible

You hit the nail right on the head. In fact, a fly on the wall related to me the entire conversation from the morning they decided to set this thing up:

Person 1: I'm bored this morning, how 'bout you?

Person 2: Yeah, me too, dewd. Let's start harassing as many innocent parties as we can!

Person 1: Yeah, dewd! That'd be way wicked cool!

Anyway, they shouldn't be blocking entire blocks of IPs. That doesn't even make sense. What does one guy on one IP out of hundreds or thousands who spammed for most of a day before he got caught have to do with my server which has run clean and reliable and secure and in good faith (including SPF and everything else) for the better part of a decade?

Blame the spammers' money and the greed of the ISPs. It used to be quite common for a spammer to run under his pink contract from an IP address until people got fed up and blocked that specific IP. Certain ISPs would then assign the spammer a new IP address knowingly full well what they were doing with the explicit intent of allowing that spammer to bypass the blocklists from people who were obviously and explicitly taking steps to avoid the spam. Unfortunately as it turned out truly innocent customers were being assigned a dirty IP address that had been previously sullied by a spammer. The moment their email server came online they were already blocked because of what had happened there before. Talk about unfair.

The spam-friendly ISPs forced the blacklisting of IP blocks: there was simply no other way to filter out the spam coming from those netblocks. Other users of that hosting service may be inconvenienced, but the system admin's right to take steps to prevent spam from gumming up the works of HIS OWN NETWORK outweights the right of anybody else to expect email originating from the same IP address used to send out three trillion ads for vgiara the week before to be received with open arms.

Does this catch innocent people in the crossfire? Unfortunately, yes. But with 4,228,250,625 possible IP addresses those who maintain the blacklists can't be expected to personally review each and every email asking to be whitelisted and spend time and effort determining who is telling the truth and who is following spam rule #1.

If widget.qqq has your domain blacklisted then your beef is with the admin of widget.qqq. Period. End of story. Beg him to whitelist you. Buy him a pizza. Send him some free (as in beer) beer. Serenade him at three in the morning. Send three billion statements of character witness. But his network, his gate, his key, his rules on granting admission.

Let's look at this another way: If I am throwing a party and, on the advice of my friend who told me that people who wear Mickey Mouse shirts are boring, I deny admission to people wearing Mickey Mouse shirts from whom will you beg entry and who shall be called nasty names for listening to somebody else?

Of course, that's the solution, isn't it? We must ban any and all people from publishing an opinion regarding the statistical probability that an email from a given IP address is spam.

Re:Definitely a bad idea...

[fm6]

...you can't argue that the process didn't put pressure on you to switch hosting providers, or at least put pressure on your hosts to ensure that they never host another spammer again...

Wrong on both counts. Blacklisters are so quick on the trigger, there are no safe providers. And how is a provider supposed to "ensure that they never host another spammer"? They can only act after a user has started spamming. Plus, they have to take some time to investigate spam complaints -- yanking someone's service without documenting their TOS violations is a good way to get sued. That delay always seems to convince blacklisters that the provider is "spam friendly".

Re:Definitely a bad idea...

[syukton]

They usually have a website and a policy telling you what is supposed to be on that list, but they NEVER block mail. By publishing a list, they give a rating. Someone else takes action based on that rating. None of your mail goes through a DNS blacklist operator's mailserver. They are simply not in the position to block anything.

Yes, I know that. They just make a list. I said that, I also said that they believe that "just making a list" absolves them from all responsibility. I also said that blacklists are implemented (by people who implement them, namely system administrators) very poorly. Were you paying attention? Do you understand?

The implementation of a blacklist is how the ISP uses it. Do they notify the customers? Do they send a weekly "You got spam from these addresses..." message? Do they enable to customers to easily edit the blacklist so that illegitimately added hosts can be removed quickly? I really don't think you understood me. heh.

It's the principle of centrally administered DNS blacklists that is at fault here, not the individual operator.

I said that a few times. Are you sure you were paying attention when you read my comment? I said that having a list maintained by people who believe themselves to be absolved of responsibility and can edit the blacklist willy-nilly without vote or consensus is bad, and we should switch to something more wiki-style that more people would have a say in.

Re:Definitely a bad idea...

[shiksaa-spamhaus]

Been blacklisted by whom, pray tell? You people who whine about Spamhaus have no clue what you're talking about. Spamhaus has editors around the globe and that means people who don't lie and who get their spam problems under control get removed - and get removed promtly. I defy any of you to show that Spamhaus has been non-responsive to anyone except spamming and spam-supporting liars.

I will thank you to stop painting everyone with the same brush. Spamhaus isn't SPEWS nor is it any other list. You don't like being listed? I wouldn't either, but then I don't spam nor do I host spammers. Deal.

And if you like spam so much, I have a metric buttload of it I'd be happy to forward to you each and every day. Send me your email addy if you've got the guts. I'm guessing you're all b.s. - IOW, you don't have the nuts or the guts to put your mailbox where your mouth is.

Re:Definitely a bad idea...

[Vainglorious Coward]

I'm with singletoned, and I think it's you that has a problem with understanding.

He(?) claimed that RBLs prevent people SENDING. He is wrong. If you agree with him that RBLs prevent sending, you are also wrong.

Reading the facts isn't enough, you need to be able to manipulate those facts and draw provable conclusions from them

Snicker. Donny Rumsfeld in da house!

I do think people should be forced to accept every email that I send.

Then you are no different than a spammer. And it's clear from the rest of your drivel that you really don't understand what happens when an RBL is in use. Hint : legitimate email suffering an RBL false-positive doesn't disappear into a black hole. That's one of the reasons why RBLs are so effective, even in an environment where some false-positives are inevitable. Or to put it another way, if the "collateral damage" from RBLs were anything other than insignificant, compared to the benefit they provide, then world+dog wouldn't be using them.

Re:Definitely a bad idea...

[trelanexiph]

Casualty of war? I think they're saying they don't want your e-mail. The internet is an even peering system. My netblock is my castle, and if I don't want you to enter you and your SMTP traffic can sit outside in the rain. You are under the misimpression that SMTP is reliable, it isn't. DNSBL's don't make it less so, they make it more so by allowing administrators to reliably filter whatever they want, whenever they want, for whatever reason they want. And if they want to use SPEWS MAPS SBL, or the AHBL they can, because guess what it's their server.

A few comments

[alanw]

From Paul Graham's original article http://paulgraham.com/spamhausblacklist.html

any filter relying on the SBL is now marking email with the url "paulgraham.com" as spam

The primary use of the SBL is to allow sysadmins to refuse e-mail coming from listed IP addresses. The mail should be rejected during the SMTP header conversation, and the senders of genuine (non-spam and non-virus) e-mails will receive a non-delivery report from their outgoing MTA.

I assume that what Paul Graham is complaining about must be SpamAssassin, or some other content filter, applying a score to articles containing URLs, which when looked up in DNS resolve to listed IP addresses. This is much less acceptable, since the sender has no way to know that their e-mail may have been classified as spam.

The details of the listing can be found at http://www.spamhaus.org/sbl/sbl.lasso?query=SBL279 45. This is a /32 - i.e. a single IP address. I don't know why Paul Graham's web site (which has that IP address) has been associated with textileshop.com, which has a completely different IP address.

The other Yahoo listing on the SBL is also a /32.

I also note in another of Paul Graham's articles http://paulgraham.com/sblbad.html he claims

The most notorious example is the MAPS RBL

As any fule kno, the most notorious spam blacklist is SPEWS. ~

Re:A few comments

[mercuryresearch]

Seeing as how this exact situation happened to me this week, I can provide some light on the /32 IP address issue.

In my case, I moved a server to a new colo facility. Most facilities have an IP block, and you get assigned an IP from it. Six months or a year ago that IP might have belonged to someone else. For me, it turned out in February a spammer installed a server at the colo, spammed from that server for a single day before the colo ISP turned them off. That IP got listed in Spamhaus; in the beginning of June I was assigned that IP.

So, I ended up with a Spamhaus listing for my mail server's IP address -- and _I_ can't get it removed. Spamhaus expects the colo operator to contact them (which they did on my request) but even there, if the blacklist operator doesn't like the ISP/colo people, they can ignore the request.

Fortunately Spamhaus listened and I got the record for my IP removed. But this showed me it was trivial for a non-spammer to inherit a blacklisted IP. I've added doing DNSBL checks on colo-assigned IP addresses for future moves to prevent any future issues.

Re:A few comments

[sloanster]

I assume that what Paul Graham is complaining about must be SpamAssassin, or some other content filter, applying a score to articles containing URLs, which when looked up in DNS resolve to listed IP addresses. This is much less acceptable, since the sender has no way to know that their e-mail may have been classified as spam.

Um, no. That's not how spamassassin works - spamassassin uses a wide spectrum approach - it can take into account whatever blacklists you want to consult, but an RBL hit in spamassassin does not automatically mark the message as spam. An RBL hit is just one of over a thousand factors taken into consideration when making the call as to whether a specific message is spam or not.

Other methods used include central clearing houses of known spam messages (razor, DCC etc), time offsets, examination of header content, message content, weighted statistical analysis, presence of buzzwords, phrases, URL patterns and more.

Using all of the methods available and making a decision based on the overall picture makes spam assassin a very effective tool, with far fewer false positives than a hard coded "RBL in the MTA" approach.

On the other hand, SA does use more machine resources than does simply rejecting a message based on an RBL result, but that's the price of intelligent behaviour - it almost always requires more effort than a knee jerk reaction.

Re:A few comments

[Desert Raven]

The most notorious example is the MAPS RBL

As any fule kno, the most notorious spam blacklist is SPEWS. ~

Actually, MAPS and ORBS are the most notorious in my book. Why? Because they got caught listing folks for reasons not specified in the listing criteria. (personal agendas) For that reason, they are the only two lists I know of to have lost legal challenges. MAPS cleaned up its act, and ORBS was shut down.

As far as I'm concerned, listing all even-numbered IP addresses is valid, so long as it is clearly stated in the list criteria. That way, sysadmins can decide whether the list is practical for them or not.

Love or hate SPEWS, they follow their own listing criteria to the letter. I have seen a few mistakes happen, but I've also seen them get cleared very quickly. Most of the folks claiming they are listed "by mistake", do fit the criteria for listing as stated in the SPEWS guidelines. Usually, because they are getting their service from an ISP that is knowingly harboring spammers. I have no sympathy for this, if you don't want to be lumped in with the spammers, don't support an ISP that allows spamming.

And I'm here to say, it's NOT impossible to get off an RBL. I got caught in a SPEWS listing, because my ISP got lax and allowed a spammer to stay on their network. It took six months for that listing to expand wide enough to cover my addresses. When I found out, I raised royal heck with my ISP, and told them in no uncertain circumstances that I would pull my service if they didn't clean up. They kicked the spammer, the Spamhaus listings were gone the next day, and within a week, the SPEWS listing covering me had been reduced so that I was no longer affected.

Having spammers on your ISP is like having a crack-house on your street. Can you blame folks for not wanting to come visit you?

Re:A few comments

[Zak3056]

When I found out, I raised royal heck with my ISP, and told them in no uncertain circumstances that I would pull my service if they didn't clean up. They kicked the spammer, the Spamhaus listings were gone the next day, and within a week, the SPEWS listing covering me had been reduced so that I was no longer affected.

This is great--IF you have the leverage to do it. If you're a large (six figures a year in spending and up) customer, you can get the ISP to jump at your command. Likewise, if you're dealing with a small local ISP, you have a significant amount of leverage even if your spending is low.

On the other hand, if you're someone with a single DS1 being provided by someone like Verio, you have NO power to negotiate or threaten. Sure, you CAN leave, but for a small organization (perhaps one with minimal or even no IT support) this kind of move is difficult, if not impossible--and in any case, is going to be really expensive. And what happens when the next time (and there will be a next time) comes around? You get to go through it all again.

RBLs (when used exclusively, instead of in some kind of weighted average ala spamassassin) are like a bad action movie--you know the ones, where the cops walk into a crowded theater and open up on the bad guys, while ignoring anyone else in the line of fire. It doesn't matter who gets taken out as long as we get our man--right?

Re:A few comments

[sjames]

For any serious stuff, don't accept an IP address which was blacklisted in the past few years (is there a service which checks this?) or is close to current blacklist entries, unless you're really really well known.

That would be hard to check (by the ISP as well), and is increasingly rare. It'll have to be outside of 0.0.0.0/0

Re:In soviet russia

[TeacherOfHeroes]

In Soviet Russia; old, tired, worn-out joke tells you

Paul is just pissed because...

[SSpade]

...his website is hosted on the same IP address as a spammer (textileshop.com) was on yesterday, and because of that he's seeing some of his mail blocked.

There's certainly a need for thoughtful and hopefully positive criticism of blacklist behaviour. This article is not it.

Re:Paul is just pissed because...

[DikSeaCup]

Is he making an accusation that Spamhaus isn't taking the IP off of the SBL? If so, maybe it's because they won't accept his word in the matter, only the word of the people who actually admin the box. Too bad - *I* wouldn't accept the word of a hosted person that the spammer is gone, only the word of the *hoster*, who, if he ends up lying, should rightfully end up with a more permanent ban. Yeah, this sucks for the hosted people, but hey - move your site. Your hoster sucks and doesn't deserve your business.

Or maybe he needs to realize that it can take some time for stuff to happen. I know so many folks who have become accustomed to immediate feedback.

Anyone know anybody who has something to do with Spamhaus? From what I understood, they were anti-spam pitbulls (this is not always a bad thing) but were also rather good at avoiding false blocks ...

Re:Paul is just pissed because...

[SSpade]

Actually the IP address that's listed is store.yahoo.com.

Yahoo hosting is riddled with spammers, and store.yahoo.com is where most of them live, and where they accept credit cards for their purchases.

The SBL lists IP addresses that are involved in spam. 66.163.161.45 is involved in a lot of spam. It's not been removed from the SBL because, well, it's still actively being used by spammers.

Because countless spammers register domains on a daily basis, yet point them at the same IP addresses some people choose to resolve the URLs in incoming email and bounce the mail if any of them resolve to particularly filthy IP addresses.

66.163.161.45 is filthy. Blocking mail that has URLs pointing there will stop a fair amount of spam. Not an approach I'd use myself, but certainly a lot more effective (in terms of spam caugh and false positives) than many, many other approaches in widespread use.

Paul chose to host his website there, despite supposedly knowing a lot about the spam issue. That was probably not a good call.

Re:Paul is just pissed because...

[deacon]

66.163.161.45 is filthy. Blocking mail that has URLs pointing there will stop a fair amount of spam. Not an approach I'd use myself, but certainly a lot more effective (in terms of spam caugh and false positives) than many, many other approaches in widespread use. Paul chose to host his website there, despite supposedly knowing a lot about the spam issue. That was probably not a good call.

Let me reword your justification of of this behaviour so others can see the flaw in it more clearly:

[66.163.161.45 is a filthy neighborhood. Lots of criminals live there. So, a group of vigilantes randomly started machine gunning people walking the street. Not something I'd do myself, I prefer to use a shotgun, but certainly more effective then using the court system. Paul chose to live there, and he should have known it's a bad area. If he gets shot at random, well, too fucking bad, he should have known better. Living there was probably not a good call.]

Some days it's hard choosing between deleting 400 spams a day and dealing with the exsistance of "spam blocking" groups. Then I read a comment from an "anti-spam" person and I think I'll be safer choosing to work that delete key.

Vigilante it ain't

[Rosco P. Coltrane]

The problem was, as vigilantes so often do, the guys at MAPS got carried away

For some reason, journalists keep calling blackmail lists "vigilantes". But there's something they don't understand: nobody forces email system administrators to use those lists.

These lists are provided by people for free. They decide to list bad email servers, but they may as well include any server they want. After all, who's to force them to provide quality of service?

The real problem, of course, is that blacklists are needed in the first place. If ISPs did their jobs a little better (aol, hotmail and the likes), the amount of spam would already decrease significantly. And don't speak to me about chinese ISPs, since most spam comes from the US.

Re:Vigilante it ain't

[Maestro4k]

For some reason, journalists keep calling blackmail lists "vigilantes". But there's something they don't understand: nobody forces email system administrators to use those lists. No, but the non-spamming sites that end up on it would certainly disagree with you, they didn't do anything to merit the block.

You seem to be confused about what a vigilante is, dictionary.com gives me this: "One who takes or advocates the taking of law enforcement into one's own hands." Note it doesn't say anything about them forcing others to agree with their views or take part in them. If you decide to take legal actions in your own hands, then you are, by definition, a vigilante. So it does apply here, just because they don't force anyone to use their lists doesn't change that.

These lists are provided by people for free. They decide to list bad email servers, but they may as well include any server they want. After all, who's to force them to provide quality of service? TFA's point was that these lists start out listing just IPs/hosts/sites they know are sending spam, then later the power corrupts ("power corrupts, absolute power corrupts absolutely") them and they start using the power they've gained by their blacklist being used by many people to start trying to force ISPs to comply with them by blocking bunches of innocents at the same ISP. That indeed has happened, although I'm really not sure if it's happened here or not. The risk of it occuring is pretty high, humans are, after all, only human and it's hard to resist that temptation, especially when you're a strong enough anti-spam advocate to run a blacklist. The real problem, of course, is that blacklists are needed in the first place. If ISPs did their jobs a little better (aol, hotmail and the likes), the amount of spam would already decrease significantly. And don't speak to me about chinese ISPs, since most spam comes from the US. The real problem is human nature in all of this. In spam existing in the first place (greed), in ISPs not blocking things they should (laziness, lack of knowledge or time), in people actually buying from spam (greed (getting something cheaper than legal means would allow), sexual desire (gotta have a longer penis!) or just simply a criminal desire to purchase illegal goods (prescription drugs for example)) as well as humans becoming corrupted by power when their blacklists get to be popular.

So basically if we can solve how to get people to stop being, well, people and giving in to baser instincts we can stop spam. Of course we'd also stop crimes of all sorts as well and we've not managed that in hundreds of years so I'm not holding my breath for it to happen.

Re:Vigilante it ain't

[Mike Markley]

This argument is horseshit. It's been horseshit for years and it will always be horseshit. The blacklists exist for the sole purpose of allowing other people to block mail based on the data contained therein. The blacklist operators don't get off the hook for having some frickin' responsibility just because they're not holding a gun to anyone's head. They publish this information with precise knowledge of what it will be used for, so this argument is basically just the administrators trying to weasel out of personal responsibility for what they list.

In case you're wondering, I do use a couple of blacklists. I use them to reject mail, as intended. I like to think that the ones I use are operated by folks who take seriously the fact that people like me are using it for that purpose.

Re:Vigilante it ain't

[hesiod]

> If you decide to take legal actions in your own hands, then you are, by definition, a vigilante

What law enforcement activities do the blacklists take into their own hands?

A Paradox?

[LegendOfLink]

A blacklist for a blacklist for a blacklist...

Personally, I find the need to disable more and more RBL's, because today a user might come thru OK, tomorrow, they're stuck in SORBS and considered a HIGH risk.

Not like people get all radical about it...

[dmorin]

Actual quote I have heard on the subject of spam blacklists: "I don't care that you're not a spammer. Your ISP allows spammers in their midst and therefore you all go on the list. Get a new ISP."

Oh, ok. Nothing like over reacting a bit.

Re:Not like people get all radical about it...

[Uruk]

No, the principle is that if ISPs know that this kind of overreaction will occur, they will make quite sure that they don't have spammers in their midst. In essence, it's an attempt to incentivize ISPs to police themselves.

What's the alternative? Having some centralized, international spam cop whose job it is to clean up every ISP on the planet? If ISPs get a completely free pass on spam and don't have to care whether their subscribers are abusing other people or not, where is their incentive to prevent the abuse? The way you avoid the tragedy of the commons is by getting people to see their individual stake in the issue.

Certainly the quote that you're pointing out isn't the most diplomatic or effective way of putting it, and I doubt this kind of thinking is behind that quote - it probably is the knee-jerk reaction that you're identifying it for. Still, the idea might have some merit.

Pure and simple...

[jellisky]

I had the unfortunate "joy" of being blocked by some of these draconian blacklists. My sister requested some information from me for a trip that she has upcoming via my yahoo.com account. After it bounced from her ISP saying that I was sending it from a "spam-hosting" ISP, I sent it from my mac.com account. Same schtick. After a couple other choices, I finally got it sent from my .edu account.

Her ISP uses SpamBag for their blacklist. SpamBag? ScamBag is more like it.

No wonder my sister is disenchanted by email. Her yahoo account got spammed to no end, then she can't get emails from most of her friends since they get bounced back by her ISP's stupid blacklist.

Blacklists are fine and dandy in principle, but practice has shown them to be useless. IT managers, just drop them. They're more annoying than anything.

-Jellisky

Re:Pure and simple...

[Anonymous Brave Guy]

Speaking of blacklists not working, the company I work for had an open relay. We discovered this when we started getting Blacklist replies one December. Management wouldn't do anything, because our admin wanted to spend $20k upgrading our server to fix the problem.

I would have thought firing the admin who left the relay open and hiring someone competent to fix it instead might have been a good thing to do. What on earth was the $20k suppose to be for?

Re:Pure and simple...

[aaronl]

Yes, fun isn't it? Trying running your own email server from a Charter business link. Then try sending email to Juno or NetZero customers. Their mail server will give you a 550 denied. Proceed to have the ISP's ignore you, and the RBL jerks ignore you.

The reason for the block? All Charter IP addresses have been put into a "residential" blocklist by one RBL nut that decided such a list was a good idea. Everyone knows that you should have to buy a T1 to send email. This is because people who really need to send email have the budget to pay 800$/mo for it, apparently. Unfortunately, Juno and NetZero both seem to agree.

Whiskey. Tango. Foxtrot. Over.

[Skye16]

So...it's okay if he goes to Federal Pound-Him-In-The-Ass penitentiary just because he rented a car from a place that also rented a car to a crack dealer?

Huh?

Sorry, but that's still bullshit. He states it clearly in his article: You can't screw over innocents just to make the guilty pay. Does the your government put a neighbor family through torture just because you got a parking ticket? No. It's YOUR fault and YOU should be punished. Not some innocent bystander.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Pay and you get removed

[tmk]

I have found an interesting offer: pay 50 bucks and you are removed immediately from the spam list. Have a look here.

Interesting: The company won't say who they are. They say this was approved by local authorities, but this is bullshit. Local authorities can not brake federal law in Germany.

Oblig. Simpsons Reference

[Mr.Progressive]

Blacklists have a structural flaw: there is no one to watch the watchers.

Lisa: If you're the police, who will police the police?
Homer: I 'unno, Coast Guard?

Who watches the Watchers?

[redelm]

... the Watched, of course! Ruel enforcement isn't a heirarchy but a loop.

Blocklists are made by people for others to use if they see fit. When they become unusable, they're no longer used. Personally, I use none. The cost to me of one false positive is greater than 1000 spams that leak through. No list is that good.

Paul Graham updates his blog

[a7244270]

OK, so PG wrote some code in the past, and is generally a smart guy, and to be honest, I actually like his writing. I like it enough that I'll even read his stuff despite the fact that he uses an excessively narrow column width for his text which makes it very annoying to read. However, there are many blogs out there written by smart programmers, some with far, far, far more geek cred than PG.

Why exactly is this a Slashdot story ?

Comment removed

[account_deleted]

Comment removed based on user account deletion

There is a problem with blacklists

[WebHostingGuy]

We deal with this all the time. Leaving any IP on a blacklist for any period of time doesn't help. Most spammers nowdays spam and run. They unload from a hacked account through a broken formmail script or a zombie computer. After 36 hours they have dumped their million emails and moved on to another IP. Blacklists generally don't get this though. They just make a bigger and bigger list. The problem with this approach is that they already missed the spammer. One time we dealt with someone who was running a blacklist and when we asked why an IP was on the list they said because it spammed years ago. When we said we have controlled the IP for the past three years they said it doesn't matter. It's like give me a break...

The solution to blacklists is to use an AOL model in which dynamic IP blocking is used. When spam is noted from an IP that IP is automatically blocked for 24-36 hours after the last spam comes in. That way the innocents are not being blocked and the spammers email doesn't make it through. There are a couple blacklists which do this but more should.

Compare this to the opposite blacklists like BLARS which requires a thousand dollars for "him" to investigate whether an IP should be removed. I have never seen an IP which is not listed with BLARS.

Re:There is a problem with blacklists

[mabu]

Spamcop's RBL does exactly what you're suggesting. Their automated system automatically "retires" IP addresses from the RBL after set amounts of time. It goes one step further though, and determines the suitability for longer-term inclusion on the list based on the IP's history of spamming. It works exceptionally well.

I have been the victim of the formmail exploit, and been RBL'd as a result. It was not difficult to get un-blocked. Yes, it was a hassle, but I suspect those that complain about being RBL'd, are the people that send nasty, vicious, "take me off or i'll sue you f'ing jerk!" e-mails and then wonder why they weren't removed. If you're polite with the RBL maintainers they're more than happy to cooperate. Anyone who's running an RBL that isn't reasonable, won't have anyone using their list so it doesn't matter.

What IP is the originating mail from?

[isn't my name]

# dig paulgraham.com MX

; <<>> DiG 9.2.4 <<>> paulgraham.com MX
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53349
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;paulgraham.com. IN MX

;; ANSWER SECTION:
paulgraham.com. 3600 IN MX 10 milter1.store.vip.sc5.yahoo.com.

;; AUTHORITY SECTION:
paulgraham.com. 3600 IN NS st-ns1.yahoo.com.
paulgraham.com. 3600 IN NS st-ns2.yahoo.com.

;; ADDITIONAL SECTION:
st-ns1.yahoo.com. 154169 IN A 216.136.225.202
st-ns2.yahoo.com. 134882 IN A 216.136.225.203

;; Query time: 228 msec
;; SERVER: 192.168.1.23#53(192.168.1.23)
;; WHEN: Thu Jun 16 14:30:43 2005
;; MSG SIZE rcvd: 150

Looking up the IP for his mail server, we get:

# nslookup milter1.store.vip.sc5.yahoo.com

Non-authoritati ve answer:
Name: milter1.store.vip.sc5.yahoo.com
Address: 216.136.232.238

A Multi-RBL check on that IP shows absolutely no black-listing in any of the many RBLs.

Is it possible that it's his outgoing cable-modem IP address that is the problem?

Is it, as the parent suggests, spam-assasin filtering?

I'm more than happy to get on the wagon of unresponsive RBLs. The only way they can actually get the response they want is if cleaning up your act results in de-listing.

However, Mr. Graham makes some big claims with nothing to back it up--and attempting to investigate on your own shows that his claims don't seem to check out.

Re:What IP is the originating mail from?

[kaarlov]

MX records don't always tell where the mail is sent from. In fact it is good idea to have separate server for sending mail. For example if your MX in some situation sends bounces to forged aol-addresses, it gets very easily blacklisted temporarily by AOL. But sending mail directly from server which hosts multiple webpages in same ip is not a good idea. But I don't thing Graham does that either.

From TFA and from parent article I got impression that he suffers from people having spam filters which run URL's in the email body through blacklists. And I think that spam filter which gives too much points for that is more broken than the concept of DNSBLs.

What's the real story?

[argent]

People switched from MAPS because the other lists were free, not because MAPS was too aggressive.

"As of this writing, any filter relying on the SBL is now marking email with the url "paulgraham.com" as spam."

Whisky Tango Foxtrot? *BLs block IP address ranges, not URLs.

"Because the guys at the SBL want to pressure Yahoo, where paulgraham.com is hosted, to delete the site of a company they believe is spamming."

1. Given that Paul's mixing up URLs and addresses of mail servers, I'm not prepared to take at face value the statement that SBL is blocking Yahoo's mail servers to pressure Yahoo to drop a "site", rather than (say) mail services Yahoo is providing the spammer.

2. If Yahoo is providing services to a spammer and Yahoo refuses to deny those services to a spammer, than Yahoo is being "spam friendly", no matter what their reputation is, and they may well be depending on the many legitimate lists they're hosting to avoid responsibility for their actions. That's exactly the situation that John Reid is referring to in Paul's quote.

I don't know what alleged spammer this is referring to, but what Paul's written is clearly not anywhere near the whole story.

Calling a spade a spade

[Valdrax]

For some reason, journalists keep calling blackmail lists "vigilantes". But there's something they don't understand: nobody forces email system administrators to use those lists.

To be honest, I like his other analogy for blacklist maintainers -- terrorists. It's much truer to the point. Vigilante in my mind at least implies an attempt to go after the bad guys and protect the innocents thanks to the pop culture influence of TV, movies, and superhero comics.

This doesn't describe blacklist maintainers.

Blacklist maintainers are cynical, bitter, little men who care nothing for the people they hurt so long as they get a spammer. They deliberately target innocents in the hopes that the innocents will complain to the higher power to get rid of the things that bothers them. This leaves little to distinguish them from terrorists other than the fact that they don't kill people. Their deeds are less dark, but their tactics are the same as the Madrid bombers who hurt innocent people to push them to choose a government more favorable to their wishes.

Sure, nobody forces email admins to use those lists. Nobody forces people in the Middle East to contribute money to Hamas either. I don't care if you think you're funding hospitals and charity for Palestinians or if you think you're fighting to keep spam off the web -- you're paying to see people get hurt too. Stop it.

Re:Calling a spade a spade

[3nd32]

Oh, come on. Do we need a new version of Godwin's Law? Blocking a website and blowing up innocent people are not comparable.

Re:Calling a spade a spade

[Valdrax]

Welllllll.... maybe. I did try to clearly deliniate that I did not see murder and extortion as morally equivalent, but I figured that I'll draw some flamebait mods anyway.

The point is still a good one. Is it morally reprehensible to target innocents for the purposes of shaping institutions of power? Is this not fundamentally the definition of terrorism? If you agree on both counts, then MAPS is an opt-in terrorist network dedicated to the destruction of spammers.

Re:Calling a spade a spade

[Valdrax]

So you mean, appart from the fact that they lack the defining characteristic of terrorists, these people are just like terrorists.

No. That's the defining characteristic of murderers. There are other ways to commit acts of terror. Kidnapping (without murder), rape, sabotage, etc. all can be acts of terrorism if intended to shape someone's opinion or vote. Really, the place where the analogy fails is that terrorism is inherently violent, where spam blacklists are not.

However, the core issue of spam blacklists deliberately targetting innocents to get them to demand change puts them in the same philosophical camp in my mind.

Guideline, not a rule

[bitflip]

I use blacklists all the time. Rather than simply rejecting the mail, if the server is on a blacklist, the initial OK is delayed by five seconds.

If you're sending a ton of mail, i.e., spam, little of it gets through. If you're only sending one or two messages, ie, likely legit mail, it goes through just fine.

Combined with more specific stuff further back (bayes, et. al), it's been quite effective at reducing the amount of spam sent, and the amount of mail that gets scanned.

The problem isn't blacklists, its how people use them.

Wrong

[autopr0n]

What they do is allow others to block email between two diffrent people, simply because they run the mail servers that sit between them. If it was only individual users who were using these blocklists, it would be a diffrent issue. But it's not.

Re:Wrong

[squiggleslash]

You're why sysdadmins and blacklists have a bad name. Just because you can do it, doesn't mean you should or even that it's particularly intelligent to do so.

If I can't receive email from a friend because my mail provider, who I pay money to, is as stupid as some of the BL-supporters here, you can bet I'll yell at them. They can whine as long as they like about how it's their equipment, *I* pay their wages.

Re:Wrong

[Linux_ho]

If I can't receive email from a friend because my mail provider, who I pay money to, is as stupid as some of the BL-supporters here, you can bet I'll yell at them.

RBL's don't kill e-mail, bad sysadmins kill e-mail. You're just demonstrating your own ignorance of spam-blocking techniques by saying "BL-supporters" are stupid. RBLs are an incredibly valuable tool. My systems, which process about 30,000 messages per day (60-70% spam), NEVER reject a message based on a single RBL hit. But if an IP is listed on three or more different reputable RBLs and doesn't have a very low Bayes score, that message is probably getting rejected. RBLs contribute a huge amount to my (currently > 99%) spam detection accuracy.

Re:Wrong

[Fulcrum of Evil]

You're why sysdadmins and blacklists have a bad name. Just because you can do it, doesn't mean you should or even that it's particularly intelligent to do so.

When you're a sysadmin, you have to weigh the flood of penis pills and mortgage scams against one or two people not getting an email because the sender is hosted by someone who can't secure their mailserver. It's really an easy call. Before you start spouting on about giving users the choice of what to receive, there's also the sheer volume of spam - accepting too much email can put a serious strain on the servers and degrade the experience for everyone.

spam blacklist blackmail?

[matt me]

Blacklisting is clearly just opening more oppurtunies for cyber-crime: spammers threatening to get companies blacklisted by major ISPs unless they pay up. Sending a few emails from fake addresses to the right places is a lot easier than organising DoS attacks from BotNets.

Loss of email hurts more too.

Re:today?

[Joe U]

"Vigilante is a very strong word "

You're right. The correct words are 'overreacting assholes'.

Most RBLs are run by assholes who have no concept of how to properly manage something as complex as a RBL.

And no, I've never been blocked by one and I weight RBL positives very low.

"Power-hungry weenies"

[slavemowgli]

Interestingly enough, the owner of the acme.com domain who was recently featured in a story due to his getting more than a million spam mails (well, attempts to send spam) a day, agrees:

DNS-RBLs - Domain Name System Realtime Black Lists. In theory the idea is fine. You have a set of sites that you blacklist, and you want to let other folks use the same list so you distribute it using DNS, which is a nice efficient de-centralized database. What's not to like?

Well, I don't know why, but in practice every single DNS-RBL eventually comes under the control of power-hungry weenies. They start listing sites unreliably, and if you complain you find yourself listed. And there's usually no way to get off the list.

A lot of people tell me I'm wrong about this. They say that certain DNS-RBLs are ok, with objective criteria for inclusion and simple procedures for getting off the list. The thing is, they give conflicting recommendations for which lists are good and which are bad. Some of these folks recommend lists which I know from personal experience are bad.

This problem is really inherent in the way DNS-RBLs are set up. You cede control of your mail system to a third party, with no real possibility of checking how they are doing. The people running the lists get overwhelmed with bogus feedback from spammers and/or idiots, to the point where they assume all their mail about the lists is from spammers and/or idiots.

If the lists you use have not yet descended into corruption and chaos, consider yourself temporarily lucky.

Do not use DNS-RBLs.

(from http://www.acme.com/mail_filtering/shame_frameset. html)

Re:"Power-hungry weenies"

[slavemowgli]

He may be referring to an older qmail version - I assume that he made the observation when he evaluated different MTAs and then didn't bother checking newer versions after he decided on one.

That being said, I think his comments about blacklists pretty much hit the nail on the head. Think about it: what you're ultimately doing is give some complete stranger near-complete control over what email is or isn't accepted by your system. Blacklists are something that might seem like a good idea in theory, but when you really think about them, they're not anymore. There's just too many ways they can be subverted in one way or another.

Gosh darn terrorists

[RickPartin]

From the article:
This is, strictly speaking, terrorism: harming innocent people as a way to pressure some central authority into doing what you want.

Can we please stop throwing the word terrorism into every sentence? Please? No? Damn.

What a clusterfuck

[maynard]

blocking spammers via a central database just doesn't work. The spammers are constantly moving from zombie client to zombie client in huge waves of hundreds of thousands of infected systems, making the RBL always filled with obsolete and incorrect information. The problem - as everyone knows - is that the protocol is fundamentally broken. It's a tragedy of the commons played out in front of our eyes.

By allowing the abuse it's outcome becomes a certainty. We're going to have to bite the bullet and dump open SMTP. And I think we're going to have to do this quickly. The levels of SPAM continue to rise. I often see ten to twenty times as many spam connections on my mail servers than legitimate connections, and this is a constant, flowing, amount of SPAM 24/7. Even with RBLs, spamassassin, etc, SPAM still gets through. The solution will not be found with another bandaid. It's time to dump SMTP and move to something that demands cryptographic authentication for users and hosts before allowing the transport session to complete. --M

'Terrorism' my behind... MAPS' side of the story

[mi]

Although MAPS did, indeed, only blacklist the actual spammers at the beginning, they changed not because they 'got carried away' (Paul Graham's words), but because the spammers adapted.

Here is the link, that responsible editors would've offered in a story like this...

So what

[Vainglorious Coward]

I reserve the right to block (or accept) any mail I choose on my own system. I also make that decision on behalf of my users, weighing the pros and cons, and especially the listing policies, of any RBLs. If I get it wrong, then yes, my users won't be happy. I'm all for doing what makes my users happy. Blocklists do make my users happy. They work. The fact that there's sqealing about the effect shows that they work. I reject utterly the contention that I should somehow be forced to accept anything I don't want to receive

Re:So what

[Chris Burke]

I reject utterly the contention that I should somehow be forced to accept anything I don't want to receive

And that means that you will readily accept someone else's decision on what you should and should not receive? You sound to individualistic for that, so I think you are probably missing the implications of these blacklists.

What if you want to receive email from someone, but their block is in the blacklist your ISP uses? Can you call up your ISP and ask them to remove it? Can you get your friend to change their ISP so they are in a non-blacklisted block? In the past, I've seen people whose ISPs would block, for example, the entire University of Michigan. That made it pretty tough to communicate with them.

You are absolutely under no obligation to accept anything. That's why I run a spam filter myself. But letting someone else's often arbitrary judgement control what you do and don't receive is contrary to the personal control that you (and I) want.

Speaking of which, I'm glad I'm not one of your users.

Re:So what

[Chris Burke]

The fact that there's sqealing about the effect shows that they work.

Um, no.

The fact that there's squealing about the effect from non-spammers shows that they don't work.

Re:So what

[Vainglorious Coward]

Okay, but I question how you can actually know how much the RBL is costing you.

Millions and millions of rejected messages versus the occasional manual intervention. It's a pretty easy judgement. I can even figure an average spam message size, multiply by the number received, compare that to my ham traffic, weight it against the cost of running my mail service and produce a dollars and cents figure of what RBLs save me (and that's before I factor in the costs associated with users having to deal with those spams if they were delivered). If I'm rejecting two thirds of all delivery attempts at the front door, I don't need to have mail systems that are three times the size and three times the cost.

If an employee sends an email asking for product information from Companies A, B, C, and D, but only gets answers from C and D, is he going to call you up assuming there's a problem or is he going to assume A and B aren't interested?

You seem to be conflating the case where I am using RBLs and the case where someone else is. If my employee attempts to send an email to a system that has us on their blocklist, my employee gets a non-delivery report from my system, advising him that the message was not delivered, including a transcript of the SMTP dialogue ("552 We don't like people with a "K" in their name"). Typically, he would then contact me and ask what was up, and I then deal with it in whatever way is appropriate. In the case where somebody elses employee tries to send to us, and we reject because of a RBL listing, that remote person gets a non-delivery report from their own system, and it is for the remote admin to deal with it as appropriate. I can only take responsibility for my own systems, I can't be postmaster for everybody else.

Shorts are no place for a hamster.

Re:Abuse my hind end

[jamie]

Obviously you feel very strongly about spam. You feel that spam is so important that websites which offer to sell spam software should be blacklisted, along with many other innocent websites hosted at the same ISP.

What else do you feel strongly about?

There are websites, I am sure, that describe in detail how to commit murder and get away with it. Some readers may find those sites, and using that knowledge, go commit violent crimes -- just as some readers of spam sites may purchase email harvesting software and then go commit the crime of sending bulk email. I assume you would support blacklisting ISPs that host violent-crime advice, since surely everyone agrees that murder is worse than spamming.

There are ISPs that host neo-Nazi propaganda calling for the murder of all non-whites. Do you think that's better or worse than offering spam software for sale? Should those ISPs be blacklisted?

Escort services? Simulated rape porn? "The Anarchist's Cookbook"? A list of abortion providers' addresses? Al Qaeda recruitment and propaganda? I want to know which of these you think is equally as bad as, or worse than, hawking a CD with a million email addresses on it. How many things do you think merit blocking all of an ISP's innocent websites?

You have your list. Others have their own lists -- and, frankly, there are a billion people who think porn is vitally important and your fixation on spam is stupid. Do you really want the internet segmented? Do you think advancing your pet cause is worth walling off the internet into warring quarters? Do you really want to wield a censor's black pen?

Maybe Paul Graham should look up "hyperbole"

[otter42]

This is, strictly speaking, terrorism: harming innnocent people as a way to pressure some central authority into doing what you want.

No. No... No, there's just something not right about that. I'm pretty sure that the definition of terrorism includes the idea of terror somewhere...

Ahhh. That's more like it: Terrorism: the unlawful use or threatened use of force or violence by a person or an organized group against people or property with the intention of intimidating or coercing societies or governments, often for ideological or political reasons.

Yeah, violence should induce terror. Not being able to send emails to my girlfriend, as hair-raising an idea as that might be, just doesn't seem to be in the same league.

And just in case Mr. Graham is too lazy to find a dictionary to look up hyperbole for himself: hyperbole - n : extravagant exaggeration

non-mail server in SBL, what about mail server?

[jdunlevy]

From TFA,

As of this writing, any filter relying on the SBL is now marking email with the url "paulgraham.com" as spam. Why? Because the guys at the SBL want to pressure Yahoo, where paulgraham.com is hosted, to delete the site of a company they believe is spamming.

E-mail w/ the 'url "paulgraham.com"'? The SBL doesn't check URLs, it'd doesn't even check domain names, it checks IP numbers. paulgraham.com resolves to [66.163.161.45], which is listed in the SBL (details for SBL27945), but since this isn't a mail server, I don't see how e-mail from paulgraham.com gets marked as spam by users of the SBL. I note that the MX record for paulgraham.com is milter1.store.vip.sc5.yahoo.com [216.136.232.238], which is not in the SBL. He never mentions what he uses as his smtp server, but I'm supsecting it either not the SBL -- or it's in for a different reason than he thinks.

Also, for what it's worth, I've found the SBL incredibly reliable (except recently, when I've found it's been increasingly unreachable at peak times), but I check it as one of many spamassassin rules -- I don't mark e-mail as spam just because it's in the SBL, though the way I have spamassassin score things, it doesn't take much more...

Home Connectivity ISP != Your Domain ISP

[billstewart]

Maybe you only have three choices of broadband ISP at home, or live somewhere sufficiently rural that there are only three choices of dial ISP - that's entirely irrelevant to how many choices you have on where you get your email, send your email, or host your web servers. Sure, it's convenient to be able to run all those things from your home Linux box, but if you want to do that, you'll probably find that your cable modem company and some of the DSL ISPs that your phone company supports might not permit that. There are hundreds or thousands of companies that run POP/IMAP mailbox services, and probably more that will host web sites, and that's not even getting into options like virtual hosting.

Re:Home Connectivity ISP != Your Domain ISP

[Skye16]

Right. So then, when those of us with a .nu domain name have to change ISPs constantly because, at any moment, someone else - that we have no control over - ruins the ability for our email to go to its intended recipient - we just get to suck up the 10$ a pop IP change for our DNS? And even aside that point - while hosting companies are a dime a dozen, good hosting companies aren't. When we do find one that is, we want to stick with it. It's not their fault someone else at the same colo decided to be a jackass.

Basically, you're just saying "too bad, I'm tired of being screwed over by spam" and I'm saying "wtf, I'm tired of being screwed over by blacklists that can't keep their shit together". Put yourself in my shoes - when a blacklist service becomes worse than spam and the spammers who spam, what does that tell you about blacklists?

Stopping spam is easy.

[jellomizer]

Just block the sub net 0.0.0.0

Collective Punishment

[edibleplastic]

What you are promoting is the tactic known in the real world as "Collective Punishment". This is the situation where retribution is meted out to anyone in the vicinity of the concerned party (innocent or not) in order to pressure that party to change. In this case, you find it acceptable that innocent users could get hurt (innocent, probably non-tech savvy users who don't know much about other ISPs or SPAM, or anything) just so that you can put pressure on ISPs to change their ways.

Now here's the fascinating part: you link to the site antiwar.com which has not 1, not 2, but 423 pages decrying the use of collective punishment.

If that's not hypocrisy, I don't know what is. Sure email's not a life and death situation, but the principle is the same in both cases. Don't like it when innocent people get their homes destroyed? You should hate it when innocent people get their IPs blacklisted.

Re:Abuse my hind end

[jp10558]

However, you seem to think it's easy to change ISPs. I can't. I have ONE broadband ISP where I live. ONE. I cannot switch.

If you suggest I move... that's rediciulous. Let's all just up and move to a different town each time a spammer comes by. Sure. Maybe if you're Bill Gates.

It is NOT easy to change ISPs, nor is it necessarily even possible. Oh, it's my fault for living here. Well excuse me - get the hell off your high horse. It's people like you making e-mail unuseable.

Re:OK, I'll go first: how is this legal?!

[Otter]

1) They're based in the UK.

That makes a defamation / slander / libel suit much easier, not harder.

This is ONE Single IP Address that's blocked.

[billstewart]

There have been spam blacklists that worked that way; they mostly weren't worth using, except as SpamAssassin weights, and mostly nobody cares. And there have been Open Relay blacklists that blacklisted every mail server at an ISP to get their attention until they cleaned up open relays, even if only some of that ISP's customers had open relays.

But this is different - this is ONE IP address - the SBL record identifies it as a /32. Virtual Hosting means that it's possible to have multiple domains all using the same IP address for their email or websites, and if you're going to blacklist based on IP addresses, it doesn't get more granular than one address (unless you want to do things like have different return codes for "address has one spammer and some non-spammers".) So if one IP address has 100 legitimate users and one spammer, and you receive email from them, is it more likely that the mail is one of the 10000 (100 users x 100 messages/day) good messages, or one of the 1,000,000 spam sent by the spammer? 99% likely that it's spam; sorry if it was Paul.

Worst. Analogy. Ever.

[Otto]

Your analogy is freakin' terrible.

Paul hasn't been shot. Emails he tried to send have not been delivered. Drawing a comparison between physical violence and the fact that a guy can't send email is rather disingenious.

What's worse is that you still got the analogy wrong. Nobody has attacked Paul. His mail server is fine. HE CAN STILL SEND EMAIL. Other people, however, can CHOOSE to reject his email because of his IP being on a list. Nobody's touched his servers.

To use your crappy analogy, nobody's shot anybody. Instead, they've put his address on a list and then people who want to know about where the bad parts of town are can read that list and think that Paul is bad because he lives there too. Then they can throw mail he sent them away based on that.

You know why they do that?

[EvilStein]

They tell people to "Get a different colo" which is just ridiculous. Or, they'll tell you to pressure your colo to stop hosting spammers.
Mine *doesn't* host spammers, and I'm in a contract. I can't pressure them to stop hosting spammers if they don't host any.

I stopped using RBLs/MAPS/SPEWS years ago and have never looked back. Even more interesting is that the volume of spam *did not* increase, but the complaints about being bounced/not getting through decreased.

Re:You know why they do that?

[prockcore]

Even more interesting is that the volume of spam *did not* increase, but the complaints about being bounced/not getting through decreased.

That's the biggest problem with RBLs... you have *no* way of knowing how effective they are. Since mail gets blocked at the server, you can't tell how many false positives or true positives there are.

How much spam are you blocking? How much legit mail are you blocking? You have no way of knowing.

Randomly denying 6 out of every 10 emails delivered would probably be just as effective as using an RBL.

Distributed List

[suwain_2]

The problem with blacklists is that -- the guy who recently had a story on spam here, at acme.com, put it nicely -- blacklists start off good, but always turn corrupt and start blacklisting excessively.

Suppose a "distributed" blacklist were created. I could blacklist the whole Internet, but I'd be the only one, so it wouldn't mean a thing. On the other hand, if 75,000 people have blacklisted an IP, there might be something there.

It needn't be totally distributed, I don't think. A community-run site, where, whenever you get obvious spam, you post the originating IP, could work. You'd post it, and that IP would have, say, 10 "points." The rating would "decay" by one point a day, so a site listed, but that went clean, would quickly leave the list: in ten days, each rating would be down to zero.

You could then simply query the site for a given IP, and it'd return the "points" a site had. This also allows you a lot more customizability: if you were obsessed with blocking all potential spam, you could block anything with more than 5 points. If you wanted to be careful, you might set it to, say, 1000 points.

Unless the people running the site keeping track of the ratings begin blatantly making up ratings, this idea means that a blacklist is much less immune to being "bad." And it allows IPs to "fade" out of the list over time.

Private blocklists.

[Pig Hogger]

There are many, many private blocklists that are not advertised anywhere.

Here is my very own private /etc/mail/access blocklist which I use on my own mail server:

#
12.217.112 550 Mediacom. Heh. What a fucking spamming cesspool. So why not eat shit and die???
12.217.113 550 Mediacom. Heh. What a fucking spamming cesspool. So why not eat shit and die???
12.217.114 550 Mediacom. Heh. What a fucking spamming cesspool. So why not eat shit and die???
12.217.115 550 Mediacom. Heh. What a fucking spamming cesspool. So why not eat shit and die???
12.217.116 550 Mediacom. Heh. What a fucking spamming cesspool. So why not eat shit and die???
12.217.117 550 Mediacom. Heh. What a fucking spamming cesspool. So why not eat shit and die???
12.217.118 550 Mediacom. Heh. What a fucking spamming cesspool. So why not eat shit and die???
12.217.119 550 Mediacom. Heh. What a fucking spamming cesspool. So why not eat shit and die???
24 550 Comcast, when you'll have cleaned your zombies, you can knock here. Not before.
24.174 550 Chuck Jones must be spinning in his grave when he see he's associated with spam. Close port 25, fuckers.
59.0 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.10 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.1 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.11 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.12 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.13 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.14 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.15 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.16 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.17 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.18 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.19 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.2 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.20 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.21 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.22 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.23 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.24 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.25 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.26 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.27 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.28 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.29 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.3 550 It's not surprizing that a country split in two like Korea would have a totally fucked-up "internet".
59.30 5

Load of FUD by Paul Graham, competitor to Spamhaus

[Steve Linford, Spamh]

Gentlemen,

You do realize that Paul Graham is in the business of pushing Bayesian anti-spam filtering, which he claims as 'the best' solution to spam. For a long time Graham has been spreading FUD about other anti-spam solutions, in particular blocklists. We're well used to hearing utter bollocks about blocklists spread by him.

Yesterday we listed on the SBL an IP of a spammer which as luck would have it is being shared by Paul Graham. We of course can not simply give the spammer carte blanche to spam our users because Paul Graham is also using the same IP. Graham has no concern for the fact he's sharing his IP with a spammer, and rather than contact his ISP to ask what a spammer is doing sharing his IP he simply sees a PR oppurtunity to bolster his "blocklists are evil, bayesian is good" campaign. I'm only surprized this actually made Slashdot.

Steve Linford, CEO, Spamhaus

Terrorism? Hardly.

[ChaosDiscord]

Graham has written some insightful and well thought out stuff, but this is just sloppy:

This is, strictly speaking, terrorism: harming innnocent people as a way to pressure some central authority into doing what you want.

I find it amazing that blacklists which mail servers must opt-in to use are somehow terrorism. Are you suggesting that these innocent people have some fundamental right to contact my mail server and send mail? They certainly don't; it's my mail server. I can use any methods I like to filter out mail, including chosing to rely on one of the IP blacklists. This can only be terrorism if random people have some sort of human right to send mail to my machine. I hardly think that's a right.

Come to think of it, apparently organizing against tangentally related people to stop another problem is terrorism? By that strange standard you could call advertiser boycotts terrorism: you're trying to influence some media outlet by negatively influencing advertisers on that outlet. They often have the same claim of innocence ("I didn't know that they would run that article! I just buy bulk advertising rates.")

(Now there are problems with blacklists, perhaps most significantly that many ISPs use them without informing their subscribers or allowing them to opt out. Blacklisting unaware users who happen to share a machine with a spammer's website is definately a complex question.)

P.S.

[That's Unpossible!]

"A much better way to cut down on spam is to use $technology_I_created."

Speaking of blacklists

[TCM]

Going away from SMTP, I am currently running a Squid HTTP proxy with a quite long blacklist of URLs and networks of "marketing" and "ad" companies.

I find myself doing for example a lookup of ad.marketingscum.com followed by a whois lookup of the IP address. If I find that they own a larger network like

NetRange: 216.73.80.0 - 216.73.95.255
CIDR: 216.73.80.0/20
NetName: DOUBLECLICK-NET

I enter the complete network into my blacklist. Are there any realtime blacklists for this purpose? This would be quite useful, wouldn't it?

Re:Load of FUD by Paul Graham, competitor to Spamh

[jmason]

hmm. What's the relationship between the user 'Steve Linford, Spamh' (who's never made any comments before this story) and 'Steve Linford' (comments made back in 2001)?

Re:Slashdot Language lesson

[DavidTC]

Vigilantes don't technically 'have' to break the law.

For example, in many places it's legal to do a citizen's arrest if you see someone actually committing a crime. If someone suspects a crime will be commited and hangs around armed with the intent of bringing the person in, that's vigilantism, and perfectly legal. Or even hanging around waiting to call the cops.

Or if, for example, people keep getting attacked in a certain part of town, so you, who happen to have a blackbelt, wander through there, waiting to be attacked so you can fight back...

It's usually not called vigilantism if it's legal, but if you are attempting to do the work of the legal system, it is being a vigilante.

However, vigilantism requires enforcing a law, be it an actual law or just a made up one. Or punishing someone who already broke the law. (Or, as sometimes happens, you merely suspect broke the law.)

Whereas spam fighting may be interacting with the results of a crime, it's no more vigilantism than picking up litter is, or rebuilding a house torched by arson. The crime already happened, no one's trying to punish or catch the criminals, they're trying to undo the harm caused.

I guess you technically could call spam reporters 'civil vigilantes', by analogy, because they are reporting a contract violation between two third parties to one of those parties. Instead of taking criminal offenses into their own hands, they're taking civil ones. But that's getting a bit silly.

Unsolicited Plug (from me) ...

[Dr.Dubious DDQ]

Considering how much my spam has been reduced by the SBL (anywhere from at least 50% up to 75%) I'd like to just say:

The mail servers under my control have always subscribed to the SBL-XBL (well, more accurately, before the XBL was established it was the SBL and cbl.abuseat.org. The latter is dedicated to short-term [72 hours, as I recall] blocking of e.g. spammers operating on DSL or cablemodem lines who are likely to appear on an IP address once or twice and then get kicked off. The CBL is now also represented in the XBL). I have so far, in the last 3-4 years or so, only been able to confirm 1 and 1/2 "false" positives in that entire time - one was from a person in China who was using a confirmed spam-haven ISP, the "1/2" from a company that, after an informative response from the CBL people, I believe were listed for appropriate reasons. In any case, the latter case cleared itself up when they were automatically re-removed from the CBL [they'd been there before] and the email lost WAS an advertisement anyway...)

I have noticed the numerous stories of overzealous blocklists, which are obviously a bad thing, but I can't think of a way to reasonably put the SBL in that category...

Besides, bayesian filtering only works AFTER the spammer has been allowed to tie up my mail server's bandwidth (and then allows them to tie up your mail server's CPU time with the bayesian analysis). I prefer to cut off known spammers before that point whenever possible. THEN I pass the remaining messages through SpamAssassin. Back in the early days of spam, I used to actually go to the effort of picking apart the mail headers and looking up the abuse addresses for the ISP whence the mail came AND the hoster of the spammers website (and on one or two occasions, even the registrar for the spammer's domain name, when I could confirm that the information was falsified). It's been a long time since I was able to keep up doing that with the volume of spam coming in, but I still can't stand the thought of allowing spammers to take ANYTHING from me that I can prevent...

Re:Wholehearted Agreement

[aaronl]

That works fine for him to keep the mail coming in. The problem is when you combine the annoying "dynamic ip range" lists with an idiotic admin that thinks using one to blindly deny is a good idea. I mentioned in another post, but Juno and Netzero do this. Neither will pay attention to you when you complain. Of course they also RBL deny their postmaster account, which is a no-no.

Re:RBL advice

[AaronLawrence]

Re Spamcop; The simple fact though, is that "misdirected bounces", though well intentioned, make the problem of spam quite significantly worse. It pushes the spam off to someone else. Sure, the system doing the bounces is not "spamming" but they are acting as a spam transfer system, a bit like open relays used to.

Still you obviously have a reasoned and generally reasonable stance on blacklists. Congratulations ;)