Spam Blocking Engine for OpenBSD

mkeke writes "In a post over at OpenBSD Journal, Theo states that he has written a spam blocker that works with pf and Spews. It looks darn cool :)"

Comments

550? 450?

[Habbie]

I assume he means a 450 reply, not a 550? 550 won't make the message stay in the queue, 450 will.

Re:550? 450?

[edgarde]

450 says there's still a mailbox there. 550 says not found. Here's a list of SMTP codes.

Incidentally, the code actually has a command line option to choose between 450 and 550.

Spews = /m\

[joeszilagyi]

Why even bother with Spews? Why not Spamcop, who doesn't block half the planet?

Re:Spews = /m\

[PacketMaster]

And spews doesn't? Spews randomly blocked a consulting company's netblock I worked for part-time simply because that our block was next to a "known spammer's" block. When they politely asked to be removed and pointed out that according to their own evidence file that their netblock had nothing to with spam, they were met with very hostile responses and told to essentially ditch their teleco provider because they'd never unlist anyone. They admitted that they simply block IPs in a form of "collateral damage" because they feel like it to hurt legitimate businesses so they flee their network provider. Look at antispews.org for more info on their flagrant abuses and why you shouldn't use spews.

... generally doesn't cause innocent third parties distress while attempting to achieve his goals.

Using spews is going to cause third-party distress.

Re:Spews = /m\

[Just Some Guy]

Your company was paying that ISP. Thus it was also supporting spam.

I understand the principle involved, and admit a fair bit of sympathy for that point of view. However, for some of us, switching ISPs isn't a luxury we have. I live in a small Midwest town. My options are:

• DSL/wireless via the local dominant ISP
• DSL via MSN
• Cable modem
• Dialup via one of those "unlimited access for only $6.95!!!!" companies

Out of that list, the first option is the only one viable for hosting servers, since the rest either block service ports, have onerous TOS contracts, or just aren't serious connections.

Say that I discover that the local ISP (which has probably a 98% market share here) has some customers with open relays. What do I do? Buy a T1 and contract with Qwest, or get out of online business altogether?

In practicality, I don't have the option to switch, regardless of my ISPs policies.

Fortunately, the provider is run by a great set of people, and employees several real system administrators, so I don't really have to worry about this hypothetical problem. That's a Good Thing, because I'm pretty well stuck where I am.

Re:Spews = /m\

[Senior Frac]

And spews doesn't? Spews randomly blocked a consulting company's netblock I worked for part-time simply because that our block was next to a "known spammer's" block.

I just went to SPEWS' website. It appears that this falls within their listing criteria. I'll take it you don't agree with their listing criteria.

When they politely asked to be removed and pointed out that according to their own evidence file that their netblock had nothing to with spam, they were met with very hostile responses and told to essentially ditch their teleco provider because they'd never unlist anyone.

They talked to SPEWS? It says here SPEWS doesn't talk to anyone. Are you sure? That statement appears highly misleading. Are you certain they didn't talk to news.admin.net-abuse.email?

They admitted that they simply block IPs in a form of "collateral damage" because they feel like it to hurt legitimate businesses so they flee their network provider.

Boy, this is so misleading as to be approaching a lie. They really, really talked to SPEWS, huh? And "spews said"...?

Look at antispews.org [antispews.org] for more info on their flagrant abuses and why you shouldn't use spews.

The fact that you disagree with their listing criteria is all fine and good; that is your right. But there seem to be lots of outright wrong information on that webpage.

My server, SPEWS recommends, my decision whether to trust them, and my decision as to their effectiveness.

Re:Spews = /m\

[Dimensio]

Antispews is run by a known hack.
SPEWS is used because it works. It is NOT the job of my ISP to tell your ISP to kick off their spammers. If your upstream is providing an open haven for criminals, don't be surprised when no one wants traffic from your upstream.

Remember, your consulting company wasn't being blocked. Your consulting company didn't own the ISPs. SPEWS wasn't blocking anything (anyone who claims that SPEWS blocks is either ignorant or lying), SPEWS was merely listing IP addresses owned by the upstream provider. It isn't SPEWS's probem that your upstream is rogue and that no one wants their traffic.

Re:difference

[fruey]

SpamAssassin uses keywords, RBL listings, et al.

This is just a lightweight SMTP server which takes over anyone who is SPEWS listed and rejects them. A decent server like Postfix + amavisd & SpamAssassin will already do this with little overhead.

More reinvention of the wheel, I fear.

Re:difference

[bconway]

SpamAssassin is nothing more than an advanced filter. This stops the spam before it gets to you and fills up the offending mail servers disk space with it.

Platform [In]dependence

[GeckoFood]

The author states that it's for OpenBSD. Any clue if he plans to port it to other flavors of Unix, such as Solaris, HP-UX, Linux, IRIX, etc? This sounds like a useful honeypot tool, I would be curious to see how well it works in actual production (translation -- I'd like some stats).

Re:Platform [In]dependence

[evilviper]

Are you not familiar with the concept of open source? Instead of saying "Gimme Gimme Gimme" you could do it yourself, or even contract someone to do it. If you aren't going to contribute, don't start complaining that others should be contributing more.

Re:difference

[Zigg]

Err, SpamAssassin isn't exactly what I'd call "low overhead". While it's pretty good at what it does, it still has potential to slow my 32MB mail server to a crawl unless I tell spamd to process only one message at a time.

And that's only filtering my mail.

Spews is NOT the right way to filter e-mail.

[Sturm]

Spews is EVIL. Plain and simple. They block IPs based soley on the fact your upstream provider hosts or has hosted in the past, someone the SPEWS "admins" (and I use that term losely) believe to be spammers. It is impossible to get off their list and if you are a customer of C&W you probably have IP space being blacklisted by them. Blocking large blocks of class Cs, just because someone happens to share IP space with an alleged spammer is the WRONG way to filter spam.
Please take a look at http://www.antispews.org for more information before using SPEWS.

Re:Spews is NOT the right way to filter e-mail.

[jamie]

"Spews is EVIL... Please take a look at http://www.antispews.org"

Thanks for the link. I'll confirm that Spews is not the way to go. Well, it depends on whether your goal is to block spam for your users, or just to piss people off.

If you're a network admin and you want to block spam for your users, try something else.

If you just want to piss people off, Spews is great. My personal mail server (very kindly hosted for me for free on a friend's network) was put on Spews' blacklist. My server has never in its lifetime sent a single spam, of course. But Spews had found four (count 'em) examples of spammer websites (not spam-sending machines) on the IP blocks owned by the people who my friend bought access from, twice removed. Because of these four claimed spam websites, Spews put FOUR CLASS A's on their list.

That's right -- a quarter-million IP numbers were blocked because they didn't like the policies at four IP numbers.

Wait, did I say four? When I checked up on them, two had already moved to other providers, one I couldn't find, and only one was still there. So my server, and a quarter-million others, were being blocked because the Spews people disagreed with one solitary website. Hosted by a company that I have no relationship with.

It goes without saying that attempts to get my server whitelisted failed.

And I do question the value of their blocking my mail server. Like I said, I was being hosted for free just because I have helpful friends... my moving to another network actually saved them money!

Somehow, I think most net administrators, if they knew that Spews' purpose was political and not technological, would be less likely to use it. There are plenty of other blacklists out there. What are the good ones that don't hijack your networks to apply political pressure?

Re:Spews is NOT the right way to filter e-mail.

[PacketMaster]

Spews randomly blocked a consulting company's netblock I worked for part-time simply because that our block was next to a "known spammer's" block. When they politely asked to be removed and pointed out that according to their own evidence file that their netblock had nothing to with spam, they were met with very hostile responses and told to essentially ditch their teleco provider because they'd never unlist anyone. They admitted that they simply block IPs in a form of "collateral damage" because they feel like it to hurt legitimate businesses so they flee their network provider. Someone mentions C&W addresses, same thing if you're getting service from Qwest. Their website makes them come off as the noble crusaders against spam, but in reality what they do is just mean-spirited, unethical and just plain wrong.

Don't use SPEWS!

See the newsgroup news.admin.net-abuse.email to see just how the spews people treat those who politely ask for erroneous entried to be removed and PROVE they have nothing to do with spammers.

Re:Spews is NOT the right way to filter e-mail.

[MrDingusMcGee]

They block IPs based soley on the fact your upstream provider hosts or has hosted in the past, someone the SPEWS "admins" (and I use that term losely) believe to be spammers.

As a sysadmin for an ISP I can assure you that this is absolutely the case. There is no human contact at Spews, the entire system is automated. Which means that when their system is alerted to a "spammer" within a particular class C, that entire class C is quickly blocked by thousands of misinformed SAs who don't understand that they are in the process going to block legitimate emails that the people within their network have every right to receive.

Blocking large blocks of class Cs, just because someone happens to share IP space with an alleged spammer is the WRONG way to filter spam.

A hosting provider should be responsible for the domains they host. But there is rarely anything a provider can do to pre-emptively stop a spammer. Just recently, my company signed up a new company for Co-Location. Within a week, this company sent out a huge spam mailing. The moment we saw spam complaints come in we called the company and demanded proof that their mailing list consisted solely of opt-in addresses. They had no proof and their contract was immediately terminated for violating our Acceptable Use Policy. However, at this point our entire class C (housing our main mail server for hundreds of websites and ten times that many individual email clients) was listed in SPEWS database. Apparently this company had, in the past, under a different name, been blacklisted as a spammer. We were now added to the list of their hosting providers and could not, despite our best effort, contact a single human at SPEWS to explain our situation. As a result, for over 3 weeks, thousands of mail servers were rejecting our clients' mail as coming from a spam-server.

I ask you, how does that make the internet a better place?

Spam is a waste of bandwidth, of time, and it's insanely annoying, as a sysadmin I realize that as much as anybody (except maybe Alan Ralsky). But SPEWS is a horrible "solution" to the problem. Too many misinformed sysadmins use SPEWS at the expense of those who use their network.

Re:Spews is NOT the right way to filter e-mail.

[Electrum]

Spews put FOUR CLASS A's on their list. That's right -- a quarter-million IP numbers were blocked because they didn't like the policies at four IP numbers.

Perhaps you meant class B's? Four class A's would have been 67 million. I doubt even SPEWS is that stupid. Wait, this is SPEWS we're talking about.

big difference: not just rejecting mail

[agshekeloh]

It doesn't reject messages. It defers them forever, telling the open relay to "try again later."

This tool is a weapon against open relays. The goal is to fill up the open relay's hard drives by deferring the incoming mail, rather than just rejecting the messages.

Yes, you can do this with other blacklists as well, but nobody seems to be actually doing that.

Re:big difference: not just rejecting mail

[dskoll]

My product CanIt can tempfail mail also. However, it can be dangerous, because you tend to get a big increase in SMTP connection attempts. If you can tempfail early (as Theo's scheme does), it's not so bad.

Our stats, however, show that most spam does not come from open relays any more. With the advent of cheap broadband, I'd say a lot of spam comes directly from DSL or cable-modem machines. Some comes from Web servers with broken formail scripts, and some from legitimate non-open relays that are abused by subscribers. Only the minority comes from open relays nowadays.

Re:Good concept - quality of execution pending

[tmark]

the only way we are going to convince them to clean up their act is to waste _their_ disk space, their time, and their network bandwidth more than they waste ours.

To me, this seems exactly the right strategy, although how well it works in practice will be interesting to watch.

To me, this is about as hypocritical a strategy I can imagine. If something is wrong, it's wrong.

It thougt it was spam though

[neurostar]

...doesn't block half the planet?

I thought half the email on the planet was spam though!

:)

Re:SPAM?

[grub]

Spam is annoying, but it isn't that big of a problem that we need Slashdot posts every day about it..

Annoying to the end-user, yes. To an ISP or firm with a large mail server it is more than that. Spam fills disks, uses bandwidth, wastes employees' time, etc etc. This is a super idea.

SPEWS is necessary & effective at hurting spam

[Charles Dodgeson]

Time and again we see case after case of some provider that

1. Let some customers spam
2. ignored abuse complaints
3. did nothing while when that particular spammer's IP was listed.
4. Only took action against a spammer when the SPEWS listing expanded to include non-spamming customers
5. Whinged that SPEWS was unfair and not the right way to do things

Every day SPEWS proves itself necessary and effective at getting otherwise unwilling providers to remove their spammers. Note that SPEWS uses an escalation process. The provider has to ignore complaints for a while to have the IP range expanded to include non-spammers

If you can suggest something that is half as effective at raising the cost for spammers as SPEWS, please suggest it. SPEWS forces providers to decide whether they want to host exclusively spammers or host exclusively non-spammers.

But if your goal is merely to filter spam (making life easier for the spammers) then you are right. SPEWS is not the way to do that.

Re:difference

[grub]

Can anyone explain why you wouldn't just use SpamAssassin?

Once the spam is in your system then your bandwidth, disk space and other resources have already been consumed by the spammer. This prevents the spam from ever coming into your network and put the burden of the load back on the spammer's shoulders.

Damn fine work.

I'm Disappointed

[TerryAtWork]

I remember when I applied for a Mead mailing list and got a nasty letter back saying 'your SPAM has been rejected!' just because I sent it from a Rogers.com address, so I know what it's like to be blacklisted like in SPEWS, and it sucks. That's not the way to do it.

Also, this new spam program retaliates and the law is very nasty about vigilantism and retaliation, perhaps because it threatens their monopoly. I don't want to see a spammer WIN in court, do you?

Also, program like popfile doe a great job of removing spam.

My advice is to forget kicking the spammers ass and just make their work vanish down a black hole like it will WHEN BAYESIAN TECHNIQUES ARE USED AT THE ISP END hint hint...

Re:I'm Disappointed

[Diabolical]

Also, this new spam program retaliates and the law is very nasty about vigilantism and retaliation,

The law has nothing to say over this. I'm at total liberty to block access to my site for whoever i want to block. If i block others in the process then that is their problem solely and not that of the lawmakers. Basicly you're stating that just because i have an email address i am not allowed to decide who may and who may not send me email.

The retaliation you're mentioning is just a message that is being sent back to the spammer who as a result has alot of errormessages in his mailbox, if they used a valid email address that is.

WHEN BAYESIAN TECHNIQUES ARE USED AT THE ISP END hint hint...

Now there's a statement i can live with.. ;-)

Use a Teergrube

[Brett Glass]

What Theo should be doing, instead of sending a 5xx response (which, by the way, won't keep the message in the spammer's queue; a 5xx is a final rejection) is to redirect spammers' connections to a Teergrube (a spam "tarpit"). If enough people do this, the spammer will be slowed down greatly.

Re:SPEWS is necessary & effective at hurting s

[jamie]

"If you can suggest something that is half as effective at raising the cost for spammers as SPEWS, please suggest it. SPEWS forces providers to decide whether they want to host exclusively spammers or host exclusively non-spammers."

First of all, I don't think most network administrators -- or their bosses -- know what they're getting into when they use Spews to police their network. If you are an admin who signs your company up for it, be prepared to have this conversation:

Boss: Hey, can you check to see if there's some kind of network trouble. I haven't gotten a reply email from a client in three days.

You: (after checking) Ah, that mail server is spam-friendly, we reject their mail.

Boss: (confused) They're not a spammer, they're our best client.

You: No, but they buy bandwidth from someone who buys bandwidth from someone who...

Boss: What?

You: We're using SPEWS, which is the most effective tool at stopping spam around the world! It forces providers to decide whether...

Boss: I don't give a damn, you work for me, not people around the world. Your job is to make the email work, not be a do-gooder. You may have cost this company a contract. Now get the damn mail working and tell me how many times you bounced my client's mail so I can decide whether you still have a job.

And -- you think Spews is effective? After being put on their list I had a grand total of one person unable to receive my mail. I have a dozen other people using my server to send and receive mail to hundreds of people, and according to my logs, among all of us, the sum total of people who couldn't get our email was two. That's the most pitiful boycott I've ever seen.

rblsmtpd + spamassassin

[Gothmolly]

Works great for me, thank you DJB! Here's a summary of the spamhouses I've blocked (with a 553 error code) over the past few hours. These never even touch spamassassin.

64.70.22.99-outbound1.lamailer.com
209.236.32.1 57-
216.19.164.127-127.opti9.com
65.126.119.178- formulatedmail.com
64.201.128.3-netblock-64-201-1 28-3.stanfordintl.co m
66.216.111.187-mail213.rm23.com
63.96.237.154-
216.109.73.35-om40.yourmailsoure.com
211.90.191 .61-
204.73.107.103-
209.189.49.102-
209.123.11 1.22-mail.dmx4.com
216.19.163.204-204.sbase30.com
63.70.105.139-ntls1.digitalriver.com
66.197.162 .15-
209.47.251.15-smtp5.rapid-e.net
209.236.57. 176-mtsbp512.email-deliveries.net
202.103.64.43-
66.216.116.78-mail153.myfunsleuth.com
65.107.195 .162-
209.213.210.18-mailer18.labeldaily.com
200 .206.207.206-200-206-207-206.terra.com.br
66.216. 115.56-mail16.justforyou-mail.com
64.119.213.95-p assionup.com
66.216.107.233-mail233.dealdelivery. com

Interesting, but here's an extra twist

[wowbagger]

I won't go into the validitiy of using SPEWS as a blocklist - there are good arguments pro and con there.

But here's a twist to the basic idea:

Given the the email sender is in $BLOCKLIST, have the filter daemon give the 450 response

v... e... r... y... ... s... l... o... w... l... y...

Combine a teergrube with the 450 response to fill up both their mail spool AND their socket connection table.

(For those who don't know, a teergrube (tarbaby) is a mail server that response slowly to a spammer, the better to tie up his connections).

Now, not only will the open relay's mail queue fill, but it will run out of (file descriptors|sockets) and choke on that too!

Re:Good concept - quality of execution pending

[Dunark]

I don't see the hypocrisy. If a neighbor of mine allows people to cross his property so they can dump garbage on my property, where do I get the obligation to accept the garbage? What's wrong with me putting up a fence and letting the garbage pile up on his side?

If someone wishes to run an open relay and be a conduit for spam, why should he be granted immunity from consequences?

Re:SPEWS is necessary & effective at hurting s

[binner1]

At my last job, that is exactly the conversation I had. My boss said: We get too much spam here, do whatever it takes to stop it. I said: Sure, I'll have qmail do some rbl polling before accepting mail. Worked great for about a month...cut roughly 50% of the spam that network received. Then, boss says: Why can't I get email from ebay seller X? I say: Oh he's rbl'd...we don't take mail from there. He says: Ok, turn off the rbl.

After that, I turned on my own bayesian filtering and said F the rest of the network/users.

-Ben

Re:Back Off

[almeida]

I don't see how it's wrong to send it back to the open relay. They are saying, "Here, have this," and you are just replying, "Not right now, thanks." That's perfectly valid use of SMTP codes. It's not like you launch an attack every time you get email from these relays, you're just telling them you don't want it right now. The idea is just to take the pain of SPAM away from the user and give it to the ones responsible (to some extent) for it. The open relays caused it, they should deal with it.

SPEWS Is Not An Open Relays List

[Anonymous Coward]

Between Theo's erroneous statements, implying that SPEWS is a list of open relays, and some of the whiners in here bitching about "don't use SPEWS because they're too aggressive," I thought it would be handy to note a couple of things.

SPEWS is not a list of open mail relays. SPEWS (Spam Prevention Early Warning System) is a list of "spam sources." Some of those spam sources may be open relays. Some of 'em may be open proxies. Some of 'em may be spammers themselves (e.g.: Topica).

Regarding those that have found yourselves SPEWSed, yet are not, themselves, spammers: I'm sorry you've found yourselves in that situation. But, you see, kinder, gentler methods have been tried for years and have not solved the problem. It only continued to grow worse. And whether you like it or not: SPEWS works. I've never, in all the years I've been battling spam, ever seen ISPs boot spammers off their networks like I have since their netblocks started getting SPEWSed. You blame SPEWS for your problems but the truth of the matter is this: you've chosen to use an irresponsible ISP for your connectivity. If your ISP had been responsive to spam complaints, their netspace wouldn't have gotten SPEWSed.

Note: my personal net space was SPEWSed once. For a short while. But my ISP is a good one. They addressed the problem promptly and got their space delisted.

Re:SPEWS Is Not An Open Relays List

[Dunark]

Wrong. Spews maintains multiple listings for various kinds of spam sources and facilitators. See their webpage at http://www.spews.org for more information.

No stooping involved

[LinuxGeek]

This is mainly intended to prevent open ( poorly configured) email servers from being used as relays by spammers. The open server's disk space being gobbled up by causing them to spool the relayed email will certainly get the admins attention. This will shift the problem away from servers that recieve the email and onto the open relay which lets the spammers spam us with no easy way to trace the mail. The problem with tracing the email is that the poorly configured relay server is maintained by someone that usually ignores the emails asking them to close their smtp setup or to please examine their logs and let us know who was using them as a relay.

I think your sympathy is misplaced due to a lack of understanding of what allows the spammers to keep sending us all of those wonderful offers. If they don't have access to open relays, then they either have to keep moving their spamming servers when accounts are terminated or buy bandwidth off the backbones directly from qwest, AT&T, worldcom, etc... Either way, the spammers costs go up.

Do you feel bad for the people you hear about in the news that get charged with maintaining a dwelling for criminal purposes when they leave an empty house to be over run with drug users? Same principle is involved here.

it looks like nobody understands the concept here

[honold]

the point is to punish open relays, not to block spam. the mail has to be retried for days, wasting network bandwidth and space.

if a signifigant number of people were to employ this, open relays would become crushed and filled with their own load.

Antispews is spam; SPEWS is good; others are too.

[Frater 219]

Please take a look at http://www.antispews.org for more information before using SPEWS.

Actually, antispews.org is likely being operated by spammers, as the Osirusoft FAQ suggests. (If nothing else, they are spammers of USENET newsgroups, since they kiboze for references to "SPEWS" and troll in response, much as Serdar Argic once did with "Turkey".) Naturally, spammers are pissed off at SPEWS, because it is simply put the most effective tool presently in the field for denying spammers access to (1) victims, and (2) willing ISPs to host them. Innumerable spammers have been terminated as a result of SPEWS listings.

There is no conceivable informed controversy as to whether or not SPEWS is effective at getting spammers off the Net. Whether or not SPEWS is a good tool for your site to use as a tool for reducing your spam count is quite another question. In my personal experience (as a security and email administrator for my site, which is a research institution) SPEWS is extremely valuable. I read my mail logs and ascertain that SPEWS usage blocks spam, with a remarkably low incidence of false positives.

In the past week, our incoming mail server has blocked 969 messages on account of SPEWS, with zero reports of false positives from our users. (To be honest, we get about one such report a month, and we whitelist the offending IP address. It's usually in China; we have several Chinese researchers.) Our locally maintained blacklist blocks about twice as much spam, and our use of sbl.spamhaus.org blocks about five times as much -- but that is biased by the fact that we consult those lists before SPEWS, and there is a good deal of overlap between them.

I would not recommend that ISPs who offer email service to their users use SPEWS by default, though it would be a valuable optional service. The DNSBLs I would recommend everyone use are:

• sbl.spamhaus.org, which lists only netblocks occupied by known repeat spam offenders
• relays.ordb.org, which lists only open mail relays; and
• proxies.relays.monkeys.com, which lists only open proxies.

These are all low-to-no-false-positives lists which I feel comfortable recommending to every ISP regardless of its stance on SPEWS.

Website is hosted by a a spamer Hurricane Electric

[dananderson]

I think what Theo did was great and I can't wait until it gets out into the mainstream.

However, I find it funny (hypercritical) that the weblog is hosted by a ISP that tolerates spam, Hurricane Electric. Specifically:

• Hurricane Electric's customers include major spammers, such as Bulk ISP Corp.
• Hurricane Electric's customers often show up in my spam trap, usually harvesting email addresses.
• Hurricane Electric's mail servers have open relays, which allows spammers to spam using their servers. Yes, I know it makes it easier for HE's customers to read email anywhere, but it allows spammers to flood others with spam also.

I'm sure others can add more, but I have other things to do . . .

Re:SpamAssassin vs Theo's Package

[realdpk]

don't believe what you read on SPEWS. some of their records are over *6 months* out of date. probably longer. worst. bl. ever.

Re:Spews is worse than the spammers

[Tackhead]

> Legitimate users like us can't keep changing IP addresses because SPEWS is too aggressive and has no organized process. If you want to use a spam advisory system, use MAPS RBL [mail-abuse.org].
>
> Spews is worse than the spammers, because at least I can ignore the spammers.

If you want an effective spam advisory system that actually lists spamhausen, use SPEWS.

SPEWS is better than MAPS, because the spammers discovered they could ignore MAPS.