Slashdot Mirror
Spam Blocking Engine for OpenBSD
mkeke writes "In a post over at OpenBSD Journal, Theo states that he has written a spam blocker that works with pf and Spews.
It looks darn cool :)"
← Back to Stories (view on slashdot.org)
← Back to Stories (view on slashdot.org)
SPEWS is lame. They do not even follow their own policies (specifically, that they will remove listings after spammers are terminated. 6+ months now, still listed. And they say you can't contact them (and posting to NANAE is _not_ an option).
SPEWS has made themselves completely irrelevant in my eyes.
Just want to pipe in and say you
are right on the money. We acquired
our IP block from a telecom only
to find that the IP range was listed
in SPEWS. We have had the IP range
for 3 years now, and it is still
listed in SPEWS.
It is unfortunate though, that many
administrators just sign up their
mail servers to all the blacklists
they can find without considering
the quality of the lists and how they
are maintained.
At risk of introducing facts to this debate, would you mind giving the listed IP/SPEWS number for the listing you're talking about?
Why should SPEWS remove a listing immediately upon removal of the spammers? The "policy" you seem to be thinking of is:
"In time" does not mean immediately. How many months did you harbor spammers? More to the point, what's the relevant IP address or range? Without specifics, your complaint rings hollow.
Of course. The NANAE regulars have seen every flavor of spammer lying and evasion. You're looking for a naive audience that might give you some sympathy.
SPEWS wasn't made to please spam hosters. It was made to keep your spammy network away from my inbox.
Again, if you think you have a legitimate complaint, post the IP range in question.
i'm not going to post the IP range.
"in time" should not mean >6 months. we were in the list for maybe 1 month before booting them. the spammer is *long gone*. IN FACT, the spammer moved before we kicked him off our network, and SPEWS recognizes this, and yet still lists us.
the reason i won't post to NANAE or here w/ the IP range is because it's pointless. SPEWS shows very clearly just how silly the anti-spam movement has become. dealing with anti-spammers is like talking to a brick wall.
I've seen plenty of claims like yours posted on NANAE. Most of the time the claimant is wrong - there is still an ongoing spam problem from the listed IPs. So without knowing more about your particular situation, the balance of probability is that you are incorrect, and there is good reason to list your IPs. Often enough, however, the claimant is right and SPEWS neglected to unlist them. In those cases, SPEWS reacts quickly, usually moving to a level 2 listing.
Here's my point: I've yet to see a single case where the IP was listed in error and SPEWS didn't immediately fix the problem.
Also, the attitude of anti-spammers on NANAE doesn't really matter. No matter how much venom they hurl at you, if they can't produce objective reasons to keep you listed SPEWS will delist. My observation is that SPEWS is not looking for a "vote" from the community - they are looking only for evidence of spam support. I've seen IP's delisted while the NANAE regulars are still out for blood.
I assume he means a 450 reply, not a 550? 550 won't make the message stay in the queue, 450 will.
Why even bother with Spews? Why not Spamcop, who doesn't block half the planet?
Dude, where's my packet?
Why drive a Ford when you've already got a Chevy available? It's a matter of choice, preference, features, etc.
-- "Other than that, how was the play Mrs. Lincoln?"
This is just a lightweight SMTP server which takes over anyone who is SPEWS listed and rejects them. A decent server like Postfix + amavisd & SpamAssassin will already do this with little overhead.
More reinvention of the wheel, I fear.
Conversion Rate Optimisation French / English consultant
SpamAssassin is nothing more than an advanced filter. This stops the spam before it gets to you and fills up the offending mail servers disk space with it.
Interested in open source engine management for your Subaru?
The author states that it's for OpenBSD. Any clue if he plans to port it to other flavors of Unix, such as Solaris, HP-UX, Linux, IRIX, etc? This sounds like a useful honeypot tool, I would be curious to see how well it works in actual production (translation -- I'd like some stats).
Be excellent to each other. And... PARTY ON, DUDES!
To me, this seems exactly the right strategy, although how well it works in practice will be interesting to watch.
Helium balloons want to be free.
Err, SpamAssassin isn't exactly what I'd call "low overhead". While it's pretty good at what it does, it still has potential to slow my 32MB mail server to a crawl unless I tell spamd to process only one message at a time.
And that's only filtering my mail.
SpamAssassin has to parse the whole message body, so you've already accepted it. I didn't mean to make it look like it was super low overhead with SpamAssassin, I meant that it's low overhead without it, and that with SpamAssassin you can do a lot more.
I oughtta Preview before Send more often. I type too fast and it gets confused
Conversion Rate Optimisation French / English consultant
Spews is EVIL. Plain and simple. They block IPs based soley on the fact your upstream provider hosts or has hosted in the past, someone the SPEWS "admins" (and I use that term losely) believe to be spammers. It is impossible to get off their list and if you are a customer of C&W you probably have IP space being blacklisted by them. Blocking large blocks of class Cs, just because someone happens to share IP space with an alleged spammer is the WRONG way to filter spam.
Please take a look at http://www.antispews.org for more information before using SPEWS.
It doesn't reject messages. It defers them forever, telling the open relay to "try again later."
This tool is a weapon against open relays. The goal is to fill up the open relay's hard drives by deferring the incoming mail, rather than just rejecting the messages.
Yes, you can do this with other blacklists as well, but nobody seems to be actually doing that.
You can setup SpamAssassin in a site-wide configuration. You could also put it together with MimeDefang and integrate it with Sendmail.
I thought half the email on the planet was spam though!
Spam is annoying, but it isn't that big of a problem that we need Slashdot posts every day about it..
Annoying to the end-user, yes. To an ISP or firm with a large mail server it is more than that. Spam fills disks, uses bandwidth, wastes employees' time, etc etc. This is a super idea.
Trolling is a art,
- Let some customers spam
- ignored abuse complaints
- did nothing while when that particular spammer's IP was listed.
- Only took action against a spammer when
the SPEWS listing expanded to include non-spamming customers
- Whinged that SPEWS was unfair and not the right
way to do things
Every day SPEWS proves itself necessary and effective at getting otherwise unwilling providers to remove their spammers. Note that SPEWS uses an escalation process. The provider has to ignore complaints for a while to have the IP range expanded to include non-spammersIf you can suggest something that is half as effective at raising the cost for spammers as SPEWS, please suggest it. SPEWS forces providers to decide whether they want to host exclusively spammers or host exclusively non-spammers.
But if your goal is merely to filter spam (making life easier for the spammers) then you are right. SPEWS is not the way to do that.
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
Can anyone explain why you wouldn't just use SpamAssassin?
Once the spam is in your system then your bandwidth, disk space and other resources have already been consumed by the spammer. This prevents the spam from ever coming into your network and put the burden of the load back on the spammer's shoulders.
Damn fine work.
Trolling is a art,
I remember when I applied for a Mead mailing list and got a nasty letter back saying 'your SPAM has been rejected!' just because I sent it from a Rogers.com address, so I know what it's like to be blacklisted like in SPEWS, and it sucks. That's not the way to do it.
Also, this new spam program retaliates and the law is very nasty about vigilantism and retaliation, perhaps because it threatens their monopoly. I don't want to see a spammer WIN in court, do you?
Also, program like popfile doe a great job of removing spam.
My advice is to forget kicking the spammers ass and just make their work vanish down a black hole like it will WHEN BAYESIAN TECHNIQUES ARE USED AT THE ISP END hint hint...
It's Christmas everyday with BitTorrent.
What Theo should be doing, instead of sending a 5xx response (which, by the way, won't keep the message in the spammer's queue; a 5xx is a final rejection) is to redirect spammers' connections to a Teergrube (a spam "tarpit"). If enough people do this, the spammer will be slowed down greatly.
uh...I can't believe this is modded as "Interesting." The mail server sending the spam will get the 450 error and save the message to try again later, not the mail server of the domain faked in the headers.
jeez, learn the basics of how email works. If all I had to do to DoS your mail server was send it tons of messages and 450 errors don't you think this would be a HUGE problem?
First of all, I don't think most network administrators -- or their bosses -- know what they're getting into when they use Spews to police their network. If you are an admin who signs your company up for it, be prepared to have this conversation:
And -- you think Spews is effective? After being put on their list I had a grand total of one person unable to receive my mail. I have a dozen other people using my server to send and receive mail to hundreds of people, and according to my logs, among all of us, the sum total of people who couldn't get our email was two. That's the most pitiful boycott I've ever seen.
Works great for me, thank you DJB! Here's a summary of the spamhouses I've blocked (with a 553 error code) over the past few hours. These never even touch spamassassin.
1 57-- formulatedmail.com1 28-3.stanfordintl.co m- 1 .61-1 1.22-mail.dmx4.comm 2 .15-. 176-mtsbp512.email-deliveries.net 5 .162-0 .206.207.206-200-206-207-206.terra.com.br. 115.56-mail16.justforyou-mail.comp assionup.com. com
64.70.22.99-outbound1.lamailer.com
209.236.32.
216.19.164.127-127.opti9.com
65.126.119.178
64.201.128.3-netblock-64-201-
66.216.111.187-mail213.rm23.com
63.96.237.154
216.109.73.35-om40.yourmailsoure.com
211.90.19
204.73.107.103-
209.189.49.102-
209.123.1
216.19.163.204-204.sbase30.co
63.70.105.139-ntls1.digitalriver.com
66.197.16
209.47.251.15-smtp5.rapid-e.net
209.236.57
202.103.64.43-
66.216.116.78-mail153.myfunsleuth.com
65.107.19
209.213.210.18-mailer18.labeldaily.com
20
66.216
64.119.213.95-
66.216.107.233-mail233.dealdelivery
I want to delete my account but Slashdot doesn't allow it.
550 is a temporary denial. 553 is a permanent failure (rblsmtpd switch is "-b"). spammers usually just move on to another host if they keep getting 553's. 550s tell them to keep on trying, which is bad on the receiving mail server if you're getting a pretty heavy load.
/23's and /22's of a network that i run, just because abuse@domain.com did not respond fast enough. The only way to get off of that list is to post to a newsgroup, and just hope they read your posting and take off the ban. That means it is a total manual process on their side to remove you.
on a side note, i would advise against using the spews.org list. it is almost impossible to get off of that list. they recently decided to put a few
in my eyes, using something like sbl.spamhaus.org or/and relays.ordb.org is a much better solution. If you are going to go the DNSBL route, and you should, i would advise you figure out how to run your own DNSBL so you can quickly add and remove hosts that are mailbombing your server.
Why read the article when I can just make up a snap judgement?
I use something very similar, MessageWall(.org). This is a smtp proxy with excellent filtering. So no need for something new.
I won't go into the validitiy of using SPEWS as a blocklist - there are good arguments pro and con there.
... s... l... o... w... l... y...
But here's a twist to the basic idea:
Given the the email sender is in $BLOCKLIST, have the filter daemon give the 450 response
v... e... r... y...
Combine a teergrube with the 450 response to fill up both their mail spool AND their socket connection table.
(For those who don't know, a teergrube (tarbaby) is a mail server that response slowly to a spammer, the better to tie up his connections).
Now, not only will the open relay's mail queue fill, but it will run out of (file descriptors|sockets) and choke on that too!
www.eFax.com are spammers
At my last job, that is exactly the conversation I had. My boss said: We get too much spam here, do whatever it takes to stop it. I said: Sure, I'll have qmail do some rbl polling before accepting mail. Worked great for about a month...cut roughly 50% of the spam that network received. Then, boss says: Why can't I get email from ebay seller X? I say: Oh he's rbl'd...we don't take mail from there. He says: Ok, turn off the rbl.
After that, I turned on my own bayesian filtering and said F the rest of the network/users.
-Ben
If message has a '!' in the title, delete.
I don't see how it's wrong to send it back to the open relay. They are saying, "Here, have this," and you are just replying, "Not right now, thanks." That's perfectly valid use of SMTP codes. It's not like you launch an attack every time you get email from these relays, you're just telling them you don't want it right now. The idea is just to take the pain of SPAM away from the user and give it to the ones responsible (to some extent) for it. The open relays caused it, they should deal with it.
SPEWS is not a list of open mail relays. SPEWS (Spam Prevention Early Warning System) is a list of "spam sources." Some of those spam sources may be open relays. Some of 'em may be open proxies. Some of 'em may be spammers themselves (e.g.: Topica).
Regarding those that have found yourselves SPEWSed, yet are not, themselves, spammers: I'm sorry you've found yourselves in that situation. But, you see, kinder, gentler methods have been tried for years and have not solved the problem. It only continued to grow worse. And whether you like it or not: SPEWS works. I've never, in all the years I've been battling spam, ever seen ISPs boot spammers off their networks like I have since their netblocks started getting SPEWSed. You blame SPEWS for your problems but the truth of the matter is this: you've chosen to use an irresponsible ISP for your connectivity. If your ISP had been responsive to spam complaints, their netspace wouldn't have gotten SPEWSed.
Note: my personal net space was SPEWSed once. For a short while. But my ISP is a good one. They addressed the problem promptly and got their space delisted.
While some of the spam detecting algorithim's are cool and innovative they are still prone to circumvention. The best spam blocker I have ever seen used whitelist blocking. If I did not send you a message you cannot send me one unless you go to a web page and entered the reason that I should see your message.
...
This blocked 100% of the spam period
Got Code?
This is mainly intended to prevent open ( poorly configured) email servers from being used as relays by spammers. The open server's disk space being gobbled up by causing them to spool the relayed email will certainly get the admins attention. This will shift the problem away from servers that recieve the email and onto the open relay which lets the spammers spam us with no easy way to trace the mail. The problem with tracing the email is that the poorly configured relay server is maintained by someone that usually ignores the emails asking them to close their smtp setup or to please examine their logs and let us know who was using them as a relay.
I think your sympathy is misplaced due to a lack of understanding of what allows the spammers to keep sending us all of those wonderful offers. If they don't have access to open relays, then they either have to keep moving their spamming servers when accounts are terminated or buy bandwidth off the backbones directly from qwest, AT&T, worldcom, etc... Either way, the spammers costs go up.
Do you feel bad for the people you hear about in the news that get charged with maintaining a dwelling for criminal purposes when they leave an empty house to be over run with drug users? Same principle is involved here.
Kindness is the language which the deaf can hear and the blind can see. - Mark Twain
What tools in Linux would one need to do the following:
Setup a pop3 server / smtp server so that email can be sent and received.
Filter spam / easily add filters to this pop3 / smtp server on the same box.
Also be able to check OTHER accounts on OTHER pop3 servers, download them, and filter out the same spam / things marked as spam.
Noobie proof is a good thing too.
PS - If BSD does it better then linux, post those tools as well. Maybe make it a chalange to see which OS can do said request better. Could win32 win (heh) ?
The ultimate network admin tool needs HELP!
They are not innocent. They are open relays, and deserve the punishment.
You are absolutely right. Although I advocate using things like SPEWS, you must make it clear that it will block mail from legitimate users. You either have to persuade people that this is right (as I believe) or not do it that way.
See this policy statement as an example of using such a policy, while making it clear that it will block mail from legitimate users.
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
1. I am customer of a small ISP. I don't send spam, and my ISP actively fights spam. Nevertheless, my ISP is on SPEWS - bad luck, wrong netblock.
2. I have zero incentive to change my ISP, and thus my ISP has zero incentive to put pressure on their upstream network operator.
3. Why ? Because I am blocked by bad luck, nothing else. I could change the ISP, but any new ISP might have the same bad luck. Changing providers will cost money, and will not secure me from future problems of that sort.
In short: the overzealous blocking by SPEWS removes any incentive to change ISP or exert any pressure on upstream providers. If it's just bad luck to be blocked, it may happen anywhere and anytime, and changing providers does not make any sense.
At my place of employment we have been filtering all incoming e-mail for ourselves and our small ISP through SPEWS and various other lists. Just now I checked and found that since 4:00am when the logs switched over we've blocked just over 2000 messages. About 1600 of them were because of SPEWS. This is a system with 6000 users and we've only had two or three complaints since we started filtering a few years ago.
That seems pretty effective to me.
Oh, and the boss loves it. As soon as we implemented the filters his spam load saw a *huge* decrease. He has even used the filters as a way to persuade a few of our more foolish clients to fix their open relays.
the point is to punish open relays, not to block spam. the mail has to be retried for days, wasting network bandwidth and space.
if a signifigant number of people were to employ this, open relays would become crushed and filled with their own load.
Actually, antispews.org is likely being operated by spammers, as the Osirusoft FAQ suggests. (If nothing else, they are spammers of USENET newsgroups, since they kiboze for references to "SPEWS" and troll in response, much as Serdar Argic once did with "Turkey".) Naturally, spammers are pissed off at SPEWS, because it is simply put the most effective tool presently in the field for denying spammers access to (1) victims, and (2) willing ISPs to host them. Innumerable spammers have been terminated as a result of SPEWS listings.
There is no conceivable informed controversy as to whether or not SPEWS is effective at getting spammers off the Net. Whether or not SPEWS is a good tool for your site to use as a tool for reducing your spam count is quite another question. In my personal experience (as a security and email administrator for my site, which is a research institution) SPEWS is extremely valuable. I read my mail logs and ascertain that SPEWS usage blocks spam, with a remarkably low incidence of false positives.
In the past week, our incoming mail server has blocked 969 messages on account of SPEWS, with zero reports of false positives from our users. (To be honest, we get about one such report a month, and we whitelist the offending IP address. It's usually in China; we have several Chinese researchers.) Our locally maintained blacklist blocks about twice as much spam, and our use of sbl.spamhaus.org blocks about five times as much -- but that is biased by the fact that we consult those lists before SPEWS, and there is a good deal of overlap between them.
I would not recommend that ISPs who offer email service to their users use SPEWS by default, though it would be a valuable optional service. The DNSBLs I would recommend everyone use are:
These are all low-to-no-false-positives lists which I feel comfortable recommending to every ISP regardless of its stance on SPEWS.
Actually, it seems spammers use the same phrases in advertising the same stuff....just by filtering subject lines alone by regular expressions I'm rejecting about 5 spams a day per account, and the rest of the spams are rejected by bouncing e-mails based on source domain of bulk e-mailers (like host4bulk.com, e-mailpromo.net, etc.) I'm now getting 2 spams a week to the 6 accounts in my domain. I may next go to filtering the body of the mail, since there are expressions that keep popping up that none of my friends would ever use
Yes, the person is at fault for not knowing what is going on with an open relay. That is the job of an admin of a system exposed to the internet. This method is only preventing them from handing the email to you. They already accepted the email, that dosen't obligate me to take it from them if I know it is spam, now does it?
They are the ones that are allowing their resources to be misused, I just wouldn't them to pass along the misuse. Are you just dense or are you a spammer trying to defend the undefendable position?
Kindness is the language which the deaf can hear and the blind can see. - Mark Twain
However, I find it funny (hypercritical) that the weblog is hosted by a ISP that tolerates spam, Hurricane Electric. Specifically:
- Hurricane Electric's customers include major spammers, such as Bulk ISP Corp.
- Hurricane Electric's customers often show up in my spam trap, usually harvesting email addresses.
- Hurricane Electric's mail servers have open relays, which allows spammers to spam using their servers. Yes, I know it makes it easier for HE's customers to read email anywhere, but it allows spammers to flood others with spam also.
I'm sure others can add more, but I have other things to do . . .Having 550 messages sent based on a bayesian filter such as bogofilter is the best/most adaptive way to handle the problem. Open relay lists have a greater statistical probability of blocking legit email. The challenge this represents is that, unlike with Spews, you have to have clients which convey back to the server which emails get marked as spam.
http://tinyurl.com/4ny52
Partial listing:
1, 65.165.237.126, HUFFNAL / underage-girls.net
1, 65.165.238.144, HUFFNAL / home-lolita.net
1, 65.165.235.230, HUFFNAL / mail.webspace4all.net
0, 65.165.239.144, HUFFNAL / dealsonpc.com (listed)
1, 65.165.235.205, HUFFNAL / trust-bill.com
1, 65.165.234.1, HUFFNAL / Spammers Perez/Walls / mortgageleads.tv
1, 65.165.232.0 - 65.165.239.255, HUFFNAL / Todd Spears/Perez/Walls (Sprint)
Looks like a pretty scummy net-neighborhood. If their ISP doesn't want to clean it up, I don't think I'd want any email from them either.
One line blog. I hear that they're called Twitters now.
What illegal act? You're just not accepting the email that they're trying to dump on you.
One line blog. I hear that they're called Twitters now.
don't believe what you read on SPEWS. some of their records are over *6 months* out of date. probably longer. worst. bl. ever.
Filtering out any email that has "This is not spam" would be a good start. :^)
One line blog. I hear that they're called Twitters now.
Spews is exactly the same.
- They disturb legitimate users: I run a business hosting an email customer support application (Neotonic.com). It is very important for us to get email support replies thorough to customers. Numerous times our IP addresses have ended up on the Spews blocklist because of some unsolicited mail sender in the same 256 address subnet. At most colocation facilities, ten or more companies share the same subnet, and it is not easy to change your IP addresses.
- There is no way to track them down: Organizations like MAPS are judicious about how they block IP addresses. They do NOT block entire subnets unless there is cause, and they have an organized appeals process to take care of their oversights. Spews has no such facilities. In fact, the only centralized item in spews is the spews.org website.
- You can't get them to stop: They block entire ISPs, and their FAQ says that I'm a victim of "rare inadvertant blocking". The trouble is, we followed their advice, we moved to a new colocation, with an entirely new bandwidth provider, and our new IPs are also spews blocked. There is no organization to appeal to, there is no way to get this fixed.
Legitimate users like us can't keep changing IP addresses because SPEWS is too aggressive and has no organized process. If you want to use a spam advisory system, use MAPS RBL.Spews is worse than the spammers, because at least I can ignore the spammers.
indeed I do: /^Subject: .*this is (no|not) spam/ REJECT 553 that WAS spam
That won't work. A 550 error has to be given before the body of the email is sent. A filter can drop the email into /dev/null once the body has been received, but you've already accepted the email from the sender.
One line blog. I hear that they're called Twitters now.
here Works for me(TM).
I want to delete my account but Slashdot doesn't allow it.
This is about as illegal as you not answering your phone when it rings. More correctly, it is like you listen to the first 3 seconds and hear a recorded 'offer' and hang up without listening to the entire telemarketing speach. Get a freaking clue.
Kindness is the language which the deaf can hear and the blind can see. - Mark Twain
IF you people would read closer, and THINK, much of the criminal activities i was talking about were an *analogy*.. I was trying to relate this to something that was concrete and not subject to interpretation ( i.e., the house burning ) as Spam is.
But that said, the last i heard, unauthorized use of computer resources IS a crime, here in the states at least.. And as far as I'm concerned what Theo is proposing constitutes unauthorized use of the so called 'offenders' systems, therefore criminal. ( this is NOT a slam on Theo btw, as i do have respect for the other things he has done )
But the main point i was trying to make was obviously lost many posts ago.. so i give up. Go ahead and justify your activities however you feel you need too.
---- Booth was a patriot ----
Please give an example of a SPEWS listing that was created because the admins didn't like someone personally.
Of course, there are plenty of people in news.admin.net-abuse.email who add to their private blocklists anyone who threatens to sue people over a SPEWS listing, but that's another matter entirely...
Real Daleks don't climb stairs - they level the building.
try spambayes. works great.
Check out this thread. Apparently realdpk harbored spammers for about a month, and then SPEWS took an additional 6 months to delist him. I'll refrain from repeating what I wrote in that thread.
I use SpamAssassin on our Qmail server using Qmail-Scanner. Works great. Out of the box it catches 95% of my spam.
All I need to do is set up a few outlook rules to parse the headers it places. Everyone else in the office seems to enjoy its results as well.
-- DrZaius - Minister of Sciences and Protector of the Faith
Seems that one day I checked and our ORDB filter was letting more and more stuff through over time. Traced it down to the fact that ORDB only blocks open relays not open proxies.
Now I read about Theo's gloating over how his s/w is going to send 550's back. Well guess what? You can't 550 an open proxy.