O'Reilly Article on Spam Defense

Dru writes "Here's an article talking about the details of implementing a network level spam defense with Qmail. It also talks a little about a new site called Trustic which uses a trust system (like Advogato) for nominating spammer/hostile IP's."

Comments

Hurrah for blacklists

[Anonymous Coward]

now all we need to ask is how long till this "community" service that they provide will take before they start charging $ for querying it just like every other blacklist, making blocking spam a privilidge for the rich (i believe MAPS is over a 1000$ a year)

Distrustful of Network Level Censorship

[werdna]

No thanks.

Your spam may be my correspondence -- I may want to get mail from those whose conduct you find abhorrent. Today, a network may responsibly be censoring only unwanted and unsolicited commercial e-mail. Next week, the powers-that-be-in-the-networks start censoring geek news.

To protect our liberties, spam control should be decentralized -- as close to the last mile as possible. Yes, of course, this means that the supposed great harm of spam -- huge volume transmissions through the network -- will not be interdicted closer to the source. In my view, an effective end-point spam model is as likely to reduce volume as a network centered model: the idea is to reduce the INCENTIVE to spam -- that will reduce the volume.

Centralized technical measures simply invite the spam wars to continue, provide centralized points of failure, will not diminish spam, and will assure that powers-that-be have ample new abilities to censor speech.

Re:Distrustful of Network Level Censorship

[kaisyain]

To protect our liberties, spam control should be decentralized -- as close to the last mile as possible.

It is. I'm the one deciding whether or not to use this service.

Re:Distrustful of Network Level Censorship

[RT Alec]

Spam control with RBLs is, in fact, decentralized. There are many RBLs to choose from, and any that are too severe will not be used for long if they generate too many false positives. As a system admin, I have my choice. I use 4 RBLs right now:

• spamhaus.relays.osirusoft.com
  (this is a mirror of the Spamhaus Block List) Well known spam operations, and is checked hourly.
• dialups.relays.osiruSoft.com
  (details at OsiruSoft) This list is of DHCP IP addresses of home users (DSL, cable, dial up).
• dnsbl.njabl.org
  (extensive details of what's on this list)
• rbl.restongeek.com
  I maintain this one myself for anything I want all my servers, primary and backup MX, to block

And there are many more to choose from. I am very happy with my results, it is a pleasure to see the reports of the mail that is blocked (see my /. journal for a sample report). If I start to think maybe one of these lists is a little too severe, or someone lets me know that there are problems with one or more of the lists, I will delete it and pick another. Or maybe not. It is my choice, I want to keep down the spam on my system, for my sake as well as my clients'.

Re:Distrustful of Network Level Censorship

[gfody]

problem is too many of you are deciding TO use it. AOL, Hotmail, MSN to name a few.. the 'want' to filter spam at the server level hurts legit email marketers, inconveniences recipients of legit email marketers, and to the parent's point - creates a target for spammers.

server side email filtering is BAD, BAD, BAD!

what if the US Post Office started throwing out your clearing house sweepstakes and credit card applications before you ever got them? problem is theres two kinds of people in the world.. those that say alright no more junk mail, and those that ask, how do you do that without getting a false positive once in a while?

Re:Distrustful of Network Level Censorship

[i.r.id10t]

So add something like the "evil bit" to a mail header, server side. Then if the end user wants, they can filter it out or not, based on that extra header info.

Re:Distrustful of Network Level Censorship

[stewby18]

You're describing a naive, trust-or-don't approach to the "evil bit" suggested, which is stupid. All a system like the bulk marker would do is add more information to use to improve an existing filtering system, with baysian analysis, whitelisting, etc.

Clearly, there are many people willing to risk false positives to filter out the crap, so why shouldn't a system which helps them at no risk to those who don't filter be implemented?

As someone pointed out... if this is voluntary, why should anyone upset about the idea of others choosing to filter their own mail?

Re:Distrustful of Network Level Censorship

[John Hasler]

> if your going to get a false positive why filter?

My spam folder gets several hundred messages each day. It is _impossible_ for me to read every one of them to determine if it is really spam. I glance over the subject lines and read the occasional borderline one, but I _guarantee_ you that I am already getting false positives. If I dropped spamassassin and allowed the spam into my other folders I would get even more false positives as I impatiently deleted every other message as obvious spam.

Re:Distrustful of Network Level Censorship

[stewby18]

spam is a real problem and filtering is not the solution.

And what is a real solution? If you say legislation, I'll just laugh.

if your[sic] going to get a false positive why filter?

What do you recommend for people who's time is too costly to read everything at the insanely high noise/signal ration? Stop using email? You talk about false positives like they happen all the time... do you have any idea how low the false positive rate is for a good filtering system? You might as well say that no-one should use mail, because your letter might get lost en-route... that's probably a hell of a lot more likely than any normal user hitting real problems with a good filter.

Unless you routinely send emails with subjects like "GET VIAGRA CHEAP NOW!!!" to 1,000 of your closest friends, you probably have nothing to worry about. But even if you do: it's always the recipient's choice whether or not to accept communications. Do you tell all of your friends to read all spam snail-mail, on the off-chance you send them a letter that they might mistake for a credit-card offer? Do you tell them to listen to every telemarketer pitch, in case it's you calling but you are slow to notice that they picked up, and call them by their last name? If so, do they listen? Why should spam be any different?

Re:Distrustful of Network Level Censorship

[waferbuster]

Where do I sign up to have the post office throw out the clearing house sweepstakes and credit card applications before I get them? Along with all the other stuff sent out to >1000 people?

As far as I care, anything sent to "Resident" can go straight into the trash can.

Re:Just like always...

[bajo77]

What the hell does IPv6 have to do with spam?
Well it makes it much harder to scan for servers that are vulnerable, either for hijacking or open smtp services.

Here's my question.

[fleppir]

Any spam measure taken at a server level could induce false positives.

I manage paid-for e-mail e-zines which I mail using PHP and sendmail (read:forged headers until I'm big enough to run my own server).

Wouldn't most server-layer anti-spam measures catch my very suspicious HTML e-zines, even if paid for?

Re:Sounds neat, but PGP'ed network sounds better.

[Mr. Sketch]

If Company X is being real lax (eg: promoting spam), write a revoke key and put it on a few OTHER machines. Thien it'll propigate throught the mail-net to disallow all connections from that MAIL server.

Just curious, how is this different from a blacklist? It sounds like the same concept, just different technology.

Re:Just junk SMTP?

[Anonymous Coward]

SMTP works for what it does. Why should I change ? I alrady can't send mail to AOL just because I have a residential cable modem account, what makes you think that not being able to send mail to hotmail or yahoo will slow me down ?

Spam doesn't come from my computer. Spam comes TO my computer. And everyone wants to change MY computer to fix THEIR spam. Whatever the details of what they do to my computer, it's not going to stop their spam because their spam comes from elsewhere.

Re:Sounds neat, but PGP'ed network sounds better.

[kaisyain]

Who gets to write a revoke key?

Re:Just junk SMTP? Not Possible

[msgmonkey]

Why not? Is n't there a time where we envisage the whole 'net will be IPv6? And thats every machine, not just servers. Eventually IPv4 will die so I dont see how SMTP deing would be a big deal.

Re:Sounds neat, but PGP'ed network sounds better.

[arth1]

Having to generate and spread keys and key revocations non-stop sounds like a very high maintenance system.

Well, at least that would give some techies back their jobs, although I'm not too sure they would like their new job...

Regards,
--
*Art

Qmail is NOT FREE

[SuperBanana]

qmail is completely free and folks that claim it isn't are just trolls.

Qmail is NOT FREE. Last I looked it was distributed without a license; now apparently it has a license, but one with oddball restrictions. If you don't believe me, do a google search with the keywords "qmail debian legal" and spend 30 minutes or so going through the various discussions.

Re:Qmail is NOT FREE

[supersudssoaker]

qmail is free, but the license is not GPL or on the list of licenses approved by the outfit that approves free licenses (OSI?).

You can download it without monetary exchange, install it where you like, modify it, etc.

You cannot modify the source and distribute it. You can distribute the unmodified source tarball with patches that modify it, like an SRPM. You can distribute binary versions as long as the files are the same as would be created by installing from an unmodified source tarball.

Re:RFC violations

[supersudssoaker]

pipelining, for one

I googled and followed a thread, don't know if it's the one you are referring to, where Matti Aarnio (Zmailer author) says

Arnt Gulbrandsen wrote:
> Uhm. If so, that would necessitate speedy reconfiguration of my
> boxes... so I tested it too, and qmail appears to handle pipelining
> okay. I verified with tcpdump that the dozen-odd RCPT TO commands in
Yes, I agree. Without knowledge of the qmail source, I have no

RFC-2821 for another. RFC 2821 and RFC 1123 for two more

qmail predates RFC2821, but there is a patch to bring it up-to-date if an adminstrator so desires. What part of RFC1123 does it not comply with?

The difference is that while Postfix CAN reject based on HELO etc...qmail seems to do so by DEFAULT.

qmail does not reject based on HELO.

Re:IP banning

[Tehrasha]

I find it hard to believe that there was no indication that they were being blocked. In my experience, most ISPs that use blacklists are more than happy to send bounces proclaiming quite clearly in no uncertain terms why the mail is being blocked. To simply drop the message in /dev/null without a bounce would do nothing to stop the flow of spam, as all email would appear to have gone through without trouble. ie: it was received, therefore keep spamming.

Spammers and the ISPs who facilitate them need to be held accountable. Blocking spam on the receiving end via filters is always going to be a losing battle. Blocking at the server by IP is the next best solution, but far from ideal. Making spamming difficult and/or expensive at its source is the only real way to stem the pink tide.

Until ISPs begin to enforce the AUPs they claim to operate under will there be any real change. Even if that means having to be forced to do so.

IP banning is bad

[Animats]

Unless you have some way to identify dynamically assigned IP addresses, IP banning hits innocent parties too often. Every time Joe Sixpack, running Windows XP Home Edition on a DSL line, gets a virus that spams, the next few people to get a lease on that IP address have mail blocked.

There's got to be a better way.

Re:IP banning is bad

[William Tanksley]

That specific example won't normally happen -- you have to repeatedly be the source of spam and do nothing about it to get on most RBLs. When you do get on it, it's more likely to be your entire ISP than just a single reassigned IP (because the ISP was a spamhaus).

HOWEVER, I dislike RBLs for the same reason you do, and I like Bayesian filtering because it prevents that problem. The problem is that the better filtering is at getting spam without killing valid use, the slower it gets. Bayesian filtering is relatively slow.

So... I've been working on defining a multi-layer client/server antispam solution. There are multiple layers of defence:

1. The blocklist: people who have been abusing the greylist system recently. Deny all communications from these IPs, but take them off the blocklist if their entries get stale (i.e. they haven't been abusing for a while).
2. The greylist: people who might be risky. This includes people with IPs on RBLs, people who sent something that a user tagged as spam, etc. Anything sent from these IPs goes into the greylist system Slashdot looked at earlier.
3. Filtering -- probably DSPAM (a very nice server-side Bayesian filter).

This is just a summary; I'm leaving out a lot of detail, like how you tell when to put someone on or take someone off of the blocklist. But I hope it gives the idea. Again, the purpose is to throw out heavy time-wasters as quickly as possible, while not wasting the time of legit users or putting their communications at risk.

-Billy

Re:RFC violations

[Electrum]

The difference is that while Postfix CAN reject based on HELO etc...qmail seems to do so by DEFAULT.

No, it does not. In fact, you don't even need a HELO with qmail.

Also, take a look at djbdns some time- it violates RFC's left and right.

Which ones and how?

A spam free world...

[Digital Dharma]

Is but an attitude shift away. All you have to do is follow Hotmail's idea of an exclusive address list. Nothing comes through for any individual user except what's from addresses in that user's personal address list. Keep the filtering feature on the client side, so all the mail server does is essentially route mail traffic, like any router should. Keep the processing load on the client. If the users want an email from a certain source, they're going to have to add the address in manually. A little unique cert generation during the initial mail client configuration, and you keep the email shotguns at bay. If someone has to reinstall their Operating System and thus has to regenerate a cert, set up an easy way for the 2 parties to re-exchange certs. Maybe utilize a website for this feature. Like public PKI... There's no reason not to do it this way with most new desktops approaching the 3 Ghz range. The users are going to have to take a proactive stance to spam, bottom line. No matter what legislation you push through, spammers will always find a way around any defenses we put up. Those who are aware of the nature of TCP/IP and programming know that whatever you implement, someone else can break. It would be trivial to force the end user to take control of their lack of spam, and thus break that particularly annoying 'feature' of open standards.

No, it's a numbers and money game

[RallyDriver]

I am the CTO at a company that provides hosted internet services, including email. We send around 3m pieces of email a week to our customers (opt-in only) lists. Speaking from the legitimate provider's viewpoint, I have a couple of observations:

1. RBL's don't work - community RBL's are used by relatively few mail systems out there; perhaps 1% of email addresses at most have RBL filtering on them at server or personal level, and the audience of any one RBL is just too small for it to have any value. Yes, using an RBL may stop *you* from receiving (some) spam, and in the short term that's all you care about, but it doesn't stop spam from being of value to the spammer. Just like the drug war, we will only win by making it unporfitable to send spam.

The biggest impact we see from RBL's is fielding individual "false-positive" complaints; we don't allow customers to send spam, so we get very few, but there's always the occasional idiot who signs up for a list and forgets, and who is too proud to click on the unsubscribe link.

What matters for delivery of my cleints' legit mailing lists, and what also a spammer cares about for delivering his spam, is delivery to the big guys - AOL, Yahoo, Hotmail, Earthlink, etc. If you're trying to email Joe Public, those guys have 50%+ of the market. Any successful spammer will have his energies focused on end-running their filters and will give a fig if RBL'ed.

2. IP-based filtering for consumer connections *does* work - ISP's and universities need to block port 25 outbound from consumer connections and desktops / 802.11 respectively. Spammers need a network connection; cut off their main source. This would stop not only transient spammers, but those who hack cable modem users.

AOL's efforts here on behalf of their users are commendable, but blocking these IP's *at source* where the blocker is making an informed decision and has the data to keep the filters accurate, is the way to go; a grassroots effort to inform ISPs about the benefits of this would be valuable.

This would leave spammers who are using business-class connections (where the ISP thus delegates the responsibility to run mail servers) which are much, much fewer in number and thus much easier to police.

Before anyone who runs their own SMTP server on tehir home Linux box cries foul, I should point out thay I do to, and I just have sendmail push everything through my ISP's SMTP relay. Big deal.

3. Money - money is they key to this. Make it uneconomic to spam, and the problem goes away.

I have one solution which I think wouls work well; like RBL's or source-end IP filtering, it suffers from the problem that it requires a large critical mass, so I think legal is the best route: I am speaking in terms of the USA, but this would work in other countries.

- anyone sending (pick a number, say 50k) pieces of email a month or more must register with the national email registry - this will cost $10k per year (this kind of price is essential to keep the spammers out, and it covers the cost of operating it). ISPs and email distributors are required both by law and defacto to sign up to be in business, and to them it's a modest cost.

- the registry will maintain an anti-spam policy and audit registrants against their track record of enforcing it; policy would need to include things like each email having clear unsubscribe info, info on where the address came from, etc.

- there will be a national "do-not-send-opt-out-mailings" list against which email marketers must clean lists which they buy; many countries have had this kind of list for phone and snail mail for quite some time, e.g. UK

- ISPs can then use the registry as a whitelist, and simply block every other IP address. Any business / individual too small to need to register can just forward their email via their upstream provider, who is then on the hook to manage their email behaviour.

Yes, it takes away some freedom to operate ones own email service, but equally I don't ru

OS flaws make technical solutions difficult

[heironymouscoward]

Spam is not just about sending unwanted email from rogue servers. Even if the Internet email system consisted of a 100% controlled network that excluded spammers' systems, there would be a serious spam problem. Why? More and more spam is sent from systems infected by viruses and trojans, and as other avenues get closed, this most promising one will be used to the maximum.
Let me race down the technology curve and predict some of the wonderful things that will happen in the war on spam:

- the majority of spam will originate from 'infected PCs'.
- some smart person will cause email to be charged, and millions of innocent users will get incredible invoices for email they 'never sent'
- as the number of infected PCs being remotely controlled by spammers increases, the volume sent from each PC will go random and low enough to be effectively undetectable.
- spammers will start modifying real email to attach their own messages.
- spammers will start modifying URLs in real email to point to their own websites.
- spammers will find ways to infect MSIE to do the same thing.
- anti-spam software will start to resemble anti-virus software, as spammers and virus writers hook-up into an organized (criminal) network.
- anti-spam software will be the main thing targetted by new viruses.

and all this time, 80% of PC users will remain blisfully unaware that their PCs are sending shiploads of spam around the world.

The basic problem is that the (Windows) PC is simply too complex, too connected, and too vulnerable to use as a secure communications device.

There is an answer somewhere... but I don't believe it lies in technological solutions, nor does it lie in making email paid, nor does it lie in attacking the servers and networks used to send spam. It is rather to understand that simplicity and transparency is the key to security. In the case of PCs, this means arriving at a OS/application combination that is immune to trojans and viruses, not thanks to the latest anti-virus scanners, but thanks to an inherently uncrackable design.

Re:IP banning

[Tadghe]

1. If you purchase IP's (actually "lease") in this day and age, you better damned well check them first... SPEWS and most of the other DNSRBL's will let you do so easily.

2. The idea of listing all or part of a class of IP's is intended to pressure the provider to change their habits of hosting/supporting spammers. Your case is a good example of why they would want to do so.

When spammer friendly ISP's stop allowing spammers to jump from IP to IP within their netblock, and start being a bit responsive to abuse complaints, this type of situation will go away.