Slashdot Mirror
The Next Step in Fighting Spam: Greylisting
Evan Harris writes "I've just published a paper on a new and unique spam blocking method called "Greylisting". The best thing about it other than achieving better than 97% effectiveness in blocking spam, is that it practically eliminates the main problem of other solutions: the false-positive. There's even source code for an example implementation written as a perl filter for sendmail, along with instructions for installing, so you can get up and running quickly."
most spam today is sent through open relays. Those relays will simply retry the delivery no matter which software the spammer uses, so the method won't work.
The Next Step in the Spam Control War: Greylisting
By Evan Harris
Copyright 2003, all rights reserved.
Introduction
This paper proposes a new and currently very effective method of enhancing the abilities of mail systems to limit the amount of spam that they recieve and deliver to their users. For the purposes of this paper, we will call this new method "Greylisting". The reason for choosing this name should become obvious as we progress.
Greylisting has been designed from the start to satisfy certain criteria:
1. Have minimal impact on users
2. Limit spammers ability to circumvent the blocking
3. Require minimal maintenance at both the user and administrator level
User-level spam blocking, while somewhat effective has a few key drawbacks that make its use in the continuing spam war undesirable. A few of these are:
1. It provides no notice to the senders of legitimate email that is falsely identified as spam.
2. It places most of the costs of processing the spam on the receivers side rather than the spammers side.
3. It provides no real disincentive to spammers to stop wasting our time and resources.
As a result, Greylisting is designed to be implemented at the MTA level, where we can cause the spammers the most amount of grief.
For the purposes of evaluating and testing Greylisting, an example implementation has been written of a filter that runs at the MTA (Message Transfer Agent) level. The source for this example implementation is available as a link below, and as other implementations or additional utility code become available, they will also be linked.
Greylisting has been tested on a few small scale mail hosts (less than 100 users, though with a fairly diverse set of senders from all over the world, and volumes over 10,000 email attempts a day), however it is designed to be scalable, as well as low impact to both administrators and users, and should be acceptable for use on a wide range of systems, including those of very large scale. Of course, performance issues are very dependent on implementation details.
The Greylisting method proposed in this paper is a complimentary method to other existing and yet-to-be-designed spam control systems, and is not intended as a replacement for those other methods. In fact, it is expected that spammers will eventually try to minimise the effectiveness of this method of blocking, and Greylisting is designed to limit options available to the spammer when attempting to do so.
The great thing about Greylisting is that the only methods of circumventing it will only make other spam control techniques just that much more effective (primarily DNS and other methods of blacklisting based on IP address) even after this adaptation by the spammers has occurred.
The Greylisting Method
High Level Overview
Greylisting got it's name because it is kind of a cross between black- and white-listing, with mostly automatic maintenance. A key element of the Greylisting method is this automatic maintenance.
The Greylisting method is very simple. It only looks at three pieces of information (which we will refer to as a "triplet" from now on) about any particular mail delivery attempt:
1. The IP address of the host attempting the delivery
2. The envelope sender address
3. The envelope recipient address
From this, we now have a unique triplet for identifying a mail "relationship". With this data, we simply follow a basic rule, which is:
If we have never seen this triplet before, then refuse this delivery and any others that may come within a certain period of time with a temporary failure.
Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec (see RFC 821). As such, any well behaved message transfer agent (MTA) should attempt retries if given an appropriate temporary failure code for a delivery attempt (see below for discussion of issues concerning non-conforming MTA's)
First I heard of it was from Landon Noll and Mel Pleasant. It is noted in brief as one of the techniques in this plan to end spam (though their plan, which did include the triplets, is not laid out in full there.)
It is a worthwhile technique for a little while, and if spammers were rational, would be worthwhile for some time to come. But spammers are not rational, and already this technique is not as useful as would be hoped.
Do a Google Search for Tempfailing especially in ASRG to see statistics etc.
Just encode your e-mail address on web pages & don't sign up to any dubious mailing lists.
I haven't received 1 single spam in recent months from doing this!
This isn't true, actually. Once one mail gets through, the system lets in subsequent mails from that sender. So there is only the initial delay, after that CEO Joe can use his email as a fat instant messenger per usual.
You're missing a big part of it though. If you have to try say 3 times to send a message [over a 5 day period or so] you're ability to mass send 100million emails is really squashed.
Legitimate people first time sending won't really mind the few day wait and most MTAs will try for upto a month.
Tom
Someday, I'll have a real sig.
Where? To me, publishing a paper means your writing appeared in some peer-reviewed journal (where the "peers" are acknowledged as domain experts). What you did was put up a web page. With a donation link at the bottom.
For others looking for a solution, try POPFile. Open source, cross platform, gives me 96% accuracy.
One more thing: "practically eliminates" is not the same as "eliminates".
not at all
Read the paper. Spammers would figure it out eventually. What it buys is what they have to do to get around it.
It means they have to do retrys...that means spam runs take longer, especially since they have to run...then wait for a locally defined timeout, and run all those addresses again
AND they have to do it from the same IP.
This raises their bandwidth profile. It wastes their time... all in all... it raises their cost of doing buisness and cuts into their profit margins.
It means they will have to upgrade their tools again. It means they get headaches. And of course, the next step is to impliment spam traps that watch activity and see that a spammer is spamming, and promotes them to a blacklist before they can even retry. (oh gee 1000 new greylist triplets from 1 IP in under 5 mins? Set the timeouts for that IP to 12 hours)
-Steve
"I opened my eyes, and everything went dark again"
I get 98-98.5% accuracy with POPfile. I get about 200 mails a day, of which around 30% spam. I get about 1 false negative a day, and maybe 2 or 3 false positives a month. It's a personal solution and as such is much more attractive to me than something server-based which has to be installed by a [typically VERY uncooperative] BOFH.
I use it experimentally for general mail classification (business/personal/a variety of mailing lists etc., all in all 7 buckets) on my home machine, and it works fine in these conditions too, although the accuracy is a bit lower (around 95%).
I've seen some comments that say (paraphrasing) "For real SPAM filtering use <POPFile|Spamassassin|...>", but these missed the point (or perhaps didn't read the paper?). This method is a "first-pass" filter, getting rid of e-mails for which no redelivery attempt will be made. The second-pass filter should still be implemented for everything that gets through the first pass. From the paper:
"The Greylisting method proposed in this paper is a complimentary method to other existing and yet-to-be-designed spam control systems, and is not intended as a replacement for those other methods. In fact, it is expected that spammers will eventually try to minimise the effectiveness of this method of blocking, and Greylisting is designed to limit options available to the spammer when attempting to do so."
sig != null
It's not the sender's mail client that connects to the server that runs the greylist system, it's the sender's SMTP server as provided by their company or ISP. Its IP address will not change regardless of the sender's connection or dynamic IP.
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
I really like the idea, and I think it will work wonders (if you are willing to accept your nearly-instant email now having approx. 1 hour delay).
However, here is how the spammers will adapt:
MailBlast 2.0 will send each mail TWICE. The first time to get the triplet on the greylist, the second time to actually send it. Or a little more sophisticated, it will run in one-hour blocks, and at the end of the hour, re-run the previous hours emails. No queue or other real-SMTP server functionality necessary.
- For the complete works of Shakespeare: cat
Aside from the obvious of getting the authorities to crack down on the existing illegal activities (relay hijacking, violation of TOS of ISPs, header forging, etc.) which is the only true solution, I think there are much better approaches than this "greylisting" method.
The problem with the greylist method is it still slows down mail service, and potentially more than the relay blacklist features. The objective here is that end-user/networks should not be penalized in the fight against spam. We already waste too many resources, and according to my latest mail server stats, more than 65% of our inbound mail is UCE. I'm fed up with more than half my e-mail bandwidth being crap my users didn't request so more resource allocation on a local level in the fight against spam is counterproductive!
Here's a very clever, much more practical method I cound recently.
A company is Canada has set up what it calls SORBS: Spam and Open Relay Blocking System.
What's different from their blacklist is that they maintain "honeypots" strategically located around the Internet. These are servers they specifically set up as inbound mail relays, but never for legitimate purposes. If the servers get [select] mail activity, it's assumed to not be legitimate and it flags the source as a potential spammer... it makes a lot of sense. You create a domain name, but don't promote it in any legitimate manner, and/or you seed spam lists with these e-mail addresses and then let the spammers send to your key systems around the internet and *bam*, they're identified in real time, and then added to a blacklist.
I really like this idea. Like any other system, it has the potential for abuse but the beauty is the identity of the honeypot systems is kept secret, so it's very difficult for anyone other than spammers to exploit the network.