Slashdot Mirror
The Next Step in Fighting Spam: Greylisting
Evan Harris writes "I've just published a paper on a new and unique spam blocking method called "Greylisting". The best thing about it other than achieving better than 97% effectiveness in blocking spam, is that it practically eliminates the main problem of other solutions: the false-positive. There's even source code for an example implementation written as a perl filter for sendmail, along with instructions for installing, so you can get up and running quickly."
I'm going to try to say this as nicely as possible and without trolling:
You have just rendered Greylisting pretty useless by making it open source. Spammers are much smarter than you think and what you have basically done is shown them what they need to do in order to get around Greylisting. That's just my take on the issue, maybe I'm wrong but I doubt it.
I have often regretted my speech, never my silence.
-Xenocrates
I like the idea though. Since SMTP is broken anyway, why not use another of it's features in a new way to help filter unwanted email. Keep up the good work!
most spam today is sent through open relays. Those relays will simply retry the delivery no matter which software the spammer uses, so the method won't work.
The Next Step in the Spam Control War: Greylisting
By Evan Harris
Copyright 2003, all rights reserved.
Introduction
This paper proposes a new and currently very effective method of enhancing the abilities of mail systems to limit the amount of spam that they recieve and deliver to their users. For the purposes of this paper, we will call this new method "Greylisting". The reason for choosing this name should become obvious as we progress.
Greylisting has been designed from the start to satisfy certain criteria:
1. Have minimal impact on users
2. Limit spammers ability to circumvent the blocking
3. Require minimal maintenance at both the user and administrator level
User-level spam blocking, while somewhat effective has a few key drawbacks that make its use in the continuing spam war undesirable. A few of these are:
1. It provides no notice to the senders of legitimate email that is falsely identified as spam.
2. It places most of the costs of processing the spam on the receivers side rather than the spammers side.
3. It provides no real disincentive to spammers to stop wasting our time and resources.
As a result, Greylisting is designed to be implemented at the MTA level, where we can cause the spammers the most amount of grief.
For the purposes of evaluating and testing Greylisting, an example implementation has been written of a filter that runs at the MTA (Message Transfer Agent) level. The source for this example implementation is available as a link below, and as other implementations or additional utility code become available, they will also be linked.
Greylisting has been tested on a few small scale mail hosts (less than 100 users, though with a fairly diverse set of senders from all over the world, and volumes over 10,000 email attempts a day), however it is designed to be scalable, as well as low impact to both administrators and users, and should be acceptable for use on a wide range of systems, including those of very large scale. Of course, performance issues are very dependent on implementation details.
The Greylisting method proposed in this paper is a complimentary method to other existing and yet-to-be-designed spam control systems, and is not intended as a replacement for those other methods. In fact, it is expected that spammers will eventually try to minimise the effectiveness of this method of blocking, and Greylisting is designed to limit options available to the spammer when attempting to do so.
The great thing about Greylisting is that the only methods of circumventing it will only make other spam control techniques just that much more effective (primarily DNS and other methods of blacklisting based on IP address) even after this adaptation by the spammers has occurred.
The Greylisting Method
High Level Overview
Greylisting got it's name because it is kind of a cross between black- and white-listing, with mostly automatic maintenance. A key element of the Greylisting method is this automatic maintenance.
The Greylisting method is very simple. It only looks at three pieces of information (which we will refer to as a "triplet" from now on) about any particular mail delivery attempt:
1. The IP address of the host attempting the delivery
2. The envelope sender address
3. The envelope recipient address
From this, we now have a unique triplet for identifying a mail "relationship". With this data, we simply follow a basic rule, which is:
If we have never seen this triplet before, then refuse this delivery and any others that may come within a certain period of time with a temporary failure.
Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec (see RFC 821). As such, any well behaved message transfer agent (MTA) should attempt retries if given an appropriate temporary failure code for a delivery attempt (see below for discussion of issues concerning non-conforming MTA's)
First I heard of it was from Landon Noll and Mel Pleasant. It is noted in brief as one of the techniques in this plan to end spam (though their plan, which did include the triplets, is not laid out in full there.)
It is a worthwhile technique for a little while, and if spammers were rational, would be worthwhile for some time to come. But spammers are not rational, and already this technique is not as useful as would be hoped.
Do a Google Search for Tempfailing especially in ASRG to see statistics etc.
Instead of filtering out email completely, we just add [spam] to the begining on anything that is potentially spam, have it forwarded to a folder, and go through it once a week. In 3 years of using it, I've only had 1 message that was accidently called spam. And I didn't care if i recieved it or not anyway.
http://use.perl.org
Short of changing my email address, is there any way I can stop them?
Just encode your e-mail address on web pages & don't sign up to any dubious mailing lists.
I haven't received 1 single spam in recent months from doing this!
ive invented pinklisting. i now just get all the good gay spam.
This isn't very reassuring:
"it practically eliminates the main problem of other solutions: the false-positive."
What does 'practically eliminates' mean? If it gives false positives at all, it is just as useless as all those 'other solutions'.
*This page intentionally left pointless*
Time critical mailing will go out the window. I can see how this might make any corporate user irate. The same thing goes for challenge-response, the time delay in the business world is unacceptable.
This would be great for personal mail, but that's about it. ISPs would have the same problems with it because their business-class users most likely use the same servers as their consumer-class users.
If they can get around it by looking at the source, then something was wrong with it, waiting to be exploited. Might as well fix it.
social sciences can never use experience to verify their statemen
with all of these solutions to spam..and all of the spam now flooding mail servers...
isn't it time to change the specification (RFC) and possibly the manner in which our current system works? i haven't come up with anything yet, but surely there must be some sort of handshaking/secure type connection that could be used - - some sort of postage (free) that is encrypted into the mail, that states that it is genuine....kind of like the hologram on those windows cds...
i dunno. file this story under redundant.
We're like rats, in some experiment! -- George Costanza
I thought it was generally understood that most spam was sent by abusing open relays, thus hiding it's origin. This could be wrong. However if it's not, those figures aren't appllicable. Nor is spam going to be diverted since an open relay is generally running a regular mta and will attempt a retry. For instance, if qmail were running on an open relay and was abused by a spammer it would try again and again with an increasing delay (calculated logarithmically if memory serves) between attempts. So the mail will still get through.
When you further consider that if a spammer hits an open relay and hammers your mailserver from it and all of the "triplet's" are new, you're increasing your traffic, because all of that mail will be attempted again.
It should work for most methods of spam. If you're Joe average using Outlook (gasp), then you can even filter with it.
Tools->Message Rules->Mail
Where the Subject line contains specific words (add words, "enlarge", "penis") - That takes care of those
Add another "horse", "cum" (or something else common) byebye.
Make the default action to move such messages to "trash" or a "spam" folder.
You can also filter by message body...
How does the effectiveness of Greylisting compare with what others are seeing with existing techniques (such as Bayesian filtering)? Is it a false positives problem, such as digests and opt-in mailing lists getting incorrectly tagged as spam?
Dr. Rick
- "It's such a fine line between clever and stupid" (Nigel Tufnel)
- Zort! (Pinky)
I parse the content before I read it (isn't php great? :-)
.exe attachments, any email with the words viagra or penis (or some other words in my list, like "second mortgage" when I don't own a home,) in it gets purged as soon as I pull it off the server.
Any email with HTML in it, any email with
It never gets to my mail program.
I could also filter on subject lines containing any word whi isn't in thdictionary but since some of my friends don't spell too well...
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
I have one filter that blocks at least 90% of my spam. If a message contains the word "offer" it's toast. Works for me anyways.
Doels this mean all public crypto algorithims are useless?
Keep the Classic Slashdot.
The best idea I've seen in YEARS was to have people start using a specific, original poem as their signatures. Then, the author granted license to anyone who WASN'T sending spam. Therefore, they could sue any spammer for copyright infringement if they used it, and you could train your mail filter to look for the signature. Once spamassassin took it up, it pretty much snowballed. See story here
-Looking for a job as a materials chemist or multivariat
Question about this system: if it sends a temporary unavailable message or whatever it does, does it log the original message? Where I'm going with this is what happens if a legitimate message is blocked but never resent? Most anti-spam software allows you to view the spam folder, or something equivalet, to check for false positives. How are false positives handled here?
This post cannot be rebroadcast without the express written constent of Major League Baseball.
I *WANT* spam.
Not shit, mailto:spammesilly@gt.rr.com
Send it to me baby!
I'm teaching my PC how to deal with it and I've been on a mission to sign up for every bullshit mailing list I can find, all the typical trash that pesters people to death.
Once I get this worked out I'll install it on my dad's PC and all my friends too. Then they can say bye-bye to spam.
I told my friend, "Your shit works because my shit is always broke!".. In other words, I am the guinna pig for everyone else. I test it then they get the working version..
So, mailto:spammesilly@gt.rr.com
How about in the spirit of RFC 3514 (the evil bit) we create a spam header in email. Spam will set this header so we can easily filter it out.
Why don't people just ask their friends to encrypt incomming email? That is one of the simplist way to eliminate spam, and it works too.
Where? To me, publishing a paper means your writing appeared in some peer-reviewed journal (where the "peers" are acknowledged as domain experts). What you did was put up a web page. With a donation link at the bottom.
For others looking for a solution, try POPFile. Open source, cross platform, gives me 96% accuracy.
One more thing: "practically eliminates" is not the same as "eliminates".
The article also doesn't measure the amount of spam coming through those relays. Even if there are only 10 open relays in the UK at any one time, it still might be possible for all of the spam to be coming through them.
Certainly, closing down open relays is a good thing, but lowering the percentage of open relays doesn't prove anything about the source of spam
Look at it this way: you can stop crank calls by unlisting your phone numbers. But you can't unlist the hospital, the ambulance service, the fire department, etc.
We're not all end-users. Some of us are the plumbers.
to stop all spam by blocking the dollar sign $ (my country doesnt use them)
i havent had any spam for months as all the spam i have ever got is from usa based spammers, works a treat
The Next Step in Fighting Spam: Death Penalty
http://use.perl.org
isn't a problem. The people I hear most often complain about SPAM are North American and UK people where SPAM isn't illegal.
Maybe its like condom's.
99% effective, 1% totally ineffective....
Spammers are notoriously resiliant. Within a few days/weeks/nanoseconds the spammers would realize they need to retry after a delay, and they would stop with the fire-forget mentality.
I wish your plan would work but I just don't think it will.
Plus the spammers can get their viagra at wholesale cost!
I don't want to be here.
I'm wondering how I'm going to explain that to a new customer over the phone who says "I'll just email that file right now so we can go over it together".
PJRC: Electronic Projects, 8051 Microcontroller Tools
According to this article (June 12), open relays at least in the corporate environment are becoming hard to find, requiring spammers to find new ways. In 1997, 91% of mail servers tested were open; as of a year ago only 1%. ISP and home machines apparently weren't tested.
This doesn't really say what's actually being used by spammers, but it's a sign of improvement. At the least, it narrows the pool of available relays. Continuing progress will increase the spam pressure on those remaining, which will in turn make it more likely that they'll be fixed.
The article also doesn't say what spammers might use as an alternative. From something else I read recently (don't recall where), mail viruses that take over users' machines are rapidly becoming the tool of choice. There are a lot more of them than mail servers, so it makes sense for the spammers. It does put them in a more dangerous position WRT the law. IMHO (IANAL), using a virus to exploit someone's machine for profit is almost certainly illegal under existing law.
It's easier to be a result of the past, but more fun to be a cause of the future! http://www.spacefinancegroup.com/
It deals with spam at the server level. All the wonderful user-level solutions don't do jack to stop spam from being sent. Look at the numbers the spammers show for return rate, and look at how fast spam programs can go, and you'll see that the only solutions that will work are those that make it expensive to send spam. Anything else will just make the spammers send more spam to try and get the hit rate they need.
This is the right place to test your anti-spam tool (is it a graylist?)
If it's "bullshit...all the typical trash that pesters people to death." that you are looking for, you found it !~
Smile, it's Friday EST.
"Whoever would overthrow the liberty of a nation must begin by subduing the freeness of speech."--Benjamin Franklin
1) Steal regular Expressions from spamassasin
2) write perl to usa a MySQl db
3) put a 1 hour timer on incoming mail (forget those important mails)
4) put up a webpage and tell slashdot
5) insert "donate" paypal button
6) Profit !!!
The registrar I use (jumpdomain.com) has a clever hack for despamming WHOIS contact email. Basically they change your published contact address once a week. The published address i automatically generated, looks like gibberish, and forwards to your real address. If someone wants to contact you by looking up your address by WHOIS and writing to you, it works fine. But if they add the address to a mailing list, it stops working in a week. That has eliminated almost all my WHOIS spam. Good scheme.
If we have never seen this triplet before, then refuse this delivery and any others that may come within a certain period of time with a temporary failure.
Since SMTP is considered an unreliable transport, the possibility of temporary failures is built into the core spec (see RFC 821). As such, any well behaved message transfer agent (MTA) should attempt retries if given an appropriate temporary failure code for a delivery attempt (see below for discussion of issues concerning non-conforming MTA's).
During the initial testing of Greylisting, it was observed that the vast majority of spam appears to be sent from applications designed specifically for spamming. These applications appear to adopt the "fire-and-forget" methodology. That is, they attempt to send the spam to one or several MX hosts for a domain, but then never attempt a true retry as a real MTA would. From our testing, this means that currently, based on a fairly conservative interpretation of testing data, we see effectiveness of over 95%, and that is with no legitimate mail ever being permanently blocked.
Any attempt to build technology to work around current spammer's techniques is pretty much a waste of time. They'll just adapt. A well written SPAM program can fetch $10,000 per license. It wouldn't be that hard for them to act like a legit MTA to get around this problem, if the patch were widespread.
Sure, it prevents 95% of spam now, but if it becomes widespread it won't.
That said, it will make spammers lives much more difficult, and requires them to identify themselves. So this could be helpful, in concert with other tools. One option would be to use this along with sender-verification, so that people who run legitimate MTAs don't need to worry about getting a verification message (for now).
autopr0n is like, down and stuff.
We should grab some of the guys who get 1000+ spams per day, point them to the physical location of the spammers, and then step back. I can guarantee you that vigilante justice is entirely appropriate here, considering we want the gov to step back from the 'net instead of entering new "secret arrests of spammers"(?) laws.
I think it should be called a brownlist.
I'm not sure I completely understand spam with how it works today. Why would a spammer want to send mail to someone who is clearly not going to buy from them? I mean, I understand their logic is to send as much spam to as many people as possible, but if we make e-mail addresses that say NOSPAM in them why would they even want to spam that address? Do these guys get together and discuss how many messages in a day they can send as if they're talking about how big their penises are?
I'm currently using Bogofilter (and looking into CRM114) and getting better than 99% accuracy (about 1 in 200 false negatives at the moment) and very very few false positives (maybe 2 in 5000 messages).
Of course these are MUA level filters (and yes, I know, I've already "paid" with bandwidth to download the spam) - however since the proposed "greylister" would have to be installed as the MTA at major ISPs (as the authors note) I'm not convinced that is more likely to get widespread adoption than the various sorts of adaptive client-based filtering now available, particularly as it requires a database to back the method up.
As far as I am concerned the major factor in a spam filter should be zero false positives - personally I don't mind reviewing one or two spams a week but I get really annoyed if I were to lose a real message (note the two false positives I have sent to date with bogofilter contained forwarded sales pitches along with a message).
-- Nothing unusual happened today
During the initial testing of Greylisting, it was observed that the vast majority of spam appears to be sent from applications designed specifically for spamming. These applications appear to adopt the "fire-and-forget" methodology.
Spam guards and spam co-evolve. Since greylisting is easy to get around by spammers, if it becomes widespread, spammers will take measures to avoid it, and the net result will be a lot of extra traffic.
In fact, the impact of this kind of system on mail could be pretty bad if widely adopted: large amounts of mail may end up being held up in delivering servers, and "informative" messages sent by helpful mail systems (about "temporary failures") may end up creating more junk mail than they avoid.
This greylisting technique CANNOT be 'perfict', it's going to send some spam through, no matter what Right now that number is 5%. Once spammers know how to get around the system, that number will go up to 100%. Another poster already mentioned, all you have to do is send the message twice. (the greylist will see that as retrying and let the msg through).
autopr0n is like, down and stuff.
Score: 5, Insightful, my ass!
So now it'll take me 1 hour every time I want to set up a Slashdot troll account.
The site seems to be fine, why not wait untill it *actualy* get's slashdotted?
autopr0n is like, down and stuff.
The biggest problem, as the greylisting paper correctly points out is the false positive. It's only happened a few times, but I have had fairly important emails end up in the POPFile spam bucket. I'd much rather have a way for it to err on the side of caution and let a few more spam through if it means almost never getting a false positive.
I basically gave up on SpamCop and the like. Most of the spammers send from off-shore and the ISPs don't care.
-S
--- What parts of "shall make no law", "shall not be infringed", and "shall not be violated" don't you understand?
I get 98-98.5% accuracy with POPfile. I get about 200 mails a day, of which around 30% spam. I get about 1 false negative a day, and maybe 2 or 3 false positives a month. It's a personal solution and as such is much more attractive to me than something server-based which has to be installed by a [typically VERY uncooperative] BOFH.
I use it experimentally for general mail classification (business/personal/a variety of mailing lists etc., all in all 7 buckets) on my home machine, and it works fine in these conditions too, although the accuracy is a bit lower (around 95%).
What I do is set my MTA (well, actualy someone I'm using someone else's mail server) to forward all the mail sent to my domain to my main account. That way, whenever I sign up for anything or give away any email address I use a unique address.
Oddly enough, I've been totaly promiscuous with these throw-away email addresses, but I've only got one SPAM from a company I actualy bought stuff for. So far no one has sold any of the address, and all the websites I've posted too either arn't scanned or are protecting the address well.
OTOH, my 'main' address are spammed constantly. rrr.
autopr0n is like, down and stuff.
You know... All of these fancy 'spam-fighting' methods are just a waste of time, when all you really need is a properly configured smtp server and some good free realtime blackhole lists. After making simple changes, I went from receiving 5-6 spams a day down to 1 in a year and a half. With no complaints of 'false-positives'. There have been instances where the senders mail server was misconfigured, but after contacting them and explaining the situation they were invariably helpful. All I did was make sure that only mail sent from fqdn to valid local accounts, and such were allowed there is an ok tutorial on basic psotfix configurattion herehere.... and a great one Here.
With all of the naysayers that come out a scrutinize every idea to the 't', it's a wonder we ever get anything done at all.
..are not an option for 97% of the internet, and 100% of the people who e-mail my company. they want e-mail because it is easy and fast and free, get them to put any work into it and they'll just start to pick up the phone or send a letter!!!
I went to battle MC Escher but drew a blank
Can I just skip 'publishing' a 'paper' and just put a paypal link to give me a donation in a story and have it posted on slashdot?
I've seen some comments that say (paraphrasing) "For real SPAM filtering use <POPFile|Spamassassin|...>", but these missed the point (or perhaps didn't read the paper?). This method is a "first-pass" filter, getting rid of e-mails for which no redelivery attempt will be made. The second-pass filter should still be implemented for everything that gets through the first pass. From the paper:
"The Greylisting method proposed in this paper is a complimentary method to other existing and yet-to-be-designed spam control systems, and is not intended as a replacement for those other methods. In fact, it is expected that spammers will eventually try to minimise the effectiveness of this method of blocking, and Greylisting is designed to limit options available to the spammer when attempting to do so."
sig != null
From: crovira
To: crovira's doctor
Subject: my penis!
Doctor, doctor! My penis has turned green, what can I do?
3 hours later
Great now it fell off
When we do our jobs well and our systems work well, email is nearly as instantaneous as IM. Unfortunately it creates this illusion that e-mail *is* IM.
I get a call from the lusers at least once a week panicking because someone in another part of the world clicked [SEND] 30 seconds ago and they didn't have the mail message.
"Was there an attachment?"
"Yeah, a pretty big powerpoint file."
"Give it a few minutes."
I check the logs later and it's a like 120 Meg file and the sending system is listed with an A record of like "joe-blow-DSL-1-2-3-4".
This is all nice and all but what about challenge/response.
. ht p
I dont see a much better model than this
and it is relatively easy to implement
Here is a link for more info
http://tmda.net/faq.cgi?req=show&file=faq01.001
Sigs are dangerous coy things
Our CanIt anti-spam product has been doing it for quite a while.
I posted an article about it in January.Personally, I only accept messages that are 10 MB in size or larger. If you want to email me, please be sure to include a huge block of random text at the end of your email or else I'll never see it.
I don't get any spam using this approach, because the spammers don't send big messages. And if *everyone* ignored small messages, spammers would have to close up shop because they could not afford to send millions of big messages.
(This is a joke. But you could do this at the SMTP level, by automatically replying to any sender who is not on your personal whitelist with the response: "Hey you, if you're real, send me back a HUGE reply!" And the SMTP server could cheerfully delete the last 99% of the first-time oversized email you get. I should write my own anti-spam paper and get mad Slashdot cred. Nah, I'm too lazy.)
You'll notice that the US is the #1 country Top 3 are:
- The United States, with over 80,000 open relays
- Korea and Japan pretty much tied at +15,000 each
- Japan, at just under 10,000
That's more than everyone else combined!I have written an essay on ending spam. The idea is to associate a second piece of information that goes along with your e-mail address. This 'secret' can be used for anything you want, such as blocking anyone who does not get the secret right.
The Next Step in Fighting Spam: Death Penalty
I'm sure his advisors are working out the details of this one as we speak.
My days of not taking you seriously are certainly coming to a middle...
153 GABON 1
ALso Population of 1.
Go figure.
I wrote an implementation of this back in November and rolled it out on my personal systems. I purposely refrained from sharing it, since as soon as someone knows how it works, it becomes trivial to circumvent.
Now it's front page news on Slashdot. So much for that trick. Thanks guys!
At least I still have my legions of spam trap accounts to snare the spamming lusers of the world. They can exhaust the open proxies by getting all of them listed by mailing my traps for all I care.
Or we could all abandon SMTP and move to my jabber-based email "solution".
What? Where is it? Oh, I'm still working on it. You can send and receive, but buddy lists are not implemented yet.
Then the target(s) have one hour to recognize you tried a spam run, and just block that IP for your second run (switching to a new IP means one new hour of waiting). That was part of the benefit of an hour delay, to give IP blockers time to react.
No, the real means of bypassing this is a little trickier.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
So far the best thing I've seen is the challenge response systems like bluebottle (free but very slow) or spamarrest (never used it, costs $34.95 per year).
I've been using bluebottle for over a month now and its so much nicer knowing new emails are really just that. Unfortunately its slow enough (at least the web interface, pop is faster) that I won't start recommending it to family and non-techy friends.
Quack, quack.
I quite like the idea of this greylisting, but it seems a lot of spam is nowadays being sent as DSPAM (cf DDOS) or Distributed Spam. A spammer infects a load of broadband machines with a simple trojan, and then calls upon a number of the trojans to send an email spam via that machines normal MTA (ie for most windows machines it uploads to the ISPs mail servers).
I know this is happening as some complete bastard seems to be doing this using my domain as a "From:" address (well, [random-word]@schmerg.com), meaning that I'm been getting about 30 or 40 bounce messages a day for the last 2 or 3 months now. And although the odd sending IP is repeated, mostly they're all from different IP addresses. And of course I'm getting perfectly valid looking bounce messages from perfectly reasonable companies (and only a couple of abusive replies so far).
Now the problem is that the email is being uploaded to thier (non-open-relay) ISP's mailserver that will retry properly, and anyone else looking at the IP address will see a perfectly reasonable IP (the spammer seems to gave infected a lot of AT+T customers, ComCast customers, etc.). So short of blocking spam on subject, this spam is harder to prevent in the first place.
I've semi-automated a process to report the infected machines (that provoked a bounce message) to the appropriate ISP, and seem to havign some success in getting the ISPs to contact their customers, but I think this new form of spamming using a distributed attack will be particularly hard to block.
Anyone with a great idea (or who knows more about this scheme, or the identity of the twat behind it) I'd love to hear from you...
--
T
I spent a lot of money on booze, birds and fast cars. The rest I just squandered. - George Best
This is Slashdot, so shouldn't this post have been repeated five times?
Is everyone has thirty second attention spans and doesn't really read anything.
The delay only happens when an MTA receives the first e-mail from that IP address within the past 36 days. Your ISP's or company's MX would have you whitelisted if you sent any e-mail in the past month and there would be no delay. Many ISPs require you to check e-mail before sending e-mail anyway and this would whitelist your address. This measure primarily affects people who mail from outside the organization or ISP who don't write mail often.
Korea and Japan pretty much tied at +15,000 each
That's Korea and China. But who cares, they all look the same to me...
My other computer is your Windows box
But by the same token, any serious fix means totally discarding all our email infrastructure and starting from scratch. Nobody seems motivated to do that.
Technically, the solution is simple enough. You create a new mail system that simply doesn't support anonymous sending. Access to the system means following procedures ("postage", throttling, whatever) designed to prevent spam.
Probably for a long time both systems would coexist for a long time. Most people would want to keep their old mailboxes so they don't lock out friends who haven't changed over. But their new spamproof addresses would be the ones they broadcast. As the level of legitimate email in the old system drops to zero, more and more people would change over, and the old system would wither away.
But developing such a system would be expensive. Perhaps the big ISPs and other providers will be motivated to spend the money as spam sucks up more and more of their resources.
In the not too distant future, when sending email is taxed, and greylisting is common, estimated time of email delivery = 3 days, eta on snail mail = 3 days; what's the difference?
Most writers regard truth as their most valuable possession, and therefore are most economical in its use - Mark Twain
Evidence has shown that email harvesters don't even *try* to parse even the simplest email obfuscation techniques (kevin at fury dot com, or changing the @ to an html entity).
These are widespread practices and yet spammers don't care about spending the effort to reach that 5% who so adamantly don't want to be reached.
Unless greymail is used by more than a quarter of *all* email readers, history has shown that the spammers just won't care.
Kevin Fox
how many people have outlook and Eudora running for 1 hour at least? this wont work unless people are often sending mail. i mean, i t will work, if you send at least 2 mails every 36hrs, but, i myself, dont even se the function in that when i almost alwyas can express myself using IM, like ICQ, MSN, or yahoo. secondly, i have never heard of a mailinglist software that would retry a failed message, and that is to consider a problem. Greylisting, will work, to block spam, but it will also work to be annoying for automated software lkike mailinglists, and service messages. i think it will be a positie thing to implement it, but only if some kind of IM service is _NOT_ implemeted, and, it already is. this problem arises when people send fewer and fewer emails, and use IM instead for their messages. im just pointing out a logical problem, please consider this. ;)
Cannot someone simply defeat this, by sending two emails with the same triplicate in a row?
It seems one side effect of this approach is that it records the sender and recipient of every email sent through a particular mail server.
Sound familiar to anyone?
A tourist wanders into a back alley antique shop in San Francisco's Chinatown. Picking through the objects on display he discovers a detailed, life-sized bronze sculpture of a rat. The sculpture is so interesting and unique that he picks it up and asks the shop owner what it costs.
"Twelve dollars for the rat, sir," says the shop owner, "and a thousand dollars more for the story behind it."
"You can keep the story, old man," he replies, "but I'll take the rat."
The transaction complete, the tourist leaves the store with the bronze rat under his arm. As he crosses the street in front of the store, two live rats emerge from a sewer drain and fall into step behind him. Nervously looking over his shoulder, he begins to walk faster, but every time he passes another sewer drain, more rats come out and follow him. By the time he's walked two blocks, at least a hundred rats are at his heels, and people begin to point and shout. He walks even faster, and soon breaks into a trot as multitudes of rats swarm from sewers, basements, vacant lots, and abandoned cars. Rats by the thousands are at his heels, and as he sees the waterfront at the bottom of the hill, he panics and starts to run full tilt.
No matter how fast he runs, the rats keep up, squealing hideously, now not just thousands but millions, so that by the time he comes rushing up to the water's edge a trail of rats twelve city blocks long is behind him. Making a mighty leap, he jumps up onto a light post, grasping it with one arm while he hurls the bronze rat into San Francisco Bay with the other, as far as he can heave it. Pulling his legs up and clinging to the light post, he watches in amazement as the seething tide of rats surges over the breakwater into the sea, where they drown.
Shaken and mumbling, he makes his way back to the antique shop.
"Ah, so you've come back for the rest of the story," says the owner.
"No," says the tourist, "I was wondering if you have a bronze spammer."
As far as I can see, despite all the "the spammers will get round it" defeatism going on is that this will make life harder and more expensive for spammers.
.sig
The key is that by using this method and collecting data on who is sending to you and at what rate then you've just bought an hour in which to blacklist that triplet - maybe you only blacklist it for a relatively small amount of time, say 24 hours - and stop that particular flood of spam. As another poster noted, 5000 mails in a very short space of time sounds like spam.
Spamassassin et al are good, but they cost in hardware. This method appears to not be a significant overhead at the recieving end so the real costs of spam are increased for the spammer.
There are clearly situations where this isn't going to be appropriate, but if my university used this for students then spam would no longer consume so much of my inbox.
Nobel
--
This is a
I think it's important to note that this should not be viewed as some kind of anti-spam panacea; it's just another tool in our anti-spam toolbox.
Sure, it's easy to get past this system - just write the spamming program so that it retries all of its mail after two hours. However, this does have one huge disadvantage for the spammer: their mail does not get through the first time. This means that things like blacklists and the like have much longer to respond than they normally do.
Anything that makes spammers do extra work seems to make it more likley that they will have patterns that can be observed and then blocked before mail really goes anywhere...
"There is more worth loving than we have strength to love." - Brian Jay Stanley
The comments in this paper about other systems ignore one of the oldest and largest SPAM filters: SpamAssassin.
SpamAssassin can also be used at the MTA-level, and while this tool might be an interesting test to integrate with SA, its claims that other systems cannot feed back to the sender that their mail has been blocked is flat-out wrong.
Most people do not do this because you are almost certainly getting this mail through a relay, and that relay is going to get the SMTP temporary error and try to send a warning to the user who sent it. Spammers regularly slam my home mail server by using my address as the "From" in an entire batch of spam. It's pretty seriously annoying to get that deluge of junk, and it's not really necessary. If your spam system just identifies spam and lets the user (or sysadmin) decide how to deal with it based on how "spamish" it is, you get a much more reasonable behavior.
I junk thousands of pieces of spam every week, and I *never* junk valid mail. Yes, I do have some spam in my inbox. Most of it is tagged as potential spam, and I delete that after cursory inspection of the from addresses. Some of it is missed, and the overhead that I suffer having to identify that myself is amazingly low compared to not being able to read my mail prior to SA.
Check out SA. The latest version is pretty impressive, and if this "new" technique (I don't think the idea of tracking connection quality is very new, it's certainly done in SA to some extent) turns out to be useful... well SA works on much the same principal as Perl: There's More Than One Way To Do It. Bayes, Blacklists, Whitelists, Obfuscation detection, Checksum trackers, you name it, SA uses it. None of these techniques gets to say "this is spam", they all just get to poke a message in the direction of being spam or non-spam. This leads to something far more reliable than any one techniqe.
Then only outlaws will spam! Where are you from with an if-its-illegal-nobody-will-do-it attitude, Europe? Geesh.
I want to delete my account but Slashdot doesn't allow it.
"With all of the naysayers that come out a scrutinize every idea to the 't', it's a wonder we ever get anything done at all."
Tell me about it. Uh, what do you see that has gotten done?
As I test once I posted 2 + 2 = 4. Naysayers volunteered. I knew they would.
Waht we need is a posting for which a naysayer's response will destroy that naysayer.
(Yeah, I know. "That will never work.")
-> MAIL FROM: <sender@somedomain.com>
<- 250 2.1.0 Sender ok
-> RCPT TO: <recipient@otherdomain.com>
<- 451 4.7.1 Please try again later
So, the suggestion is to hold the session open at "rcpt to:" while searching a database for the triplet? And then, depending on what is found, continuing the session or killing it? For each email? How much impact will that delay have on the capabilities of a busy server?
I once heard a story once that is probably false (you never know), but contains an interesting idea on how to end spam.
It says that Mississippi tried to outlaw "flag burning", and the law was struck down as unconstitutional. So the Mississippi legislature responded by limiting the maximum penalty for assaulting a person who is engaged in flag burning to a $25 fine.
This sounds like a fine idea on how to handle the spam problem.
Create a law that states that the maximum penalty for physically assaulting a spammer is a $100 fine. I know more than a few people who'd be willing to pony up and take a whack at them.
Though this law would probably be far more satisfying than sensible.
At work our external e-mail went down a few weekends ago, and came back up around 8 am on monday. It took about 9 hours for e-mail sent on the monday morning at 9:30 to reach me. My plans to meet someone were in that e-mail.
:)
SO I PHONED THEM. Wow.
As I wrote earlier, we've been doing this for quite a while. You can see our statistics here.
Greylisting will not catch anwhere near 97% of spam. Our statistics show it catches anywhere from 15% to 30%. Nevertheless, the fact that it uses hardly any resources on your computer makes it worth doing.
You can also mitigate the delay problem by having a secondary MX record. Rather than waiting an hour, a legitimate SMTP host will retry the message on the seondary MX and it will get through almost immediately.
I know this is slightly off topic but please bear with me. I'm sure some of you have been wronged in someway by some pathetic person over the internet and wanted to get back at them without breaking the law. Recently a "friend" of mine managed to gain control of one of my online gaming accounts and drag my good name through the mud. The kind reps at Blizzard said there was nothing they could to about it and suggested I use some "frontier justice". So since I have my "friend's" email address I was thinking about signing him up for some spam. What would be the easiest way to get him signed up to receive tons of the stuff?
I really like the idea, and I think it will work wonders (if you are willing to accept your nearly-instant email now having approx. 1 hour delay).
However, here is how the spammers will adapt:
MailBlast 2.0 will send each mail TWICE. The first time to get the triplet on the greylist, the second time to actually send it. Or a little more sophisticated, it will run in one-hour blocks, and at the end of the hour, re-run the previous hours emails. No queue or other real-SMTP server functionality necessary.
- For the complete works of Shakespeare: cat
Greylisting mainly relies on this (quote) :
"These applications appear to adopt the "fire-and-forget" methodology. That is, they attempt to send the spam to one or several MX hosts for a domain, but then never attempt a true retry as a real MTA would."
I strongly disagree. A vaste majority of spammers actually use real mail servers like Qmail. Or strange spam-specific software with support for retries.
Apart from Spam Assassin, I'm using OpenBSD built-in "spamd" ip-based filter. A quick look at the spamd log files shows that the same spammers retry over and over, usually during 7 days.
What I like in Greylistings is that it actually prioritizes mails. A mail coming from a known source will be processed before a mail coming from an unknown source (that will have to wait for the next try) . Not really an antispam feature, but still nice to have.
{{.sig}}
If every time you email a colleague for the first time (or for the first time after 30 days or whatever the timeout is) it takes an hour, that makes many, many types of communication impractical.
Example: Someone gives you his business card and wants to hire you. You send him your resume - it takes an hour. He replies with a job description - it takes an hour. By then you can email instantly, but you've both given up in frustration by that point! Fax or ftp would be faster.
sulli
RTFJ.
All of you naysayers out there (I'd be one too if I said it but I won't, read on to find out why) are making a terrible, terrible assumption: that every mail system admin out there will jump on the greylisting bandwagon and implement this.
Back in reality, a lot less than 0.01% will actually implement this technique, especially after reading this thread. So, it's a non-issue. Greylisting is dead.
Must-not-watch TV!
Would you be willing to share a pre-print of your paper, or at least tell me the reference? My research focus is on e-mail classification and I try to keep up with what everyone's doing in the area.
If you'd rather talk privately, my e-mail address is public.
the sender level.
Like, say, putting a maximum limit on the number and size of e-mails that can be sent out a day.
Gather studies on how many people send 1 to n e-mails a day, and how many people send out e-mails of 1 byte to n bytes in size.
My guess is, it's a pretty distorted curve, with maybe a few thousand people -- of all those online -- sending out millions of e-mails a day. The maximum most "normal" people will send a day is probably 100 (and that's a large over-estimate).
social sciences can never use experience to verify their statemen
Well maybe I'm wrong but it would be nice if there was a subscription based email system done by a large company or even the government, then we could be guaranteed that the person that is sending to us is not trying to hide themselves and could be accessible for prosecution just like telephone spamming.
But of course this opens up a whole new slew of privacy arguments... but, for that I would say that this is one place where I would not mind. It's done good for my phone anyways, I haven't gotten any salespeople callong fo a long while now.
Seriously folks, the DHD has huge assets and the ability to get away with shit left and right. Let them spend a few billion on finding the actual spammers, arresting their asses with the PATRIOT Act and holding them for a few months without charges and let them get sodomized by some nice gentelmen at Guantanamo Bay. That should decrease the urge to spam or program spamming tools.
It's not stupid. It's advanced.
Sorry for the typo there, guess it's a good thing I included a link :-)
Here are my latest thoughts on winning the spam war.
:]
I've submitted it to Slashdot. They rejected it. Tell me what you think. I'd like reactive approaches to get discussed a bit more. If you do too, submit this to Slashdot
Veronica: That was "Snowball."
Dante: Why do you call him that?
Veronica: Sylvan made it up, it's a -- blowjob thing.
Dante: What do you mean?
Veronica: After he gets a blowjob, he likes to have it spit back in his mouth while kissing -- it's called 'snowballing.'
...which it couldn't, IMHO, due to the ... tenacity...of the spam community, I still think people would take issues with the proposed "1 hour delay". There are plenty of times when a company will send you an e-mail while you're on the phone with them (for example, RMA requests for damaged equipment), or perhaps you've forgotten the password to a particular news site and need to have it sent to you. Having to wait an hour for something that used to be near instantaneous is a less than ideal solution. My vote's for an overhaul of STMP. We'll be better off in the long run.
When their numbers dwindled from 50 to 8, the dwarves began to suspect Hungry.
Pardon me for being clueless, but I don't see in this concept a description of what happens when a posting from a listserv or other e-mail list goes to a new subscriber. Mail bounces back to the listserv, right?
Well, the first e-mail to a news subscriber is often the e-mail required to confirm subscription. No reply, and the subscriber is plonked.
Sounds suboptimal to me.
So, you whitelist the listserv machines... until one of them has to change IP addresses. Whoops! No umpteen bazillions of e-mail messages go no where.
I'm sure that listserv admin would find the idea suboptimal about this time.
Nice idea, but No Slack, as far as I can see.
There is nothing wrong with yr Internet. Do not attempt to adjust the picture. We are controlling the transmission - NSA
My impression is that spammers don't send mail through ISP relays; they send it directly from the originating machine to target mail servers. Unless ISPs want to play nasty tricks such as firewalling port 25, they can't control how much mail their users send.
Aside from the obvious of getting the authorities to crack down on the existing illegal activities (relay hijacking, violation of TOS of ISPs, header forging, etc.) which is the only true solution, I think there are much better approaches than this "greylisting" method.
The problem with the greylist method is it still slows down mail service, and potentially more than the relay blacklist features. The objective here is that end-user/networks should not be penalized in the fight against spam. We already waste too many resources, and according to my latest mail server stats, more than 65% of our inbound mail is UCE. I'm fed up with more than half my e-mail bandwidth being crap my users didn't request so more resource allocation on a local level in the fight against spam is counterproductive!
Here's a very clever, much more practical method I cound recently.
A company is Canada has set up what it calls SORBS: Spam and Open Relay Blocking System.
What's different from their blacklist is that they maintain "honeypots" strategically located around the Internet. These are servers they specifically set up as inbound mail relays, but never for legitimate purposes. If the servers get [select] mail activity, it's assumed to not be legitimate and it flags the source as a potential spammer... it makes a lot of sense. You create a domain name, but don't promote it in any legitimate manner, and/or you seed spam lists with these e-mail addresses and then let the spammers send to your key systems around the internet and *bam*, they're identified in real time, and then added to a blacklist.
I really like this idea. Like any other system, it has the potential for abuse but the beauty is the identity of the honeypot systems is kept secret, so it's very difficult for anyone other than spammers to exploit the network.
The entire system seems to revolve around the assumption that spam programs are being used, and they are only attempting to send the mail once. But don't spammers still take over open relays and send their mail though those? If so, those relays would hold the spam and continue to retry sending it, thus passing by this filter.
But if you were to combine this program with a well-maintaned blacklist, it would probably cut down on a great deal of spam.
All in all, I think this guy has the right idea. Spam must be fought at the server level.
-R
Comment removed based on user account deletion
Is the 1.5% difference worth you paying for the bandwidth and CPU cycles it takes to identify those spam emails on your client, or would you rather make the spammer waste hours/days trying to get a single piece of email to you without costing you much more than a "Try again later" message?
...and that's the way the cookie crumbles.
Personally I think this is the most hopeful solution out there for real world email use.
http://assp.sourceforge.net/
It auto whitelists people you send mail to and uses bayesian filtering for parsing mail from uknown senders.
# Korea and Japan pretty much tied at +15,000 each
# Japan, at just under 10,000
Don't forget about Japan!
Let us assume that the timeout is 't'. The spammer can send 'x' emails in time 't'. With greylisting a spammer can now only send 'x' emails in time '2t'. It's that fucking simple.
-- HG Pennypacker, wealthy industrialist and philanthropist
If the spammer receives a temporary failure, this confirms that the address really exists, right?
Open-source crypto works because the secret isn't the algorithm, it's the keys. In this case, the secret is the algorithm. The entire scheme can be circumvented by someone who knows how it works.
how did you japanese text to work?
äãå©æããoeã
cool it works again.
åãã!
of a guy spamming slashdot with his "published" spam-blocking paper.
If publishing papers were as easy as putting them up on your webpage, I would have gotten my PhD like umm a decade ago.
Ok, back to writing thesis and 'em darn journal papers.
What would happen if you combined the temp fail implementation with a subscription token sent transparantly from the smtp server. If there is a real person on the other side, the token is quickly replied so communication can occur. If you would then give this token a certain cost (number of bytes), depending on maybe domain name or spammyness of the sendername, wouldn't it be possible to avoid spam at the cost of a 'handshake' mail?
There have been legal noises made about whether spam is actually illegal - the consensus seems to be that it should be (for some suitable definition of spam), but it'd be difficult to prosecute anyone for it in most places without new laws.
On the other hand, knowingly obtaining unauthorised access to a computer system (e.g. with a trojan) is unquestionably illegal, so if spammers are taking a risk like that, they're getting pretty desparate. A spammer who does this and gets caught will *not* be in a good situation.
Part of the plan is to cost spammers more money, driving more of the smaller ones out of the market. Once the spammer market consolidates enough that only a few mechanisms are effective, those individuals will be easier to attack, pickaxe, and make an example of.
Having your car up on blocks when all the other neighbors are doing it is one thing. When you're tho only one, your neighbors suddenly start focusing a bit.
I'm kidding. A little.
I forget what 8 was for.
I think the spirit of that RFC would require that all traffic being used to send spam would have the evil bit set already. Clearly there hasn't been enough uptake of this feature yet, or the spam problem would have solved itself by now.
proof, n. A demonstration that a conclusion is implied by certain premises and axioms.
But Virus/Trojan Relays are a lower risk, because they're less likely to implement a full-scale correct SMTP, especially because spammers know that large numbers of their addresses will be bogus. Much more likely that they simply try relaying once, though if they do report success/failure back to the spammer's master machine, perhaps the spammer can try again.
Virus proxies could be more of a risk, if they're giving their users full capabilities to pass packets across - does anybody know how much of this kind of malware is relay vs. proxy?
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
(Sorry, but it was _such_ an obvious parallel...)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
...where do you think this retrying will lead? More bandwidth. If the spammers don't use the fire-and-forget method, or say they are using a perfectly valid mail server as some kind of proxy (not just open relays - think inter-network ISP mail servers open to all their customers), then the retry will eventually succeed. This means that now you're using up more than twice the same bandwidth for sending spam than is now taken up. Yeah, perfect solution.
For years i've been saying SMTP Sucks and we should develop a new protocol, and nobody wants to go to the effort. It's like the losers on AOL who won't give up their shitty service because they're scared they'll fucking lose that ONE email address and then their world will end without the 50+ lesbian midget donkey screwing porno spams each day to an address that should have been retired long ago. The truth is that it's quite easy to switch addresses. Everybody's just too lazy to take the effort to create a system which isn't so easily abused.
First of all, we need to make our personal messages STAY personal and junk mail take a different route. If a business asks you for an email address, don't give it to them. If they need it for confirmation of some shitty service, tell them to take a hike. I know nobody wants to change the system but whining and moaning and giving people $50,000 fines won't fix the god damn problem either. Considering the fact that it's easier to track down a murderer than a good spammer and the fact that even under penalty of death people still murder other people (quite ironic i think - tell someone "don't do this or we'll do it to you! doesn't apply to us if we warned ya first!"), it doesn't seem likely that a lesser charge which is harder to enforce will do much good. But I digress. A simple client-side system which would make spams a little less prevalent (and won't require changes to the system - hurrah for lazy bastards) would be to fit an email client with a setting that checks an email based on a similar "graylisting" filter and have it delete emails within 6 or 12 hours if it came from a user or domain or something the person had never gotten an email from before. That would only make spammers use "support@microsoft.com" for every spam or something but then you could make the filter match on outgoing emails to see if this incoming address had ever been emailed, but then they'd just find some address everyoen emails or find your address book or something, etc etc. It doesn't really end with all these hacks. You need a whole new god damn wheel because you can only patch it so many times before the wheel becomes one giant fix-er-uper.
Another idea (i'm brainstorming) would be a filter on the MTA which says "if i'm receiving an email from the same #_INSERT_UNIQUE_IDENTIFIER_HERE_# more than x times now, put it into a flagged database for analysis and if it turns out that this identifier is a popular place for these many emails to be coming from for one single account (or the same message to all the accounts) then send it to the admin who then puts an X next to it signifying this new identifier should be matched and blocked for all incoming emails".
And yet another idea is "make all email unique". That is to say, make it a service and put in a web of trust. Make it so every user of email has to be given a unique identifier by some organization which can only be given out by people deemed cool by this organization and when the MTA recieves the mail it does some algorithm on the identifier to authenticate it, like with SHTTP. Ok thats a bad idea I admit. But the web of trust I think could work. If you wanted one person and one person only to have your email address, you give them a business card or address or whatever that they need to go to on the web. They fill out a "join my web of trust" form for the user they wanna send email to. This is all loosely based on friendster btw. The person who gave out this address recieves a request to join from user X. They know they gave that person their info and they accept them. N
In my other posting, I'd put in an unquoted joke about and it was interpreted as a broken piece of HTML and deleted
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
For most other email business applications, the first contact isn't all that time-critical - you might not even be in the same timezone or working at the same time - and most of the exceptions are knowable in advance, e.g. email to abuse@ and support@ where you'd obviously turn off greylisting.
There are exceptions - you're on the road using dialup, or you're having a conference call and somebody needs to send some document you're working on. They may be annoying enough that you wouldn't use it, but for most companies, many of the companies you'd be in that situation with are Usual Suspects that you'd whitelist anyway. And besides, your email staff can do it in all the free time they have because your spam got reduced by a few percent :-) Yeah, ok, this also means there'd be increased forgery of email purporting to be from corporate sites, but really, who'd send out email claiming to be from support@microsoft.com - nobody'd believe that :-)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Which is why I think SPEWS rocks.
Part of the pain is that ISPs have their IPA ranges listed more and more the longer the spammer stays. This causes pain and suffering on an ISP that is too clueless to respond to complaints or is in cahoots with the spammer. SPEWS attitude seems to be "As long as you take the spammers money, we don't want your traffic."
Personally, I think it's time and past to do more than just block the spammy ISP's mail. Time to block EVERYTHING from them.
For some insight into just how fast a major ISP can kick a spammer when it wants to,, see this thread in News.Admin.Net-Abuse.Email.
Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves.
If you had read the original paper, you'd know that possible spammers' reaction and adaption strategies have been well taken into account.
That's not a problem. It will simply take longer to deliver the message. The error isn't "No such user", it's a temporary failure. The list server will simply try again later. The timeout for responding to ML subscription confirmations is typically in excess of 24 hours, that is more than long enough.
I may be wrong, but wouldn't Greylisting affect automated emails sent by e.g. PHP scripts? The major sites use emails to remind for passwords or send confirmations. Those systems, too, work with a "fire and forget" premise. If you start greylisting, you could render - for an example - slashdot subscription useless, because the user will never get the confirmation mail ...
Correct me if I'm wrong
Screw the FSM - Real geeks believe in the Invisible Pink Unicorn
Reading the code raises the suspicion that the author is usually a C-coder.
Someone should rewrite this since it's pretty tough to follow. Goto-statements are extremely rarely necessary in Perl. Use subroutines or "ordinary" flowcontrol.
If this is going to be used it needs to be maintainable. Because of this it would likely be wise to reimplement it as module(s). Greater abstraction would perhaps add a lot more readability and make further development easier.
Fight for your digital freedom, join the EFF *now*: http://www.eff.org/support/
Why is it that firewalling port 25 would be a nasty trick?
social sciences can never use experience to verify their statemen
I was alerted to this article by a post on the Declude Junkmail list. My response to that list included an alternative implementation of the delay-before-receipt methodology that avoids some of the pit-falls.
The premise of my response seems to be well covered: specifically that the spammers can easily adapt to "Greylisting" as it is described and that the resulting adaptation is worse than the original problem... Having said that I have an alternative to offer from our ongoing research.
Here is part of that response/description:
---
I've considered a similar protocol that would force the required complexity on the spammers but since it would require broad deployment to be effective the methodology has been shelved (at least for now). Here is a short description of the IRRQ protocol. (IRRQ = Intelligent Retry Request).
IRRQ adjusts the SMTP protocol by enforcing a lightweight authentication (automated challenge) for new senders to an MTA. IRRQ is still in consideration for later deployment as part of a COT protocol suite in SortMonster. (COT = Circle Of Trust).
1. A new sender (SMTP SENDER + SOURCE MTA) attempts to deliver a message.
2. The new sender is not in the local COT reference so their message is initially rejected with a temporary failure. However the temporary failure messages is modified to include an authentication code (in the form of a special email address) for future retries of this message. The authentication code is a secure one-time-pad.
3. The sending MTA stores the authentication code with the message to be retried. The receiving MTA stores the authentication message and envelope information with an expiration time and a guard time.
4. The receiving MTA may perform other operations to automatically or manually white-list the new sender. For example, in a COT model the peers in the local COT might be queried for a rating of the sender or an acceptance policy. As a result any sending MTAs that are not implementing the protocol can be accommodated after the delay dependent upon the policies of the receiving MTA and the services (such as COT, RBLs, etc.) at it's disposal.
5. The sending MTA retries the message after a given guard time (established by policies, but typically between 15 and 120 minutes). In this retry the authentication code is added to the envelope and sent as the first recipient.
6. The receiving MTA recognizes the authentication code and retrieves it's stored copy of the envelope for verification. The sending MTA completes the envelope and if it matches the stored version the message is accepted normally.
7. The authentication message is retired and may be stored for a period in order to detect hack attempts. A particular authentication code can only be used once legitimately.
NOTE 1: Sending MTAs that do not implement IRRQ are not effected by the adjustments to the protocol and may still be accepted based on policy decisions evaluated in step 4.
NOTE 2: Attempts to hack IRRQ by sending falsified authentication codes, reusing codes, or altering the envelope associated with the code will result in strong negative ratings within a COT.
NOTE 3: This methodology strongly biases the mail system against spammers by forcing legitimate senders to properly implement the retry protocol. Spammers typically use systems which transmit messages (on the fly) without regard to bounces or other response messages. Simple re-transmission counters to the IRRQ protocol will not be effective.
I like this very much. The concept of graylisting/tempfailing is so simple that one wonders why it wasn't thought of and widely implemented earlier!
It would work very well when combined with Bayesian and tarpitting techniques.
Most email servers have a few well-known addresses that would have graylisting turned off, so that their mail could be received immediately: abuse@, support@, sales@, and so on. These addresses typically are read by better software that splits the inbox load up among multiple people on a team, so that no one person has to bear the burden of reading and dealing with it all.
Because these addresses have no graylisting, they would receive all messages, including spam. Bayesian filters could then be applied. Whenever someone reads incoming mail, they could decide whether or not it is spam. Eventually, the software will be able to make good Bayesian guesses. The person will be free to concentrate first on the messages that are determined not to be spam.
When this is done, the Bayesian score of each incoming message can then be calculated. Combined with tarpitting, this would be a very good thing to apply to incoming mailboxes. After the DATA command is received by SMTP, the Bayesian score of the message could be calculated, based on its contents. If a message has a high probability of being spam, an intentional delay could be inserted before the SMTP server returns a success code to the sender! If the client software disconnects earlier, the mail would be treated as not sent, because the SMTP transaction was not yet complete. This would force senders of what looks like spam to wait a while before sending each message, perhaps with timeouts up to 30 seconds. This would greatly cut down on their ability to quickly send many messages.
So, there's 3 layers here:
1) Graylisting/tempfailing of all email boxes that are read by an ISP's end users. The actual email contents are never read by people at the ISP.
2) Bayesian sorting of all incoming mail that is read by the ISP (abuse@, support@, and so on). This builds up a good Bayesian database of incoming email.
3) Tarpitting incoming SMTP connections when incoming mail is determined to most likely be spam, by using the Bayesian score. This would be applied to all email boxes.
Note that nowhere in any of these layers, incoming email is refused! These systems are great, because there wouldn't be the worry of accidentally refusing an important email because it was misclassified as spam.
I'd like to be a customer of an ISP that applies these three layers to incoming email....
Dr. Demento On The 'Net!
Most common single-server mail transports can sustain ~10k-100k deliveries per hour under ideal conditions, with this delivery rate frequently saturating available bandwidth. Issues such as MX and DNS resolution become significant at these volumes. Thus, 100m mails is 1,000 server-hours of time.
Sending more mail requires multiple servers and mulitple pipes. Both of these are resources which are only available to the spamhaus at additional cost or reduced control.
The mitigating issue is that multiple drops (cramming hundreds or thousands of local deliveries to a receiving MTA at once) can reduce the total outbound time. Again, anything that reduces this capability (allowing, say, no more than 10 local deliveries on a single connection) increases the spamhaus's need for resources: servers, time, or specialized software.
See:
What part of "gestalt" don't you understand?
Yes, when the phone company hosed the PREPnet SMDS cloud back in the 90s and the BGP routes in the Internet core started flapping around like Harpo Marx with a wet fish. Some kind soul on the outside noticed, and sent a note to my non-PREPnet address so that I could report the problem to the router weenies and get the Commonwealth of Pennsylvania restored to the Internet.
Dear Ghod, I hope you are not in charge of anything important....
Ah, I see, you are recommending a challenge/response system. You're right, that's reasonably easy to implement; I will certainly consider it.
Thanks for the suggestion!