Slashdot Mirror
Novel Method for Universal Email Authentication
MKaplan writes "Most spam is sent using spoofed domains. Email authentication schemes such as SPF attempt to foil spoofing by having domain administrators publish a list of their approved outgoing mail servers. SPF is sharply limited by incomplete domain participation and failure to authenticate forwarded email. A paper describes a novel method to rapidly generate a near-perfect global SPF database independent of the participation of domain administrators. A single email from an unauthenticated domain is bounced and then resent — this previously unauthenticated domain and the server listed in the return path of the resent bounce are entered into a globally accessible database. All future emails sent from this domain via this server will be authenticated after checking this new database. Mechanisms to authenticate forwarded email and to nullify subversion of this anti-spam system are also described."
Mail servers are authenticated by Spamcop and forward spam automatically to Spamcop which adds it to their database. When using reject_rbl_client bl.spamcop.net SPAM is blocked.
Works like a charm!
So what happens when you receive an email from a big site like Sympatico, Hotmail, or any number of other places that have farms of SMTP servers, where your message isn't guaranteed to be resent from the same IP?
This also requires users to install software to use effectively, and features CAPTCHAs which are a usability nightmare and not nearly as impregnable as the author thinks.
All that effort instead of just adding a TXT record to their domains.
No, not at all. If you don't want to read the article, just keep guessing how it works, and we'll let you know if you are getting warm.
The author may be an anti-spam kook but the paper is so badly written I can't be bothered identifying which.
...but this had to be posted.
Your post advocates a
(X) technical ( ) legislative ( ) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)
( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
(X) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business
Specifically, your plan fails to account for
( ) Laws expressly prohibiting it
(X) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
(X) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook
and the following philosophical objections may also apply:
(X) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
(X) Blacklists suck
(X) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
(X) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough
Furthermore, this is what I think about you:
(X) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!
Then he talks about having people install software:
Yeah, installing new software is a great solution.
The proposed scheme ignores one thing: the majority of bounce messages today are false bounces caused by spammer joe-jobs, therefore they themselves get flagged as spam and deleted/ignored. In addition, it also increases the annoyance of greylist authentication schemes, since a spammer forging my address in the From field will cause every host participating in this scheme to send me a verification e-mail for a message I didn't send which I'll have to deal with. The proposed scheme makes a very fundamental mistake: assuming that you can trust the sender's address in a message to be the true sender's address. You can do that only after you've determined the message is authentic and not spam, at which point you don't need this scheme anymore.
Your post advocates a
(*) technical ( ) legislative ( ) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)
( ) Spammers can easily use it to harvest email addresses
(*) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
(*) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
(*) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business
Specifically, your plan fails to account for
( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
(*) Open relays in foreign countries
(*) Features in MTA software that can be disabled, such as MDNs
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
(*) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
(*) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
(*) Dishonesty on the part of spammers themselves
(*) Bandwidth costs that are unaffected by client filtering
( ) Outlook
and the following philosophical objections may also apply:
(*) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
(*) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough
Furthermore, this is what I think about you:
(*) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!
I didn't spend too much time looking through the options, so go easy if I got it wrong. Will that do?
Resistance is futile. Reactance buggers it up.
Which is kind of like greylisting. The FIRST problem is that the spammers have adapted to this and retry.
The SECOND problem with this is he's saying:
Huh? So this is also about SENDING email?
And it doesn't address the issue of "fast flux" where the domains are "legit" in that they exist and point to the IP address of the sending machine
So he's talking about "bouncing" messages
No fucking way is this going to work.
Is MS windows boxes that are comprised and doing this - you can see this where the spam mails get 'chinese whispered from one box to another and end up incoherent (to say the least).
Any ISP should/could get suspicious of thousands of mails sent from one 'home user' source at anytime. But when you have thousands of 'users' doing the same thing, it gets lost in the noise.
One simple solution is:
if account == home user & running MS
if mails sent > 10 per minute
block it
fi
fi
etc.
Very easy.
"SPF is sharply limited by incomplete domain participation"
That's not a big problem. 99% of non-participating domains fit in default SPF record "a/24 mx/24 ptr -all", we use it in qmail for few years. Together with Spamassassin it results in 99,8% antispam accuracy (warning: one big exception is yahoo.com, you should use domainkeys or add ptr:yahoo.com to default spf rule)
A Google search revealed this intelligent discussion of the scheme.
http://www1.ietf.org/mail-archive/web/asrg/current/msg12403.html
As I stood at a kiosk at a trade show this week, and waded through my spam-filled email on a few services (work email, hotmail, and gmail), the young woman at the kiosk next to me accessed her myspace and facebook accounts and responded to friends only.
She turned and said that only old people use email. And she was a VENDOR at the conference.... Things that make you go hmmmmmmmm......
The key is to reject the obvious nonsense before invoking your cpu-intensive analysis. I reject on the order of 90+% of everything that my mail server sees (even more at the last place I worked where they were using the same system). False positives on my home mail server are near 0. The ones that are mistakenly flagged, are simply flagged as spam, so I still see them, they weren't rejected or discarded. More at work got through, but that is because we have to be more conservative due to not having a good way to do bayesian filtering for individuals (I left before I had the time to run that project with the internal mail admins).
Then call spamassassin on anything that is left (SA will increase/decreas scores based again on RBLs that we don't outright reject, SPF records, etc):
This is just an additional layer over automatic whitelisting of addresses using tagged responses.
Some years ago I set up for my family a pretty simple set of procmail rules and scripts that bounced messages that hadn't otherwise been classified as spam or been whitelisted with requests that they be resent with a certain keyword in the subject line. For example:
"Hello, you just sent me the following message. Could you send me the message again with the word 'leisure' in the subject line? You can reply to this message if you like, just be sure to add 'leisure' to the subject line."
Over a period of several years the only spam that's gotten through this has been from a 419er.
The advantage of a subject line token like this is that you can tell people the token to use, or put the token in the subject line when you send the message so it's usually there when the recipient replies.
Whether you take the resulting message and whitelist the sender address, or some other information in the header that you consider reasonable, that's up to you. It's not really the same thing as the SPF database, though, even if you choose to make the same kind of information the key you use for whitelisting. The point of SPF is that it's supposed to be authoritative for the organizations involved, and doesn't include things like "I sent something with my work address from Earthlink and now you're accepting mail from my work domain through Earthlink's servers".
And using this to whitelist the sender rather than their whole domain gives you a lot finer control.
Nope. We need a solution involving cruise missiles though bedroom windows late at night.
We need Spam Assasin Ninjas clad in impregable black carbon-fibre capes with the knives of cutting edge technology and the deadly intent of artificial intelligence enhanced mania.
We need mountains of spammer bodies piled high on the forefront of technological .
We need chain gangs of spammers publicly televised chanting "The Only Good Spammer is a dead Spammer" to the sound of hammers hitting rocks.
IN Summary: Cruel and inhuman tortue is not enough for these guys
Sent from my ASR33 using ASCII
This is clearly Challenge/Response with automated whitelisting. The following Wikipedia entry addresses every facet of this system:
http://en.wikipedia.org/wiki/Challenge-response_spam_filteringSo? They don't care. They have, effectively, limitless bandwidth and limitless processor power.
Greylisting WAS effective
No. It fails when they implement (as they have) a process to resend any temp rejections after X minutes.
Greylisting had THREE features:
#1. It could temp reject spam and if the spammer never tried again
#2. It could temp reject spam and if the spammer randomized the "From:" username/domain
#3. It could temp reject spam and if the IP addresses was listed in a blacklist within the temp reject time frame
Now all that is left is #3. It costs the spammers NOTHING to upgrade the zombies. And if they get the spam through, the spammer wins.
Now, the zombie can appear MORE legit than a lot of the real mail servers out there.
I don't know, I didn't get that far. The article and the concept is bullshit.
The 'From' field is the keystone of their identification process. Well, I got news for you if you bothered to read the RFC. 'From' does not have to represent the real sender. I can forge it up all I want into anything I want and you can't tell. I didn't get past section 3 where this is before I determined the rest isn't worth reading.
Once again we have another company trying to come up the next Big Thing and they don't know what the hell they are talking about. SPF is cute -- but relies too much on people setting it up and correctly. I suppose you could pay a service to act as a third party validator, but that's turning into a boondoggle too.
I don't think bouncing email at valid senders is going to win any friends.
Perhaps there is a way to do it successfully and with great accuracy. I would love to say I'm working on it. But quite frankly, if I do figure it out I probably won't mention to anyone since I really don't want the legal hassle of trying to defend my idea against someone else's billions. I can block spam. I can block spam to the tune of 99+%. The rest is trivial. I was even surprised to hear them say 94% was the average. Perhaps people would be better off if they stopped using SpamAssassin.
Sorry, my opinion is that statistical filtering is more than sufficient if it's managed well. I think few people are willing to do the work required of them to make them spam free. Kind of like locking the door to keep out the crooks.
How many times have we heard the "this will fix Spam real good" claim? First it was "close those open relays, ye bastards", and lo, that worked for about a week. Then it was "Well, we'll just keep these black lists, and that'll fix things", until of course the complexity of maintaining such lists and the harsh consequences for any poor bastard who somehow found himself the victim of a false positive tried to get himself off said lists. Then there was "We'll just tarpit consumer IPs based upon some nifty string-matching" and the matching "we'll check reverse IPs, and if they don't match, fuck ya!" which of course buggered up all those poor guys using their cable and DSL connections to run small personal mail servers, or anyone with a retarded or miserable provider who refused to alter reverse DNS entries. Then there was "Hey, you don't have an MX record for that IP, so down the shitter ye go!", which nailed anyone who might be sending from sort of a proxy, and didn't want their actual mail servers advertised as such so that they didn't become victims of joe jobs and distributed dictionary attacks. Then there came greylisting, which actually worked for a while, but seriously screwed with "immediate delivery" that all those in the post UUCP world had become accustomed to with email, not to mention the smart spammers learning from the trick and just retrying. SPF was then heralded as the end-all and be-all, but of course has its own problems (particularly with message forwarding, which requires rewriting the header), not to mention that everyone came into compliance with neutral records, so at least the big guys wouldn't jettison mail from their server due to lack of an SPF record.
At the end of the day, you're right. Statistical filtering, with the careful use of all of the above solutions (though I think whitelists/blacklists are as bad as the problem they attempt to solve) is the only way to reliably filter spam. You're never going to catch it all, but the ISP I worked at was catching, by my estimate, about 90% to 95%, which meant that a guy getting about fifty spam a day was down to three or four, and in many cases less than that. It does mean work, there's no solution that doesn't require monitoring, management and tweaking, because the spammers are smart bastards who learn the tricks as fast we can come up with them.
The world's burning. Moped Jesus spotted on I50. Details at 11.
I don't know about the GP, but for me greylisting is very effective. I have a personal domain for my wife and myself. I have a catchall mail address.
Here are some stats for part of last week:
Start Date 23/09/07 04:02
End Date 28/09/07 17:00
5.54 days
Total spam: 4624
Spam blocked with greylisting: 4478 (96.8%)
spam via backup MX: 69 (1.5%)
spam retried (got past greylisting): 77 (1.7%)
Total through to end user: 146
Identified as spam (SpamAssassin): 123 (84.2%)
backup MX marked as spam: 50 (72.5%)
direct marked as spam: 72 (93.5%)
Total to end user not marked as spam: 23 (0.5%)
NB. Up until about a month ago, ~25% of SPAM came via my backup MX, which doesn't have greylisting. I don't know why it dropped, but I'm happy it did.
Ever stop to think