Earthlink Deploying Challenge-Response Anti-Spam System

deliasee writes "The Washington Post reports that Earthlink is preparing to offer new spam filter technology that requires sender authentication. AOL is still concerned that such technologies will put too much burden on consumers." The day after it's deployed, every legitimate mailing list on the planet will get challenges from all the Earthlink subscribers...

Comments

Nice moves

[hendridm]

I was hoping more ISPs would adopt the challenge-response system, like MailBlocks, previously featured on Slashdot. Way to go Earthlink! If I was interested in dialup, this would be a big selling point for me. I'm still waiting for a service that offers the challenge-response feature of MailBlocks but allows me to forward to my existing provider. I mean, a 12MB inbox is pretty lame. There are free providers that can give me that much space...

Re:Nice moves

[apoc.famine]

I dunno. This may be painful for a bit, and increase the amount of mail, but in the long run it might be worthwhile. While I agree that it makes some peoples' jobs harder, those people probably aren't using the major ISPs/mail-services. If the major players do this, it makes it that much less profitable for spammers to do business.

I mean, if you're a spammer, a brute force mailing to joeuser.org is MUCH less profitable than mailing the same million messages to hotmail.com. Go big guys, go! It won't bother me at all.

Re:Nice moves

[d_lesage]

It drives network traffic as well up to the sky

But wouldn't the added traffic be more than compensated by the reduction in traffic that would ensue when the spammers go out of "business"?

Re:Nice moves

[darien]

Er, what?

eMail was not designed for such a challenge

So what? This system works within the standard. Who cares whether or not the designers foresaw it?

It drives network traffic as well up to the sky.

Hardly. If you're on Earthlink and decide to opt-in for this, it simply means that everybody you know has to send you one extra email once. Earthlink's traffic may be a bit higher for the first few days, but once people get their whitelists in order it'll drop back to where it is now - and below, because there'll be less spam floating around.

However, I do hope (the article didn't say) they've come up with a smart solution to the problem of spammers putting real (but stolen) addresses as their From: address. Otherwise people unlucky enough to have their addresses stolen may indeed find their network traffic increases, thanks to a million challenges from Earthlink.

Re:Nice moves

[tacocat]

These systems don't work that well. I have been designing and building my own for about 8 months now and have come to the following conclusions.

They are easily bypassed using a smart enough auto-responder. If all you do is fire back the original message then you're on their list.

They sometimes fail to pick up the human response. I have several cases where people will simply respond to the email, removing enough of the critical content, to render the reply useless. This comes in two flavors. Email clients will strip out the Header information needed, or people will strip out the Body information needed.

To impliment this upon a very large system like this is going to be a nightmare not only for their email administrators, but for everyone that they touch.

One of the biggest problems that these systems have is that they are totally incapable of handling Solicited email from a Bot. Examples include:

• Payment Confirmations (amazon.com)
• mailing list confirmations
• Profile Update Notifications (paypal, ebay..)
• Password changes or resets

It's going to be a pretty ugly system of implimentation.

Re:Nice moves

[phat_joe23]

It drives network traffic as well up to the sky.

Hardly. If you're on Earthlink and decide to opt-in for this, it simply means that everybody you know has to send you one extra email once.

And that every time you get spammed from a new address (read: constantly), the system fires off another confirmation email from you. It effectively doubles the number of network connections spam generates. /joe

Too drastic?

[mao che minh]

Drastic times call for drastic measures. The situation caused by the relentless onslaught of SPAM (which supposedly is rendering "damages" in the billions annually) can certainly be categorized as drastic. Is Earthlink's counter attack too drastic a measure, though?

On one hand it (Earthlink's new "technology") seems reasonable enough to the every-day-joe. I'm sure that the majority of Earthlink subscribers don't utilize news or mailing lists, and don't bother paying their bills online. For these people, it's fine. On the other hand, many others use online banking and other such automated tools (even account control mechanisms for online games will be affected). How quickly will all of these vendors conform to Earthlink's new technology and make the needed changes in their automated systems? Will Earthlink simply render many of these domains exempt?

The answer to solving SPAM resides in the current mechanisms used for the actual transmission and delivery, the mechanisms that all participants must use, not just Earthlink. This is of course the mail servers themselves.

Re:Too drastic?

[iangoldby]

People who want to continue to receive messages from mailing lists, online banking, etc, will have to add these sources to their whitelist.

It's a bit of a faf though, and I suspect many people will either not understand how to, not bother, or forget at least one address.

The solution is to have the incoming messages moved into a 'holding' folder that the recipient can see, and check in just the same way as checking through a 'spam' folder. This would remind the user to add false positives in the 'holding' folder to the whitelist. After a while, you can safely stop checking your 'holding' folder. Wouldn't it be good if this is what Earthlink are doing?

I think a scheme like this could be made to work, at least for webmail. For POP3, it could be a bit more tricky...

Re:Too drastic?

[Binestar]

Too drastic? I don't think so. This is something that is off by default, and needs to be turned on by the user. That user can also pre-approve e-mail addresses from his address book and mailing lists that he is on so that the challange never reaches those people.

This is just an added feature that users can use if they choose to.

As for the automated systems: It is the users responcibility to add those addresses to the accept list when (s)he signs up for the services.

Since this challange responce system has to be turned on by the user, it is only the user's fault if (s)he forgets to whitelist the address of places (s)he gives his e-mail account out to.

All in all it's definately a good option to have, but it's also a good thing that it is off by default, with the option to turn it on left upto the user.

Re:Too drastic?

[letxa2000]

Challenge-Response is bogus. I don't know of any such systems that have been deployed without significant problems for their users, the people that send mail to their users, and especially mailing lists.

If challenge-response is largely deployed, I suspect spammers will just unite such that one spammer sends a message, gets the challenge, answers it and is then "unlocked" to send message. He'll then distribute that email address in real-time to dozens or hundreds of other spammers who will send their spam immediately with the same newly-unlocked address.

Or, perhaps, spammers will change their tactic from spamming millions of users with 1 spam at a time to spamming 1 user at a time with dozens or hundreds of spam. You unlock the system with a valid response to the challenge and then flood them with spam until the user blocks that address.

I just don't see where challenge-response is anything more than a very stopgap measure. It's not particuarly "clean" now and will become more and more useless in the future.

Almost a year after Paul Graham's "A Plan For Spam" Bayesian is still the easiest system to develop as well as the easiest for the user to use. It is extremely effective (99.5%+) with very few false positives and doesn't require any additional effort for the sender and only requires that the user report false positives and false negatives--and that is mostly only needed at the beginning. Once it is initially tuned it's not necessary to do much of anything--it just keeps learning and working.

Re:Too drastic?

[Tackhead]

> People who want to continue to receive messages from mailing lists, online banking, etc, will have to add these sources to their whitelist.

Problem is, you don't know what that email is necessarily going to be.

I ordered something from foo.com and got order number 12345.

A few seconds later, I got a confirmation mail from confirm-12345@foo.com telling me what I bought and when to expect delivery. (Or worse, from order-12345@foo.com telling me there was a problem, and that I needed to fix something!)

If challenge-response becomes widespread, foo.com will say "Now you must whitelist the address confirm-12345@foo.com" when processing the order. (Or switch their order-processing back-end software to use something more sane, like "confirm@foo.com" and put the damn "Order 12345" in the Subject: header where it belongs!)

Problem is, until then, some vendors and some users using challenge-response are gonna be up the proverbial estuary without a utensil for propulsion.

If foo.com is disreputable, of course, challenge-response solves the donkey pr0n spam problem, but not the mainsleaze part of the spam problem. A mainsleazer at foo.com will simply start spamming his customer list with a From: of "confirm@foo.com" - Subject: "New Dealz from foo.com!" *sigh*)

Re:Too drastic?

[letxa2000]

I don't know why more people/ISPs aren't using this. This system seems to be the most effective because it doesn't have silly little measures

I agree. It's so simple yet so effective. It really makes me wonder why people invest time and money in silly, less-friendly and potentially less-effective solutions such as C/R.

it seems to rate the spam based on its content, which no spammer can get around.

They're starting to try. When they start breaking up words so that "cock" is "c.o.c.k" they're making an effort to avoid filters, but also are addressing the Bayesian filters since that will normally get broken up into 4 tokens, one for each letter. Of course, if they do it enough then a single token "c" might actually become a commomn characteristic of spam for that user.

Anyway, Bayesian works great now. I think spammers will evolve to deal with it, but all that is necessary is to implement new token-identifying logic in the Bayesian filter... the Bayesian approach itself is very solid.

It's a hell of a lot faster to do than actually placing calls to people and talking to them, and people

I agree. I suspect you will see spammers actually analyzing the C/R responses. If it's something the software has seen before and is capable of responding automatically, it will. Those that it can't will be forwarded to someone to quickly deal with it. If some of the megaspammers make as much as they supposedly do, hiring a teenage kid at $6/hr to spend the day answering C/R responses is not a huge investment.

Re:Too drastic?

[lommer]

Yes, I think the Earthlink measure is FAR too drastic, and whitelisting (with a holding folder), while it does solve many problems, is very inconvenient.

I am currently in the process of applying to universities as I am graduating this year. Many universities contact me by email. If I miss ONE important email from these universities, I am in danger of losing my application. Further, some emails that the universities send me are time sensitive, so that mandates checking my holding folder daily. Finally, many universities use auto-mailers to send out announcements and such that have an invalid return address, so confirmation emails don't have a hope in hell of getting through.

Combine all of this with the fact that many people at a university, with many different email address (sometimes in different domains even) may have to deal with my file and you can see my problem. Spam needs to be stopped at the source, not at my inbox because the consequences of even one false positive are just too high for me. Yes, this will mean that legislative measures will be required, not just technical measures. I realize that many slashdotters are not in favour of this, but this is the only way the spam problem will be solved IMHO.

Re:Too drastic?

[BlackHawk-666]

TMDA utlises shortlived email addresses for this purpose. It will create an email alias that anyone can send to...but just for x (5 for example) days. Give this to the company as you sign up and you will receive their confirmations. You can either leave it like that and then 5 days later they can't spam you, or whitelist them and give them your permanent address.

Re:Too drastic?

[creideiki]

It's a bit of a faf though, and I suspect many people will either not understand how to, not bother, or forget at least one address.

Agreed. I think the optimal solution to allow for independently certified e-mail. Certification authorities would raise the bar (by requiring REAL forms of ID) for getting a user id which would need to map to a public key. Normal users could have this taken care of by their ISP, after all, they know who's paying for the service. This id would be guaranteed by the certification authority to map to a person or business, though, to protect privacy, no personal information would be stored - only for creating an ID hash.

Recipients should be able to file a complaint once per message per sender. The rating of a person or business would be cumulative (though possibly normalizing toward zero over time as old ratings "drop off"), recipients could just set a maximum evil amount or whitelist specific ids/keys that'd otherwise be considered too evil. This makes it very easy for recipients as they don't have to do much work and they can still recieve mailings that they just signed up for.

If a spammer or some other malicious type sends out a million messages and everyone complains, he'd have to wait until his rating normalized before he could reasonably expect people to recieve his messages again. Additionally, due to the requirements of proving who you are before getting an address, one couldn't just create another account (which also has the side-effect of ruining his other business ventures or his personal life as his only recourses would be a legal name change for himself or his business, or using non-certified e-mail).

Just my two cents, but I firmly believe that it's the ease of getting an e-mail address and the vunerability of implicit trust that allow spam to be rampant. Phone companies just don't give out numbers, a similar model for e-mail would be beneficial (though it would require the collaboration of ISPs and possibly independent certification authorities). Furthermore, spam is a technical problem and needs a technical solution not a legal one.

Re:Too drastic?

[letxa2000]

ISP's don't use it because it massively increases the load on their mail servers,

I've recently implemented my own Bayesian system on my server. While my first-cut was very CPU intensive, very straight-forward techniques can be made to make it extremely CPU-friendly. In fact, I'll bet my current Bayesian system is less CPU-intensive than a simple keyword-filter that has 5000 "keywords" in its database.

I don't use SpamAssassin and can't comment on its toll on the CPU, but there is no inherent reason why a Bayesian system can't be deployed by ISPs. About the only drawback I see is that you have to store a corpus for each user and that ends up being between 1MB and 2MB per user. But disk space is cheap...

Re:Too drastic?

[letxa2000]

This is moot anyway since spammers don't actually provide return email addresses.

Oh, I'm sure they'd start using actual return addresses... at yahoo, hotmail, etc. As long as they (the free email accounts) last long enough to collect some challenges that's all they need. Even if the accounts are closed by hotmail you can still send email "from" that account.

C/R doesn't even have a chance of working large-scale while there are free email providers such as Yahoo.

And even if it does, as someone else said, you just start sending spam with email addresses that have a high chance of being whitelisted. orders@amazon.com, orders@cdnow.com. So now instead of sending 1 spam to each user they'll send the same message 100+ different times from different addresses that they have concluded are more likely to be whitelisted in the hopes that one of them actually is whitelisted.

At best, C/R doubles spam traffic by generating a C/R request for each spam sent--now instead of just getting bounces sent to some poor innocent victim, the innocent victim will get bounces plus thousands of C/R requests. At worst, spammers will take the brute-force approach mentioned above of sending hundreds of copies of the same spam to every user using different "common" whitelisted email addresses. Either way the spam problem arguably gets worse, not better.

Re:Too drastic?

[corz]

Almost a year after Paul Graham's "A Plan For Spam" Bayesian is still the easiest system to develop as well as the easiest for the user to use. It is extremely effective (99.5%+) with very few false positives and doesn't require any additional effort for the sender and only requires that the user report false positives and false negatives--and that is mostly only needed at the beginning. Once it is initially tuned it's not necessary to do much of anything--it just keeps learning and working.

Personally I use a combination of SpamAssassin's bayesian abilities along with TMDA, a challenge/response system. I only require confirmation for messages that SpamAssassin identifies as being over my threshold of 5. In my .tmda/filters/incoming file I have the following rule:

pipe "/usr/bin/spamc -c" ok

That means that if SpamAssassin says its clean, then no confirmation is required and TMDA delivers the message to my inbox.

Simple, effective, the best of both worlds.

How do two people with C/R communicate?

[corsec67]

How do two people with challenge and response communicate?
If the challenge always gets thrugh, then the spammer will just issue cahllenges as spam.
If they don't get through, then you would have a nasty mail loop.

Re:How do two people with C/R communicate?

[Nutcase]

very good point. I would mod you up if I could.

You can't have an automated challenge/response system, because that defeats the point.

You can't have a non C/R address for the challenges to be sent to, because it would end up getting spammed.

Basically, there is a no communications barrier in place until they communicate.. which makes no sense.

Re:How do two people with C/R communicate?

[IIEFreeMan]

> How do two people with challenge and response communicate?
> If the challenge always gets thrugh, then the spammer will just issue cahllenges as spam.
> If they don't get through, then you would have a nasty mail loop.

In TMDA (a challenge response system in python) at least, when you send a email to somebody, they don't get a challenge when they answer. It's logical because if you send him an email, you know he will not spam you :)
So i assume earthlink system will act the same.

Re:How do two people with C/R communicate?

[stratjakt]

The way I read it, earthlink, up on recieving an e-mail, sends a challenge to the email sender. If the e-mail sender responds, it delivers the mail.

From the article:

When someone sends an e-mail to a challenge-response user, he or she gets an e-mail back asking to verify that the sender is a live person.

Once the sender does that by replicating a word or picture displayed on the screen, the original e-mail is allowed through. The system automatically recognizes future e-mails from the same sender, so the verification needs only to be performed once. Without the verification, the e-mail is not delivered.

So if earthlink people are on your mailing list, you'll get a challenge next time you send it out. It should only happen once, and from then on, you're email addy is "legit".

It's not like you get 9000000 challenges from everyone on the list. But if every ISP did it, you'd get a challenge from every ISP on the list.

This is the first step towards email being such a pain in the ass, that people just no longer bother using it.

Kiss SMTP and POP3 goodbye.

Re:How do two people with C/R communicate?

[Garion911]

One idea: Any emails you send out, the recpt is automaticly added to the "ok, let through" list.

Re:How do two people with C/R communicate?

[Chester K]

How do two people with challenge and response communicate?

My C/R setup (TMDA) automatically put anyone I send email to on my whitelist; therefore I'd get their challenge message.

Re:How do two people with C/R communicate?

[SomeoneGotMyNick]

The challenge is probably a randomly generated code to be returned before the original e-mail gets sent to the intended recipient.

Most spammers use fake return addresses anyway. The challenge will never arrive and the mail gets tossed. Thus, it never gets to the recipient. Voila, one less potential viagra purchase.

Re:How do two people with C/R communicate?

[esme]

Here's how it works:

1. Alice sends an email to Bob.
2. Bob is automatically added to her access list (b/c she's sending him mail, he's not a spammer).
3. Bob's mail server sends a confirmation request.
4. Alice recieves the confirmation requestand responds.
5. Original message is delivered to Bob.

-Esme

Re:How do two people with C/R communicate?

[1729]

You can't have an automated challenge/response system, because that defeats the point.

That's not true. There is an approach where you show a "proof of computational effort"; that is, your computer spends 10 or so seconds computing the response to a challenge. Here's a paper on the subject.

Re:How do two people with C/R communicate?

[tacocat]

True. But now the mail administrator has to deal with thousands of spam mail that doesn't get a reply.

And how long are they supposed to wait for a response. Remember, email is not supposed to be a Real Time system. Email servers frequently have a delivery retry schedule of about 4 days. That would mean that Earthlink has to carry the entire spam volume of four days in some kind of mail pending queue and to periodically attempt a redelivery.

I've tried this myself. When you can easily run 100+ spams per day per account, imagine what you are going to be dealing with for an entire ISP. You can easily scale into the million email queue.

Their servers will not be able to handle their entire population and the resulting network load on themselves and everyone else will be prohibitive.

Consider this. AOL and HOTMAIL are the largest spam address sources, real or imaginary. So, when they get spam from AOL, they have to attempt a delivery. If AOL's system doesn't allow for immediate failures based on "address unknown" then EarthLink will hit AOL with thousands of bogus email delivery attempts. Now the two goliaths are beating each other to death over bandwidth.

Someone will be suing for a DOS attach.

Re:How do two people with C/R communicate?

[platypus]

And what happens if ReplyTo != From ?

Re:How do two people with C/R communicate?

[inbox]

Hrm... I think that yes, in fact, you do get 9000000 challenges from everybody on the list. The sender's e-mail address is not whitelisted at the Earthlink mail server, it is whitelisted at each e-mail account.

Otherwise, a spammer just sends one message from an address, responds to the challenge, and then spams away.

Or am I misunderstanding it?

Forged Headers

[Anonymous Coward]

I think forged headers are the calamity of the inprocess SMTP transfer mechanism. If we can liberate the dynamic IPs saturated on the IPlanet web matrix, then we could perform 3-way LDAP POP3 authentication with a digital certificate.

The other way this could be accomplished is to triangulate a 801.11b WAP source into an array of POSIX message headers that would reflect the consistency of the mail protocol.

What do you think?

too much hassle

[chabegger]

I think this will create way too much hassle. There are some people who wouldn't mind, but others (such as grandma) who have to be told three times where the power switch is won't really know what is going on. At least now when I don't reply I'll have a decent excuse... "but grandma, you forget to send it twice, so i didn't get it"

Now the spammers get address validation for free

[chefbimbo]

Seriously, what are they thinking? TMDA might seem like a nice idea in theory, in practice, it's a pain to use and not exactly safe either. Once this gets widescale usage, the spammers will simply start responding to the challenges (after all, it's not like that couldn't be easily automated).

Good idea, but...

[onemorehour]

This seems like it might be a good step, but it's missing the point. The only thing that will truly curb spam is to rework the SMTP protocol to not implicitly trust every host, as was mentioned in an earlier /. article.

Michael's comment

[Rev.LoveJoy]

This is true, but perhaps it illustrates an opportunity for developers of mailing list software more than it exposes a flaw in Earthlink's plan to thwart spam?

As a network admin, many of the remote users I support (sales reps, on-the-road types) use Earthlink dial-up while travelling. At times, some of the program's that Earthlink has used to stop people from using their services to spam have make my job harder. However, I do not begrudge Eartlink for these inconviences, at least they, as a major ISP, are doing *something* about this problem.

My two cents,
-- RLJ

Correction

[robbyjo]

every legitimate mailing list on the planet will get challenges from all the Earthlink subscribers

Not exactly right. It happens only for the first time to detect whether the sender is legitimate or not. Quote the article:

The system automatically recognizes future e-mails from the same sender, so the verification needs only to be performed once.

The problem with this system is that the spammer can still spam using legitimate e-mail accounts as a camouflage (or expired e-mail accounts). Once the legitimate e-mail address is procured, the spam still goes on. It is futile, IMHO.

Re:Correction

[Ed Avis]

Spammers seem to be sending a whole bunch of crap from my address (ed@membled.com) even now. At least, I keep seeing what appear to be genuine delivery failure notifications of Russian spam sent from my address. Any system which trusts individual email addresses, without relying on some real authentication such as PGP signatures, is broken.

A simple rule is: Headers can be forged. Don't trust anything in the headers for antispam purposes. This includes the sender and recipient.

Re:Correction

[errxn]

That's why I prefer my anti-spam system, known as "Firing Squad". Use it once, and all spam will stop.

Warning: Infinite loop detected

[Marx_Mrvelous]

Ha! I can just see it... Alice@me.com send and e-mail to Bob@you.com. Bob@ send a challenge to Alice. Alice, never having heard from Bob, send a challenge back to Bob. Either Bob ignores the second e-mail, or sends another challence. Of course, if the e-mail software allows any outgoing e-mail address to reply without challenge, this wouldn't be a problem.

Re:Now the spammers get address validation for fre

[PerlGuru]

the article implies that an image would be part of the response, such as ticketmaster's please type the word in the picture into the box.

Good idea, bad idea.

[numbski]

How to set up SpamAssassin Milter on OSX <- Easily adapted for other platforms. I wrote it.
Squirrel Mail
SpamAssassin Config for Squirrel Mail <- Register Globals must be turned on in php.ini to use this.

Now, that being said, I run an ISP in St. Louis, and spam is a problem, but for the precise reason mentioned on the submission, I can't use a challenge-response system. The reason is that our support staff equals myself plus 1. If I want to answer phone calls all day from people complaining about not being able to get mail from their daily spamming of mailing lists, I best allow all. The problem is that these same people complain about all the spam they get...ugh. The above solution is elegant and leaves the ability to control the filter to the end user via webmail. If they don't like it, set the threshold high and it's 'off'. Been using this for months without a complaint.

Now if you don't use lists, and it's for your own mail server...go for it. That has to be the most effective method available, but not appropriate for wide scale use.

They should offer it with new email address

[dnoyeb]

me@challenge.earthlink.com

something like that. So that it allows users to gradually changeover to the system. That would allow them to be more extreme in their refusal to accept emails and much less compromising.

I like it.

Needs to be 'hard' in some way

[Ed Avis]

Of course it is no good if the spammers can set up automated systems to respond to the challenge. There are only two ways around this:

- Make the challenge 'AI-complete', that is, to give a correct answer you must be a thinking human being and not a computer. But then how can the other end check that the answer is correct? Having humans generate a fixed number of questions and provide sample answers also isn't going to work, since spammers will learn the correct answers. You need a way to generate an unlimited number of questions and to mark the answers automatically, and clearly this can't be done if the questions are intended to be too hard for a computer.

- Make the response computationally burdensome, so a computer can do it but only at the cost of some CPU power (so large bulk mailings would be impractical). This is what Hash Cash and similar systems suggest.

It looks like Earthlink's system will rely on sending pictures you have to look at. Apart from the practical problems of clogging the wires with image files, I worry about OCR potential. The examples of this stuff I've seen on Yahoo, where you have to type in a number shown in a partially 'obscured' image, wouldn't have been difficult to develop OCR software for if you were so minded.

There's also the question of the spammer taking the challenge and sending it out to some other user. That user, by now used to replying to challenges from Earthlink and other addresses, will respond to the question and send the correct answer back to the spammer. D'oh!

Re:Needs to be 'hard' in some way

[Cirvam]

So how do you respond to a challenge if you are just using a terminal or are blind? Obviously if the characters are obscured, the screen reading program can't read it, and they would have to be a graphic of some sort. Unless they just make an alt tag that tells you what it is. :)

Oh great, now spam has its own protocol

[Anonymous Coward]

"...the spam client MUST provide a Accept-Topics: header, where the value is one of 'penis-enlargment', 'make-money-fast', 'repair-credit', or 'any'. The server MUST reply with a Spam-Type: header, specifying the type of spam transferred. In addition, the server MUST respond with a Spam-Encoding: header, where the value is one of the options 'all-caps', 'many-exclamation-points', or 'broken-english'..."

why challenge-response won't work

[X_Bones]

What if I'm registering at eBay or PayPal or some other site which sends an automatically-generated email when I complete the first step? What if I subscribe to a mailing list where I can't get a response from a human to a challenge? What if I'm applying for a job online and the company sends me an email saying they've received my resume, which I will not be able to see?
I think this kind of scheme is only useful when the message sender is human and you know who they are, in which case the system is pointless anyway. What I think we need is to phase in a new, secure version of SMTP where emails aren't relayed unless the sender's ID can be verified.

Re:why challenge-response won't work

[NanoGator]

"What if I'm registering at eBay or PayPal or some other site which sends an automatically-generated email when I complete the first step?"

That's a good point, but the solution is simple: throw-away addresses.

If you are an earthlink subscriber, you get an email address like nanogator@earthlink.net. (Hey, that useta be my address!) Then, Earthlink could provide a service where you create a unique address that expires after x amount of time. so nanogator.dkaf3fj39@earthlink.net becomes active, and that's the one you use. From there, you can add them to your whitelist.

It's a bit round-about, but that's the beauty of Earthlink. They're a major ISP. Surely places like Ebay will have to stand up to comply with the upcoming standard. It'll never happen if some people don't have issues like this.

Fill up the ISP servers

[nuggz]

So when a spammer fires a few hundred or thousand emails to an ISP, they will sit on the mailserver waiting for him to respond.
Since the from address is faked, that same ISP will launch an acknowledgement flood against a third user.
Excellent.

I just see so many tricky things that someone somewhere will screw up.

Re:Fill up the ISP servers

[stratjakt]

The ISP sends only one challenge. You respond once, and henceforth are allowed to send as much as you want.

Now if I wanted to Joe Job some guy, I just pick someone who's chances are good that he's already allowed through earthlink. Say the maintainer of a mailing list with earthlink subscribers.

I've said it before. This is just a step towards making SMTP a pain in the ass, and obsolete. We can look forward to a high tech pay-per-use replacement in the future. Yay! Paying to send e-mail, I cant wait. But at least the two or three spams I get a month will be gone.

Challenge - Response doesn't work

[tshak]

What happens when the customer orders something from Amazon - the purchase confirmation email comes from a non-human address.

Just the other day I got an email from a company that I ordered software from describing a free upgrade that I could download. It came from donotreply@[host].com, meaning, if I was using Earthlink's system I probably wouldn't have received it.

The problem with Challenge - Response is that it makes the assumption that if there's not a human behind the email that it's spam. In practice, there are many legit emails that are not individually sent by a human.

bad protocal: SMTP

[JDizzy]

The answer is not attaching more bad ideas to an already bad protocol. The ultimate answer is in the protocol designers. A government/state can pass as many laws governing the interaction of people/things with the bad protocols, but the IETF/IEEE will still create them, and certify them. People should just wake up and realize that SMTP is to blame for this big mess. ISP's should stop offering SMTP outright, and think of ways to replace it. Chat programs are probably a better way to pass messages anyways. SMTP has become a massive bazaar that is full over everyone on earth, and since it is completely open, its also completely ok to send bulk mail. Forging headers is another issue, but simply spewing email is intrinsically allowed by the protocol, and thus taken advantage of. If everyone one on earth had a computer, and everyone on earth sent email to everyone else on earth every day, would that be spam? No, because it would cross the line into accepted practice, and that is what we are starting to see due to the sheer bulk of spam sent to everyone on a daily basis. The point is that as long as SMTP exists, so will spam. The answer is to replace SMTP with something that doesn't allow spam to exist by removing the ability to anonymously send people messages.

Folks, It's Opt In

[davewill]

The article clearly states that the user turns this on or off. So it seems unlikely that a large number of challenges will start going out. As far as Grandma is concerned, you can add her email address to the OK list yourself so that she never sees a challenge. The only minor problem I see is receiving email from text only people, (Pine, etc..), or portable devices that might not render the bitmap correctly. But it seems a minor complaint, really.

There's a whitelist

[Spittoon]

Jeez people, read the whole article, it's not that long:

The challenge-response system will be optional and free for EarthLink subscribers, Anderson said. It will allow users to automatically clear the e-mail addresses of friends, family members and other associates in their electronic address books, so those people would not receive the challenge e-mail.

That's called a "white list"-- a list of addresses you know are legitimate.

When someone responds to a challenge and you accept their response, they go on your whitelist.

When you turn on this gadget, add your mailing list addresses to your white list. If you suddenly stop getting a list, go find out if they changed their sending address and add it to your white list.

If that's too much of a burden, feel free not to use the service, and go back to complaining about spam.

Wow, nobody understands this!

[MrPerfekt]

I see a slew of people saying "blah blah blah, they'll automate the response blah blah blah". And apparently, to alot of you, this is all new.

This is something that's been around for a few years and gee, spammers haven't gotten around it yet. C/R antispam systems work because spammers don't use valid Reply-to: or To: addresses.

If they did and the spam gets through the system, then great! There's one more point where we can nail them on when/if we go to hunt them down. Oh, you used your dialup with an SMTP server to auto-respond to the challenge (which is probably alot of work for the average evil spammer), great, email abuse@isp and have his account shutdown.

Since I have started using ASK to C/R my email. -zero- spams have gotten in my Inbox (which is what annoyed me the most about spam, the false positive I got when the little sound would ring telling me I had new mail.)

Intrusive? PLEASE! How lazy are you? Hit reply -once- and you'll never have to see it again when sending email to me. I'd say getting pelted with 200 spams a day is slightly more intrusive to me than what you're going to have to do to send an email to me.

How has this problem escaped me?

[Cobralisk]

But spammers have found ways to defeat them and spam accounts for 40 percent of all e-mail

Is this true?

Of all my email accounts, the only one I ever get spam on is my yahoo account, which I set up pretty much to get spam on, since any websites I visit that require registration, I always give them the "spam" address I got for free. I don't even check that email for anything. Human beings are the only recipients of my paid email addresses. I am for measures like this though, because even though I'm not affected directly by spam, increased traffic on the net is bad for everyone.

We need to punish the sensless posting of one's own email address to anonymous sources. These are the same people that give out their address and phone numbers when they buy batteries from radio shack. Use your head, they don't want to know where you live so they can send you a case of scotch. They want to drink your beer, crash on your couch, sleep with your daughter, and have you pay them for the privelege.

You can do this yourself.

[Malcontent]

Take a look at this

Re:You can do this yourself.

[WetCat]

Well, imagine you have no job and selling yourself
You posted the resume, and waiting for emails.
Do you seriously expect that prospective employer will have time to respond to "confirmation" message?

Re:You can do this yourself.

[StarOwl]

I use TMDA to provide a challenge/response mechanism in my antispam filter.

When I first started using TMDA, I had problems with people not understanding the mechanism. My grandmother, for example, complained about "bounces" (how she interpreted the challenges).

So, to avoid those problems, I:

• Actively manage my whitelist. For example, if I needed to send a resume, I would make darned sure that the prospective employer's domain was on the list.
• Use challenge-response only in conjunction with other antispam tools. My system is roughly: if I know it's spam (tagged address known to be in spammers databases), it gets trashed. If spamassassin or spamoracle thing it's spam, I refer to tmda for possible challenge/response. Otherwise, the mail gets delivered.
• Warn people about the system. If I know that someone new is about to send me email, I warn them: "You might get an autoresponse back. If you do, just hit 'reply'."
• Use some care in writing the challenge email. Trying to craft a letter that is understandable to non-geeks wasn't that easy.

I still have the odd piece of spam leak through that process, but it's nowhere near the quantity that's actually sent to me.

The only problem with the scheme: there are some spammers who are dumb enough to not get the hint, and respond to the challenge. They don't seem to realize that their response probably constitutes harassment via 'net, which is a crime in the U.S. (Spammer go to jail. Do not pass go. Do not collect $200.)

Re:You can do this yourself.

[BlackHawk-666]

I also use TMDA and I can tell you it has vastly reduced the amount of spam I receive from approximately 20-30/day to 1 in the last two months. I've never been happier ;-)

Whitelisting is important, and easy too. Just export your address book to a text file and copy the results to your whitelist (which is also text).

It's worth noting that you can also auto-whitelist anyone you send mail to by using their nifty little mail proxy. It sits and proxies for SMTP and adds all outgoing mail automatically to your whitelist, so whoever you sent that resume to will never see a challenge...neat!

P.S. Can't recommend the product enough.

Re:You can do this yourself.

[lobotomy]

Or better yet, what happens when a confirmation message is sent to confirm your confirmation message? Is there any looping message detection built in? Maybe if both sides are using the same program, but this could be disasterous if two users have different challange-response systems that don't know about each other.

Re:You can do this yourself.

[mazor]

Yes, TMDA has loop detection built-in, both for TMDA responses and for other mail agent autoresponses. Mail storms are caused by people who don't follow the RFC standards for mail processing.

-mazor

Re:Now the spammers get address validation for fre

[PerlGuru]

I don't know about earthlink but ticketmaster's sys uses random different patterns obscuring the text. As for the text, the fonts they use vary, size varies, lines are not straight, and most of the fonts look like they are hand written (with even a single letter appearing differently in the same image)

I'd guess there system is pretty effective.

Proper scenario, better way

[phorm]

Nope, more like:

Alice@me.com sends an email to Bob@you.com

Mailing program adds "Bob@you.com" to Alice's list of valid emails (after all, you're not often going to send email to somebody that you don't want responding, right?).

Bob@you.com sends a challenge to Alice@me.com

Alice@me.com accepts the challenge, since she already sent the original email to "Bob" and had him added as an authorized user

Alice authenticates to Bob's system, and all is good


Another way would be to make all "challenge" type emails follow a specific pattern - with little to no allowance for anything other than the challenge. Then, challenges will be accepted as legit without bouncing back-and-forth, and spammers cannot simply send a message as a challenge with extra spamcrap attached - and still cannot send non-challenging email.
Now, an ignorant spammer could send a flood of challenges just to be annoying, but this isn't very profitable as they wouldn't be able to contain penis/viagara/etc ads.

I assume

[ceswiedler]

I assume that the challenge-response is intended for messages already tagged as potential spam. In other words, low-scoring messages (spam-wise) wouldn't get the challenge. I certainly wouldn't expect a perfectly not-spam message to require the CR. Earthlink's (and other) spam-rating systems are pretty good, I think using it for the 'grey-area' emails would work well. And block the obvious spam without hesitation.

One question: shouldn't it be REALLY OBVIOUS to ISPs what is spam and what isn't? It seems that if a nearly-identical message gets sent to a large enough percentage of their users, it's clearly spam. Is this hard to do? Are spammers clever enough to distribute emails to avoid this?

Which planet are you from?

[mccrew]

Education is the way to go for spammers.

Other than using a cow prod or a red hot poker, how on earth do you "educate" a spammer? Send them to Spammer School? Enroll them in self esteem classes? D00d, this is just about the stupidest thing I have heard in in a loooooonnnnnnngggg time.

Perhaps education is the way to go for Slashdot posters...

Sue them if you're richt (read: AOL), complain about them if you're poor (read: everyone else)

Sue them if your rich? Perhaps you can enlighten the techno-elite here how exactly you find a spammer who is sending e-mails with forged headers, connecting through open HTTP proxies? If you're going to sue them, you gotta find 'em first, right?

and be happy if they loose your DSL connection because of you as one guy dig who pissed me of days ago.

Ohhhh great job, kiddie! Sounds like you did a denial of service on some average home user who didn't happen to know that he had an open web proxy server. Whoo hoo! You da man!

Re:Which planet are you from?

[shawn.fox]

how on earth do you "educate" a spammer?

Haven't you ever seen Clockwork Orange?

Relative speed

[SunPin]

Way to go Earthlink! If I was interested in dialup, this would be a big selling point for me.

Earthlink offers DSL and cable. I'm using it right now.

I am definitely in favor of a little pain up front in increased traffic from challenge-response to get the spam boys off the net.

I suspect that when the spammers stop sucking up so much bandwidth, net speeds will increase for everyone--including dial up users.

Remember when 14.4K was fast? So do I. And I think with a correction in the system, it can be a decent speed.

Re:Relative speed

[dasunt]

The parent poster writes:
Remember when 14.4K was fast? So do I. And I think with a correction in the system, it can be a decent speed.

Nope. Sorry. There are 2 reasons why 14.4K will never be fast again:

1. Graphics. There are plenty of web pages that are not optimizing for graphics, and plenty of web pages that are using more complicated technologies (such as flash) where simple technologies (such as gif) will work.
2. HTML Mail. Isn't it wonderful how a simple "Meet you at 5" can end up being bloated to half a meg with a "pretty" html background?

Re:Relative speed

[evilviper]

Heh... My first response when reading this was "Good for them..." That was until I remembered that Earthlink is my ISP... I just don't happen to use their E-Mail service. Guess I'll have to pop over to their website now and figure out what their e-mail settings are.

Remember when 14.4K was fast? So do I. And I think with a correction in the system, it can be a decent speed.

Well, the solution can be implimented on the user's end... I personally use Privoxy to filter out just about every ad and flash animation out there.

What I would like to see, is browsers giving preference to content, rather than bloat. Just imagine, you have an incredibly slow modem, but web-pages open-up instantly. You open 10 links at the same time, and they load right away...

The only thing browsers have to do is load the HTML first, then, only after each HTML page has been fetched, should it begin to fetch the images (smaller ones first, preferably), and flash animations or other embedded content last. That would be a great way to counter web-site bloat, and I'd consider it rather fair too.

If you look at the page for a seconds, and decide it isn't what you want, the bloat won't even be loaded... If you read it for a few minutes, the ads will be loaded eventually. Text ads, will be loaded instantly.

Re:Relative speed

[batobin]

As a web host AND web designer, I can say that larger web pages aren't the fault of poor design. Page sizes are simply larger these days. Take for example loading this thread at +2 or +3. It would take minutes to load on a 14.4. Is that the fault of large images? Of inefficient code? Nope.

I have a feeling if you saw pages designed for 14.4 today, you'd be deeply disappointed.

Re:Relative speed

[jazman_777]

Bouncing HTML mail back to the lusers who send it takes care of that problem 95% of the time. HTML mail is nearly as annoying as top-posting [demon.co.uk] to Usenet.

I'm digressing (well, _you_ brought it up), but I found this little blurb once about top-posting:

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

Filtering instead of Blocking

[thehun101]

It would be useful if the system could be used to filter instead of block, at least for the first few months. Perhaps, if there is not response to a challenge after 72 hours, and email could be redirected to a 'Spam' or 'Bulk' filder.
This way, If I get monthly newsletters from donotreply@... and I want to keep getting it, i can approve that email. After about 3 months of this type of filtering and I would probably have approved everything I want to receive. Then, I could turn it back to blocking instead of filtering.

-the Hun

Adaptive teergrubing anyone?

[Tackhead]

Instead of challenge-response (putting the burden onto the end user), why not put the burden on the inbound mailserver?

A residential broadband customer mailing through his ISP's mail server is whitelisted (most stuff from that server is nonspam). An rr.com luzer with an open proxy is tarpitted into oblivion (everything else in 24.0.0.0/8 is spam). Yes, Joe Linux running (non-relaying) Sendmail on his Linux box is also tarpitted, but he's not trying to send a million mails a day. So he's not hurtin'.

I can see a scaling problem in that you'd have to run some sort of adaptive filtering process on the receiving end, which might be prohibitive CPU-wise. OTOH, if you only scanned 1% of all inbound mails for "spamminess", you'd still rapidly figure out that for a US ISP, 24.0.0.0/8 is an ocean of spam with a few islands of real email, and 200.0.0.0/7 is a shitstorm of spam. You don't need to analyze every inbound mail - you only need a statistically-valid sampling of the inbound mail queue to figure out which netblocks are teh sux0r.

Having it be adaptive would be cool - because a South American ISP (which probably has less of a problem with 200.0.0.0/7 than, say, Earthlink does, because they have legitimate users emailing each other from within those netblocks). So an ISP in .mx would end up with a different set of teergrubing weights. They might end up letting most of 200.0.0.0/7 in, only tarpitting the worst /24s, and teergrubing all 24.0.0.0/8 because so few of their users get anything but spam from rr.com netblocks.

Think of it as combining the best part of SPEWS (naughty netblocks are noticed semi-automatically), without as much collateral damage (if you're an ISP, a 10 second delay to anyone emailing one of your customers from a naughty netblock will never be noticed, but it'll *kill* some dirtball trying to spam to 10000 of your users through an open proxy.)

Re:Adaptive teergrubing anyone?

[Nonsanity]

Tackhead said:

They might end up letting most of 200.0.0.0/7 in, only tarpitting the worst /24s, and teergrubing all 24.0.0.0/8 because so few of their users get anything but spam from rr.com netblocks.

I'm sorry, but Babelfish isn't doing anything for this post. Anyone have a translation? It SOUNDS interesting... :)

~ Nonsanity

Re:Adaptive teergrubing anyone?

[Tackhead]

> I'm sorry, but Babelfish isn't doing anything for this post. Anyone have a translation? It SOUNDS interesting... :)

ROFLMAO.

"teergrube" - German word for "tarpit".

Teergrubing FAQ

Teergrubing is a good idea, but it dates back from the days when open relays, not open proxies, were sending the emails. One spammer (with dialup) would hit you from one relay (with broadband) from the spammer's own (dialup) connection, and the goal was to slow down the open relay so that the open relay wouldn't be able to spew as many emails. Eventually, the admin of the open relay would wonder why his outbound queue was so huge, or why Sendmail fell over and died because /var/spool got full, and secure his server. In the old environment (spammer has narrowband, must hunt down broadband by finding open relays to steal from), one teergrube could "fix" one open relay at best, and at worst, would at least prevent delivery of several hundred thousand spams.

Doesn't really work as well in a world with millions of open broadband proxies. The spammer no longer cares if any individual open proxy hits a teergrube, because there's plenty more bandwidth where that came from. (And because open proxy luzers tend to be clueless twits, they're less likely to notice even if their machine crashes.) In today's environment (plenty of bandwidth on both the spammer's end, and plenty of proxies to steal bandwidth from), teergrubing in its original form is somewhat less effective.

Re:Adaptive teergrubing anyone?

[milo_Gwalthny]

Take a look at the front page article in the WSJ today... about one of Earthlink's most virulent spammers. He used 300+ dial-up accounts, set up with fraudulent/stolen billing info and was sending (they say) 1 million+ spams per day. Took them like a year and a John Doe lawsuit to finally figure out who he was and stop him. Interestingly, one of the ways they were tracking his accounts was by which passwords he used (he tended to use just a few for all of his accounts)--thought he would catch on to that.

Great article, wish I could post a link. To your point... wouldn't this guy have been automatically whitelisted?

Challenge-response works as part of a whole

[koreth]

I have a homegrown challenge-response system on my mailbox and it's done wonders for my spam flow. The trick, though, is that it doesn't send a challenge to everyone -- it looks at incoming mail and determines how likely it is to be spam (using Bayesian analysis, collaborative filtering, some keyword filtering, and a couple other things). Mail that doesn't trip any of the checks goes through without a challenge. Mailing lists I subscribe to are also whitelisted, as are addresses I send outgoing mail to.

In theory, someone could send me a spamlike message and would have to reply to the autoresponder. In theory, a spammer could validate himself. In practice, those two things almost never happen. The system catches about 150 spams a day and over 90% of its autoreplies immediately bounce. Last time I analyzed it, only about 2% of my legitimate correspondents had hit the autoresponder (note, that's a fraction of a percent of my total legitimate email, since a given correspondent only has to validate once.)

I have yet to see a notification from Amazon, my bank, or other similar email trip the filter. Haven't had any of my correspondents complain yet, but I have had a couple of them ask how they can set up the same thing for themselves.

So if it's implemented carefully, I think this could be a big win for Earthlink subscribers and more or less invisible to everyone who communicates with them.

It can work - if implemented correctly

[dracol1ch]

I've been using Mailblocks since they opened publicly. I can't speak for the implementation that Earthlink is planning on utilizing but the Mailblocks system works very well.

First it is important to note that the challenge system at Mailblocks is not something that can be automatically replied to. Much like the signup verifications for many forum systems out there the Mailblocks challenge email is simply a link to a web site. On that web site is a dynamically generated .gif of a number. The image is formatted in such a way so as to make it difficult for screen scrapers to write an algorithm which can decipher the numbers in the image (multiple fonts, different colors, background noise). If ever a spammer figured out how to programatically decipher the image then Mailblocks simply has to rework their image generation system and stay one step ahead of the spammers.

Next you have throw away addresses. Maiblocks calls these trackers. When you create a tracker a number and short ID are appended to the end of your username. This email address is then immune to the challenge response and can either be delivered to a purpose built folder or directly to your inbox. So if you wanted to have an address to get receipts from you simply make a tracker named say [username]+receipts4325@mailblocks.com. Then any email to this address can be delivered to the +receipts folder in your inbox. If you start getting spam at that address you just delete the address and create [username]+receipts5563@mailblocks.com and start giving this out. It can be a little bit of work to maintain your trackers but compared to deleting 20-30+ spam mails from my accounts each day it's well worth it.

When an email is successfully delivered to your main address the originating address is entered into your address book including the reason why this address was validated (completed puzzle, user added). Mailblocks also adds the address of any outgoing mail you write to your address book so that responses can be properly delivered without challenge. Finally, if you are expecting something to appear in your email that doesn't the 'pending' folder holds all email that hasn't been validated for a certain amount of time before deleting. If you really want to you can go back and dig through the email there to find the one you want, validate it, and it will be delivered to your inbox. If something gets validated you don't want simply go to your address book and either delete it or check 'do not deliver mail from this address'. Viola. Also of interest is the fact that Mailblocks can provide the same security to any other mail account you have. It can check POP3, IMAP, accept forwards, and even screen scrape web mail to bring all of your mail to a central location. When it does it provides the same callenge-response capability through these other accounts.

Re:Now the spammers get address validation for fre

[S.Lemmon]

I guess blind people will just have to give up on using email then? Sounds like an ADA lawsuit in the making.

I used to use this...

[All Names Have Been]

I was using this until I realized I was spending more time enabling/disabling the C/R system or screwing with the whitelist that I was dealing with SPAM. Everytime I wanted to sign up for some mailing list (it it coming from company.com or parentcompany.com or ???) or a user would sign up for some service that sent an email automatically, which, of course, would never appear, causing complaints and yet another trip to vi to modify the whitelist.

Don't even get me started on all those damn email card companies - lots of missing Easter cards because dumbassonlinecards.com wasn't in the whitelist and again, noone is going to send confirmation mails from an automated system.

The whole thing got dumped. Back to SpamAssassin, which causes far fewer headaches. Fortunately, this Earthlink deal is an opt-in system. I couldn't stand to use it myself and I bet few customers will live with this long-term.

Re:Now the spammers get address validation for fre

[PerlGuru]

It would also be a problem for people with text based email clients

Um, the blind?

[cnoocy]

So does this mean that if you're blind, you don't get to send mail to C/R users? Another hurdle for blind users is just what the net needs.

micro payments

[goombah99]

Challenge response is going to be effective but intrusive since a human must read the challenge and reply. this will suck when I sent the family newsletter to 40 friends I havent written to in a couple years and get 40 fresh challenges because my presence on their whitelist had expired. likewise even for automated things I sign up for like like slashdot updates or t rowe price stock reports

I'd like to suggest a way this could all be done automatically, so transparently your an AOL grandma could do it, and almost non-intrusively. Like the lessig-style stamp, all users would be charged say 0.01 cents to send ME an e-mail. but I would automatically refund this payment if either 1) the sender was in my addressbook/whitelist or 2) I did not file the e-mail in my junk mailbox.

what is needed is some sort of distributed postal service to handle the actual micropayments. And this is the main problem--how to collect these. I think the least intrusive method is that when you get an e-mail account you put down a pre-payment, lets say $10 on account at the postal service. when you send messages that are welcome your account is not depleted. when you send messages that aren't it slowly drains.

the cost of the postal service ditributed servers could probably be paid for by
1) the charges for unwanted e-mail
2) interest on the deposits on account.
thus people would be willing to set up these servers.

the final missing ingredient is a centralized server that coordinated the actual postal servers. all this would be would be like a DNS that told all of the remote servers the names of the other ones so they could communicate account info.

the transactions themselves would be in number about twice as the number of e-mails handled (one to the post office from the first ISP to receive the mail to validate the payment code in the header, one from to the postal service me to authorize refund/no refund), and the accounting message size very small.

Perhaps this is a rotten idea. its main benefits are 1) its not intrusive and is nearly transparent 2) it pays for itself 3) requires changes only at the browser level.

I does not stop spam from showing up in my inbox, but makes it very expensive to mass mail.

flame on! or suggests problems and their solution.

Calling all perl wizards and poor college kids!

[MattGWU]

Perl gurus, start your editors!
How many lines will it take to write a script to automatically reply to challanges? As long as the messages have predictable structure, you should be able to write a parser to pick out the word or picture they want, then throw it back.

College kids: Are you bored, broke, and of weak moral fiber? You too can make money while sitting on your ass by replying to email challanges for the princely sum of 3 cents per message! Combine the first suggestion with the second, and you've got yourself a money machine.

It's great to see an ISP take some decisive steps, but this scheme has weaknesses. Interesting to see how it goes. Despite the concerns, I'm cautiously optimistic.
As a twist, it would be interesting to see how that anti-spam vs. spam lawsuit with the copyrighted haiku goes (don't recall the parties names, but it's gotten coverage here). Maybe something similar could be combined with the challange-response system to make it illegal to respond to the challange under false pretenses. Raises a few slippery-slope legal issues that if you're going to touch, you might as well criminalize spam outright (which would be fine, of course).

Re:Now the spammers get address validation for fre

[Chester K]

Once this gets widescale usage, the spammers will simply start responding to the challenges (after all, it's not like that couldn't be easily automated).

In order to send responses to the challenges, it means the spammer has to provide at least a valid return address, and dedicate resources to responding to those requests (even if it is automated). It raises the cost of sending spam, and increases accountability due to the valid return address requirement, which is the best we can hope for with a SMTP-based solution for the time being. It's not perfect, but nothing is.

Bayesian Filter + Challenge Response

[juggler314]

A number of folks have pointed out how this really doesn't work so well in a real world situation. This is pretty much true, there are myriad problems. What can work fantastically is a two tiered approach though: 1) Use a Bayesian filter to sort your mail however you want (for simplicity lets just say spam/not spam). 2) Forward all filtered mail marked as spam to your CR prog of choice - this chunk of mail should already be confirmed in the high 90%'s to be spam - the few false positives should get caught. The reason this works so well is that the Bayesian filter approach is pretty solid, but there's always a worry of a few important false positives sifting through. This gets rid of those. If you really want to go balls-out you could make use of a service such as spamgourmet.com for ordering processes. Whenever you order something where you are expecting some automoted return mail that might hit the Bayesian filter AND also not respond to the CR use one of the self destruct e-mails. You should never get more than 5 or so e-mails from an order anyway. You can then just filter everything from your bogus self destruct e-mails into a generic "orders" folder.

Precedence: Bulk

[Euphonious Coward]

All they need to do to handle legitimate mailing lists, at least at first, is to challenge only mail that is not explicitly labeled with "Precedence: bulk". Legitimate mailing lists carry that label, but spam never does.

Once the spammers are obliged to label their stuff "bulk", half the battle is won. Then they start collecting a "white list" of legitimate mailing list sources, and label every bulk message not on it as "suspected spam" and dump it in a separate folder.

Thoughts and observations

[cr@ckwhore]

First of all, the system is completely optional for earthlink users. For the users that are stupid enough to opt-in, they deserve the extra hassles they'll receive.

But here's what it means to me, a publisher of a popular website...

When a new user signs up for an account, they get a confirmation email. Since I'm not about to check the server's return-path for C-R messages, C-R users will be out of luck. This means that at the very least I'll have to update my site with a special notice during the sign-up process that will notify earthlink users to expect problems.

The crux of the matter, there are automated emails that will fall victim to this C-R paradigm that AREN'T spam!

So, what is earthlink's "fix" for this problem? Well, it appears as though they will assign special addresses that users can use for sign-ups, sales receipts, etc. that will bypass the regular C-R system. Ok, great. Two problems with that ...

1. If the special bypass addresses are only temporary, then my users' accounts will become invalid because their email address is no longer valid and I don't allow ghost accounts.

2. If the special bypass addresses are permanent, and they're used for sign-ups and sales receipts, well fsck! Thats where SPAM comes from. duh. Great ... all their spam will arrive via bypass addresses. Awesome!

Re:Now the spammers get address validation for fre

[StarOwl]

Once [TMDA] gets widescale usage, the spammers will simply start responding to the challenges (after all, it's not like that couldn't be easily automated).

There are currently three defenses to this:

1. Most spammers dummy up their headers. The challenge never gets delivered to them, and therefore the spam goes undelivered.
2. Spammers who use legit email addresses usually see their inboxes fill quickly to the point of bouncing mail. Again, they don't see the challenge, so the spam goes undelivered.
3. Spammers who use legit addresses and have large inboxes are likely to be trackable. If they're in your country, and if your challenge message is worded correctly, there is some legal exposure on their part.

Admittedly it's not foolproof. There is no 100% effective way to combat spam (short of abandoning SMTP). There's always going to be a risk that some spam will leak through or that some legit email will bounce.

See "Guarded Email" paper

[dwheeler]

For more details on a challenge-response system, see my paper on "Guarded Email" at: http://www.dwheeler/guarded-email.

Guarded email completely deals with some of the problems noted in these comments:

1. How do you receive challenges? Yes - if you SEND a message to someone, then you can set things up to automatically RECEIVE messages from that someone.
2. Can blind people send email? Yes - the challenge should be human-readable, but not computer-processable. That's easy.
3. Can you prevent loops? Yes - you have to think about it, but there are simple loop-prevention techniques so that EVERYONE can use these kinds of systems.

Pre-emptive Anti-Spam Measures

[akedia]

I've used Earthlink as an ISP for going on 6 years now, and I must say, I've never dealt with better. For one thing, in the years that I've had my earthlink address, I'd say I never get more than 3 or 4 spams per week. What is my secret? For starters, if I need to provide an e-mail address for something that may result in unsolicited messages, I use one of the free webmail providers (Hotmail, Yahoo!, etc.) I can check those to confirm what I wanted, then never check it again, and my Outlook (with my primary e-mail) doesn't fill up with useless crap.

Another way to stop the spam before it starts is to keep your e-mail address from getting on those lists in the first place. When posting to Usenet, BBSes, forums, even Slashdot, use some sort of clever cloaking (Slashcode does this already), or even a fake email. Encryption for e-mail such as using a free personal certificate from Thawte or a GPL encryption such as GNU Privacy Guard is always a good idea.

In addition, Earthlink's Spaminator is a Godsend. With that baby enabled, I'm lucky if I get one spam a month. Case in point: my mother has an Earthlink address that she uses for her business contact. She complained that she's getting hundreds of porn spam and "enlarge your penis"-type e-mails (no idea how these got here.) Setting up a few Outlook Express filters and enabling Spaminator cut the dirty messages by about 90%, and she is grateful she no longer has to wade through such filth to get to her real mesages.

The bottom line is, the fewer spammers that have your address, the fewer spams you're gonna get. I have a Hotmail that gets 1000+ spams a day. My real e-mails get next to none. It's just like telemarketers, they get your number from companies who need a contact info for whatever reason. However, Hotmail address are free, whereas extra phone numbers to give the telemarketers, and then never answer, are not. Well, we do have Caller-ID for that, but that's another post...

Could help slow some worms, viruses. . .

[GeorgieBoy]

. . .as long as people aren't getting them from their buddies. Even so, if emails are scanned for viruses/worms in attachments before they get to the user, there can be more wins than just stopping spam.

How Earthlink's system actually works.

[Gendou]

I'm using the beta-test of this system now, so I know the news article doesn't describe it very well.

Here's the internal description of the service, which, by the way, is always going to be optional -- users have to turn it on manually. So fears of mass confusion from users when Earthlink turns this system on are a bit unfounded.

What is Suspect Email?

With some messages, only you can decide whether they are junk. When you turn on Suspect Email Blocking in addition to Known spam Blocking, you'll only receive messages from senders who are in your TotalAccess or Web Mail Address Book. Other messages will be temporarily held in your Suspect Email folder, and the unknown senders will receive an automatic reply message telling them how to ask to be added to your Allowed Senders list.

This is what the automated reply looks like:

From: automated-response@earthlink.net
To: user@somedomain.net
Subject: Re: How are you doing?

This is an automatic reply to your e-mail message to earthlinker@earthlink.net.

This email address is protected by Earthlink spamBlocker. Before earthlinker@earthlink.net can receive your message, your email address must be added to a list of allowed senders.

Click the link below to ask earthlinker@earthlink.net to add you to this list:
http://webmail.earthlink.net/wam/addme?a=ea rthlink er@earthlink.net&id=xxxyyyzzz

And finally a more detailed description they supply:

Suspect Email Blocking is disabled by default, and includes Known spam Blocking. You must activate it yourself if you wish to use it.

With Suspect Email Blocking, spamBlocker examines any message that Known spam Blocking has not intercepted. If the sender's email address or Company (Domain) (i.e., the portion of the email address after the @ symbol, such as earthlink.net) appears in your Address Book, spamBlocker allows the message to reach your Inbox normally.

If the sender's address or Company (Domain) does not appear in your Address Book, spamBlocker does three things:

Intercepts the message and stores it online in your Suspect Email folder (which you can open by clicking the Suspect Email tab in the spamBlocker interface).
Automatically replies to the sender with instructions on how to ask to be added to your Address Book
Notifies you about the intercepted message in a summary you'll receive periodically via email (see spamBlocker Settings for more about email summaries)
Note: Messages in your Suspect Email folder remain on EarthLink's incoming email server and count toward your 10MB mailbox storage limit. spamBlocker automatically deletes Suspect Email messages that are more than 14 days old.

Suspect Email Blocking practically ensures that your Inbox will be spam-free. To be effective, however, Suspect Email Blocking requires that you maintain a list of email addresses and Companies (Domains) you want to receive email from in your Address Book.

Suspect Email Blocking works in conjunction with Known spam Blocking. You cannot use Suspect Email Blocking by itself.

Procmail...

[Brew Bird]

Don't know where I found this at, but it's pretty old... Share and Enjoy!
.procmailrc

----------Cut Here-------------

#Define the password
PASSWD_=PASSWORD

#Whatever other recipes in between.

# Email is not challanged from:
:0
* ^From: myfriend@aol\.com
${DEFAULT}

#Return email if the password is not there
:0:passwd.lock
#
# Check for (the lack of) the password
* $ ! ^Subject:.*${PASSWD_}
#
# Avoid email loops
* ! ^X-Loop: your-addrs@mail\.isp\.net
* ! ^From:.*your-addrs@([-a-z0-9_]+\.)mail\.isp\.net
#
# Prepare and send the notification
# Be sure to customize your sendmail path
| (formail -r \
-i"Subject: Returned email: Password or privileges required" \
-A"X-Loop: your-addrs@mail.isp.net" ; \

echo "* This is a computer-generated response message *" ; \

echo ; \
echo "Email password required!" ; \
echo "Please include (${PASSWD_}) anywhere on your subject line." ; \
echo "Then kindly resend your email to your-addrs@isp.net") \
| /usr/sbin/sendmail -t

Blindness

[druske]

If the challenge is based on an image ("please respond with the fuzzy word in the subject line" or somesuch), where does that leave vision impaired email users? How do they respond to a challenge to get their email delivered?

One problem with this system.

[illumin8]

Did anyone notice that in order to workaround automated systems that need to send legitimate email, such as Amazon when you buy something, or mailing lists you subscribe to, they give you a second email address that will not be protected by Challenge/Response?

I can see this being a big problem. In my experience, people only get spam if they have done one of several things:

1. Published their email address on a web page to be picked up by harvesters.
2. Given their email address to an online retailer that sells it.
3. Signed up for some spyware scam where they again give their email address to someone that will add it to a spam list.
4. Opened a Hotmail account, which, it seems is automatically sold to all the various spam providers.

In almost all of these cases, the act that caused spam to be received was the user giving out their email address to a non-trustworthy source.

How is having a second email address that people will just type into any webpage that promises free porn and bypasses Challenge/Response going to curb the spam problem? I give this system only 1-2 months before spam is back at it's initial volume, just using the new email address instead of the old.

You need to also educate users about the problems of giving their email address out to unreputable places on the net. A lot of users don't correlate their spam problem with the fact that they typed their email address into some website to get a free porno password the night before.

Re:One problem with this system.

[datavortex]

Then, if you added a dozen more equally clever features, and a nifty web interface availible, you would have TMDA

:)

Having written a similar system, I have questions.

[kaoshin]

If someone from earthlink emails someone else from earthlink, how would challenge response handled then? Do they make all mail that is sent returnable without challenge responses, and if so is this a temporary rule or are the addresses of all mail you send permanently whitelisted?

If the challenge response triggers a mail daemon reply, is it filtered or do you get flooded with those replies caused by all the spammers with forged addresses? If they are filtered, how do you know when mail you send doesn't go through without the use of message reciepts since mailer daemon replies are all different.

If I mass email tons of earthlink addresses with a forge from address, would it mailbomb the fake address, or do they have flood protection to prevent this?

Re:Having written a similar system, I have questio

[datavortex]

If the challenge response triggers a mail daemon reply, is it filtered or do you get flooded with those replies caused by all the spammers with forged addresses?

As you will find to be the case with most C/R systems, the challenge is sent with a null envelope.

If I mass email tons of earthlink addresses with a forge from address, would it mailbomb the fake address, or do they have flood protection to prevent this?

Yes. There are daily (and other) limits to how many challenges are sent to an address or server.

Re:How about another approach...

[grishnav]

I run a legitimate e-mail server for my family, but cannot afford an SSL certificate for it. I instead use a self-signed one.

If self-signed certificates would be allowed, then spammers would make their own. So that can't be allowed.

If they are prompted, as you suggested earlier, it would inevitably lead to people who just ignore invalid ones, because they are sick of being prompted. My little mail server gets creamed.

Nice idea, but unless you get Verisign to give away free certs, I can't see it working.

This DOES work.

[Anonymous Psychopath]

I've been using TMDA (http://www.tmda.net) for well over a year now, had maybe five or six spam emails sneak through the system in that entire time. Twice a day it sends me a list of "pending" emails so I can manually release and/or whitelist a message.

Challenge/response systems DO work, and they work extremely well. I think those who have not used one should give it a try before throwing rocks.

An alternative solution?

[NanoProf]

A fundamental problem of Spam is that the sender of an email cannot be identified and verified with 100% accuracy, so it is difficulty to filter 100% effectively. However, there is one and only one part of an incoming message that must of necessity be accurate- the To: address. So use the To: address to identify the sender! Publish your public address: "foo@bar.com". Any email to foo generates a reply "Thanks for the note. Mr. Foo loves you so much that he's generated a special personal email address just for you to use: 'foo_RANDOMSTRING@bar.com'. Please use this address in the future- sorry but you'll need to resend the message just sent to this new address. Don't ever give out this secial address to any else, because if Mr. Foo begins to receive spam on this To: address, he will automatically filter all future messages to foo_RANDOMSTRING straight to the trash." Every sender gets a unique RANDOMSTRING, so you can filter on the To: address. It's similar to throw-away email addresses, but coupled to a public address that triggers auto-generation of new RANDOMSTRING addresses. The sender has the inconvenience of adding foo_RANDOMSTRING@bar.com to their address book. Also, spammers can read the auto-reply and then add foo_RANDOMSTRING to their spam list, but this could be made difficult by putting it in a distorted gif image. The email client would also need to be configured to set Reply-To: correctly on folowups. One nice thing is that for user-requested bot-generated emails, one can simply give them a new RANDOMSTRING-based email address right off in the registration form or whatever. The ever-expanding number of foo_RANDOMSTRING@bar.com addresses adds to the overall load on the servers, but is that handle-able (nasty things could happen if your inbox got Dos'd)? In such a world, people would get used to pinging new people with just a short message to obtain their personalized RANDOMSTRING address. Kind of a weird system but maybe it's interesting to think about?