Earthlink Deploying Challenge-Response Anti-Spam System

deliasee writes "The Washington Post reports that Earthlink is preparing to offer new spam filter technology that requires sender authentication. AOL is still concerned that such technologies will put too much burden on consumers." The day after it's deployed, every legitimate mailing list on the planet will get challenges from all the Earthlink subscribers...

Comments

Nice moves

[hendridm]

I was hoping more ISPs would adopt the challenge-response system, like MailBlocks, previously featured on Slashdot. Way to go Earthlink! If I was interested in dialup, this would be a big selling point for me. I'm still waiting for a service that offers the challenge-response feature of MailBlocks but allows me to forward to my existing provider. I mean, a 12MB inbox is pretty lame. There are free providers that can give me that much space...

Re:Nice moves

[apoc.famine]

I dunno. This may be painful for a bit, and increase the amount of mail, but in the long run it might be worthwhile. While I agree that it makes some peoples' jobs harder, those people probably aren't using the major ISPs/mail-services. If the major players do this, it makes it that much less profitable for spammers to do business.

I mean, if you're a spammer, a brute force mailing to joeuser.org is MUCH less profitable than mailing the same million messages to hotmail.com. Go big guys, go! It won't bother me at all.

Re:Nice moves

[darien]

Er, what?

eMail was not designed for such a challenge

So what? This system works within the standard. Who cares whether or not the designers foresaw it?

It drives network traffic as well up to the sky.

Hardly. If you're on Earthlink and decide to opt-in for this, it simply means that everybody you know has to send you one extra email once. Earthlink's traffic may be a bit higher for the first few days, but once people get their whitelists in order it'll drop back to where it is now - and below, because there'll be less spam floating around.

However, I do hope (the article didn't say) they've come up with a smart solution to the problem of spammers putting real (but stolen) addresses as their From: address. Otherwise people unlucky enough to have their addresses stolen may indeed find their network traffic increases, thanks to a million challenges from Earthlink.

Too drastic?

[mao che minh]

Drastic times call for drastic measures. The situation caused by the relentless onslaught of SPAM (which supposedly is rendering "damages" in the billions annually) can certainly be categorized as drastic. Is Earthlink's counter attack too drastic a measure, though?

On one hand it (Earthlink's new "technology") seems reasonable enough to the every-day-joe. I'm sure that the majority of Earthlink subscribers don't utilize news or mailing lists, and don't bother paying their bills online. For these people, it's fine. On the other hand, many others use online banking and other such automated tools (even account control mechanisms for online games will be affected). How quickly will all of these vendors conform to Earthlink's new technology and make the needed changes in their automated systems? Will Earthlink simply render many of these domains exempt?

The answer to solving SPAM resides in the current mechanisms used for the actual transmission and delivery, the mechanisms that all participants must use, not just Earthlink. This is of course the mail servers themselves.

Re:Too drastic?

[iangoldby]

People who want to continue to receive messages from mailing lists, online banking, etc, will have to add these sources to their whitelist.

It's a bit of a faf though, and I suspect many people will either not understand how to, not bother, or forget at least one address.

The solution is to have the incoming messages moved into a 'holding' folder that the recipient can see, and check in just the same way as checking through a 'spam' folder. This would remind the user to add false positives in the 'holding' folder to the whitelist. After a while, you can safely stop checking your 'holding' folder. Wouldn't it be good if this is what Earthlink are doing?

I think a scheme like this could be made to work, at least for webmail. For POP3, it could be a bit more tricky...

Re:Too drastic?

[letxa2000]

Challenge-Response is bogus. I don't know of any such systems that have been deployed without significant problems for their users, the people that send mail to their users, and especially mailing lists.

If challenge-response is largely deployed, I suspect spammers will just unite such that one spammer sends a message, gets the challenge, answers it and is then "unlocked" to send message. He'll then distribute that email address in real-time to dozens or hundreds of other spammers who will send their spam immediately with the same newly-unlocked address.

Or, perhaps, spammers will change their tactic from spamming millions of users with 1 spam at a time to spamming 1 user at a time with dozens or hundreds of spam. You unlock the system with a valid response to the challenge and then flood them with spam until the user blocks that address.

I just don't see where challenge-response is anything more than a very stopgap measure. It's not particuarly "clean" now and will become more and more useless in the future.

Almost a year after Paul Graham's "A Plan For Spam" Bayesian is still the easiest system to develop as well as the easiest for the user to use. It is extremely effective (99.5%+) with very few false positives and doesn't require any additional effort for the sender and only requires that the user report false positives and false negatives--and that is mostly only needed at the beginning. Once it is initially tuned it's not necessary to do much of anything--it just keeps learning and working.

Re:Too drastic?

[Tackhead]

> People who want to continue to receive messages from mailing lists, online banking, etc, will have to add these sources to their whitelist.

Problem is, you don't know what that email is necessarily going to be.

I ordered something from foo.com and got order number 12345.

A few seconds later, I got a confirmation mail from confirm-12345@foo.com telling me what I bought and when to expect delivery. (Or worse, from order-12345@foo.com telling me there was a problem, and that I needed to fix something!)

If challenge-response becomes widespread, foo.com will say "Now you must whitelist the address confirm-12345@foo.com" when processing the order. (Or switch their order-processing back-end software to use something more sane, like "confirm@foo.com" and put the damn "Order 12345" in the Subject: header where it belongs!)

Problem is, until then, some vendors and some users using challenge-response are gonna be up the proverbial estuary without a utensil for propulsion.

If foo.com is disreputable, of course, challenge-response solves the donkey pr0n spam problem, but not the mainsleaze part of the spam problem. A mainsleazer at foo.com will simply start spamming his customer list with a From: of "confirm@foo.com" - Subject: "New Dealz from foo.com!" *sigh*)

How do two people with C/R communicate?

[corsec67]

How do two people with challenge and response communicate?
If the challenge always gets thrugh, then the spammer will just issue cahllenges as spam.
If they don't get through, then you would have a nasty mail loop.

Re:How do two people with C/R communicate?

[stratjakt]

The way I read it, earthlink, up on recieving an e-mail, sends a challenge to the email sender. If the e-mail sender responds, it delivers the mail.

From the article:

When someone sends an e-mail to a challenge-response user, he or she gets an e-mail back asking to verify that the sender is a live person.

Once the sender does that by replicating a word or picture displayed on the screen, the original e-mail is allowed through. The system automatically recognizes future e-mails from the same sender, so the verification needs only to be performed once. Without the verification, the e-mail is not delivered.

So if earthlink people are on your mailing list, you'll get a challenge next time you send it out. It should only happen once, and from then on, you're email addy is "legit".

It's not like you get 9000000 challenges from everyone on the list. But if every ISP did it, you'd get a challenge from every ISP on the list.

This is the first step towards email being such a pain in the ass, that people just no longer bother using it.

Kiss SMTP and POP3 goodbye.

Re:How do two people with C/R communicate?

[Chester K]

How do two people with challenge and response communicate?

My C/R setup (TMDA) automatically put anyone I send email to on my whitelist; therefore I'd get their challenge message.

Re:How do two people with C/R communicate?

[esme]

Here's how it works:

1. Alice sends an email to Bob.
2. Bob is automatically added to her access list (b/c she's sending him mail, he's not a spammer).
3. Bob's mail server sends a confirmation request.
4. Alice recieves the confirmation requestand responds.
5. Original message is delivered to Bob.

-Esme

Re:How do two people with C/R communicate?

[1729]

You can't have an automated challenge/response system, because that defeats the point.

That's not true. There is an approach where you show a "proof of computational effort"; that is, your computer spends 10 or so seconds computing the response to a challenge. Here's a paper on the subject.

Forged Headers

[Anonymous Coward]

I think forged headers are the calamity of the inprocess SMTP transfer mechanism. If we can liberate the dynamic IPs saturated on the IPlanet web matrix, then we could perform 3-way LDAP POP3 authentication with a digital certificate.

The other way this could be accomplished is to triangulate a 801.11b WAP source into an array of POSIX message headers that would reflect the consistency of the mail protocol.

What do you think?

Now the spammers get address validation for free

[chefbimbo]

Seriously, what are they thinking? TMDA might seem like a nice idea in theory, in practice, it's a pain to use and not exactly safe either. Once this gets widescale usage, the spammers will simply start responding to the challenges (after all, it's not like that couldn't be easily automated).

Michael's comment

[Rev.LoveJoy]

This is true, but perhaps it illustrates an opportunity for developers of mailing list software more than it exposes a flaw in Earthlink's plan to thwart spam?

As a network admin, many of the remote users I support (sales reps, on-the-road types) use Earthlink dial-up while travelling. At times, some of the program's that Earthlink has used to stop people from using their services to spam have make my job harder. However, I do not begrudge Eartlink for these inconviences, at least they, as a major ISP, are doing *something* about this problem.

My two cents,
-- RLJ

Correction

[robbyjo]

every legitimate mailing list on the planet will get challenges from all the Earthlink subscribers

Not exactly right. It happens only for the first time to detect whether the sender is legitimate or not. Quote the article:

The system automatically recognizes future e-mails from the same sender, so the verification needs only to be performed once.

The problem with this system is that the spammer can still spam using legitimate e-mail accounts as a camouflage (or expired e-mail accounts). Once the legitimate e-mail address is procured, the spam still goes on. It is futile, IMHO.

Re:Correction

[Ed Avis]

Spammers seem to be sending a whole bunch of crap from my address (ed@membled.com) even now. At least, I keep seeing what appear to be genuine delivery failure notifications of Russian spam sent from my address. Any system which trusts individual email addresses, without relying on some real authentication such as PGP signatures, is broken.

A simple rule is: Headers can be forged. Don't trust anything in the headers for antispam purposes. This includes the sender and recipient.

Good idea, bad idea.

[numbski]

How to set up SpamAssassin Milter on OSX <- Easily adapted for other platforms. I wrote it.
Squirrel Mail
SpamAssassin Config for Squirrel Mail <- Register Globals must be turned on in php.ini to use this.

Now, that being said, I run an ISP in St. Louis, and spam is a problem, but for the precise reason mentioned on the submission, I can't use a challenge-response system. The reason is that our support staff equals myself plus 1. If I want to answer phone calls all day from people complaining about not being able to get mail from their daily spamming of mailing lists, I best allow all. The problem is that these same people complain about all the spam they get...ugh. The above solution is elegant and leaves the ability to control the filter to the end user via webmail. If they don't like it, set the threshold high and it's 'off'. Been using this for months without a complaint.

Now if you don't use lists, and it's for your own mail server...go for it. That has to be the most effective method available, but not appropriate for wide scale use.

They should offer it with new email address

[dnoyeb]

me@challenge.earthlink.com

something like that. So that it allows users to gradually changeover to the system. That would allow them to be more extreme in their refusal to accept emails and much less compromising.

I like it.

Oh great, now spam has its own protocol

[Anonymous Coward]

"...the spam client MUST provide a Accept-Topics: header, where the value is one of 'penis-enlargment', 'make-money-fast', 'repair-credit', or 'any'. The server MUST reply with a Spam-Type: header, specifying the type of spam transferred. In addition, the server MUST respond with a Spam-Encoding: header, where the value is one of the options 'all-caps', 'many-exclamation-points', or 'broken-english'..."

Fill up the ISP servers

[nuggz]

So when a spammer fires a few hundred or thousand emails to an ISP, they will sit on the mailserver waiting for him to respond.
Since the from address is faked, that same ISP will launch an acknowledgement flood against a third user.
Excellent.

I just see so many tricky things that someone somewhere will screw up.

Challenge - Response doesn't work

[tshak]

What happens when the customer orders something from Amazon - the purchase confirmation email comes from a non-human address.

Just the other day I got an email from a company that I ordered software from describing a free upgrade that I could download. It came from donotreply@[host].com, meaning, if I was using Earthlink's system I probably wouldn't have received it.

The problem with Challenge - Response is that it makes the assumption that if there's not a human behind the email that it's spam. In practice, there are many legit emails that are not individually sent by a human.

bad protocal: SMTP

[JDizzy]

The answer is not attaching more bad ideas to an already bad protocol. The ultimate answer is in the protocol designers. A government/state can pass as many laws governing the interaction of people/things with the bad protocols, but the IETF/IEEE will still create them, and certify them. People should just wake up and realize that SMTP is to blame for this big mess. ISP's should stop offering SMTP outright, and think of ways to replace it. Chat programs are probably a better way to pass messages anyways. SMTP has become a massive bazaar that is full over everyone on earth, and since it is completely open, its also completely ok to send bulk mail. Forging headers is another issue, but simply spewing email is intrinsically allowed by the protocol, and thus taken advantage of. If everyone one on earth had a computer, and everyone on earth sent email to everyone else on earth every day, would that be spam? No, because it would cross the line into accepted practice, and that is what we are starting to see due to the sheer bulk of spam sent to everyone on a daily basis. The point is that as long as SMTP exists, so will spam. The answer is to replace SMTP with something that doesn't allow spam to exist by removing the ability to anonymously send people messages.

There's a whitelist

[Spittoon]

Jeez people, read the whole article, it's not that long:

The challenge-response system will be optional and free for EarthLink subscribers, Anderson said. It will allow users to automatically clear the e-mail addresses of friends, family members and other associates in their electronic address books, so those people would not receive the challenge e-mail.

That's called a "white list"-- a list of addresses you know are legitimate.

When someone responds to a challenge and you accept their response, they go on your whitelist.

When you turn on this gadget, add your mailing list addresses to your white list. If you suddenly stop getting a list, go find out if they changed their sending address and add it to your white list.

If that's too much of a burden, feel free not to use the service, and go back to complaining about spam.

Wow, nobody understands this!

[MrPerfekt]

I see a slew of people saying "blah blah blah, they'll automate the response blah blah blah". And apparently, to alot of you, this is all new.

This is something that's been around for a few years and gee, spammers haven't gotten around it yet. C/R antispam systems work because spammers don't use valid Reply-to: or To: addresses.

If they did and the spam gets through the system, then great! There's one more point where we can nail them on when/if we go to hunt them down. Oh, you used your dialup with an SMTP server to auto-respond to the challenge (which is probably alot of work for the average evil spammer), great, email abuse@isp and have his account shutdown.

Since I have started using ASK to C/R my email. -zero- spams have gotten in my Inbox (which is what annoyed me the most about spam, the false positive I got when the little sound would ring telling me I had new mail.)

Intrusive? PLEASE! How lazy are you? Hit reply -once- and you'll never have to see it again when sending email to me. I'd say getting pelted with 200 spams a day is slightly more intrusive to me than what you're going to have to do to send an email to me.

You can do this yourself.

[Malcontent]

Take a look at this

Re:You can do this yourself.

[StarOwl]

I use TMDA to provide a challenge/response mechanism in my antispam filter.

When I first started using TMDA, I had problems with people not understanding the mechanism. My grandmother, for example, complained about "bounces" (how she interpreted the challenges).

So, to avoid those problems, I:

• Actively manage my whitelist. For example, if I needed to send a resume, I would make darned sure that the prospective employer's domain was on the list.
• Use challenge-response only in conjunction with other antispam tools. My system is roughly: if I know it's spam (tagged address known to be in spammers databases), it gets trashed. If spamassassin or spamoracle thing it's spam, I refer to tmda for possible challenge/response. Otherwise, the mail gets delivered.
• Warn people about the system. If I know that someone new is about to send me email, I warn them: "You might get an autoresponse back. If you do, just hit 'reply'."
• Use some care in writing the challenge email. Trying to craft a letter that is understandable to non-geeks wasn't that easy.

I still have the odd piece of spam leak through that process, but it's nowhere near the quantity that's actually sent to me.

The only problem with the scheme: there are some spammers who are dumb enough to not get the hint, and respond to the challenge. They don't seem to realize that their response probably constitutes harassment via 'net, which is a crime in the U.S. (Spammer go to jail. Do not pass go. Do not collect $200.)

Re:You can do this yourself.

[BlackHawk-666]

I also use TMDA and I can tell you it has vastly reduced the amount of spam I receive from approximately 20-30/day to 1 in the last two months. I've never been happier ;-)

Whitelisting is important, and easy too. Just export your address book to a text file and copy the results to your whitelist (which is also text).

It's worth noting that you can also auto-whitelist anyone you send mail to by using their nifty little mail proxy. It sits and proxies for SMTP and adds all outgoing mail automatically to your whitelist, so whoever you sent that resume to will never see a challenge...neat!

P.S. Can't recommend the product enough.

Which planet are you from?

[mccrew]

Education is the way to go for spammers.

Other than using a cow prod or a red hot poker, how on earth do you "educate" a spammer? Send them to Spammer School? Enroll them in self esteem classes? D00d, this is just about the stupidest thing I have heard in in a loooooonnnnnnngggg time.

Perhaps education is the way to go for Slashdot posters...

Sue them if you're richt (read: AOL), complain about them if you're poor (read: everyone else)

Sue them if your rich? Perhaps you can enlighten the techno-elite here how exactly you find a spammer who is sending e-mails with forged headers, connecting through open HTTP proxies? If you're going to sue them, you gotta find 'em first, right?

and be happy if they loose your DSL connection because of you as one guy dig who pissed me of days ago.

Ohhhh great job, kiddie! Sounds like you did a denial of service on some average home user who didn't happen to know that he had an open web proxy server. Whoo hoo! You da man!

Re:Which planet are you from?

[shawn.fox]

how on earth do you "educate" a spammer?

Haven't you ever seen Clockwork Orange?

Adaptive teergrubing anyone?

[Tackhead]

Instead of challenge-response (putting the burden onto the end user), why not put the burden on the inbound mailserver?

A residential broadband customer mailing through his ISP's mail server is whitelisted (most stuff from that server is nonspam). An rr.com luzer with an open proxy is tarpitted into oblivion (everything else in 24.0.0.0/8 is spam). Yes, Joe Linux running (non-relaying) Sendmail on his Linux box is also tarpitted, but he's not trying to send a million mails a day. So he's not hurtin'.

I can see a scaling problem in that you'd have to run some sort of adaptive filtering process on the receiving end, which might be prohibitive CPU-wise. OTOH, if you only scanned 1% of all inbound mails for "spamminess", you'd still rapidly figure out that for a US ISP, 24.0.0.0/8 is an ocean of spam with a few islands of real email, and 200.0.0.0/7 is a shitstorm of spam. You don't need to analyze every inbound mail - you only need a statistically-valid sampling of the inbound mail queue to figure out which netblocks are teh sux0r.

Having it be adaptive would be cool - because a South American ISP (which probably has less of a problem with 200.0.0.0/7 than, say, Earthlink does, because they have legitimate users emailing each other from within those netblocks). So an ISP in .mx would end up with a different set of teergrubing weights. They might end up letting most of 200.0.0.0/7 in, only tarpitting the worst /24s, and teergrubing all 24.0.0.0/8 because so few of their users get anything but spam from rr.com netblocks.

Think of it as combining the best part of SPEWS (naughty netblocks are noticed semi-automatically), without as much collateral damage (if you're an ISP, a 10 second delay to anyone emailing one of your customers from a naughty netblock will never be noticed, but it'll *kill* some dirtball trying to spam to 10000 of your users through an open proxy.)

Re:Adaptive teergrubing anyone?

[Nonsanity]

Tackhead said:

They might end up letting most of 200.0.0.0/7 in, only tarpitting the worst /24s, and teergrubing all 24.0.0.0/8 because so few of their users get anything but spam from rr.com netblocks.

I'm sorry, but Babelfish isn't doing anything for this post. Anyone have a translation? It SOUNDS interesting... :)

~ Nonsanity

It can work - if implemented correctly

[dracol1ch]

I've been using Mailblocks since they opened publicly. I can't speak for the implementation that Earthlink is planning on utilizing but the Mailblocks system works very well.

First it is important to note that the challenge system at Mailblocks is not something that can be automatically replied to. Much like the signup verifications for many forum systems out there the Mailblocks challenge email is simply a link to a web site. On that web site is a dynamically generated .gif of a number. The image is formatted in such a way so as to make it difficult for screen scrapers to write an algorithm which can decipher the numbers in the image (multiple fonts, different colors, background noise). If ever a spammer figured out how to programatically decipher the image then Mailblocks simply has to rework their image generation system and stay one step ahead of the spammers.

Next you have throw away addresses. Maiblocks calls these trackers. When you create a tracker a number and short ID are appended to the end of your username. This email address is then immune to the challenge response and can either be delivered to a purpose built folder or directly to your inbox. So if you wanted to have an address to get receipts from you simply make a tracker named say [username]+receipts4325@mailblocks.com. Then any email to this address can be delivered to the +receipts folder in your inbox. If you start getting spam at that address you just delete the address and create [username]+receipts5563@mailblocks.com and start giving this out. It can be a little bit of work to maintain your trackers but compared to deleting 20-30+ spam mails from my accounts each day it's well worth it.

When an email is successfully delivered to your main address the originating address is entered into your address book including the reason why this address was validated (completed puzzle, user added). Mailblocks also adds the address of any outgoing mail you write to your address book so that responses can be properly delivered without challenge. Finally, if you are expecting something to appear in your email that doesn't the 'pending' folder holds all email that hasn't been validated for a certain amount of time before deleting. If you really want to you can go back and dig through the email there to find the one you want, validate it, and it will be delivered to your inbox. If something gets validated you don't want simply go to your address book and either delete it or check 'do not deliver mail from this address'. Viola. Also of interest is the fact that Mailblocks can provide the same security to any other mail account you have. It can check POP3, IMAP, accept forwards, and even screen scrape web mail to bring all of your mail to a central location. When it does it provides the same callenge-response capability through these other accounts.

Um, the blind?

[cnoocy]

So does this mean that if you're blind, you don't get to send mail to C/R users? Another hurdle for blind users is just what the net needs.

Re:Now the spammers get address validation for fre

[Chester K]

Once this gets widescale usage, the spammers will simply start responding to the challenges (after all, it's not like that couldn't be easily automated).

In order to send responses to the challenges, it means the spammer has to provide at least a valid return address, and dedicate resources to responding to those requests (even if it is automated). It raises the cost of sending spam, and increases accountability due to the valid return address requirement, which is the best we can hope for with a SMTP-based solution for the time being. It's not perfect, but nothing is.

Blindness

[druske]

If the challenge is based on an image ("please respond with the fuzzy word in the subject line" or somesuch), where does that leave vision impaired email users? How do they respond to a challenge to get their email delivered?

Having written a similar system, I have questions.

[kaoshin]

If someone from earthlink emails someone else from earthlink, how would challenge response handled then? Do they make all mail that is sent returnable without challenge responses, and if so is this a temporary rule or are the addresses of all mail you send permanently whitelisted?

If the challenge response triggers a mail daemon reply, is it filtered or do you get flooded with those replies caused by all the spammers with forged addresses? If they are filtered, how do you know when mail you send doesn't go through without the use of message reciepts since mailer daemon replies are all different.

If I mass email tons of earthlink addresses with a forge from address, would it mailbomb the fake address, or do they have flood protection to prevent this?

Re:Relative speed

[jazman_777]

Bouncing HTML mail back to the lusers who send it takes care of that problem 95% of the time. HTML mail is nearly as annoying as top-posting [demon.co.uk] to Usenet.

I'm digressing (well, _you_ brought it up), but I found this little blurb once about top-posting:

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?