Earthlink Deploying Challenge-Response Anti-Spam System

deliasee writes "The Washington Post reports that Earthlink is preparing to offer new spam filter technology that requires sender authentication. AOL is still concerned that such technologies will put too much burden on consumers." The day after it's deployed, every legitimate mailing list on the planet will get challenges from all the Earthlink subscribers...

Comments

Too drastic?

[mao che minh]

Drastic times call for drastic measures. The situation caused by the relentless onslaught of SPAM (which supposedly is rendering "damages" in the billions annually) can certainly be categorized as drastic. Is Earthlink's counter attack too drastic a measure, though?

On one hand it (Earthlink's new "technology") seems reasonable enough to the every-day-joe. I'm sure that the majority of Earthlink subscribers don't utilize news or mailing lists, and don't bother paying their bills online. For these people, it's fine. On the other hand, many others use online banking and other such automated tools (even account control mechanisms for online games will be affected). How quickly will all of these vendors conform to Earthlink's new technology and make the needed changes in their automated systems? Will Earthlink simply render many of these domains exempt?

The answer to solving SPAM resides in the current mechanisms used for the actual transmission and delivery, the mechanisms that all participants must use, not just Earthlink. This is of course the mail servers themselves.

Re:Too drastic?

[iangoldby]

People who want to continue to receive messages from mailing lists, online banking, etc, will have to add these sources to their whitelist.

It's a bit of a faf though, and I suspect many people will either not understand how to, not bother, or forget at least one address.

The solution is to have the incoming messages moved into a 'holding' folder that the recipient can see, and check in just the same way as checking through a 'spam' folder. This would remind the user to add false positives in the 'holding' folder to the whitelist. After a while, you can safely stop checking your 'holding' folder. Wouldn't it be good if this is what Earthlink are doing?

I think a scheme like this could be made to work, at least for webmail. For POP3, it could be a bit more tricky...

Re:Too drastic?

[Binestar]

Too drastic? I don't think so. This is something that is off by default, and needs to be turned on by the user. That user can also pre-approve e-mail addresses from his address book and mailing lists that he is on so that the challange never reaches those people.

This is just an added feature that users can use if they choose to.

As for the automated systems: It is the users responcibility to add those addresses to the accept list when (s)he signs up for the services.

Since this challange responce system has to be turned on by the user, it is only the user's fault if (s)he forgets to whitelist the address of places (s)he gives his e-mail account out to.

All in all it's definately a good option to have, but it's also a good thing that it is off by default, with the option to turn it on left upto the user.

Re:Too drastic?

[letxa2000]

Challenge-Response is bogus. I don't know of any such systems that have been deployed without significant problems for their users, the people that send mail to their users, and especially mailing lists.

If challenge-response is largely deployed, I suspect spammers will just unite such that one spammer sends a message, gets the challenge, answers it and is then "unlocked" to send message. He'll then distribute that email address in real-time to dozens or hundreds of other spammers who will send their spam immediately with the same newly-unlocked address.

Or, perhaps, spammers will change their tactic from spamming millions of users with 1 spam at a time to spamming 1 user at a time with dozens or hundreds of spam. You unlock the system with a valid response to the challenge and then flood them with spam until the user blocks that address.

I just don't see where challenge-response is anything more than a very stopgap measure. It's not particuarly "clean" now and will become more and more useless in the future.

Almost a year after Paul Graham's "A Plan For Spam" Bayesian is still the easiest system to develop as well as the easiest for the user to use. It is extremely effective (99.5%+) with very few false positives and doesn't require any additional effort for the sender and only requires that the user report false positives and false negatives--and that is mostly only needed at the beginning. Once it is initially tuned it's not necessary to do much of anything--it just keeps learning and working.

Re:Too drastic?

[Tackhead]

> People who want to continue to receive messages from mailing lists, online banking, etc, will have to add these sources to their whitelist.

Problem is, you don't know what that email is necessarily going to be.

I ordered something from foo.com and got order number 12345.

A few seconds later, I got a confirmation mail from confirm-12345@foo.com telling me what I bought and when to expect delivery. (Or worse, from order-12345@foo.com telling me there was a problem, and that I needed to fix something!)

If challenge-response becomes widespread, foo.com will say "Now you must whitelist the address confirm-12345@foo.com" when processing the order. (Or switch their order-processing back-end software to use something more sane, like "confirm@foo.com" and put the damn "Order 12345" in the Subject: header where it belongs!)

Problem is, until then, some vendors and some users using challenge-response are gonna be up the proverbial estuary without a utensil for propulsion.

If foo.com is disreputable, of course, challenge-response solves the donkey pr0n spam problem, but not the mainsleaze part of the spam problem. A mainsleazer at foo.com will simply start spamming his customer list with a From: of "confirm@foo.com" - Subject: "New Dealz from foo.com!" *sigh*)

Re:Too drastic?

[lommer]

Yes, I think the Earthlink measure is FAR too drastic, and whitelisting (with a holding folder), while it does solve many problems, is very inconvenient.

I am currently in the process of applying to universities as I am graduating this year. Many universities contact me by email. If I miss ONE important email from these universities, I am in danger of losing my application. Further, some emails that the universities send me are time sensitive, so that mandates checking my holding folder daily. Finally, many universities use auto-mailers to send out announcements and such that have an invalid return address, so confirmation emails don't have a hope in hell of getting through.

Combine all of this with the fact that many people at a university, with many different email address (sometimes in different domains even) may have to deal with my file and you can see my problem. Spam needs to be stopped at the source, not at my inbox because the consequences of even one false positive are just too high for me. Yes, this will mean that legislative measures will be required, not just technical measures. I realize that many slashdotters are not in favour of this, but this is the only way the spam problem will be solved IMHO.

Re:Too drastic?

[creideiki]

It's a bit of a faf though, and I suspect many people will either not understand how to, not bother, or forget at least one address.

Agreed. I think the optimal solution to allow for independently certified e-mail. Certification authorities would raise the bar (by requiring REAL forms of ID) for getting a user id which would need to map to a public key. Normal users could have this taken care of by their ISP, after all, they know who's paying for the service. This id would be guaranteed by the certification authority to map to a person or business, though, to protect privacy, no personal information would be stored - only for creating an ID hash.

Recipients should be able to file a complaint once per message per sender. The rating of a person or business would be cumulative (though possibly normalizing toward zero over time as old ratings "drop off"), recipients could just set a maximum evil amount or whitelist specific ids/keys that'd otherwise be considered too evil. This makes it very easy for recipients as they don't have to do much work and they can still recieve mailings that they just signed up for.

If a spammer or some other malicious type sends out a million messages and everyone complains, he'd have to wait until his rating normalized before he could reasonably expect people to recieve his messages again. Additionally, due to the requirements of proving who you are before getting an address, one couldn't just create another account (which also has the side-effect of ruining his other business ventures or his personal life as his only recourses would be a legal name change for himself or his business, or using non-certified e-mail).

Just my two cents, but I firmly believe that it's the ease of getting an e-mail address and the vunerability of implicit trust that allow spam to be rampant. Phone companies just don't give out numbers, a similar model for e-mail would be beneficial (though it would require the collaboration of ISPs and possibly independent certification authorities). Furthermore, spam is a technical problem and needs a technical solution not a legal one.

How do two people with C/R communicate?

[corsec67]

How do two people with challenge and response communicate?
If the challenge always gets thrugh, then the spammer will just issue cahllenges as spam.
If they don't get through, then you would have a nasty mail loop.

Re:How do two people with C/R communicate?

[Nutcase]

very good point. I would mod you up if I could.

You can't have an automated challenge/response system, because that defeats the point.

You can't have a non C/R address for the challenges to be sent to, because it would end up getting spammed.

Basically, there is a no communications barrier in place until they communicate.. which makes no sense.

Re:How do two people with C/R communicate?

[SomeoneGotMyNick]

The challenge is probably a randomly generated code to be returned before the original e-mail gets sent to the intended recipient.

Most spammers use fake return addresses anyway. The challenge will never arrive and the mail gets tossed. Thus, it never gets to the recipient. Voila, one less potential viagra purchase.

Re:How do two people with C/R communicate?

[tacocat]

True. But now the mail administrator has to deal with thousands of spam mail that doesn't get a reply.

And how long are they supposed to wait for a response. Remember, email is not supposed to be a Real Time system. Email servers frequently have a delivery retry schedule of about 4 days. That would mean that Earthlink has to carry the entire spam volume of four days in some kind of mail pending queue and to periodically attempt a redelivery.

I've tried this myself. When you can easily run 100+ spams per day per account, imagine what you are going to be dealing with for an entire ISP. You can easily scale into the million email queue.

Their servers will not be able to handle their entire population and the resulting network load on themselves and everyone else will be prohibitive.

Consider this. AOL and HOTMAIL are the largest spam address sources, real or imaginary. So, when they get spam from AOL, they have to attempt a delivery. If AOL's system doesn't allow for immediate failures based on "address unknown" then EarthLink will hit AOL with thousands of bogus email delivery attempts. Now the two goliaths are beating each other to death over bandwidth.

Someone will be suing for a DOS attach.

too much hassle

[chabegger]

I think this will create way too much hassle. There are some people who wouldn't mind, but others (such as grandma) who have to be told three times where the power switch is won't really know what is going on. At least now when I don't reply I'll have a decent excuse... "but grandma, you forget to send it twice, so i didn't get it"

Now the spammers get address validation for free

[chefbimbo]

Seriously, what are they thinking? TMDA might seem like a nice idea in theory, in practice, it's a pain to use and not exactly safe either. Once this gets widescale usage, the spammers will simply start responding to the challenges (after all, it's not like that couldn't be easily automated).

Good idea, but...

[onemorehour]

This seems like it might be a good step, but it's missing the point. The only thing that will truly curb spam is to rework the SMTP protocol to not implicitly trust every host, as was mentioned in an earlier /. article.

They should offer it with new email address

[dnoyeb]

me@challenge.earthlink.com

something like that. So that it allows users to gradually changeover to the system. That would allow them to be more extreme in their refusal to accept emails and much less compromising.

I like it.

Re:Nice moves

[apoc.famine]

I dunno. This may be painful for a bit, and increase the amount of mail, but in the long run it might be worthwhile. While I agree that it makes some peoples' jobs harder, those people probably aren't using the major ISPs/mail-services. If the major players do this, it makes it that much less profitable for spammers to do business.

I mean, if you're a spammer, a brute force mailing to joeuser.org is MUCH less profitable than mailing the same million messages to hotmail.com. Go big guys, go! It won't bother me at all.

Fill up the ISP servers

[nuggz]

So when a spammer fires a few hundred or thousand emails to an ISP, they will sit on the mailserver waiting for him to respond.
Since the from address is faked, that same ISP will launch an acknowledgement flood against a third user.
Excellent.

I just see so many tricky things that someone somewhere will screw up.

Re:Fill up the ISP servers

[stratjakt]

The ISP sends only one challenge. You respond once, and henceforth are allowed to send as much as you want.

Now if I wanted to Joe Job some guy, I just pick someone who's chances are good that he's already allowed through earthlink. Say the maintainer of a mailing list with earthlink subscribers.

I've said it before. This is just a step towards making SMTP a pain in the ass, and obsolete. We can look forward to a high tech pay-per-use replacement in the future. Yay! Paying to send e-mail, I cant wait. But at least the two or three spams I get a month will be gone.

Challenge - Response doesn't work

[tshak]

What happens when the customer orders something from Amazon - the purchase confirmation email comes from a non-human address.

Just the other day I got an email from a company that I ordered software from describing a free upgrade that I could download. It came from donotreply@[host].com, meaning, if I was using Earthlink's system I probably wouldn't have received it.

The problem with Challenge - Response is that it makes the assumption that if there's not a human behind the email that it's spam. In practice, there are many legit emails that are not individually sent by a human.

bad protocal: SMTP

[JDizzy]

The answer is not attaching more bad ideas to an already bad protocol. The ultimate answer is in the protocol designers. A government/state can pass as many laws governing the interaction of people/things with the bad protocols, but the IETF/IEEE will still create them, and certify them. People should just wake up and realize that SMTP is to blame for this big mess. ISP's should stop offering SMTP outright, and think of ways to replace it. Chat programs are probably a better way to pass messages anyways. SMTP has become a massive bazaar that is full over everyone on earth, and since it is completely open, its also completely ok to send bulk mail. Forging headers is another issue, but simply spewing email is intrinsically allowed by the protocol, and thus taken advantage of. If everyone one on earth had a computer, and everyone on earth sent email to everyone else on earth every day, would that be spam? No, because it would cross the line into accepted practice, and that is what we are starting to see due to the sheer bulk of spam sent to everyone on a daily basis. The point is that as long as SMTP exists, so will spam. The answer is to replace SMTP with something that doesn't allow spam to exist by removing the ability to anonymously send people messages.

Re:Nice moves

[d_lesage]

It drives network traffic as well up to the sky

But wouldn't the added traffic be more than compensated by the reduction in traffic that would ensue when the spammers go out of "business"?

Wow, nobody understands this!

[MrPerfekt]

I see a slew of people saying "blah blah blah, they'll automate the response blah blah blah". And apparently, to alot of you, this is all new.

This is something that's been around for a few years and gee, spammers haven't gotten around it yet. C/R antispam systems work because spammers don't use valid Reply-to: or To: addresses.

If they did and the spam gets through the system, then great! There's one more point where we can nail them on when/if we go to hunt them down. Oh, you used your dialup with an SMTP server to auto-respond to the challenge (which is probably alot of work for the average evil spammer), great, email abuse@isp and have his account shutdown.

Since I have started using ASK to C/R my email. -zero- spams have gotten in my Inbox (which is what annoyed me the most about spam, the false positive I got when the little sound would ring telling me I had new mail.)

Intrusive? PLEASE! How lazy are you? Hit reply -once- and you'll never have to see it again when sending email to me. I'd say getting pelted with 200 spams a day is slightly more intrusive to me than what you're going to have to do to send an email to me.

Re:Nice moves

[darien]

Er, what?

eMail was not designed for such a challenge

So what? This system works within the standard. Who cares whether or not the designers foresaw it?

It drives network traffic as well up to the sky.

Hardly. If you're on Earthlink and decide to opt-in for this, it simply means that everybody you know has to send you one extra email once. Earthlink's traffic may be a bit higher for the first few days, but once people get their whitelists in order it'll drop back to where it is now - and below, because there'll be less spam floating around.

However, I do hope (the article didn't say) they've come up with a smart solution to the problem of spammers putting real (but stolen) addresses as their From: address. Otherwise people unlucky enough to have their addresses stolen may indeed find their network traffic increases, thanks to a million challenges from Earthlink.

Re:Needs to be 'hard' in some way

[Cirvam]

So how do you respond to a challenge if you are just using a terminal or are blind? Obviously if the characters are obscured, the screen reading program can't read it, and they would have to be a graphic of some sort. Unless they just make an alt tag that tells you what it is. :)

Re:Nice moves

[tacocat]

These systems don't work that well. I have been designing and building my own for about 8 months now and have come to the following conclusions.

They are easily bypassed using a smart enough auto-responder. If all you do is fire back the original message then you're on their list.

They sometimes fail to pick up the human response. I have several cases where people will simply respond to the email, removing enough of the critical content, to render the reply useless. This comes in two flavors. Email clients will strip out the Header information needed, or people will strip out the Body information needed.

To impliment this upon a very large system like this is going to be a nightmare not only for their email administrators, but for everyone that they touch.

One of the biggest problems that these systems have is that they are totally incapable of handling Solicited email from a Bot. Examples include:

• Payment Confirmations (amazon.com)
• mailing list confirmations
• Profile Update Notifications (paypal, ebay..)
• Password changes or resets

It's going to be a pretty ugly system of implimentation.

Re:Now the spammers get address validation for fre

[PerlGuru]

It would also be a problem for people with text based email clients

Re:You can do this yourself.

[WetCat]

Well, imagine you have no job and selling yourself
You posted the resume, and waiting for emails.
Do you seriously expect that prospective employer will have time to respond to "confirmation" message?

Re:Relative speed

[dasunt]

The parent poster writes:
Remember when 14.4K was fast? So do I. And I think with a correction in the system, it can be a decent speed.

Nope. Sorry. There are 2 reasons why 14.4K will never be fast again:

1. Graphics. There are plenty of web pages that are not optimizing for graphics, and plenty of web pages that are using more complicated technologies (such as flash) where simple technologies (such as gif) will work.
2. HTML Mail. Isn't it wonderful how a simple "Meet you at 5" can end up being bloated to half a meg with a "pretty" html background?

Re:Now the spammers get address validation for fre

[Chester K]

Once this gets widescale usage, the spammers will simply start responding to the challenges (after all, it's not like that couldn't be easily automated).

In order to send responses to the challenges, it means the spammer has to provide at least a valid return address, and dedicate resources to responding to those requests (even if it is automated). It raises the cost of sending spam, and increases accountability due to the valid return address requirement, which is the best we can hope for with a SMTP-based solution for the time being. It's not perfect, but nothing is.

que?

[Anonymous Coward]

They are easily bypassed using a smart enough auto-responder. If all you do is fire back the original message then you're on their list.

Did you read the article? A picture of a word is sent to the sender. The sender then has to TYPE the word in a response email.
The autoresponder would have to be able to analyze a picture and interpret what 'word' was being shown. There are ways to make this more difficult for an AI to do.

They sometimes fail to pick up the human response. I have several cases where people will simply respond to the email, removing enough of the critical content, to render the reply useless. This comes in two flavors. Email clients will strip out the Header information needed, or people will strip out the Body information needed.

Maybe the system YOU designed words that way, but there should be NO reason why a response email should be rejected if the respondee followed directions.

One of the biggest problems that these systems have is that they are totally incapable of handling Solicited email from a Bot

You have a point here.
The fix would be for the enduser to be able to manually enter approved addresses. I.e.: I manually add in the rule that says mail from amazon.com is allowed.

ac

Thoughts and observations

[cr@ckwhore]

First of all, the system is completely optional for earthlink users. For the users that are stupid enough to opt-in, they deserve the extra hassles they'll receive.

But here's what it means to me, a publisher of a popular website...

When a new user signs up for an account, they get a confirmation email. Since I'm not about to check the server's return-path for C-R messages, C-R users will be out of luck. This means that at the very least I'll have to update my site with a special notice during the sign-up process that will notify earthlink users to expect problems.

The crux of the matter, there are automated emails that will fall victim to this C-R paradigm that AREN'T spam!

So, what is earthlink's "fix" for this problem? Well, it appears as though they will assign special addresses that users can use for sign-ups, sales receipts, etc. that will bypass the regular C-R system. Ok, great. Two problems with that ...

1. If the special bypass addresses are only temporary, then my users' accounts will become invalid because their email address is no longer valid and I don't allow ghost accounts.

2. If the special bypass addresses are permanent, and they're used for sign-ups and sales receipts, well fsck! Thats where SPAM comes from. duh. Great ... all their spam will arrive via bypass addresses. Awesome!

Re:Now the spammers get address validation for fre

[StarOwl]

Once [TMDA] gets widescale usage, the spammers will simply start responding to the challenges (after all, it's not like that couldn't be easily automated).

There are currently three defenses to this:

1. Most spammers dummy up their headers. The challenge never gets delivered to them, and therefore the spam goes undelivered.
2. Spammers who use legit email addresses usually see their inboxes fill quickly to the point of bouncing mail. Again, they don't see the challenge, so the spam goes undelivered.
3. Spammers who use legit addresses and have large inboxes are likely to be trackable. If they're in your country, and if your challenge message is worded correctly, there is some legal exposure on their part.

Admittedly it's not foolproof. There is no 100% effective way to combat spam (short of abandoning SMTP). There's always going to be a risk that some spam will leak through or that some legit email will bounce.

Re:Relative speed

[evilviper]

Heh... My first response when reading this was "Good for them..." That was until I remembered that Earthlink is my ISP... I just don't happen to use their E-Mail service. Guess I'll have to pop over to their website now and figure out what their e-mail settings are.

Remember when 14.4K was fast? So do I. And I think with a correction in the system, it can be a decent speed.

Well, the solution can be implimented on the user's end... I personally use Privoxy to filter out just about every ad and flash animation out there.

What I would like to see, is browsers giving preference to content, rather than bloat. Just imagine, you have an incredibly slow modem, but web-pages open-up instantly. You open 10 links at the same time, and they load right away...

The only thing browsers have to do is load the HTML first, then, only after each HTML page has been fetched, should it begin to fetch the images (smaller ones first, preferably), and flash animations or other embedded content last. That would be a great way to counter web-site bloat, and I'd consider it rather fair too.

If you look at the page for a seconds, and decide it isn't what you want, the bloat won't even be loaded... If you read it for a few minutes, the ads will be loaded eventually. Text ads, will be loaded instantly.

Pre-emptive Anti-Spam Measures

[akedia]

I've used Earthlink as an ISP for going on 6 years now, and I must say, I've never dealt with better. For one thing, in the years that I've had my earthlink address, I'd say I never get more than 3 or 4 spams per week. What is my secret? For starters, if I need to provide an e-mail address for something that may result in unsolicited messages, I use one of the free webmail providers (Hotmail, Yahoo!, etc.) I can check those to confirm what I wanted, then never check it again, and my Outlook (with my primary e-mail) doesn't fill up with useless crap.

Another way to stop the spam before it starts is to keep your e-mail address from getting on those lists in the first place. When posting to Usenet, BBSes, forums, even Slashdot, use some sort of clever cloaking (Slashcode does this already), or even a fake email. Encryption for e-mail such as using a free personal certificate from Thawte or a GPL encryption such as GNU Privacy Guard is always a good idea.

In addition, Earthlink's Spaminator is a Godsend. With that baby enabled, I'm lucky if I get one spam a month. Case in point: my mother has an Earthlink address that she uses for her business contact. She complained that she's getting hundreds of porn spam and "enlarge your penis"-type e-mails (no idea how these got here.) Setting up a few Outlook Express filters and enabling Spaminator cut the dirty messages by about 90%, and she is grateful she no longer has to wade through such filth to get to her real mesages.

The bottom line is, the fewer spammers that have your address, the fewer spams you're gonna get. I have a Hotmail that gets 1000+ spams a day. My real e-mails get next to none. It's just like telemarketers, they get your number from companies who need a contact info for whatever reason. However, Hotmail address are free, whereas extra phone numbers to give the telemarketers, and then never answer, are not. Well, we do have Caller-ID for that, but that's another post...

Having written a similar system, I have questions.

[kaoshin]

If someone from earthlink emails someone else from earthlink, how would challenge response handled then? Do they make all mail that is sent returnable without challenge responses, and if so is this a temporary rule or are the addresses of all mail you send permanently whitelisted?

If the challenge response triggers a mail daemon reply, is it filtered or do you get flooded with those replies caused by all the spammers with forged addresses? If they are filtered, how do you know when mail you send doesn't go through without the use of message reciepts since mailer daemon replies are all different.

If I mass email tons of earthlink addresses with a forge from address, would it mailbomb the fake address, or do they have flood protection to prevent this?