Email Authentication Schemes - Friends or Foes?

jtprice writes "At a time when spam levels have exceeded 80%, there's growing momentum behind Microsoft's email CallerID, the SPF effort, Yahoo!'s DomainKeys, and the IETF's new MARID Working Group initiatives to address various email abuse problems including spam, joe-jobbing, phishing, and so on. Sendmail has already implemented DomainKeys and CallerID. 10,000+ domains have turned on SPF now. Where the heck are we going with this? Are these efforts at cross purposes, confusing at best or likely to be consolidated? Seems to be less about the end of spam and more about the end of open, uniform, standards-based email as we know it. Apparently the people behind these initiatives are getting together for the first time for something called the Open Email Accountability Symposium next month, at the Inbox Email Conference in San Jose, with the intent of outlining their proposals and answering questions. Any thoughts about all of this, or hard questions that should be asked of these people? Is the email dilemma creating just another monopoly opportunity to force email into proprietary territory?"

Comments

It's worth it...

[danielrm26]

"Is the email dilemma creating just another monopoly opportunity to force email into proprietary territory?"

Perhaps, but this doesn't make it a bad idea. Any good idea or technology can be taken advantage of; that in itself shouldn't keep those with good intentions from trying to bring about change for the better.

Re:It's worth it...

[alchemistkevin]

you might say that today, but i'm sure you'll be shouting the loudest to put it under GPL when microsoft or other 'perceived' monopolies get away with it.

This would have been first post..

[slittle]

..but my corporate overlords got hit by Sasser and I couldn't get a new certificate :(

It still won't work

[Anonymous Coward]

Because millions of small-company and household e-mail servers will never have the funds to implement any proprietary system. Make it public domain, and it will be worse- the spammers will just hack it.

Re:It still won't work

[Nasarius]

Right. Just like they hacked Apache or PGP or SSL or...
Open standards and peer review are profoundly *good* things.

Re:It still won't work

[Electrum]

Most email currently goes through Apache . . . I think that the open sorce community has done a pretty good job of creating the email server of choice.

Umm...

Re:It still won't work

[jonesvery]

Most email currently goes through Apache . . . I think that the open sorce community has done a pretty good job of creating the email server of choice. I think that they're probably the right group to also make it more secure.

To clarify someone's "ummmmmm" comment -- this is some sort of weird troll, right?

The Apache Software Foundation does support a project known as James, a "pure Java SMTP and POP3 Mail server and NNTP News server, but ummmmm...well, not a whole lot of people use it.

Are you perhaps thinking of qmail or postfix?

no real solution on the orizon

[AnalogFile]

I've been thinking about the problem. And have looked around for the different proposals. There's been a mailing list for ng mail with many interesting ruminations. But then it was sinked with spam :-(

IMO there main alternative is:
1) a solution compatible with original RFC (that is it does not rule out any sender that the original spec would permit)
2) a completely new and different system. Redesigned from scratch.

Obviously a solution is not a solution if it may have a false positive (block nonspam).
False negatives are just a matter of efficiency.

Methink option 1 is not possible. And this has the added bonus of giving us the chance for a visionary change. But it's unclear if we can afford the time it takes. As the problem is really becoming urgent (much more urgent than the 32bit limitation in IP adress space. Expecially because NAT is addressing it very well.)

There are MANY proposals that use SMTP and add up on the requirements actually ruling out cases that were originally legal. These I really think should be avoided. But I'm affraid that's were many will likely go because they are fast to deploy.

Re:no real solution on the orizon

[CatLord42]

IMO there main alternative is:
1) a solution compatible with original RFC (that is it does not rule out any sender that the original spec would permit)
2) a completely new and different system. Redesigned from scratch.

Even if we could completely revamp SMTP, it still sits on top of TCP/IP (etc.), and there will still be ways to get around any protections we could add to SMTP.

Unfortunately, I think it will take some major overhauling of the Internet and its core protocols to solve this problem. And that means lots of work, lots of new equipment and lots of new applications, all at enormous expense.

So, what's worse, loss of bandwidth, over-burdened mail servers and everyone spending time deleting junk out of their inboxes, or everyone spending a significant amount of money, users for new e-mail programs, companies for the same programs, new mail servers and routers, ISPs and backbone providers for expensive new infrastructure, and none of it possible until all the protocols are reworked, let's say, five years from now?

Re:no real solution on the orizon

[AnalogFile]

Even if we could completely revamp SMTP, it still sits on top of TCP/IP (etc.)

Not exact. If we are revamping there's no need to sit on TCP. It may be TCP or UDP or something completely new. Or it may be even just be a non problem.
- If the protocol assumes a connection and does not depend on it being anything in particular (technically: if it's an appllication level protocol), than it'll sit on any connection oriented protocol. That's exactly what the ISO layering is supposed to mean.
- It is possible to design a completelly new connection layer protocol. TCP is having it's own problems. True most of these have been addressed with a handfull of extensions. Reno is good and Vegas is even better. But big speed*RTT links are still problematic. And links are going to become much faster and with possibly bigger RTTs. We should not abandon TCP. But maybe we could start thinking alternatives.

and there will still be ways to get around any protections we could add to SMTP.

There'll always be ways to get around anything, probably. Down this line of thinking there's no solution at all. But we may come to a point where getting around is not worth doing.

I think it will take some major overhauling of the Internet and its core protocols to solve this problem.

Ageed. That is exactly the point in my post. But if we take that lane, we may get extra benefits. Mail-ng need not just be mail. We may think messaging here, instead of mail. Mailing lists can be designed in upfront. And news too. Maybe even chatting and instant messaging. And did you notice people is now using SMTP to do what FTP was designed for (remember FTP supports push and even sidewise transfers, even if today it's mostly used in pull mode)?

And that means lots of work, lots of new equipment and lots of new applications, all at enormous expense.

Maybe. Maybe not. We should keep those possible consequences in mind. Lots of work in SW development may be a non problem (not for the F/OSS community, at least. I do not care what that means for an individual company that's not going to share). Lots of equipment I doubt. If we can sit on IP and care not what version of it is below us, than the routing infrastructure need not change. The firewalling/natting/tunneling part may need some fixes. But these are mostly SW and generally very close to the endpoints, not really a big deal if we are doing a revamp. Expenses? Again: SW upgrade at the endpoints is not a big cost. Not if you are on the sharing side of the fences. It is not zero (not for large companies with thousands of servers and more clients, for example). But it needn't cost more than any other SW upgrade.

none of it possible until all the protocols are reworked, let's say, five years from now

This sentence is the one I agree upon. That's what I worried about in my post. This may take time. And the problem is now so much urgent that people may be unwilling to wait. The worse that can happen is a partial solution. It would slow down a revamp (NAT slowing IPv6 is an example. And we risk the mail 'solution' is much worse than NAT is).

Re:no real solution on the orizon

[T-Ranger]

If the protocol assumes a connection and does not depend on it being anything in particular (technically: if it's an appllication level protocol), than it'll sit on any connection oriented protocol. That's exactly what the ISO layering is supposed to mean.
Prehaps that is what it is supposed to mean, but two things
- IP and friends came out long before ISO and their layer model. IP is based on the DOD model; and while DOD has less layers they dont exactly match up. Not really important, however
- SMTP uses lots of things that assume IP. HELO/EHLO wants a hostname, and usually one that can be resolved to the address that the connection is from. From (in SMTP), From, To, CC, etc in RFC822 expect either UUCP routes, or IP hostnames. Any MUA written after 1990 would likely get confused if presented with a UUCP route.

New stuff
Cisco came out with IPv6 around 3 years after everyone else did, including Microsoft (Research). While a p120 with Linux is just fine in your apartment, and Cisco has competitors, Cisco not supporting something (relevent to their product line) means it wont become wildly used.
- You don't necessaraly need new hardware, but routers are heavily optimized for dealing with IP sized data chunks.
Upgrading endpoints v. backbones: lots of people are running IPv6 over v4 tunnles. Im willing to bet there are exactly 0 production, commercial, IPv6 backbones. New things happen on the edges.

Five years is a very short period of time. The reason why IP is in use is because it works. There are more then a few system that are "better" on paper. IP has been proven to work, and is the only protocol to acheive its level of sucuess... v6 is in the process of being proven (and refined)

Re:no real solution on the orizon

[Ramses0]

Check this one:

http://cr.yp.to/im2000.html ..."Internet Mail 2000" ... no really. :^) Basically, you (user) don't receive the email. You receive a notification: "your email is available". Now: you have to download the email from the site that notified you.

Imagine: I am a spammer. I must now host a server which has the capability to receive $millions of hits from all my wonderful spam-receiving customers. This is the first thing which begins shifting the burden of sending of email back onto the sender.

Obviously it's not perfect, if I were a spammer, I'd probably have all my malware-infected robots be the hosts for sent emails. Buuut... they'd only be active when that malware-infected computer is on. Notifications could obviously be cheaper to send (b/c it's not so bandwidth intensive, just "hey, joe@joe.com has an email available at bob@spammer.com/msgid/12345/hash-password/a1e2ff" ...but, this scheme has the potential to conserve enormous amounts of bandwidth internet wide (mail is "fetched if desired", not "sent no matter what"). If too many reports of spam are coming from a particular server, blacklist that server. Boom. There are a finite # of IP addresses on the internet, and forcing a spammer to maintain servers so their messages are successfully delivered should eventually drive them further and further out of address space.

This speaks really to "trust v. ip address", how much do you trust a residential DSL IP to deliver mail? (not much). How much do you trust Yahoo Mail's IP? (a lot, all kidding aside).

So, what's the transition? Since it is such a drastic change (rather than sending mail, sending notifications), and directly impacts $zillions of installed mail clients, I couldn't wrap my head around a valid transition path until it hit me ... use email! ;^)

Since most mailers nowadays support highlighting of http links, just send "Subject: regular", but make the body: "Internet Mail 2000 message. http://sending.server.com/msg/12345/hash/abc123" ... totally painless transition (apart from vulnerable browsers), until mail programs can be re-architected to fix it. ( if: is_im2k(): fetch( $url_from_message )

Anyway, think about it.

--Robert

Re:no real solution on the orizon

[smcv]

IM2000 does sound like a good idea; it's basically the way I send inconveniently large attachments, in fact (zip, upload to temp directory on web server, send an email "covering note" with the URL, ask recipient to let me know when they've downloaded the file so I can delete it).

The immediate down side I can think of is that the sender knows (by observing their web server logs) that you received and read that message (or at least that you received it with a POP3-equivalent offline mail reader, which would presumably just have to download everything that wasn't blacklisted). This is possibly a good thing (debatable) if it's legit email, but is a bad thing if it's spam (the spammer now knows that joe@joe.com is a valid address which is read by a human).

Re:no real solution on the orizon

[SagSaw]

Imagine: I am a spammer. I must now host a server which has the capability to receive $millions of hits from all my wonderful spam-receiving customers. This is the first thing which begins shifting the burden of sending of email back onto the sender.

The problem with this idea is that, on a per e-mail basis, spammers actually need fewer resources than non-spammers. Compare a spammer whose domain sends 1 Million e-mails a day, and a legitimate domain who also sends 1 Million e-mails a day. For the legitimate domain, most of the e-mails sent will be unique, most will be read, and it is probably vitally importatnt that the readers be able to access the sending server at all uptimes. This means the legitimate domain must maintain a mail server which can store millions of messages, which has a very fast connection, and which has a very high availaibility. The spammer, on the otherhand, is going to send out many copies of the same message, only a few of which will actually be read, and really doesn't care if his server is down for a few hours (after all, the same suckers who read the e-mail the first time will probably read another spam sent to their address). As a result, the spammer can get by with much more basic hardware. Which brings us to the second point:

if I were a spammer, I'd probably have all my malware-infected robots be the hosts for sent emails.

Here we have the basic hardware. Since the spammer doesn't need much storage, the user won't notice their computer hard-drive is filling up. All they will notice is that (if the spammer is successful) their connection is slower than normal. This will probably be blamed on the p2p software of the week the user's teenage daughter installed.

Re:no real solution on the orizon

[AnalogFile]

HELO/EHLO wants a hostname, and usually one that can be resolved to the address that the connection is from

What about this ?

From (in SMTP), From, To, CC, etc in RFC822 expect either UUCP routes

What about this ?

Of course those links may be wrong. I'm no big expert. Surely they seem convincing.

Cisco not supporting something (relevent to their product line) means it wont become wildly used

This is sadly true. And that's probably a sign CISCO should either make an extremely strong public commitment into early availability (even before the standard is actually a standard) of new protocols or be ready to face antitrust the way MS is doing. Or both. And wether the aforementioned commitment is done making the platforms totally open or investing in-house for development that may never turn out to be required is not my problem (but I have an opinion).

New things happen on the edges.

Amen!

Five years is a very short period of time

True. I did not make that number. Just refused to comment. But since we are at it: I think that 10 to 15 years for the new generation mail infrastructure is more likely. And that's exactly one of the problems as many will not be willing to spend 4 or 5 years just for preliminary talks. They want spam killed now. And will, sadly, be happy with half baked (non)solutions.

Off topic, but...

[Nasarius]

Expecially because NAT is addressing it very well.

No, no it's not. NAT is a quick-fix that just complicates matters.

The question answers itself

[kimba]

The person that asked the question provided the answer. The IETF MARID working group is designed to take all the existing proposals, find the best, and settle on it.

This article is just making something out of nothing.

Client-side filtering should be last step

[TBone]

The problem is, in many places, people still pay per quantity of bandwidth or time online. Saying "filter it at the client" doesn't do anything to stop the spam from being sent to the user, and still requires the user to retreive and parse the message before deciding it's spam and filing it in the circular bin.

No, client-side spam filtering should be the last line of defense against spam. Spam should be killed off before it ever reaches a mailbox, final or intermediate, by the servers that handle the mail.

CallerID for email boycott

[Anonymous Coward]

The concerns presented by the boycott of Microsoft's Caller ID still haven't been addressed by Microsoft. It's still patent-encumbered, still far too verbose and still not used by anyone besides Microsoft and Amazon.

Stick to SPF, give DomainKeys a try once someone actually publishes some info about it. Skip caller ID.

SPF is Email Authentication

[jgardn]

SPF only authenticates mail as being approved mail from a domain. In itself, this only prevents joe jobbing and phishing, but domains can still send spam.

As SPF adoption grows, there will be two types of email: authenticated and unauthenticated. Authenticated mail will consist of both spam and legitimate mail. Unauthenticated mail will be just like the mail we are sending around today.

What does authenticated mail get us? As we can track mail down to the owners, we will begin to set up a trust system. DNS block lists will become viable. The owners of domain names can protect or abuse their domain names as they see fit.

Eventually, there will be a system where domain names will have value again. If I don't abuse my home domain, and only use it for legitimate purposes, people will not add my name to black lists. If my domain has sent a large number of emails with a very low score of spam, it will be more legitimate than one who has sent only a few emails or has sent mostly spam.

SPF is only the first step in stopping unsolicited email. Once it is in place, the next step -- accountability -- is easy to implement.

The beauty of SPF is that it doesn't invalidate email as it is now. Participation is optional. Those who are early adopters get an early boost, so the incentive is there to adopt it early on. But email as it is now will not be stopped.

Re:SPF is Email Authentication

[John Hasler]

> What happens when $VIRUS turns your domain name
> into a spamfest?

You get blacklisted, as you should be.

> If you're supporting any normal users at all,
> you're likely going to find it hard to maintain
> that reputation.

Securing your domain is your responsibility.

Alternate Solutions

[lachlan76]

Backwards compatibility and security can't be combined. Just like you can't simultaneously find the position and momentum of a particle with perfect accuracy, any attempt to make a certified mail system backwards compatible with existing systems means that spam will still exist. So far the most promising method of slowing spam is cryptographic challenges. By sending the client a simple cryptographic challenge which can be solved in anywhere from 10 seconds to 1 minute, spam can be slowed significantly, since the limiting factor is a technical one, which can only be bypassed by running the servers on a *HUGE* server farm in order to keep the spam flowing at the rate it is today. The other problem is that if the key size is too small, then the challenges can be precomputed, although using a different IV would increase the storage space need to precompute a list of RC4 values by a factor of 2^24. Or, a distributed computing task could be used (you must text X RC5 keys or OGR nodes to send Y emails) to not only slow spam, but provide practical information about what amount of time it would take to crack a given encryption algorithm. Other distributed computing tests could be used of course. The only way to stop spam is to create a controlled (timewise) mail system completely incompatible with existing SMTP clients, as SMTP is anonymous and uncontrolled. A public/private key system could also be used to verify the identity of a sender, and spammers can easily be identified (ie. you get a spam, you verify it with the public key included in the mail header, and send the key to a central server. Want to send mail, but haven't got a key? Too bad, since your mail can't be traced back to you. What's that you say? Make your own key pair? Well then your mail is rejected by the server too, since you do not have a valid key) bye their registered public key. SMTP and similar, unauthenticated mail systems, though, can not be beaten for reliability, since there are no central servers to go down, and as long as the sending and receiving mail servers are working, what is sent is always received.