Yahoo! Develops Anti-Spam Architecture

prostoalex writes "Yahoo!, the owner of one of the largest e-mail systems in the world, is said to be developing a cryptographic product that will be offered freely to mail servers. 'Domain Keys,' according to the Reuters article, would require the message sender to authenticate in order for message to come across a trusted e-mail network. The idea has been around for ages, however, it required someone from the big league like Yahoo! to step in." While Yahoo! isn't the first name that comes to mind when I think of trusted email, it's still a step in the right direction.

Comments

Re:Oh yeah it seems like a good idea right now....

[pdaoust007]

I can't see how they can "lock" anything since it is clearly stated that the initiative will be open sourced...

Of course, Microsoft will probably figure out a way to break it so that it only works with their products but that's a different story...

Open standards?

[satyap]

As long as it's an open standard that eventually becomes RFC3821, I'll be okay with it. But if it's one of those proprietary "pay us to participate" schemes, they can go jump. Oh, and there should be no scope for someone to say "pay us or we won't accept email from you.

Must be missing something

[Space cowboy]

The text of the article has to be wrong - they say the private key is delivered as a message header! Hmm, not very private...

I'm assuming that what is sent out is an encypted token for which the public key can be used to decrpyt, so:

• Alice wants to send an email to Bob.
  
• Alice encrypts the MD5 checksum of the mail body content (or some other representative text, probably longer than 32 bytes!) using her private key, and embeds the resulting encoded string into a mail header
  
• Bob receives the mail, and looks up Alice's public key to decrypt the token
  
• Bob compares the decrypted token with the same representative text to see if they match.
  
• Match => Read. No match => Put into 'Junk' folder
  

So, the token to be encoded will change from mail to mail, thus making replay techniques pretty much impossible, I think. At least, that's the way I'd do it, and I'm pretty sure I've seen it presented before as well...

On the other hand, I ain't a security expert, so there's probably a gaping hole in the above :-)

Simon

Re:Oh yeah it seems like a good idea right now....

[swb]

It can be open sourced, but that doesn't mean anything about preventing lock-in.

Presumably a 'domain key' is some cryptographic element that authenticates that your domain is who it claims to be. To me this sounds an awful lot like SSL where a third party issues the keys, or acts as a clearinghouse for self-issued keys.

Either way, Yahoo could be the man in the middle acting as either issuer or clearinghouse. Think of it this way, OpenSSL is open sourced, but that doesn't keep the SSL issuers from having a lock on that market.

Re:So now...

[gbjbaanb]

yes, but now you'll know for sure that the email came from Yahoo - and not some forged return-to that dumps on some ordinary Joe's server.

step, by step, the spam problem can be solved. That doesn't mean that you should not take the first step simply because it doesn't provide a total cure.

One solution

[FonkiE]

when you think about it, BUT this should come from IETF or some other body not from a company. A few important points:

1) Who will issue the keys?

2) Is anonymous mail possible if the receiver allows it?

Furthermore spamming is a social problem emerging from our commercial world and technical solutions can never be 100%. What if:

a) I send spam from a "secure" domain?

b) forge certificates?

c) the certificates are too expensive? (like SSL, I think it should be included with a domain)

I like the "Bayes" spam filters best. You get 99.5% spam protection and keep anonymous mail.

We all see the need for authenticated senders (biz communication, etc.), but we should be careful ...

Re:One solution

[the uNF cola]

... this should come from IETF or some other body not from a company.

We should expect something like this to come from the IETF, but big corps do good things all the time. What makes you uncomfortable about it? The privacy issue? If it's on the net and you want privacy, encrypt the content. But if you want to hit my network w/ SMTP, much less an ICMP package, I want to at least know who you are.

Are you worrying who will govern the entire thing? Who do you trust? Some .org run by someone? Some corp? The gov't? All-in-all, you have to trust SOMEONE.

Re:One solution

[jaybna]

There is a simple solution for this - make senders pay a fair price per K of email sent. Charge commerical email senders and exempt isp-to-isp emails - then only legitimate companies can afford to broadcast. This works very well in the direct mail world. Postage is a barrier to entry.

Re:Trusted email?

[hey]

I use Yahoo mail and its very good.

They have a pretty good spam catching service.
It puts suspected spam in a "Bulk" folder. You can
review this folder or just like it get purged after 30 days. Nice. You can also click on the "its not spam" / "this is spam" buttons to help them tune.

They offer a SSL login and it was discuessed recently on Slashdot that they use the Javascriptcrypto library to calculate MD5's on the client side and send the digiest for seduvcity (maybe when you are not logging in with SSL).

You can check your POP3/IMAP mailboxes. The resources come back color-coded.

Good uptime. Always available.

It's free. You can enought resources for reseaonable use. But you can buy more if you want.

All this sounds exactly like a crypto-nerd and slashdotter would design a mail service. And this new thing is going to be opensourced!

User account verification

[pe1chl]

First let them implement some user account verification, so that a RCPT TO: results in a 550 reply when that user does not exist.
This enables SMTP callbacks to stop spam being spoofed "from yahoo", just like everyone else does.

Re:Not necessarily

[GordoSlasher]

While some ISPs might quickly jump on the bandwagon to be able to accept mail from yahoo.com, most corporations will not. Even if Microsoft updates Exchange Server to support this, how many corporations are going to upgrade? It's a major deal to upgrade the email servers at a big company, and corporations that don't deal directly with consumers probably get an insignificant amount of mail from yahoo.com, so what's the business motivation? If it's so I can receive a personal email from a friend, my company will tell me to stop using the company servers for personal use.

Re:Not necessarily

[zzxc]

Not accepting it would be the wrong answer. It should be an option on an account to have a secure inbox with known-good mail, regular inbox that may have spam, and bulk which is mail known to come from spammers. This would be perfect to use as a spam assassin complete bypass. Regular mail could still come through, but would be subject to your filtering. This is definately a Good Idea.

Only for GPL players?

[Johnny Mnemonic]

From the article: Yahoo said its "Domain Keys" software, which it hopes to launch in 2004, will be made available freely to the developers of the Web's major open-source e-mail software and systems.

But later: Garlinghouse also argued that Yahoo's proposal should be attractive to other e-mail providers because it is free and comes with no special restrictions. Is the GPL considered a "special restriction"? Will it not actually be GPL, just available to open systems?

I'm guessing that you'll need to be a GPL mail server to both require the private key for receipt, and to be able to use the system to give the email the private key for sending. So, what will this do to non-open mail systems?

• You could presumably send to a non-open system, as they will simply ignore the key if present, but will still accept email if absent.
• Open systems that require the key to receive will presumably refuse email without the key (otherwise what's the point), which means that a mail system that's open that uses this methodology might gain the perception of "being broken" from the end users point of view. Of course, the admin setting up such a system would be well aware that some email will be refused, and will be prepared to handle refusals, either with a "bounce message", a phase in period that just gives a warning, etc.
• Senders that use a non-open system that can't use this technology will find an increasing amount of their email being refused; at first they'll blame the recipient, but as this gets more widespread, they'll blame their own sending service. Is that the sound of IIS's mail server being obsoleted?
  
• The end result will be that users of open systems will receive less spam, whereas users of closed systems will find themselves still receiving spam, and increasingly unable to send to others.

Is Yahoo trying to break MicroSoft's mail service? Will this work? What's MSFT's option--reverse this and include it in their system anyways? Switch to an open system for a mail server, like, say, something based on a BSD license? Or ignore it, in an attempt to deprive it of critical mass?

Indeed, this might all be moot; Yahoo might make it free and available to everyone, either on a free system or a non-free system; the article isn't clear as it says both. It could also be that MSFT already uses an OSS mailserver in IIS for all I know about MSFT product. But I suspect this is a power-grab, like everything else these days. And, I have to say, if it is I wish Yahoo the best of luck--this would be another demonstration of the power of OSS; it allows the community to change together on a dime and play well together. Whereas makers of proprietary systems each have to modify their own systems with their own coders.

Lock-in isn't necessarily an issue

[RevMike]

It can be open sourced, but that doesn't mean anything about preventing lock-in.

Presumably a 'domain key' is some cryptographic element that authenticates that your domain is who it claims to be. To me this sounds an awful lot like SSL where a third party issues the keys, or acts as a clearinghouse for self-issued keys.

Either way, Yahoo could be the man in the middle acting as either issuer or clearinghouse. Think of it this way, OpenSSL is open sourced, but that doesn't keep the SSL issuers from having a lock on that market.

I don't see how lock in will be an issue. Imagine the following scenario:

1. Originating mail software sends a message, including some token in the header that is encrypted using the sending mail server's private key.
2. Zero or more intermediate mail server pass along the message.
3. The destination mail server receives the message.
4. The destination mail server looks up the domain of the message originator and requests that domain's public key.
5. The destination mail server attempts to decrypt the token.
6. If the token is successfully decrypted, the mail is delivered. The receiver knows the identify of the sending system with certainty. Email domains can't be spoofed.
7. Otherwise the message is dropped.

I can't see how this would neccesitate a clearinghouse.

Re:Lock-in isn't necessarily an issue

[letxa2000]

First, there is nothing preventing spammers from registering their own domains (e.g. legitimatemail1.com, legitimatemail2.com), putting perfectly valid public keys on their nameserver, and sending mail which will be accepted.

It seems to me that all this does is more or less prove that the mail being received is coming from where it purports to come from. So, yes, a spammer can still create a mail server with keys and everything but at least when he sends a spam the message will be signed as having come from their server which makes it easier to filter on their server. It also causes their server to have to spend CPU cycles generating the encrypted key for each spam--which I assume would have to be separately generated for each copy of the spam which increases the cost of sending spam.

Finally, I think this is most useful in that if you know that every message that comes from Yahoo.com is signed with this scheme and you receive a message that purports to be from Yahoo.com that DOESN'T have the signature, it's spam. You can start creating a list of servers that you know use it--and if a message purports to come from one of those servers then you know it's spam. Yahoo probably has an interest in this because there are probably a lot of people and mail servers that are just filtering on Yahoo.com these days, even though we all know most of that spam doesn't actually come from Yahoo.

This mostly looks like an attempt to attack the problem of spammers forging email addresses that don't belong to them in the spam they send. It doesn't solve the spam problem, but it solves an annoying part of it--especially when some spammer forges YOUR email address as the "From" address in a spam sent to millions of people (and bounced from thousands).

Re:Not necessarily

[JohnFluxx]

They add this module, and get a reduction in spam.

Seems like a big business case to me - last I heard business didn't like spam. (.. except the spam business I suppose)

BEWARE THE BIG RED Y!

[poofmeisterp]

If they're offering it for free, BEWARE. IT'S A TRICK. There's some hidden patent they're going to decide to enforce once the entire world adopts the architecture.
*waves hands ominously*

Too resource intensive, and broken anyway

[RealProgrammer]

Under Yahoo's new architecture, a system sending an e-mail message would embed a secure, private key in a message header. The receiving system would check the Internet's Domain Name System for the public key registered to the sending domain.

If the public key is able to decrypt the private key embedded in the message, then the e-mail is considered authentic and can be delivered. If not, then the message is assumed not to be an authentic one from the sender and is blocked.

For every message, I have to check and unpack the header, go out to some PK server, and validate the keys, before I decide to accept/reject? That introduces a big latency into SMTP.

Also, this doesn't do anything to stop 'legitimate email marketers'. There's a death penalty (blacklist) for a site or particular sender's key, but nothing to stop a spammer from changing keys and starting over.

Or will everyone have to get their own key pair? Who's going to validate them, and at what cost per key pair?

This won't do a thing to stop spam, and imposes too big a burden on the infrastructure and on the 99% of us who don't spam.

Not for me

[bigberk]

Require the message sender to authenticate in order for message to come across a trusted e-mail network

Read: trusted network == commercial network

Why do you think this is in the "Money & Investing" department (see the linked article). No, this isn't for me. Businesses may well choose to use something like this for their communications, but they will not have the pleasure of communicating with me. While SMTP has its flaws, it still allows any IP host to send mail to any other IP host and that is a good thing.

To gain insight into what's going to happen with email and Internet communications in general over the next couple of years, you have to adopt a business mindset to see it from their eyes. There is a big problem (spam) hence a potential to make money. Various companies are going to try and cash in on this situation by offering a solution that might very well decrease spam -- some sort of commercially controlled communication network -- but this is definitely not in the best interest of the Internet. Of course, it's in the best interest of the company that's peddling the solution (duh!)

The Internet isn't Compuserve, or AOL. It's a network of IP hosts, and those are the entities which should have a facility for sending communications back and forth. There is no need for a central carrier for communications

Re:Not necessarily

[bokmann]

Your comments are close... for better adoption, although over a longer time span:

1) Software needs to be based on open standards. RFC90210 or something like that... Others need to be able to make implementations.

2) Yahoo's implmentation should do ONE THING WELL. It shouldn't try to stick an advertisement on the bottom of my emails the way their groups tools do.

3) Give the software a few months to propagate to a few major ISPs.

4) On a given date, all email going through those servers that are not 'signed' as this system specifies get some kind of flaf in the header that users can filter on. Appending something like, "Warning: This sender of this message has not been authenticated. It may not have come from the person you think it came from" to all emails that aren't authenticated. I guarantee that the 'peer pressure' from a label like that on all email originating from MY company would force us to upgrade.

5) let that bake for a year or three. By that time, everyone will be clamoring for non-authenticated email to be blocked.

Leading to a standard

[Offwhite98]

The way the IETF and other standards bodies have worked is that some organization wouldtry out a new concept for a technology and once they feel the concept is working, they will create a Request For Comments (RFC) which allows others to implement and offer feedback. Over time the RFC gains support and ultimately becomes a recommendation.

This process was used to create the internet today, including all of the network protocols and services that run on top of it. Even SMTP was an RFC first.

This is a large, stinking pile of bullshit.

[McDutchie]

1. This is the classic confusion of authentication with security. Authentication does not protect against spammers. The spammers will simply authenticate and keep right on spamming, and now they won't have to do tricks to circumvent the filters because the cert makes them "trusted". (One other example of this is the illusion of security caused by cryptographic authentication on the web. That hasn't stopped spyware sleazebags such as Gator/Claria; they just get their own certs.)
2. Yahoo is an unrepentant spammer and spam support service itself. They reset your marketing preferences at their whim. Abuse reports routinely go to /dev/null. Any "anti-spam" solution coming from a spammer and spam supporter is necessarily a scam.

Re:Why does no one seem to get it?

[IIH]

The proposal is very simple and most of the posts are just plain wrong about what it means.

I get it, because it sounds like an idea I've been bouncing around for a while (e.g. See previous comments of mine)

The mail server could add a header saying if the domain was verified and spamassassin could then adjust the spam rating of the message appropriately. Eventually servers would be configured to refuse mail from unverified domains.

Exactly, and the main advantage of this is the network effect - if yahoo.com "mail verifys" its domain, all mail servers will know that unverified email "from" yahoo.com is spam, and hence have a good reason to upgrade, and reject all forged yahoo emails. So spammers will have to use "otherisp.com" as the return address, and otherisp gets increased bounces. If OtherIsp change to a verified domain, spammers have to move until finally, the from address actually trustworthly, and banning individual isps on the SMTP from becomes feasible, and complaints will go to the corrct isp.

Aside from the possible computational requirements for all the crypto work, I don't see any downsides to this.

The only downsides are that people will complain about not being able to set their from address when they are using different isps. Personally, I don't see that as a problem, I belive the "from" address should be the equilavent of an electonic postmark, and if you want to set the return address you should used the sender or reply-to field instead.