Hashing Email Addresses For Web Considered Harmful

cce writes "The MicroID standard, despite getting thrashed soundly by Ben Laurie two years ago, has since been recommended by the DataPortability Project and published on the user profiles of millions of users at Digg and Last.fm. MicroID is basically a hash calculated using a user's profile page URL and registered email address, producing a token that makes the email address vulnerable to dictionary attacks. To see how easy it was to crack these tokens, I conducted a small study, choosing 56,775 random Digg users, and cracking the email addresses of 14,294 of them (25%) using just their MicroID, username, and a list of popular email domains. Digg has more than 2 million users, and that means half a million of them — mostly people who had never heard of MicroID, and had probably not logged in for a long time — had their email addresses exposed to this trivial attack. I also applied this attack to Last.fm (19%) and ClaimID (34%). Digg and Last.fm have since removed support for MicroID, but the lesson is clear: don't publish a hash of my email address online, guys!"

Comments

Solution: salt your emails

[pwnies]

I suppose this is yet another reason why it's nice that a few email services (most notably gmail) allow you to append a string to your email address using the + symbol (e.g. youremail+string@gmail.com will go to the inbox of youremail@gmail.com). In effect it allows you to "salt" your email, which adds a layer of complexity when trying to match these hashes with valid email (not to mention it allows you to check which site compromised your email if you use different 'salts' for each site you use your address on). If more email services start to allow this (doubtful), more sites start realizing that a + in your email is still a valid email (more doubtful), and more users start using it effectively (even more doubtful still), then I don't think the MicroID will be a huge problem.

Re:Solution: salt your emails

[nblender]

+ is a bad delimiter. Many web-forms don't accept email addresses with '+' in the username portion. Attempts to educate webmasters to the information in the relevant RFC's is usually met with silence or worse... I did manage to get a FOAF to fix dell.com though.

Re:Solution: salt your emails

[Anonymous Coward]

Except that once the salted email is found, everything between the @ and the + will just be discarded.

Re:Solution: salt your emails

[Rinisari]

Maybe that FOAF could attack ESPN.com, too. I tried registering there for a fantasy football league at work and used myaddress+espn@gmail.com. The damned system took the + out, making the address invalid!

Re:Solution: salt your emails

[geekgirlandrea]

Except that lots and lots of web sites fail at RFC 822 and think + isn't a valid character in an e-mail address. Usually the same sort of maldesigned horrors that make you type your e-mail address twice even though, unlike your password, you can read it as you type to make sure it's correct, or have a single free-form blank for credit card numbers and enforce some idiosyncratic rule on separators (really, is $cc =~ s/-//g; that hard?), or enforce strong passwords and then cripple them with mandatory 'security' questions that allow anyone who knows you halfway well to reset your password.

Yeah, I use them too, and if web designers were a whole lot smarter they would be a better solution to things like this, but in practice lots of web sites just refuse to accept addresses like that. I should get around to making sendmail let me use an underscore instead of a + for that purpose.

Re:Solution: salt your emails

[statemachine]

Giving out e-mails with "+something" is worthless for spam. The malicious spammers will just strip the "+something" from address, as both can be delivered, but the short form will be less likely filtered, and you won't know which service it was sold/stolen from.

I actually make a separate alias for each site eg. name-something@example.com. If you shorten my alias to the part before the hyphen, it won't deliver. Yes, spammers have tried.

If you're using "+something" just know that you might as well not append that onto your e-mail address, for all the good that it does, as you're giving out your primary address anyway. Cat, bag, already open.

Re:Solution: salt your emails

[geekgirlandrea]

Yeah, this can happen, but I dunno that this is as big a problem as you think. Spammers just plain aren't all that bright, and they don't care very much if they miss the tiny proportion of addresses that geeks try to protect like this when there are so many totally unprotected addresses so easy to obtain. It seems like a lot of the time, when they try to harvest addresses, the harvester doesn't realize + is a valid character in an address and only gets the part after the plus sign. I bounce a lot of spam sent to addresses like slashdot@persephoneslair.org and usenet@persephoneslair.org.

Re:Solution: salt your emails

[aj50]

Except that some web forms (and some mail servers) won't accept an email address with a '+' in it.

We use these types of addresses at work to organise replies to tickets and some people's mail set-ups really screw things up.

Re:Solution: salt your emails

[statemachine]

And the few times a harvester is correctly written? What then? That's the address that gets spread around. Obscurity doesn't work on the Internet. Just don't post it at all.

But you seem fine with it because you're also posting your personal domain name here, which links to your name and your photo, along with a street address and phone number (which I hope are only P.O. box and a voicemail-only phone service). You're a hell of a lot more comfortable with it than I am. (At least I hope you knew that all that info was very publicly available.)

Re:Solution: salt your emails

[cduffy]

Obscurity doesn't work on the Internet.

So why bother?

Someone who was serious could get into public records and get my address anyhow (owning a house generates lots of public records). Someone who isn't serious presumably doesn't pose a threat. I think the worst thing that's actually likely to happen is 4chan-style harassment, and (1) it's not particularly likely, as I don't hang around those types enough for them to care about me, and (2) if it did happen, countermeasures are certainly available. And, again, (3) if anyone were serious enough about it, they could find all the relevant information through other channels anyhow.

Being nymous online is a Good Thing -- it means people I know IRL can recognize me (I've run into ex-coworkers and old friends I didn't think I'd see again) and it gives me a chance to build a reputation that follows me into Real Life (so potential employers find plenty to recommend me when googling my name). Further, it acts counter to the tendency for anonymous communication to degrade into... well, you're on slashdot; you know exactly what I'm talking about. :)

Re:Solution: salt your emails

[mi]

+ is a bad delimiter.

It is the delimiter, originally created as such by the authors of the very first MTA... There is no other character, that:

1. Can be part of an e-mail address.
2. Can not be part of a username.

Many web-forms don't accept email addresses with '+' in the username portion. Attempts to educate webmasters to the information in the relevant RFC's is usually met with silence or worse...

This is, unfortunately, the truth... Far too many programmer wannabees around... It is a good fight, however, and kudos to GMail for keeping support for it (unlike Yahoo! Mail).

I use this whenever I can, when giving my address to web-sites (including Slashdot)...

Re:Solution: salt your emails

[daeg]

Spammers aren't bright? So spam filtering is easy, right?

One (partial) solution is to have large providers provide alternate domains that you can register throw-away addresses. For instance, under Google Account settings, you might have the option to generate an address from cephelo@gmail.com and assign d785jd47fj@southeast.gmail.com and allow you to record a note that you intend to use d785jd47fj@southeast.gmail.com as your Amazon.com user ID.

As time progresses, Gmail can show you stats that, for example, 100% of e-mail on d785jd47fj@southeast.gmail.com is spam - "Do you want to delete this account?" and poof - the spam stops. Now that address automatically becomes a honey pot.

Re:Solution: salt your emails

[cduffy]

Hey -- we didn't arrange our schedules that way on purpose; it just happened as a happy accident. Likewise, I mess with Asterisk first and foremost because I think it's fun, and only secondarily because I dislike phone spam. (We did decide to do the large-dog thing as a security measure, but that was for late-night walks outside, not protection of the household proper -- and any weaponry we may have usable for home defense would have been purchased primary for recreational hunting; that said, I don't disclose the presence or lack of such online). I don't put myself through a whole bunch of hassle because I'm paranoid about security, and I'd probably still decide to be as easily identifiable online as I am had things not worked out that way, on account of the benefits I gave earlier (ability to translate online reputation-building into real-life interactions, which I really do think is a serious and compelling advantage)... that said, when it comes down to defending my decision, the set of happy accidents comes in handy.

I agree with you that paranoia is contrary to happiness -- that's part of why I'm comfortable with having my identity online; if I had to live in a mental state such that I believed people as a whole to be an irresponsible set (or such irresponsible people to be numerous enough to be worth thinking about), that mode of thought would, in and of itself, make me less happy.

Re:Solution: salt your emails

[mcrbids]

Let's see... Large email provider, throwaway addresses, access until you don't want it anymore...

You mean, kinda like Mailinator??

There are others, Mailinator is the easiest.

Re:Solution: salt your emails

[skeeto]

This is, unfortunately, the truth... Far too many programmer wannabees around...

It is also unfortunate that perfect e-mail parsing is extremely complex. The Perl regexp for e-mail address validation according to RFC 822 is about 6.3 kilobytes. If you try to do it yourself you are pretty much guaranteed to get it wrong.

Those crappy programmers could still make things much better with liberal validation, allowing some invalid addresses to make validation simpler. Something simple like /[^@]+@[^@]+\.[^@]+/, will match all valid e-mail addresses (I think, and the /. filter won't let me write anything more complex than that anyway) plus a bunch of invalid ones.

They already have your email address

[RevDigger]

This concern that you may have your email address *discovered* by spammers because you post it on a web page is so 5-years-ago. They already have your email address, and they probably didn't get it by scraping web pages.

When you have sent a couple emails out with a given address, you can figure that at least one of them will to sit around in someone's Outlook mailstore for the next couple years. (Someone you know uses Windows!) When that person's computer gets infected with spam gang malware (as they all do), they have your address.

Once of them has it, they probably all have it.

Re:They already have your email address

[John Hasler]

> Once of them has it, they probably all have it.

But they don't know that it is yours. They can spam you with it but they can't use it for anything else.

Re:They already have your email address

[cce]

I'd argue that the added value of a spammer getting an email address connected to your online "identity" -- your user profile, recently-played Last.fm songs, favorite Digg articles, etc -- makes getting your email from a MicroID a little more valuable than the ordinary harvested email address. Plus, they don't have to bother confirming the address to see if it's still active (Digg already did).

Re:They already have your email address

[oldspewey]

They can spam you with it but they can't use it for anything else

Actually, in addition to spamming you, they can use your email address in the from and reply-to field for their next spam run.


Ask me how I know.

Re:They already have your email address

[coryking]

The spammer (or actually, botnet owner who wrote a spam program) has already figured that out by putting a shim inbetween you and your network card. They just sniff your traffic for anything that looks interesting. In fact, I wouldn't be suprised at all that the botnet software will "turn on" when you use hit up gmail.com and can screen scrape the page while you check your email. I would even bet that it can update its screen scraping rules from some kind of distributed network.

Somebody in this thread said spammers are dumb. That might have been the case five years ago but it is not the case now. The "spam industry" has really evolved to the "botnet industry". These botnet people are smart, smart people. Almost as smart as the P2P people in terms of getting around "damage". Shame they couldn't apply their skill and talent to doing something positive for our society though.

superparanoid? regexp

[Tmack]

If you are superparanoid, you can run your own mta, like qmail or postfix, and specify your own delimiter to regexp out of the address in one of the pre-processing filters. With qmail, I believe you could even just edit the qmail-smtpd config/run file (iirc, been a while) and add a pipe through sed to do the dirty work with the addy before the normal pipe through qmail.

tm

Re:What does MicroID actually do for the user?

[Fred Ferrigno]

I read up on it and I'm still confused, but I think this is the idea:

1. You set up an account at website Alpha.
2. You have a publicly-viewable profile page at Alpha. On the page is your MicroID.
3. You set up an account at website Beta.
4. You tell Beta about your Alpha profile page.
5. Beta verifies that your Alpha profile page is really yours by checking the MicroID.

Beta can't really do anything with your Alpha page except link to it. I guess the point would be to prevent people who aren't you from linking to your Alpha page on their Beta pages. That way, other people can be sure that the same person owns both accounts.

The attack mentioned in the article doesn't compromise the proper use of the MicroID, since Beta is assumed to have verified that you own your email address and you wouldn't link to a profile page claiming to be yours that wasn't. All it does is make it possible for spammers to harvest your email.

even the spec admits it is retarded

[hdon]

I wrote about this earlier this year. My conclusion, more or less, was to carefully read the specification, which Iâ(TM)ll excerpt here:

By itself, a MicroID has no inherent meaning, since it is simply a string created from two URIs. Any entity can generate a MicroID even if it has not verified the identity of the resources associated with one or both URIs. Furthermore, a MicroID is easily copied by an entity that did not generate it. Finally, a MicroID is not digitally signed by the entity that generated it and therefore cannot be cryptographically associated with the generating entity.

Re:Okay...

[WuphonsReach]

Do they still do that? I know from a distant past they tried it with smaller providers too, but haven't seen them for a long time. As far as I can tell, spammers do still use malware which harvests/sniffs email-address directly from peoples computers.

This is a definite tactic. I see it all the time on a mail server that I administer. From the results, there are definitely spammers that monitor user's e-mail, address book, or other sources of e-mail addresses on their computer. (Basically, on a brand new e-mail address, the user started getting spam within a few hours of contacting someone else.)

But we still see dictionary attacks on our mail server, so that's a popular tactic too.

Re:A better solution?

[Firehed]

Use gmail. I'll get a thousand or so spams a month, but I've had maybe four make it to my inbox in the past three years.

It obviously doesn't eliminate the problem of spam, but in theory if it didn't make it to anyone's inbox, idiots would stop acting on it and suddenly spam wouldn't be profitable and would fizzle away.

Flawed study?

[dmuir]

What's the difference between attacking the MicroID to collect email addresses, and running a dictionary attack on email servers using people's usernames?

Re:Flawed study?

[QuantumG]

Offline attacks are better because they:

1. can't be monitored
2. can't be blocked
3. are not limited by bandwidth
4. can be sped up by throwing more hardware at them

This is basically why salting was added to the unix password file. And that failed.. so /etc/shadow was introduced. Revealing hashes is just unnecessary, so don't do it.

Postfix Solution

[bill_mcgonigle]

Assuming you're using postfix and virtual, you can do something like this:

main.cf:

recipient_delimiter = +
virtual_alias_maps = hash:/etc/postfix/virtual, regexp:/etc/postfix/virtual-regexp

virtual-regexp: /(.*)\-(.*)@example.com/ ${1}+${2}@example.com

and then you can do:

    bob-somesite.com@example.com

this works for every site I've tried but oracle.com, who apparently doesn't want you tracking their mail. :)

It's not a secret: jeffrey@goldmark.org

[Charles Dodgeson]

I fully agree with the parent. The idea of keeping an email address that you actually use private is several orders of magnitude sillier than thinking your credit card number and social security number hasn't been stolen a dozen times already.

But there is one place I won't "publish" my email address (jeffrey@goldmark.org), and that is in the From line of a Usenet posting. Reply-to is fine, and there absolutely no problem in the body of messages, but tests have shown that putting something in the From line of a Usenet posting will give you a very noticeable increase in spam.

Just use dots, then

[Cow Jones]

Apart from the fact "+" is a perfectly valid character in an email address, if you're using Gmail, you can insert random dots in your address, and your mail will still get delivered.

my.name@gmail.com

is equivalent to

my.na.me@gmail.com
my....name@gmail.com
m.y.n.a.m.e@gmail.com
etc

Re:Just use dots, then

[funfail]

This is completely different. What the grandparent said that "username.ml@gmail.com" would automatically go to "usernameml@gmail.com". Gmail just ignores dots in e-mail addresses.

Why is this a big deal?

[Ed Avis]

You are worried because someone, if they really wanted to send you some mail, could go to the trouble of doing a CPU-intensive search against some hash shown on a website and find out that ultimate, embarassing secret: your *email address*??

What gives? Email addresses are designed to be public. If you don't want people you do not know to be able to contact you, then you are free to drop all mail from unrecognized addresses. If you want to set up some kind of secret knowledge that people must have in order to contact you, then ask them to put a particular word in the subject line when first sending you a message. Either of these does not rely on keeping the address secret, which just isn't likely to happen.

The only thing more broken than trying to keep an email address secret is trying to make a 'private' web page by keeping the URI secret. Again, the system is designed so that the address itself is not sensitive, but other information such as a password or PGP key can be.

Actually, what it reminds me of most is the crazy situation in the US where a basically public identifier, the social security number, is abused as some kind of secret token. Hence all the fuss made when it is possible to find out someone's SSN. The answer is not to add more and more baroque means to stop the SSN from leaking out: one breach, and it's no longer a secret.

I understand the desire to stop spam address harvesters, but really, there are hundreds of web sites which display email addresses with only light obfuscation, enough to stop a harvester bot but not a determined human being (or someone determined enough to use an OCR engine). The kind of hashing talked about here is way more difficult to undo than that. If you are even more paranoid, you need to revisit your assumptions of what is public and what is secret.

microid doesn't seem to factor into it

[MisterBad]

It seems like the attack is just taking user names and other publicly-known data trying to determine an email address from them. Spammers don't need microid to confirm that their guess is correct; they'll just send to all 50 or 100 top email domains, hoping to get a hit.

The whole point of MicroID is that if someone knows your email address, they can tell that you are the author of the page. If your email address is easy to guess, then your email address will be revealed, _whether_or_not_ there's a microid here, there, or anywhere.

If an email address is easy to guess, then the email address is easy to guess. Not clear what new ground we're covering here.