Slashdot Mirror


Thousands of Email Addresses Accidentally Disclosed By Let's Encrypt (letsencrypt.org)

An anonymous reader writes "Let's Encrypt, the certificate authority best known for offering free SSL/TLS certificates, has reported that it accidentally disclosed thousands of user email addresses due to a bug with an automated emailing system." Executive Director Josh Aas posted this announcement: On June 11 2016 (UTC), we started sending an email to all active subscribers who provided an email address, informing them of an update to our subscriber agreement. This was done via an automated system which contained a bug that mistakenly prepended between 0 and 7,618 other email addresses to the body of the email... The problem was noticed and the system was stopped after 7,618 out of approximately 383,000 emails (1.9%) were sent. Each email mistakenly contained the email addresses from the emails sent prior to it, so earlier emails contained fewer addresses than later ones.

We take our relationship with our users very seriously and apologize for the error... If you received one of these emails we ask that you not post lists of email addresses publicly.

45 of 81 comments (clear)

  1. Why the heck by NotInHere · · Score: 1

    Why the heck do they actually require to store e-mail addresses the first place? ACME works with public keys and cryptography, no? Or was it the email addresses for some forum or something, unrelated from the core service?

    1. Re:Why the heck by NotInHere · · Score: 1

      Wasn't ACME partly about making renewing certs automatic?

    2. Re:Why the heck by Anonymous Coward · · Score: 1

      Wasn't ACME partly about making renewing certs automatic?

      No, it was a company that sold absolutely anything and everything to Wile E. Coyote. The executives of this company were very smart and truly understood their market. That's why they never offered to sell raw or cooked road runners, nor any other poultry, to said coyote. To do that would destroy their much more lucrative crazy gadget market.

    3. Re:Why the heck by Lennie · · Score: 1

      Yes, but because it's all new (and people don't trust the tooling yet) they allow you to specify an e-mail address.

      --
      New things are always on the horizon
    4. Re:Why the heck by cryptizard · · Score: 4, Informative

      Modern web servers support hot swapping certificates so downtime is not usually an issue. The most important reason for short expirations is that certificates with short expiration times are more secure against attackers that might be able to steal your certificate. A cert that is valid for a year will be much more valuable than one which expires in 90 days. The same holds true even for the Let's Encrypt credentials themselves. If Let's Encrypt refuses to grant any certificates longer than 90 days then your credentials are actually NOT as valuable as they would be otherwise. They also participate in public certificate transparency logs so it is easy to detect a situation where someone gets another certificate issued for your site.

      Additionally, Let's Encrypt WANTS to make certificate renewal an automated process, and short expirations force users to do that like you said. There should be no reason to "think about the cert creation process" because CAs should never issue you a certificate with a cipher or key length that is not secure. The idea that web site administrators should all be up on the cutting edge of cryptographic attacks is pretty crazy actually. Standards should be enforced at the bottlenecks (in this case CAs) so that as few fuck ups as possible can happen.

    5. Re:Why the heck by Anonymous Coward · · Score: 1

      No sane person is going to be willing to babysit a cert upgrade four times a year.

      That's the point. They want to strongly discourage babysitting cert upgrades. They only want to support automatic upgrades. They would like to make the period as short as possible.

    6. Re:Why the heck by CRC'99 · · Score: 1

      I can get free 1-year certs from StartSSL, so why in the world does anybody even use those?

      And I'll keep mentioning StartAPI every time this BS about Lets Encrypt comes out - and point to my implementation of it in perl: https://github.com/CRCinAU/sta...

      It takes 90% of the pain out of SSL renewals.

      --
      Sendmail is like emacs: A nice operating system, but missing an editor and a MTA.
    7. Re:Why the heck by cryptizard · · Score: 2

      This is where I realize you don't know what you're talking about because SSLv3 has been disabled in modern browsers for 2 years now. Have a good day with your uninformed, knee-jerk opinions.

    8. Re:Why the heck by CRC'99 · · Score: 1

      This is where I realize you don't know what you're talking about because SSLv3 has been disabled in modern browsers for 2 years now. Have a good day with your uninformed, knee-jerk opinions.

      So you're going to tell a guy with a 5 digit slashdot ID that SSLv3 problems don't exist because the browser disabled it?

      You realise that unless you disable it *ON THE SERVER* it is still offered, right?

      --
      Sendmail is like emacs: A nice operating system, but missing an editor and a MTA.
    9. Re:Why the heck by cryptizard · · Score: 1

      Appeal to slashdot ID, haven't heard that one before lol. Who cares what the server supports if the clients don't allow it? All the client has to know is if they see the green lock (as you said) then their communications are secure. It would only matter if the client is using an extremely old browser, in which case SSLv3 is probably the least of your worries considering the huge number of unpatched security vulnerabilities you probably have going on. So yes, I am going to tell you that SSLv3 problems do not exist for the vast majority of people.

    10. Re:Why the heck by dgatwood · · Score: 1

      And my point is that it isn't their place to discourage sysadmins from monitoring their server cert upgrades. It's fine for them to encourage the various web servers to make the automatic upgrades easy so that people with no experience have an easier time, but it is crossing a line to insist that we shouldn't be allowed to be cautious.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    11. Re:Why the heck by dgatwood · · Score: 1

      Fun fact: All the big players use short-lived certificates.

      Fun fact: All the big players use short-lived servers. They deploy things to cloud servers frequently, un-deploy them frequently, and they run their own CAs. That's great if you're a giant cloud provider. After all, if one of those servers fails to come up because of a re-cert, the server transparently fails over to another server, and as long as the configuration doesn't propagate too far, there's no impact on anything.

      That's completely different from the types of users who are in the market for a free certificate provider. Those folks usually have only a single server that has to "just work". Every extra bit of automation adds risk. Every configuration change is a risk. That risk is much higher because their services are not massively distributed. For them, limiting the rate of change is more important than the infinitesimal risk of someone compromising their server, stealing their key, and using it to impersonate their hobby server.

      So no, 90 days is not a better choice. It is just a choice. You're trading higher risk of something going wrong during re-certing against a shorter period of vulnerability for clients that don't support OCSP and are actively being attacked by someone who successfully compromised your server at some point in the recent past, up until the end of that cert's validity window. IMO, the decision of certificate length should be entirely in the hands of the site admins, not forced upon them by a heavy-handed certificate authority.

      Once upon a time the "certs that people pay for" lasted up to 10 years. That stopped. Can you guess why? You're probably thinking "So they could charge a lot more money" right? Nope. Because hardly any of those 10 year certificates remained in use after just FIVE years.

      That's in large part because most companies don't keep servers running that long. Heck, a statistical majority of companies aren't around that long. This is not the case for personal servers, of course, but they were historically never secured with HTTPS because of the cost. That critical difference makes any statistics from that period wholly invalid when determining proper lengths for certs in this day and age.

      Also, folks periodically find weaknesses in crypto technology that requires you to rekey your servers. For this reason, periods should be long enough to not be annoying, but short enough that people have to manually intervene every two or three years, to ensure that sysadmins realize that their keys are no longer strong enough and upgrade things accordingly. IMO, automated upgrades actively hinder security by making the process "out of sight, out of mind". Whether that difference matters or not remains to be seen, but it is at least cause for concern.

      --

      Check out my sci-fi/humor trilogy at PatriotsBooks.

    12. Re:Why the heck by allo · · Score: 1

      its optional for access recovery (revocing certs when you lost your access key) and notification mails about certs about to expire (to see if your automatic renewal failed).

    13. Re:Why the heck by allo · · Score: 1

      > Having such a short expiration period weakens security
      This is bullshit. Google uses very short lived certificates for many of its services.

      google.com right now (seen from here):
      begin: 06/01/2016
      end: 08/24/2016

    14. Re:Why the heck by allo · · Score: 1

      IF you're hacked, you want to minimize the damage.

    15. Re:Why the heck by dave420 · · Score: 1

      So be cautious. Babysit your certificate upgrades if you really can't get it together enough to automate them. Clearly babysitting certificate upgrades is not scalable, and is prone to introducing errors. No one will stop you from being overly paranoid and under performing.

  2. This, from a "security" organization? by Anonymous Coward · · Score: 1

    I guess this demonstrates how seriously they take security. I wonder how they protect their root keys.

  3. I'm worried how it's being brushed off at HN! by Anonymous Coward · · Score: 3, Insightful

    I first learned about this awful incident at Hacker News.

    What scares me the most is some of the responses there which just brush it off as no big deal! There are comments there like:

    It sucks this happened but I don't really care.

    and

    Interesting. Slightly embarrassing. Not a huge deal. Handled well.

    and

    Things like this happens all the time. Give them a break.

    The responses are just about as bad over at reddit:

    Stupid bug, but the consequences of this bug is... nada. zip.

    and

    this is non-news, emails are public info

    To make matters worse I'm seeing comments from people pointing out that this is not acceptable getting downvoted!

    It scares the living hell out me that people can think that somehow this incident was acceptable or excusable, especially when it was an organization that has to put security, privacy and trust paramount that was responsible.

    This incident was not acceptable. It should be considered a total disaster.

    1. Re:I'm worried how it's being brushed off at HN! by Anonymous Coward · · Score: 1, Insightful

      To make matters worse I'm seeing comments from people pointing out that this is not acceptable getting downvoted!

      Yes, the generic term for this phenomenon is "echo chamber". The same thing happens here at Slashdot if you say certain things that are unpopular and back them up with solid evidence so they can't easily be hand-waved away. It's just the way small minded people react at the point where they should say "I was mistaken".

      It scares the living hell out me that people can think that somehow this incident was acceptable or excusable, especially when it was an organization that has to put security, privacy and trust paramount that was responsible. This incident was not acceptable. It should be considered a total disaster.

      Speaking of small minded people, they generally look at particulars and not at principles. So it's all insignificant until they personally get hurt by it. When it's a SSN or a bank account number, and it happens to them, suddenly it matters and not a moment before. They don't seem to understand that security really matters at all times. Tolerating this kind of slack approach to security just encourages more of it and eventually it WILL be something a lot worse than e-mail addresses.

    2. Re:I'm worried how it's being brushed off at HN! by Cyberax · · Score: 1

      Uhm... Why are they incorrect? It's not a total disaster - nothing is compromised except the email addresses. You know, the ones that are attached to any emails that you send out.

      It's a remote possibility that somebody used a unique email that encodes some important information in the address. But such people are their own enemies.

    3. Re:I'm worried how it's being brushed off at HN! by sacrilicious · · Score: 1

      Just so you know: when you post a sky-is-falling position about something that seems pretty harmless, and the org in question is one that's up-ending a long-standing cartel that's held the reigns on something tightly, and you don't give any expansion on why (according to you) it is such a catastrophe, AND you post as Anonymous Coward... well, it makes you look pretty likely to be a shill for the industry that's being up-ended. Just wanted you to know how it comes across.

      --
      - First they ignore you, then they laugh at you, then ???, then profit.
  4. Re:I wanted to believe by pepsikid · · Score: 1

    Damn, but you're mad about this 100% free service!

    The install script doesn't stay resident or seize resources from your server. It performs the automated confirmation that you're the owner of the website you're applying for. You actually don't have to use it, but you'd have a lot of work on your hands otherwise.
    I use Let's Encrypt certs and couldn't be happier. The only problems I've had with it are where I link to content on my servers, and the forum/host that I wanted it to appear on is unable to fetch it. In the long run, that's an added bonus since I've had problems with hotlink abuse anyway.

  5. Re:I wanted to believe by NotInHere · · Score: 2

    The process of dealing with certificates was shitty to begin with, but at least I figured it out already and it's relatively simple, now I'm forced to deal with another layer of crap on top of that?

    In fact they've tried to make it easier. Most people just want to get the job done, nothing more. The "layer" of crap removes one big problem with SSL certificates: manual renewal. Usually you have to renew certificates manually, but with the program from Let's Encrypt, this happens automatically.

    Maybe in the future this is even built into the HTTP servers, so that you don't have to install third party software. Its all just checking whether the cert is expired, and running the ACME protocol to get a new one automatically.

  6. Re:I wanted to believe by ledow · · Score: 1

    Only the default "press and go" option actually sucks in your Apache configs, finds all the domains, makes certs for them all, then plugs those certs back into your Apache config. That's the "dumb user" mode.

    You can have it just create the certs you specify for you, save them into a given location, and just put the command that does that into, say, a cron job or do it manually.

    Same utility, different command line parameters.

  7. Sooo... by dejitaru · · Score: 1

    No good deed goes unpunished? Sounds about right... sadly.

  8. Amateur-level scripting by gweihir · · Score: 1

    This is a basic coding mistake made worse by a set of basic testing mistakes. The state of practical IT is truly amazingly bad when mistakes like these are made routinely. Does not instill any confidence in this specific group of people either.

    --
    Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    1. Re:Amateur-level scripting by Sax+Russell+5449D29A · · Score: 1

      Who needs test management, right? I mean, if your code is always *awesome*, why throw money down the well! :-)

      Test management is and always has been an uphill battle. You have to be constantly proving yourself and telling the management how this-and-that bug saved the company a $100K+ down the road. Quality has this funny trait of being rather invisible when it's present and people only really notice it when it's not there.

      --
      -SR
    2. Re:Amateur-level scripting by gweihir · · Score: 1

      More like "who needs tests" in this case. Test management is a bit overblown for a simple small script. (At least that is what I would use. And I would send it out via a relay and check what is in the queue before activating that relay. Of course, I may just be overly cautious.)

      You do have a point of course. Skimping on testing, test management and developer skills is about as stupid as if firing the sysadmins because "everything is working well".

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
  9. SSL scam by JcMorin · · Score: 1

    I think you get it correctly, but do the way browsers are build you need a valid not-self signed certificate and Lets encrypt do exactly that... If you get hacked or have your system not working your certificate will expired after 90 days limited the problem of a forgotten certificate (for instance if you switch server). I'm a user for Lets Encrypts and as my existing certificate expired I'm replacing them all with this awesome services.

  10. Re:I wanted to believe by dgatwood · · Score: 1

    In fact they've tried to make it easier. Most people just want to get the job done, nothing more. The "layer" of crap removes one big problem with SSL certificates: manual renewal. Usually you have to renew certificates manually, but with the program from Let's Encrypt, this happens automatically.

    That's downright terrifying to me. I've had at least a few situations where a seemingly innocuous cert upgrade went horribly wrong and I had to roll back to a previous configuration to get back up and running. The last thing I want is for that process to be fully automated. There's a reason I don't automate my TLS cert upgrades, and it isn't because it isn't easy to do. It is because if anything goes wrong, I want to be abso-freaking-lutely sure that I'm right there beside the machine when it fails so that I can fix the problem quickly.

    --

    Check out my sci-fi/humor trilogy at PatriotsBooks.

  11. Re:A centralized 'free' repository of certs by NotInHere · · Score: 1

    I agree, having a centralized repository is bad, but what is even worse is what we are currently having: hundreds of CAs world-wide being able to sign certs for any domain in the planet without any oversight.

    So while it is a good reflex to be upset about something centralized, in the case of CAs its the opposite: the more CAs you trust, the bigger the chance that one of them sold a signing cert to some shady country or something.

    Yes, a system where multiple entities have to agree before giving out a cert would help the system alot.

    Your criticism of "free" looks misplaced. Handing out certificates barely require any manual intervention, nor any resources (if you disregard for the server costs). So its only straightforward to make obtaining them free.

  12. Re:Because it's a scam by NotInHere · · Score: 1

    In the case of these people, it's to pretend that you need them WAY more than you actually do.

    Not really. There are multiple reasons why long term certs are bad: https://letsencrypt.org/2015/1...

    Manual renewal is a bad habit.

  13. Re:I wanted to believe by cryptizard · · Score: 1

    But ideally that process should not be volatile. If it is that is a whole other issue. That is like saying I don't want my website to automatically respond to HTTP requests, what if it makes a mistake? I have to be sitting next to it in case that happens.

  14. Re:A centralized 'free' repository of certs by cryptizard · · Score: 1

    Let's Encrypt also participates in certificate transparency programs. Every certificate that it issues is put in a public log so if you ever encounter one signed by them you can verify that it is on there. On the other side, site owners can verify that all the certificates they asked for are on there and no extra ones for any sites they own. If enough people audit this, then maliciously granted certificates would be discovered and there would be indisputable proof that something bad was happening. Google Chrome automatically does some of this auditing whenever you visit an HTTPS site I believe, so it would be pretty difficult to get away with anything more impactful than an extremely targeted (toward only a handful of users) bogus certificate attack.

  15. Re:Because it's a scam by Anonymous Coward · · Score: 1

    Try sourcing something that isn't also Mozilla and isn't gathering TLS stats from a single browser (gasp, Firefox!).

    Automatic renewal opens yourself up to bugs like TFA and MitM attacks, among other possibilities. Manual renewal can suffer from similar problems, but the difference is that a human is in the driver's seat and has the ability to assess the situation. Trusting an automated system to be secure is asking for trouble. This is all discounting the fact that the certificate authority model is broken to begin with.

  16. Re:I wanted to believe by pepsikid · · Score: 2

    AC, you are spectacularly bad at composing analogies.

  17. Re:I wanted to believe by I'm+New+Around+Here · · Score: 1

    AC, you are spectacularly bad at composing analogies.

    Really. He didn't even mention your car. How can you have an analogy without a car?

    --
    If you think I voted for Trump because of this post, you're wrong. I voted for Dr. Jill Stein of the Green Party. Again.
  18. Re:I wanted to believe by pepsikid · · Score: 1

    You are not a zener diode.

  19. In other news by JustAnotherOldGuy · · Score: 1

    In other news, "Let's Encrypt" has changed their name to "Let's Disclose".

    --
    Just cruising through this digital world at 33 1/3 rpm...
  20. Re:A centralized 'free' repository of certs by NotInHere · · Score: 1

    And even if, you still have to trust the certificate log.

    The only real way to be sure your domain is yours is by getting your cert pinned by google and shipped in the browsers.

  21. Re:I wanted to believe by ShaunC · · Score: 1

    The last thing I want is for that process to be fully automated.

    I agree, but there's nothing about Let's Encrypt that forces you to do everything in an automated manner. The certbot client offers various levels of automation; there are other clients that probably have similar options. I choose to automate issuance/retrieval of the certificates only, and then install them into Apache manually. I also run the renewal process manually instead of having cron do it. There's a --dry-run option when using the command line that will identify potential problems. If for some reason a rollback is necessary, I'm already in the shell and the old certs are sitting in /etc/letsencrypt/archive/.

    --
    Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
  22. Re:How do I find.... by ShaunC · · Score: 1

    If you received an email from Let's Encrypt yesterday that had other peoples' email addresses prepended to the message, you were affected. If you didn't get such an email, your address wasn't leaked.

    --
    Thanks to the War on Drugs, it's easier to buy meth than it is to buy cold medicine!
  23. Re:Because it's a scam by allo · · Score: 1

    If you really want to make self signed default, as encrypted but not signed is better than nothing, you do not need certs for the default case. Just encrypt and use certs when you need to certify something (like an identity or a domain ownership for the ip owner).

  24. Re:I wanted to believe by allo · · Score: 1

    even with the official tool it's easy without touching your webserver.
    letsencrypt certonly --webroot --webroot-path /var/www/ -d yourdomain
    and then point your webserver to /etc/letsencrypt/live/yourdomain/fullchain.pem

    you may even generate your own csr, which allows you to keep the key instead of generating a new one for each certificate. If you see a point in keeping the key (maybe when your use HPKP?)
    Probably there is some cmdline option to keep the first key generated as well. would need to look it up.

  25. Re:This is why automated solutions won't work by allo · · Score: 1

    nothing happened, which couldn't happen for other CAs as well.