Slashdot Mirror

← Back to Stories (view on slashdot.org)

Heartbleed Disclosure Timeline Revealed

Posted by samzenpus on from the when-did-you-know dept.

bennyboy64 (1437419) writes "Ever since the Heartbleed flaw in OpenSSL was made public there have been various questions about who knew what and when. The Sydney Morning Herald has done some analysis of public mailing lists and talked to those involved with disclosing the bug to get the bottom of it. The newspaper finds that Google discovered Heartbleed on or before March 21 and notified OpenSSL on April 1. Other key dates include Finnish security testing firm Codenomicon discovering the flaw independently of Google at 23:30 PDT, April 3. SuSE, Debian, FreeBSD and AltLinux all got a heads up from Red Hat about the flaw in the early hours of April 7 — a few hours before it was made public. Ubuntu, Gentoo and Chromium attempted to get a heads up by responding to an email with few details about it but didn't, as the guy at Red Hat sending the disclosure messages out in India went to bed. By the time he woke up, Codenomicon had reported the bug to OpenSSL."

62 comments

  1. End result: mass panic by Anonymous Coward · · Score: 0

    Almost nobody had a patch ready by the time the news was made public.

    1. Re:End result: mass panic by Anonymous Coward · · Score: 1

      They exist; it's just that the vast majority of the people who belong to those eyes are really not qualified to be working on software that will be used in such important roles, and now we're paying the price. You don't use your Fisher Price tool set when you are building a real house. You just don't.

    2. Re:End result: mass panic by Anonymous Coward · · Score: 2, Insightful

      And you also see this same type of thing in proprietary software, where tons of losers are hired to work on the code, with predictably terrible results. The thing about open source is that anyone can see the source code, and people not part of the group that wrote the code can check it, so you at least have some chance of understanding what's going on.

      Anyone who claims that open source advocates claim that open source is 100% immune from all flaws is just spewing forth straw men.

    3. Re:End result: mass panic by Anonymous Coward · · Score: 0

      Anyone who claims that open source advocates claim that open source is 100% immune from all flaws is just spewing forth straw men.

      I suggest you spend more time reading open source forums and blogs because it is not a strawman.

  2. "Independent" discovery? by Red+Herring · · Score: 5, Interesting

    > Google discovered Heartbleed on or before March 21 and notified OpenSSL on April 1. Other key dates include Finnish security testing firm Codenomicon discovering the flaw independently of Google at 23:30 PDT, April 2.

    Doesn't it seem strange that the flaw has existed for a long, long time (years?) but Codenomicon happens to find it less than a day after Google notified OpenSSL, and, per the article, "some infrastructure providers under embargo"? That just seems... unlikely. Not impossible, but it kind of makes you wonder who is leaking information...

    --
    #include "standard_disclaimer.h"
    1. Re:"Independent" discovery? by Albanach · · Score: 5, Interesting

      Not necessarily. It may be that the bug was known to others and that Google and Codenomicon were both monitoring channels used by more nefarious types. Both organizations may have independently 'discovered' the bug after each becoming aware that an exploit existed without having full details of the exploit.

    2. Re:"Independent" discovery? by AdhSeidh · · Score: 5, Interesting

      perhaps you have already forgotten about CVE-2014-1266 the Apple SSL/TLS bug from Februrary this is why every security group on the planet was looking for other encryption related loopholes

    3. Re:"Independent" discovery? by rmdingler · · Score: 4, Interesting
      In all likelihood, there was a "discovery" by Google that led to a sharing of information with Codenomicon... someone told an old college buddy or former co-worker.

      There were almost certainly folks who were aware of the vulnerability before Google.

      Were these folks criminals or government employees? And yes, there's a small difference... generally found in the probability for prosecution.

      --
      Happiness in intelligent people is the rarest thing I know.

      Ernest Hemingway

    4. Re:"Independent" discovery? by briancox2 · · Score: 4, Funny

      That's what Newton said to Leibniz.

      --
      We should learn what we need to know about issues, before we decide what we need to feel about them.
    5. Re:"Independent" discovery? by Anonymous Coward · · Score: 1

      No, this is not uncommon at all in research. The idea that two groups are both looking into how [X] works, and how [Y] responds to [X] is quite common. Being a security researcher myself (slightly different sub-field, but still reason for anon posting) I can say that it is quite an easy possibility that both teams were checking the ENTIRE ISO~TCP/IP stack from lvl 0 up to lvl (whatever 'top' is in your outlook/naming scheme) And that they both found it around the same time.

      Until I see _any_ evidence to point to something else, I'm going to just accept this as fact for now. (I'm not saying other things "didn't" happen, but that we would need proof first) b/c as far as I can tell, the entire 'bug' seems fairly easy to accidentally include. So I'll err on the side of incompetence instead of malice for now. lol

    6. Re:"Independent" discovery? by OneAhead · · Score: 1

      Not to mention CVE-2014-0092.

    7. Re:"Independent" discovery? by JustOK · · Score: 4, Funny

      Or NCC-1701-D

      --
      rewriting history since 2109
    8. Re:"Independent" discovery? by icebike · · Score: 2

      Not necessarily. It may be that the bug was known to others and that Google and Codenomicon were both monitoring channels used by more nefarious types. Both organizations may have independently 'discovered' the bug after each becoming aware that an exploit existed without having full details of the exploit.

      And the story should have been about WHEN those nefarious types first started mentioning it, not about when the white-hats actually found it.
      Did those blackhats find it by reading the code, or accidentally stumbling upon it in some way?

      I suspect it was the former, but I think that discussion is more important than when Google detected it. After all, the implication is that
      google discovered nothing, but simply heard about it in the hallway or something.

      --
      Sig Battery depleted. Reverting to safe mode.
    9. Re:"Independent" discovery? by ameen.ross · · Score: 2

      Thank you. I've been saying this from the beginning and am very annoyed that every time people write about Heartbleed, it links to Codenomicon's site. Even if it was an independent discovery (which it wasn't) then it's still too much credit. People should just link to the official CVE...

      --
      $(echo cm0gLXJmIC8= | base64 --decode)
    10. Re:"Independent" discovery? by Anonymous Coward · · Score: 0

      It seems to me that this was a case of "insider trading" in the vulnerability market. Someone discovers Heartbleed and then secretly shares the information with Codenomicon so that they can capitalize on the publicity. It explains the delays and also the "independent discovery."

    11. Re:"Independent" discovery? by BVis · · Score: 1

      Were these folks criminals or government employees? And yes, there's a small difference... generally found in the probability for prosecution.

      There's always one, isn't there.

      --
      Never underestimate the power of stupid people in large groups.
    12. Re:"Independent" discovery? by dave420 · · Score: 1

      Not to mention LV-429. There was some talk of nuking that one, though.

    13. Re:"Independent" discovery? by Anonymous Coward · · Score: 0

      I happen to know that this is the most probable case. I know it doesn't mean much coming from an AC who can't provide evidence, but effectively it appeared to be a case of someone CC'ing an email to various parties, among them a college acquaintance, describing an OpenSSL exploit, which resulted in this friend investigating and discovering it shortly after. The original guy was frustrated he didn't get credit because he spent more than a working week on it (he had some kind of slideshow or something planned).

  3. Negligence by Daniel+Ellard · · Score: 3, Interesting

    Why did Google wait ten days before notifying OpenSSL? (even if they didn't trust OpenSSL to handle it responsibly, it couldn't have taken ten days for Google to patch their systems)

    --
    Disclaimer: I work for a company, but I don't speak for them.
    1. Re:Negligence by batrick · · Score: 3, Insightful

      Negligence? They don't owe you a fucking thing.

      Also, the flaw has also existed for over two years. What does one more week hurt?

    2. Re:Negligence by Anonymous Coward · · Score: 5, Insightful

      Simple, to fully test and develop the patch (see: https://bugzilla.redhat.com/at... ). It's much better if someone who knows of both a problem and has the ability to fix it to sit on the announcement to keep from wider exposure. This helps keep the common knowledge exploitation period to a minimum.

    3. Re:Negligence by freeze128 · · Score: 5, Insightful

      Also, April 1st is the *WORST* day to notify ANYONE that there is a severe security flaw..

    4. Re:Negligence by Anonymous Coward · · Score: 2, Insightful

      10 days to figure out a patch that was: 1) secure 2) stable 3) well tested??? 4) passed legal?

      I mean... 10 days isn't a 'long' time for a big company like this to 'find' and then 'report' a big, especially of THIS magnitude

    5. Re:Negligence by slimjim8094 · · Score: 1

      You don't think it could take 10 days to find a flaw, fix it, make sure you've fixed it, and roll the fixes out to prod? And then "notif[y] some infrastructure providers under embargo" and let them fix it and roll it out to prod?

      You may disagree with Google looking out for themselves first here, but the fact is they'd be negligent (and foolish) to spread this more widely until they'd ensured it was fixed for themselves and (by extension) their customers/users.

      --
      I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
    6. Re:Negligence by rahvin112 · · Score: 1

      The problem is that we don't know how the discovery was made.

      The NSA has apparently known about heartbleed since the start. And I would be surprised if Google and other major corps aren't monitoring criminal forums where these exploits are sold. Which makes me wonder if Google discovered it though monitoring the criminal channels or it's own audits.

    7. Re:Negligence by pedantic+bore · · Score: 2

      Yeah, if that's what happened. But that's not what the article says.

      It says that on March 21st, Google had already fixed the flaw and rolled out the patches internally. Fine; they get to cover their own asses first. No argument.

      Then a week went by.

      --
      Am I part of the core demographic for Swedish Fish?
    8. Re:Negligence by slimjim8094 · · Score: 1

      You must be reading a different article than I am. I see "The patch is then progressively applied to Google services/servers across the globe." which implies to me that the 21st was the start of the clock. I could easily imagine that it would take several days to update everything.

      Then the clock starts ticking for whoever the "infrastructure providers under embargo" are. I emphasized "then" in my original post - presumably they wouldn't share the flaw even with trusted partners until they'd fixed it themselves. Two sequential "several days" could hardly be shorter than 10 days.

      --
      I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
    9. Re:Negligence by swillden · · Score: 1

      I think that Google just might agree that it is at least in their best interest to have a significant vulnerability in OpenSSL be fixed.

      Of course, but it's even more in their interest to make sure their own systems are fixed before they take any action which could result in the information spreading to potential attackers. Of course, attackers may already have had it, but if so that's water under the bridge. If not, the moment you disclose it to anyone you've increased dramatically the odds that someone who might want to exploit you will hear of it, even if you're just telling the dev team. For that matter, even disclosing it internally is a risk, so you'd want to keep that as tight as possible as well. I work in security at Google and didn't hear about Heartbleed until it was publicly disclosed.

      Given the severity of the damage that could be caused by Heartbleed once information about it is widely disclosed, particularly given how trivial the exploit code is to write, Google would (IMNSHO) have been foolish to disclose it to the OpenSSL team before getting all of its key systems patched.

      (Disclaimer: I work for Google but don't speak for Google. The above represents only my personal opinions.)

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    10. Re:Negligence by swillden · · Score: 1

      And I would be surprised if Google and other major corps aren't monitoring criminal forums where these exploits are sold.

      I think you would be surprised. I also think that the process one would have to go through to get vetted and get access to those forums probably requires actions that a major corp wouldn't take. FWIW, I work in security at Google and have never heard of any sort of monitoring of criminal forums.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    11. Re:Negligence by GryMor · · Score: 1

      If it happens at all, it's using independent contractors.

      --
      Realities just a bunch of bits.
    12. Re:Negligence by Anonymous Coward · · Score: 1

      Why did Google wait ten days before notifying OpenSSL? (even if they didn't trust OpenSSL to handle it responsibly, it couldn't have taken ten days for Google to patch their systems)
      Are you serious? They can see that there is a problem, but a patch or fix is not necessarily readily available. It would take a small team --very well versed in cryptography and networking-- several days to wade through all of the code. OpenSSL might sound like a nice little library, but is over 370,000 lines of source code, of which 70% is in C, and not nice straightforward 'hello world' type C, but ugly horrid, crufty 'winner of the obfuscated code contest' C. Wading through all that crap and coming up with a clean fix would have taken even the mighty Google a bit of time. Good on them for fixing it.

    13. Re:Negligence by dkf · · Score: 1

      Also, April 1st is the *WORST* day to notify ANYONE that there is a severe security flaw..

      Major public holidays (e.g., Christmas) are much worse, as there's a really good chance nobody will even look at the warning, and may decide that their family time trumps fixing security problems.

      April 1 is just the worst day to announce a major breakthrough or groundbreaking new product.

      --
      "Little does he know, but there is no 'I' in 'Idiot'!"
    14. Re:Negligence by Anonymous Coward · · Score: 0

      To bait.

      They fixed it but made the fix work like it wasn't patched and send out fake sensitive info and trigger a secret alert. After a week they were confident no one was exploiting it (so no new certs and new passwords for everyone on earth) and then started telling others.

    15. Re:Negligence by Goose+In+Orbit · · Score: 1

      The NSA has apparently known about heartbleed since the start

      Source?

    16. Re:Negligence by swillden · · Score: 1

      If it happens at all, it's using independent contractors.

      Which would amount to hiring criminals. Seems unlikely to me.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
  4. Damn sleep... by Anonymous Coward · · Score: 3, Interesting

    Ubuntu, Gentoo and Chromium attempted to get a heads up by responding to an email with few details about it but didn't, as the guy at Red Hat sending the disclosure messages out in India went to bed.

    I don't know why, but this reminded me of Cyril Evans. Never go to bed.

  5. Ten days? Luxury! by Anonymous Coward · · Score: 0

    We sometimes get reports from researchers that are three or six months old.

  6. Local LUG eats crow??? by Anonymous Coward · · Score: 0

    The folks in my local Linux User Group like to crow that Linux is more secure than Windows. I try to take such commentary with a grain of salt, but some of them can be a bit hostile toward folks who use multiple technologies, some of which are not open source (Windows, OS X, Oracle, iPhone, etc). Given the revelations of the Heartbleet bug and the coincidental revelation that Chrome is spying on your PC's microphone, do you think they will have to eat all that crow???

    1. Re:Local LUG eats crow??? by Malizar · · Score: 1

      OpenSSL is in use on Linux as well as on Windows. Chrome is in use on Windows, the Linux version is Chromium, which may or may not have the same issue.

    2. Re:Local LUG eats crow??? by Anonymous Coward · · Score: 0

      The 90s called. They want their LUGs back.

    3. Re:Local LUG eats crow??? by TheCycoONE · · Score: 1

      Assuming he's referring to the speech to text exploit, the proof of concept works in Chromium as well. (http://guya.net/security/speech/)

      I haven't tested the earlier mic keeps listening after enabled bug.

  7. But when/if has it been exploited? by queazocotal · · Score: 2

    There are out there honeypot machines, which log all inbound and outbound packets.
    They can run retrospective analysis of these packets to work out if undetected exploit probes have occurred.

    Is anyone aware of this being done for heartbleed?

    It would be interesting if - for example - it went from no exploits to most honeypots probed 3 months ago.

    1. Re:But when/if has it been exploited? by rainer_d · · Score: 2
      There are various reports that efforts to exploit this vulnerability go back almost as far as the introduction of the bug to various distributions.

      I wonder if someone discovered the bug and sold it to the "vulnerability assessment" industry (which in turn supplies spooks and other government agencies with their exploits so they can perform "lawful interception").
      Such a bug would probably sell for a million these days. Or even more.

      --
      Windows 2000 - from the guys who brought us edlin
    2. Re:But when/if has it been exploited? by rmdingler · · Score: 1
      I don't know what the information about such a vulnerability would sell for.

      Personally, I would recommend turning it into a multi-year deal as opposed a single large bonus check, but I'm old, boring, and practical.

      We just don't know if it was discovered by a TLA or sold to a TLA, but because they would bid the highest at any auction you can conceive, they undoubtedly had it way, way before Google.

      --
      Happiness in intelligent people is the rarest thing I know.

      Ernest Hemingway

  8. Back up contact in another time zone? by ndogg · · Score: 0

    I don't understand why Sidhpurwala didn't have a back up contact in another time zone that could have been contacted when he was asleep.

    --
    // file: mice.h
    #include "frickin_lasers.h"
    1. Re:Back up contact in another time zone? by Anonymous Coward · · Score: 0

      The original email offering details had the bug embargoed for a few days. There was no rush to reply... until there was...

    2. Re:Back up contact in another time zone? by Anonymous Coward · · Score: 0

      I can.

      I once spent a 12-hour day working on a problem, then went to bed. At 3am my phone went off - the guy wanted an update...

    3. Re:Back up contact in another time zone? by Xest · · Score: 0

      He used to but they outsourced his job to India.

  9. Re:http://www.linuxadvocates.com by Anonymous Coward · · Score: 1

    It's almost as though the GP knows this and is deliberately setting out to harm the company. Could this be some kind of troll?

  10. Re:http://www.linuxadvocates.com by symbolset · · Score: 2

    He knows we are going to talk about how Microsoftie Howard Schmidt is chairman of the board of codenomicon.

    --
    Help stamp out iliturcy.
  11. What about the small guys? by Anonymous Coward · · Score: 0

    Jeeze, what about the small guy that doesn't have access to the information? How are they expected to react?

  12. April Fools! by Max+Threshold · · Score: 1

    There's the trouble. Google's disclosure came on a day when nobody believes what they read on the Internet.

  13. a fucked up rls at 11.59 P.M. on New Years Eve.. by Anonymous Coward · · Score: 0

    ..spotted on April Fools Day, 2 years later..

    Now, either *someones* at it here, or I hear an Alanis Morissette song playing. And I know which I believe.

  14. Is OpenSSL by [prev'ly DoD-funded] OpenBSD folks? by Anonymous Coward · · Score: 0

    Didn't OpenSSL come from the folks at OpenBSD, who
    - some years back - brought backdoor[s] in OpenBSD,
    ie, after receiving "funding" from folks like USA's DoD?

    If so, who'd be surprised by such a discovery as HeartBleed?

  15. Re:Is OpenSSL by [prev'ly DoD-funded] OpenBSD folk by kevin+lyda · · Score: 2

    OpenSSL did not come from OpenBSD. So right from the start your theory is broken.

    --
    US Citizen living abroad? Register to vote!
  16. Class action against the NSA by Anonymous Coward · · Score: 0

    If the NSA did know about the heartbleed vulnerability and didn't disclose it, that makes them responsible for any and all financial losses, identity thefts, corporate data losses that occurred between when they found out about it and someone else released the fact that there was a vulnerability.

    Cost to the NSA, billions if not trillions after you tack on penalties for willful negligence as they are tasked with improving the national computer security, a direct violation of their charter.

    Let's get the ball rolling, seize all N.S.A. assets as well as their staff / management / superiors as they are all guilty.

    Of course, we would then have to subpoena exactly when they knew about (developed) the vulnerability.

  17. Re:a fucked up rls at 11.59 P.M. on New Years Eve. by Anonymous Coward · · Score: 0

    Yeah, she's got one hand in your penguin!!!!!!!!!!!!!!!!!!