Heartbleed Disclosure Timeline Revealed

bennyboy64 (1437419) writes "Ever since the Heartbleed flaw in OpenSSL was made public there have been various questions about who knew what and when. The Sydney Morning Herald has done some analysis of public mailing lists and talked to those involved with disclosing the bug to get the bottom of it. The newspaper finds that Google discovered Heartbleed on or before March 21 and notified OpenSSL on April 1. Other key dates include Finnish security testing firm Codenomicon discovering the flaw independently of Google at 23:30 PDT, April 3. SuSE, Debian, FreeBSD and AltLinux all got a heads up from Red Hat about the flaw in the early hours of April 7 — a few hours before it was made public. Ubuntu, Gentoo and Chromium attempted to get a heads up by responding to an email with few details about it but didn't, as the guy at Red Hat sending the disclosure messages out in India went to bed. By the time he woke up, Codenomicon had reported the bug to OpenSSL."

Comments

"Independent" discovery?

[Red Herring]

> Google discovered Heartbleed on or before March 21 and notified OpenSSL on April 1. Other key dates include Finnish security testing firm Codenomicon discovering the flaw independently of Google at 23:30 PDT, April 2.

Doesn't it seem strange that the flaw has existed for a long, long time (years?) but Codenomicon happens to find it less than a day after Google notified OpenSSL, and, per the article, "some infrastructure providers under embargo"? That just seems... unlikely. Not impossible, but it kind of makes you wonder who is leaking information...

Re:"Independent" discovery?

[Albanach]

Not necessarily. It may be that the bug was known to others and that Google and Codenomicon were both monitoring channels used by more nefarious types. Both organizations may have independently 'discovered' the bug after each becoming aware that an exploit existed without having full details of the exploit.

Re:"Independent" discovery?

[AdhSeidh]

perhaps you have already forgotten about CVE-2014-1266 the Apple SSL/TLS bug from Februrary this is why every security group on the planet was looking for other encryption related loopholes

Re:"Independent" discovery?

[rmdingler]

In all likelihood, there was a "discovery" by Google that led to a sharing of information with Codenomicon... someone told an old college buddy or former co-worker.

There were almost certainly folks who were aware of the vulnerability before Google.

Were these folks criminals or government employees? And yes, there's a small difference... generally found in the probability for prosecution.

Re:"Independent" discovery?

[briancox2]

That's what Newton said to Leibniz.

Re:"Independent" discovery?

[Anonymous Coward]

No, this is not uncommon at all in research. The idea that two groups are both looking into how [X] works, and how [Y] responds to [X] is quite common. Being a security researcher myself (slightly different sub-field, but still reason for anon posting) I can say that it is quite an easy possibility that both teams were checking the ENTIRE ISO~TCP/IP stack from lvl 0 up to lvl (whatever 'top' is in your outlook/naming scheme) And that they both found it around the same time.

Until I see _any_ evidence to point to something else, I'm going to just accept this as fact for now. (I'm not saying other things "didn't" happen, but that we would need proof first) b/c as far as I can tell, the entire 'bug' seems fairly easy to accidentally include. So I'll err on the side of incompetence instead of malice for now. lol

Re:"Independent" discovery?

[OneAhead]

Not to mention CVE-2014-0092.

Re:"Independent" discovery?

[JustOK]

Or NCC-1701-D

Re:"Independent" discovery?

[icebike]

Not necessarily. It may be that the bug was known to others and that Google and Codenomicon were both monitoring channels used by more nefarious types. Both organizations may have independently 'discovered' the bug after each becoming aware that an exploit existed without having full details of the exploit.

And the story should have been about WHEN those nefarious types first started mentioning it, not about when the white-hats actually found it.
Did those blackhats find it by reading the code, or accidentally stumbling upon it in some way?

I suspect it was the former, but I think that discussion is more important than when Google detected it. After all, the implication is that
google discovered nothing, but simply heard about it in the hallway or something.

Re:"Independent" discovery?

[ameen.ross]

Thank you. I've been saying this from the beginning and am very annoyed that every time people write about Heartbleed, it links to Codenomicon's site. Even if it was an independent discovery (which it wasn't) then it's still too much credit. People should just link to the official CVE...

Re:"Independent" discovery?

[BVis]

Were these folks criminals or government employees? And yes, there's a small difference... generally found in the probability for prosecution.

There's always one, isn't there.

Re:"Independent" discovery?

[dave420]

Not to mention LV-429. There was some talk of nuking that one, though.

Negligence

[Daniel Ellard]

Why did Google wait ten days before notifying OpenSSL? (even if they didn't trust OpenSSL to handle it responsibly, it couldn't have taken ten days for Google to patch their systems)

Re:Negligence

[batrick]

Negligence? They don't owe you a fucking thing.

Also, the flaw has also existed for over two years. What does one more week hurt?

Re:Negligence

[Anonymous Coward]

Simple, to fully test and develop the patch (see: https://bugzilla.redhat.com/at... ). It's much better if someone who knows of both a problem and has the ability to fix it to sit on the announcement to keep from wider exposure. This helps keep the common knowledge exploitation period to a minimum.

Re:Negligence

[freeze128]

Also, April 1st is the *WORST* day to notify ANYONE that there is a severe security flaw..

Re:Negligence

[Anonymous Coward]

10 days to figure out a patch that was: 1) secure 2) stable 3) well tested??? 4) passed legal?

I mean... 10 days isn't a 'long' time for a big company like this to 'find' and then 'report' a big, especially of THIS magnitude

Re:Negligence

[slimjim8094]

You don't think it could take 10 days to find a flaw, fix it, make sure you've fixed it, and roll the fixes out to prod? And then "notif[y] some infrastructure providers under embargo" and let them fix it and roll it out to prod?

You may disagree with Google looking out for themselves first here, but the fact is they'd be negligent (and foolish) to spread this more widely until they'd ensured it was fixed for themselves and (by extension) their customers/users.

Re:Negligence

[rahvin112]

The problem is that we don't know how the discovery was made.

The NSA has apparently known about heartbleed since the start. And I would be surprised if Google and other major corps aren't monitoring criminal forums where these exploits are sold. Which makes me wonder if Google discovered it though monitoring the criminal channels or it's own audits.

Re:Negligence

[pedantic bore]

Yeah, if that's what happened. But that's not what the article says.

It says that on March 21st, Google had already fixed the flaw and rolled out the patches internally. Fine; they get to cover their own asses first. No argument.

Then a week went by.

Re:Negligence

[slimjim8094]

You must be reading a different article than I am. I see "The patch is then progressively applied to Google services/servers across the globe." which implies to me that the 21st was the start of the clock. I could easily imagine that it would take several days to update everything.

Then the clock starts ticking for whoever the "infrastructure providers under embargo" are. I emphasized "then" in my original post - presumably they wouldn't share the flaw even with trusted partners until they'd fixed it themselves. Two sequential "several days" could hardly be shorter than 10 days.

Re:Negligence

[swillden]

I think that Google just might agree that it is at least in their best interest to have a significant vulnerability in OpenSSL be fixed.

Of course, but it's even more in their interest to make sure their own systems are fixed before they take any action which could result in the information spreading to potential attackers. Of course, attackers may already have had it, but if so that's water under the bridge. If not, the moment you disclose it to anyone you've increased dramatically the odds that someone who might want to exploit you will hear of it, even if you're just telling the dev team. For that matter, even disclosing it internally is a risk, so you'd want to keep that as tight as possible as well. I work in security at Google and didn't hear about Heartbleed until it was publicly disclosed.

Given the severity of the damage that could be caused by Heartbleed once information about it is widely disclosed, particularly given how trivial the exploit code is to write, Google would (IMNSHO) have been foolish to disclose it to the OpenSSL team before getting all of its key systems patched.

(Disclaimer: I work for Google but don't speak for Google. The above represents only my personal opinions.)

Re:Negligence

[swillden]

And I would be surprised if Google and other major corps aren't monitoring criminal forums where these exploits are sold.

I think you would be surprised. I also think that the process one would have to go through to get vetted and get access to those forums probably requires actions that a major corp wouldn't take. FWIW, I work in security at Google and have never heard of any sort of monitoring of criminal forums.

Re:Negligence

[GryMor]

If it happens at all, it's using independent contractors.

Re:Negligence

[Anonymous Coward]

Why did Google wait ten days before notifying OpenSSL? (even if they didn't trust OpenSSL to handle it responsibly, it couldn't have taken ten days for Google to patch their systems)
Are you serious? They can see that there is a problem, but a patch or fix is not necessarily readily available. It would take a small team --very well versed in cryptography and networking-- several days to wade through all of the code. OpenSSL might sound like a nice little library, but is over 370,000 lines of source code, of which 70% is in C, and not nice straightforward 'hello world' type C, but ugly horrid, crufty 'winner of the obfuscated code contest' C. Wading through all that crap and coming up with a clean fix would have taken even the mighty Google a bit of time. Good on them for fixing it.

Re:Negligence

[dkf]

Also, April 1st is the *WORST* day to notify ANYONE that there is a severe security flaw..

Major public holidays (e.g., Christmas) are much worse, as there's a really good chance nobody will even look at the warning, and may decide that their family time trumps fixing security problems.

April 1 is just the worst day to announce a major breakthrough or groundbreaking new product.

Re:Negligence

[Goose In Orbit]

The NSA has apparently known about heartbleed since the start

Source?

Re:Negligence

[swillden]

If it happens at all, it's using independent contractors.

Which would amount to hiring criminals. Seems unlikely to me.

Damn sleep...

[Anonymous Coward]

Ubuntu, Gentoo and Chromium attempted to get a heads up by responding to an email with few details about it but didn't, as the guy at Red Hat sending the disclosure messages out in India went to bed.

I don't know why, but this reminded me of Cyril Evans. Never go to bed.

Re:End result: mass panic

[Anonymous Coward]

They exist; it's just that the vast majority of the people who belong to those eyes are really not qualified to be working on software that will be used in such important roles, and now we're paying the price. You don't use your Fisher Price tool set when you are building a real house. You just don't.

Re:End result: mass panic

[Anonymous Coward]

And you also see this same type of thing in proprietary software, where tons of losers are hired to work on the code, with predictably terrible results. The thing about open source is that anyone can see the source code, and people not part of the group that wrote the code can check it, so you at least have some chance of understanding what's going on.

Anyone who claims that open source advocates claim that open source is 100% immune from all flaws is just spewing forth straw men.

But when/if has it been exploited?

[queazocotal]

There are out there honeypot machines, which log all inbound and outbound packets.
They can run retrospective analysis of these packets to work out if undetected exploit probes have occurred.

Is anyone aware of this being done for heartbleed?

It would be interesting if - for example - it went from no exploits to most honeypots probed 3 months ago.

Re:But when/if has it been exploited?

[rainer_d]

There are various reports that efforts to exploit this vulnerability go back almost as far as the introduction of the bug to various distributions.

I wonder if someone discovered the bug and sold it to the "vulnerability assessment" industry (which in turn supplies spooks and other government agencies with their exploits so they can perform "lawful interception").
Such a bug would probably sell for a million these days. Or even more.

Re:But when/if has it been exploited?

[rmdingler]

I don't know what the information about such a vulnerability would sell for.

Personally, I would recommend turning it into a multi-year deal as opposed a single large bonus check, but I'm old, boring, and practical.

We just don't know if it was discovered by a TLA or sold to a TLA, but because they would bid the highest at any auction you can conceive, they undoubtedly had it way, way before Google.

Re:http://www.linuxadvocates.com

[Anonymous Coward]

It's almost as though the GP knows this and is deliberately setting out to harm the company. Could this be some kind of troll?

Re:Local LUG eats crow???

[Malizar]

OpenSSL is in use on Linux as well as on Windows. Chrome is in use on Windows, the Linux version is Chromium, which may or may not have the same issue.

Re:http://www.linuxadvocates.com

[symbolset]

He knows we are going to talk about how Microsoftie Howard Schmidt is chairman of the board of codenomicon.

April Fools!

[Max Threshold]

There's the trouble. Google's disclosure came on a day when nobody believes what they read on the Internet.

Re:Is OpenSSL by [prev'ly DoD-funded] OpenBSD folk

[kevin lyda]

OpenSSL did not come from OpenBSD. So right from the start your theory is broken.

Re:Local LUG eats crow???

[TheCycoONE]

Assuming he's referring to the speech to text exploit, the proof of concept works in Chromium as well. (http://guya.net/security/speech/)

I haven't tested the earlier mic keeps listening after enabled bug.