Slashdot Mirror
Theo De Raadt's Small Rant On OpenSSL
New submitter raides (881987) writes "Theo De Raadt has been on a better roll as of late. Since his rant about FreeBSD playing catch up, he has something to say about OpenSSL. It is worth the 5 second read because it is how a few thousand of us feel about the whole thing and the stupidity that caused this panic."
Update: 04/10 15:20 GMT by U L : Reader badger.foo pointed out Ted Unangst (the Ted in the mailing list post) wrote two posts on the issue: "heartbleed vs malloc.conf and "analysis of openssl freelist reuse" for those seeking more detail.
This could get a lot more ugly...
Once upon a time, SSL certificates were signed against a single root certificate, each SSL cert issuer had a single root certificate authority for each of its product lines. Now all corps issue an SSL certificate that is signed against and INTERMEDIATE certificate, which in turn is signed against the root certificate.
What happens if a provider's server has this exploit and the intermediate certificate is compromised? EVERY certificate signed against that intermediate must be revoked. Or put another way, the ENTIRE PRODUCT LINE must be tossed into the garbage and all certs reissued.
So if Verisign or Thawte discover new their intermediate certificate MIGHT have been exploited, would they say anything? The servers implementing those certs are in the hands of a select few - it would be easy to hide the possibility they might have been compromised.
When the foot seeks the place of the head, the line is crossed. Know your place. Keep your place. Be a shoe.
When Slashdot comments become full front page stories? This was already posted a few times as comments in the last OpenSSL post.
Years ago the BSD guys added safeguards to malloc and mmap, but they were disabled for all platforms in OpenSSL only because they caused performance problems on some platforms. He finishes by saying that OpenSSL is not developed by a responsible team.
heartbleed vs malloc.conf
and
analysis of openssl freelist reuse. Short articles with a lot of good information.
-- That grumpy BSD guy - http://bsdly.blogspot.com/
So as far as I can tell, his rant is essentially that people should not use custom allocators and instead rely on the general purpose one built into libc because they can add system wide tools there.
I can see the argument for most cases, that is kinda the point of a general purpose allocator, but encryption (esp if you are doing lots of it) really strikes me as a case where you can really benefit from having explicit control over the behavior. I have worked on a number of applications where custom allocators had significant (user facing, not just benchmarks) impacts on performance. Ironically it also meant we were able to do better checking then the general exploit detection kits since we could bake more specific knowledge into the validator.
Why OpenSSL is so popular? It has FIPS-certified module, and this becomes important for selling your product to the government.
So what could be done to prevent something like this from happening in the future? People will keep writing bad code, this is unavoidable, but what automated tests could be run to make sure to avoid the worst of it? Someone with direct development experience please educate the rest of us.
So, it's always great fun bashing "obvious" bugs, especially when they have such an impact, but let it be noted that thousands of implementers used openssl to build systems taking the package at face value despite these now "obvious" deficiencies in development process. If you were that concerned about security, they would have done what Google did, and audit the code. There are of course many practical reasons why people can't do that, but regardless, the blame arrow here points both ways.
The trick is not to read TFA in the first place. Also, you must be new here.
May the Maths Be with you!
That code is almost a text book example of material that is submitted to the Underhanded C contest...
http://en.wikipedia.org/wiki/Underhanded_C_Contest
GnuTLS, which recently people were being told to avoid in favor of OpenSSL. You see, there was this bug...
He's just trying to be a bigger dick than Linus..
My understanding is Theo said: Developers on a security product made a conscious decision to turn off last line of defence security for all platforms in the interest of performance on some systems. That does not sound like and unfortunate error to me, it sounds outright irresponsible.
Because who in the world has ever heard of OpenSSH, right?
Pretty good is actually pretty bad.
"it is how a few thousand of us feel about the whole thing"
Then maybe you thousands should stop complaining and start contributing to the project, which is so under-resourced problems like this are pretty much inevitable.
So.. it has come to this
He said that, but is that what happened? Were OpenSSL's developers aware that malloc()/free() have special security concerns that OpenBSD's developers had specifically addressed (I assume that's what meant by "a conscious decision to turn off last-line-of-defense-security")
I understand Theo's point, to a certain degree I kinda understand it, but I'm more inclined to feel the problem is with OpenSSL's developers clearly not understanding the security concerns about malloc(). That is, if they were aware that OpenBSD's malloc() contained code to ensure against data leakage, it would seem to me to be highly probable they would have implemented the same deal in OpenSSL given, you know, their entire point is security. The fact they didn't makes me think they didn't know OpenBSD's malloc() had these measures in the first place.
Should they have done? And how should they have known? Genuine question, and finger pointing would be inappropriate right now: how do we make sure that certain security strategies and issues are as well known as, say, stack pointer issues are today.
You are not alone. This is not normal. None of this is normal.
If you've never heard of him, you're not part of any important "tech community". Period.
Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
That's pretty scathing. I'd hate to be THOSE guys...
In addition, the mitigation countermeasures also prevent memory debuggers like Valgrind from finding the problem (Valgrind find use-before-init for malloc'ed blocks, but not if there is a wrapper in between that re-uses blocks), and may also neutralize code-security scanners like Fortify.
I have to admit that while my original intuition was "screwup", this looks more and more like some parts of the OpenSSL team have been compromised and did things that make this kind of bug far more likely. Like their own insecure memory allocation. Like not requiring time-of-use boundary checks or having any secure coding guidelines in the first place. Like documenting everything badly so potential reviewers get turned away. Like not having working review for patched or a working fuzz-testing set-up (which would have found bug this easily).
After all, the NSA does not have to sabotage FOSS crypto software. They just have to make sure the quality is low enough. The bugs they can exploit will follow. And the current mess is just a plain classic. Do enough things wrong and eventually stuff breaks spectacularly.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
My impression is OpenBSD's hardened allocator is relatively common knowledge and definitely should be among people writing security software. And that's not even remotely the only such allocator out there that does that sort of thing too, though it's probably the most well-known from the industrial side.
A lot of large performance-sensitive projects implement custom allocators in the form of arenas and freelists. Lots of platforms have a fast malloc implementation these days, but none of them will be as fast as this for the simple reason that the program knows more about its memory usage patterns than any general-purpose allocator ever could.
Not to say I can't understand Theo's point of view -- if he wants maximum security, then a program which bypasses one of his layers in the name of performance might not be the best for him.
On the flip side, the standards have no notion of such security layers and I feel it is perfectly reasonable for a team to not throw away performance in the interests of some platform-specific behavior. This was a bug, pure and simple. There's nothing wrong with using custom allocators. To say that "OpenSSL is not developed by a responsible team" is simply nonsense.
This is not a problem with OpenSSL, or the C Language or the Malloc implementation, this is a problem because everyone is relying on the same black box they do not understand. Because this is "standard" and common practice to use it. The only long term defense against this kind of vulnerability is software (and hardware?) diversity. Software built on custom SSL implementations may have even worse vulnerabilities, but nobody will discover them, and even if they do, it won't affect everyone on this planet. When I read Theo De Raadt, I fear his "solution" may only worsen the problem. We can't have all our secrets protected by the exact same door, no matter how strong the door is, once it's broken...
is not helping!! What would be useful is a list of popular services affected by this issue the BBC is at least making a start here http://www.bbc.co.uk/news/tech...
Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.
That is, if they were aware that OpenBSD's malloc() contained code to ensure against data leakage, it would seem to me to be highly probable they would have implemented the same deal in OpenSSL given, you know, their entire point is security. The fact they didn't makes me think they didn't know OpenBSD's malloc() had these measures in the first place.
Not just OpenBSD's malloc(). glibc can do the same thing if you set MALLOC_PERTURB.
Watch this Heartland Institute video
De Raadt wrote "OpenSSL is not developed by a responsible team".
On the contrary, I believe it was developed by a responsible team, that unfortunately made an error.
Most everyone have made errors, even if most go unnoticed and are essentially harmless. This one appears different, but I don't think it justifies De Raadt's moronic comment.
Not so sure they're responsible.
Did you read this?
This bug would have been utterly trivial to detect when introduced had the OpenSSL developers bothered testing with a normal malloc (not even a security focused malloc, just one that frees memory every now and again). Instead, it lay dormant for years until I went looking for a way to disable their Heartbleed accelerating custom allocator.
Building exploit mitigations isn’t easy. It’s difficult because the attackers are relentlessly clever. And it’s aggravating because there’s so much shitty software that doesn’t run properly even when it’s not under attack, meaning that many mitigations cannot be fully enabled. But it’s absolutely infuriating when developers of security sensitive software are actively thwarting those efforts by using the world’s most exploitable allocation policy and then not even testing that one can disable it.
The OpenSSL team doesn't fully test their product.
That's pretty much as good an example of incompetence that you can probably find.
A lot of large performance-sensitive projects implement custom allocators in the form of arenas and freelists. Lots of platforms have a fast malloc implementation these days, but none of them will be as fast as this for the simple reason that the program knows more about its memory usage patterns than any general-purpose allocator ever could.
This is security software. You don't sacrifice the library's core functionality to make it run a bit faster on the old Celeron 300 running Windows 98.
Oh, and read this: http://www.tedunangst.com/flak/post/analysis-of-openssl-freelist-reuse
In effect at some points OpenSSL does:
free (rec);
rec = malloc (...);
and assumes that rec is the same.
Eeew,
Watch this Heartland Institute video
Well, what you are pointing out is that a CA is a single point of failure -- Something actual security conscious engineers avoid like the plague. What you may not realize is that collectively the entire CA system is compromised by ANY ONE of those single points of failure because any CA can create a cert for ANY domain without the domain owner's permission. See also: The Diginotar Debacle.
The thing is, nobody actually checks the the cert chain, AND there's really no way to do so. How do I know if my email provider switched from Verisign to DigiCert? I don't, and there's no way to find out that's not susceptible to the same MITM attack.
So, let's take a step back for a second. Symmetric stream ciphers need a key. If you have a password as the key then you need to transmit that key back and forth without anyone knowing what it is. You have to transmit the secret, and that's where Public Key Crypto comes in, however it doesn't authenticate the identity of the endpoints, that's what the CA system is supposed to do. Don't you see? All this CA PKI system is just moving the problem of sharing a secret from being the password, to being which cert the endpoint is using -- That becomes the essential "secret" you need to know, and it's far less entropy than a passphrase!
At this time I would like to point out that if we ONLY used public key crypto between an client and server to establish a shared secret upon account creation, then we could use a minor tweak to the existing HTTP Auth Hashed Message Authentication Code (HMAC) proof of knowledge protocol (whereby one endpoint provides a nonce, then the nonce is HMAC'd with the passphrase and the unique-per-session resultant hash provides proof that the endpoints know the same secret without revealing it) to secure all the connections quite simply: Server and client exchange Nonces & available protocols for negotiation, the nonces are concatenated and HMAC'd with the shared secret stored at both ends, then fed to your key-stretching / key expansion system AND THAT KEYS THE SYMMETRIC STREAM CIPHER SIMULTANEOUSLY AT BOTH ENDS so the connection proceeds immediately with the efficient symmetric encryption without any PKI CA system required.
PKI doesn't really authenticate the endpoint, it just obfuscates the fact that it doesn't by going through the motions and pretending to do so. It's a security theater. SSL/TLS and PKI are essentially the Emperor's New Secure Clothes. At least with the shared secret model I mention above, there's just that one-time small window of PK crypto for secret exchange at worst (failing to intercept account creation means no MITM) and at best you would actually have the CHANCE to go exchange your secret key out of band -- Visit your bank in person and exchange the passphrase, etc. then NO MITM could intercept the data. HTTP Auth asks for the password in a native browser dialog BEFORE showing you any page to login (and it could remember the PW in a list, or even generate them via hashing the domain name with a master PW and some salt so you could have one password for the entire Internet). That's how ALL security should work, it ALL relies on a shared secret, so you want the MOST entropic keyspace not the least entropic selection (which CA did they use). If you're typing a password into a form field on a web page, it's ALREADY game over.
Do this: Check the root certs in your browser. For Firefox > Preferences > Advanced > Certificates > View. See that CNNIC one? What about the Hong Kong Post? Those are Known bad actors that your country is probably at cyber war with, and THEY ARE TRUSTED ROOTS IN YOUR FUCKING BROWSER?! Not to mention all the other Russian ones or Turkish, etc. ones that are on the USA's official "enemy" list. Now, ANY of those can pretend to be whatever domain's CA they want, and if your traffic bounces through their neck of the woods they can MITM you and you'll be n
Ouch. Serious ouch. Thank you. That suggests that the situation is considerably worse than De Raadt said.
You are not alone. This is not normal. None of this is normal.
Theo De Raadt is the king of tinfoil hats, and behind OpenBSD -- a version of BSD designed to be as secure as possible.
Hell yes they should have known, because the people responsible for one of the most important security applications in the entire world damn well ought to be experts!
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
This is simply not true, stop spinning it.
Even if OpenSSL is using system's malloc, with all its mitigation features, the bug still works. The attacker just has to be more careful, lest he should read freed() and unmapped memory, and so cause a crash and (supposedly) leave some kind of meaningful trail.
Allocators in this case make no significant difference with regards to severity of the problem.
What is or is not in process free list makes no difference when you can arbitrarily request any block of memory you please.. only slightly effects chance of success when it becomes necessary to shoot in the dark. Lets not forget most OS provided optimized allocators keep freed memory in their heaps for some time as well and may still not throw anything when referenced.
Looking at code for this bug I am amazed any of this garbage was accepted in the first place. There is no effort at all to minimize chance of error with redundant + 3's and 1 + 2's sprinkled everywhere complete with unchecked allocation for good measure.
Suppose I should be glad 1 + 2 = 3 today and they have not used signed integers when dealing with lengths.
... oh dear god ...
Well at least they learned their lesson and have stopped sprinkling redundant and error prone type + length + padding garbage everywhere... see..
and here ..
... oh well .. Looks like plenty of low hanging fruit to be had for anyone with a little spare time.
To say that "OpenSSL is not developed by a responsible team" is simply nonsense.
Unless you look at the code, and notice they are using unvalidated data for......anything. That's a rookie mistake.
"First they came for the slanderers and i said nothing."
As I read his analysis, OpenSSL relies on releasing a buffer, reallocating it, and getting the PREVIOUS contents of that buffer back -- or else it will abort the connection. (Search for the string "On line 1059, we find a call to ssl3_release_read_buffer after we have read the header, which will free the current buffer." in his article referenced by the parent post).
Now, IMO, this goes way beyond sloppy. Releasing a buffer before you're done with it, and relying on a wacky LIFO reallocation scheme giving you back that very same buffer so you can process it, is either 1) an utterly incompetent coding blunder that just happened to work when combined with an utterly terrible, insecure custom allocation scheme, or 2) specifically designed to ensure that this insecure combination is widely deployed to provide a custom-made back door, as it works only with the leaky custom allocator.
If 1), then I must agree with Theo that the OpenSSL team were indeed irresponsible, since at least one of these two cooperating blunders ought to have shown up in a decent security audit of the code, and any decent set of security-oriented coding standards would forbid them both.
If 2), then it was deliberate, and the tinfoil-hat crowd is right for once.
"My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
Also, they didn't need to use OpenBSD's stuff - any standard malloc would have caused OpenSSL to abort the connection in this case, as Ted shows in his blog post.
I think that might scare people from upgrading. Going from 7.10322 to to 8 is scarier than going from 7.10322 to 7.10322g It suggests that upgrading will break things.
mod parent up
Let's not miss the opportunity to point a finger of blame at the RFC, which says " to make the extension as versatile as possible, an arbitrary payload and a random padding is preferred, ". https://tools.ietf.org/html/rf... Arbitrary payload and a random padding for a heartbeat instead of a specified sequence of bits? This is very suspicious.
"Mit der Dummheit kaempfen Goetter selbst vergebens." - Schiller
I agree with Theo on the broader point, but disagree here: "their product" is the code they wrote with their custom memory allocator. It's not the code they wrote with some changes someone outside the product made. Their custom allocator isn't disabled or enabled depending on configuration checks or something like that, it's always part of the product.
It was still a pretty stupid decision.
Ted Unangst merely used a different configuration option - one provided in the standard OpenSSL software.
Given some of the other bugs he found, we know for a fact that the OpenSSL team does not fully test their product.
Period.
Unangst even found OpenSSL basically assumes that "free(prt); ptr=malloc()" will return the same pointer. Holy fucking shit, you gotta be kidding me. What kind of incompetent boob doesn't test a C/C++ application for that?
Maybe the OpenSSL folks need to begin a Kickstart campaign to pay for a Purify license or two.
Can't say I'm surprised. OpenSSL is a pile of dung. It's nothing to do with being written in any language, it's just horrible.
There's not even any documentation. I mean, literally, none. Nothing vaguely useful. How do I programmatically load a certificate into the store, along with a chain of related trusted certificates, and then set my requirements (must be in-date, must be validly signed, etc.) and get out a "It's fine" / "Something's not right" response? The only answers I could ever find were to follow published examples and tweak.
And when it comes to working out where in the published examples structure X comes from, or how to convert it to structure Y, you're on your own unless you happen to have picked a comprehensive (and almost certainly not OpenSSL-supplied) example.
It's just that bad. I was writing a pseudo-DRM for a game / Steam-like distribution platform as a hobbyist project. It was literally horrible to even try to self-sign some certificate and then see if it all panned out later from another computer to guarantee integrity. In the end, I had to "imagine" every possible case and find a way to counter it (i.e. client cert expired, client cert invalid, server cert not signed client cert, server cert has bad chain of trust, client cert not signable for that purpose, etc.) - and almost always there was NOTHING to indicate what the recommended way to do it was.
There is no decent OpenSSL documentation at all. Not even a decent overview of the process of checking certificates. It scared me at the time, knowing how important the library is, and it can only lead to bad code.
In the end, I'm quite glad I don't have to program against it for a living. If I did, I'd be seriously looking for something else.
That would have been an appropriate boilerplate response to your typical Theo De Raadt comment, but... did you actually read that "analysis of openssl freelist reuse" link? It's hard for me to imagine that having been done by mistake; it's more of an instance of a coder cleverly cutting a corner. And some might argue "don't we all?", but if one is writing such security-critical software, one should at least have some notion that clever corner-cutting is a gigantic no-no.
Now, the freelist reuse is not the cause of the heartbleed bug; it merely frustrates what would otherwise have been a relatively straightforward mitigation strategy. But it's a symptom of an attitude that is, well, irresponsible.
I understand Theo's point, to a certain degree I kinda understand it, but I'm more inclined to feel the problem is with OpenSSL's developers clearly not understanding the security concerns about malloc(). That is, if they were aware that OpenBSD's malloc() contained code to ensure against data leakage, it would seem to me to be highly probable they would have implemented the same deal in OpenSSL given, you know, their entire point is security. The fact they didn't makes me think they didn't know OpenBSD's malloc() had these measures in the first place.
Here is what happens, as far as I understand: a client sends two bytes of data to the server and asks the server to send the identical two bytes back to the client, to check that the server is still alive. That's how it works normally. A client could send 65,000 bytes and ask for the 65,000 bytes to be sent back, except that would be inefficient.
Instead, an attacker sends two bytes of data to the server and asks for the same 65,000 bytes back. The server stores the two bytes with a bit of overhead into a malloc block, creates a huge malloc block for the results, and memcpy's 65,000 bytes from the small malloc block to the huge one. 64,998 bytes that are copied are just whatever was in memory after that malloc block.
All the usual measures against buffer overwrites don't help, because there is no buffer overwrite. Nothing is destroyed on the server, instead it is tricked into giving information it didn't want to give. What could malloc do about that? A "free" and "realloc" implementation that sets memory to zero wouldn't hurt. Of course that doesn't help if the memory after the small malloc block is actually currently used. You'd need a malloc that will crash if you read past the end of a malloc buffer. That's hard to do efficiently.
Theo's "rant" isn't brought about by a typical bug in the sense of a mistake in the code. His rant is brought about by the fact that OpenSSL deliberately introduced wrapper functions for malloc/free, making it impossible for a system to provide hardened ones, then made those wrappers default. He further rants about the fact that, because it's default, they later introduced a bug that means you can't turn the wrappers off.
The heartbeat bug itself was a mistake and a bug in the traditional sense. The "hey let's replace malloc/free" is much closer to "bad decision" than "mistake."
OpenBSD's "usual measures" would have helped, because the OpenBSD allocator will often put allocated memory in isolated pages. So it's quite likely that 64K read will cross into an unmapped page, and boom, you have your crash.
This level of overread protection is close to free.
Even if OpenSSL is using system's malloc, with all its mitigation features, the bug still works. The attacker just has to be more careful, lest he should read freed() and unmapped memory, and so cause a crash and (supposedly) leave some kind of meaningful trail.
You got it exactly right. He's complaining that because they provided their own malloc() wrapper that the read of freed() memory is NOT causing a crash. If they had used the system malloc() then there would be crashes, the issues would be detected, and they would have been fixed.
That's not true. A malicious server can exploit a vulnerable client and read the contents of its memory.
Theo has been a strong proponent of hard hats/steel helmets, not tinfoil hats. Most people thought that approach was overkill for general walking around, at least up until Snowden showed there were lots of national actors firing lots of rocks into the air.
Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde. - Voltaire