Slashdot Mirror


Theo De Raadt's Small Rant On OpenSSL

New submitter raides (881987) writes "Theo De Raadt has been on a better roll as of late. Since his rant about FreeBSD playing catch up, he has something to say about OpenSSL. It is worth the 5 second read because it is how a few thousand of us feel about the whole thing and the stupidity that caused this panic." Update: 04/10 15:20 GMT by U L : Reader badger.foo pointed out Ted Unangst (the Ted in the mailing list post) wrote two posts on the issue: "heartbleed vs malloc.conf and "analysis of openssl freelist reuse" for those seeking more detail.

301 comments

  1. "It's Not a Tumor" - Oh Wait, It Is by alphatel · · Score: 4, Interesting

    This could get a lot more ugly...

    Once upon a time, SSL certificates were signed against a single root certificate, each SSL cert issuer had a single root certificate authority for each of its product lines. Now all corps issue an SSL certificate that is signed against and INTERMEDIATE certificate, which in turn is signed against the root certificate.

    What happens if a provider's server has this exploit and the intermediate certificate is compromised? EVERY certificate signed against that intermediate must be revoked. Or put another way, the ENTIRE PRODUCT LINE must be tossed into the garbage and all certs reissued.

    So if Verisign or Thawte discover new their intermediate certificate MIGHT have been exploited, would they say anything? The servers implementing those certs are in the hands of a select few - it would be easy to hide the possibility they might have been compromised.

    --
    When the foot seeks the place of the head, the line is crossed. Know your place. Keep your place. Be a shoe.
    1. Re:"It's Not a Tumor" - Oh Wait, It Is by putaro · · Score: 5, Insightful

      If your intermediate certificate's signing keys are on the internet facing web servers you're doing it wrong. That intermediate signing key should be treated with the same level of security you would treat a root key with.

    2. Re:"It's Not a Tumor" - Oh Wait, It Is by Anonymous Coward · · Score: 5, Informative

      It is good practice to sign against an intermediate certificate. That way if it is compromised you can reject it and issue a new intermediate certificate signed by your root certificate. You can push the new certificates as updates since they would be validated against the root certificate.

      You need to read up on authenticating the entire chain of certificates.

    3. Re:"It's Not a Tumor" - Oh Wait, It Is by Bill,+Shooter+of+Bul · · Score: 2

      I don't understand what you are trying to imply by contrasting the old way without intermediate vs the new way with intermediate certs. Surely, its better with the intermediates as the root cert that signs it should be and can be used less often and stored in cold storage somewhere.

      --
      Well.. maybe. Or Maybe not. But Definitely not sort of.
    4. Re:"It's Not a Tumor" - Oh Wait, It Is by Anonymous Coward · · Score: 5, Informative

      I don't get what you're saying, and I think that's probably because you don't know what you're talking about. Having certificate chains is only a plus, the flat structure was crap. Here's how it works:

      I have a root certificate that's universally trusted. It is used *only* to sign intermediate certificates. Having the public cert in the public is fine since it only contains the public key part of the asymmetric public/private key pair. The private key, sits on a server which is physically isolated from the world. By that, I mean that the root certificate and private keys are literally on servers with *no* network connections. When you want to generate a new intermediate certificate, you put the CSR on a USB stick, plug it in, and sign it from that machine. In this way you never have to worry about external threats potentially gaining access to your private key (internal are an ever constant threat, and you put in good safeguards to prevent against that).

      Now that you have a chained hierarchy, you can use different intermediates to sign different end user certificates. Remember that both the root and intermediate have their own certificate revocation lists: the root can revoke intermediates (which means anything signed by them is null and void) and intermediates can revoke server or subordinate intermediate certs.

      As a result of my chained hierarchy, if an intermediate is compromised, I can revoke it, without revoking every single end user / server certificate out there. This gives me finer grained control.

      Now, I said that the root isn't even connected to the internet. The intermediate ideally is not either. Ideally the user / server signing intermediate is behind a set of firewalls, and *pulls* signing requests from frontends, that it then processes and posts the resulting signed certs to. That way if you compromise the front end, you have to the compromise the firewall (which ideally is blocking all inbound connection requests / unconnected sockets / anything that is not communication required for the intermediate server(s) to send pulls to the frontends) in order to get to the intermediate.

      Your flat view of the world is draconian, wrong, uneducated, and probably hurts everyone who reads it by making them a little less educated.

    5. Re:"It's Not a Tumor" - Oh Wait, It Is by Krojack · · Score: 3, Interesting

      As we all know, most high level hacks these days come from an internal computer getting infected with something.

    6. Re:"It's Not a Tumor" - Oh Wait, It Is by Anonymous Coward · · Score: 1

      Exactly. Even air gapping is not a 100% effective defense.

    7. Re:"It's Not a Tumor" - Oh Wait, It Is by Anonymous Coward · · Score: 1

      Root and intermediate certificates are not used to encrypt data. They're used to sign other certificates. If/When a root or intermediate is compromised, only trust is compromised (I can get my own personal microsoft.com cert 'signed' by a trusted CA). The real microsoft.com's encryption is not compromised, as the data is encrypted with their private key.
       
      SSL is horribly broken, but not in the manner you indicate.

    8. Re:"It's Not a Tumor" - Oh Wait, It Is by lgw · · Score: 1, Insightful

      put the CSR on a USB stick, plug it in, and sign it from that machine. In this way you never have to worry about external threats potentially gaining access to your private key

      Ever heard of Stuxnet? Connecting via sneakernet is much like connecting via a normal network: you need the machine itself to be secure. Sure, limiting connectivity limits attack vectors, and that helps. But it won't help against an attacker who understands your architecture, and targets you specifically.

      You're never invincible. Don't get cocky, kid.

      --
      Socialism: a lie told by totalitarians and believed by fools.
    9. Re:"It's Not a Tumor" - Oh Wait, It Is by DarkOx · · Score: 4, Informative

      Far more important is that if the intermediate certificate is compromised, you as the CA have ability to act. You know from your records who your customers are. What you need to do is:

      1) Fix the glitch
      2) Get the media that stores the trusted root certs private key out of the vault
      3) Issue new intermediate certificates
      4) return the root certs private key to vault
      5) Start contacting your certificate customers and issuing them new certs, revoking the old ones along the way as customers reported they have switched, or if there *is* indication of a compromised cert, revoke immediately.
      6). Revoke the old intermediate certificates as soon as 5 is complete.

      If were signing client certificates directly with the trusted root like they once did you (as the CA) would be screwed royally. You would need to somehow get every client device to update their trusted roots. Or you'd have upset customers crying about how their reissued certs are untrusted by 3/4ths of the clients out there that nobody bothers to update and nobody who understands these things manages directly..

      --
      Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
    10. Re:"It's Not a Tumor" - Oh Wait, It Is by Lanboy · · Score: 1

      The private keys for the CA provider roots are not exposed to any internet connectivity.

      I know for Verisign, for instance they need to get 5 or 6 people with access to physically be at the servers in question to issue certificates that require manual configuration.

    11. Re:"It's Not a Tumor" - Oh Wait, It Is by Anonymous Coward · · Score: 0

      No.

      By design the private keys at such providers are extremely heavily guarded.

      They are not publically accessible, and internally likely via a private API not HTTPS so without the Heartbeat SSL feature.

      At no point will the private key therefore be in memory of a server that can be exploited by an attacker. Without that key the certificate is secure.

    12. Re:"It's Not a Tumor" - Oh Wait, It Is by Anonymous Coward · · Score: 0

      CA private keys should not be stored in computers at all. They should be stored in hardware security modules. If the HSM fails, fine, you can't issue any more certificates with that CA, generate a new one.

    13. Re:"It's Not a Tumor" - Oh Wait, It Is by mysidia · · Score: 1

      That intermediate signing key should be treated with the same level of security you would treat a root key with.

      No... the Root key should be designed with physical security safeguards and an airgap requiring multiple authorized humans to conduct each signing.

      The intermediates should be used for automated signing needs. When an end user requests a certificate, they require a prompt turnaround.

    14. Re:"It's Not a Tumor" - Oh Wait, It Is by mysidia · · Score: 1

      6). Revoke the old intermediate certificates as soon as 5 is complete.

      I believe the CA/Browser forum and Mozilla program rules require you to do this immediately. You don't get to wait until you've reissued all customer certs; time is of the essence, with regards to the revocation, and failing to do it promptly can get BOTH your intermediate AND your Root permanently removed from the browser trust stores..

    15. Re:"It's Not a Tumor" - Oh Wait, It Is by putaro · · Score: 1

      It depends on where you are in the chain.

      If you're a CA, then yes, the intermediate key would be used for automated signing. It STILL shouldn't be on hosts that are directly connected to the Internet.

      If you're a company that is not a CA, then the intermediate key signed by the CA is pretty much your root key. It shouldn't be on your web servers, you should keep it offline if possible and you should be generating another layer of keys that are used to sign actual server certificates.

    16. Re:"It's Not a Tumor" - Oh Wait, It Is by Sabriel · · Score: 1

      I'm surprised you didn't notice the word "external" in the sentence you partially quoted, or that the sentence continued with "(internal are an ever constant threat, and you put in good safeguards to prevent against that)."

    17. Re:"It's Not a Tumor" - Oh Wait, It Is by mysidia · · Score: 1

      If you're a company that is not a CA, then the intermediate key signed by the CA is pretty much your root key.

      If you're a company that is not a CA, then a CA is not supposed to be issuing you a certificate that subordinate certificates can be chained under in any circumstances, in the first place.

      For each certificate you need, you apply to your CA and present the Certificate Signing Request to be validated and completed.

    18. Re:"It's Not a Tumor" - Oh Wait, It Is by putaro · · Score: 1

      How about this https://www.globalsign.com/cer...?

      I haven't tried setting up a large PKI infrastructure so I'm curious if you know more. Technically it's possible but I could see why a CA wouldn't do it. The info for this GlobalSign "Trusted Root" seems to imply that you get to sign keys with your own existing root CA but that GlobalSign will sign it as well so you don't need to distribute your own root cert. Am I reading it wrong?

    19. Re:"It's Not a Tumor" - Oh Wait, It Is by mysidia · · Score: 1

      Correct. If your organization engages Geotrust with that service, then you can setup a certificate authority within your own organization chained to their certificate.

      However, you have to follow rules that are even more restrictive than what that a CA has to follow with their root certificate, and you have to be audited like a CA.

      This is very expensive, and it is not immediately clear: what organizations would be willing to go through the tremendous expenses, and not take the additional few steps to get on the browser trust lists.

      It is certainly not something you will see Mom and Pop firms doing. Perhaps some companies in the top 50 of the Fortune 500.

  2. When comments... by darkain · · Score: 2

    When Slashdot comments become full front page stories? This was already posted a few times as comments in the last OpenSSL post.

    1. Re:When comments... by Anonymous Coward · · Score: 5, Insightful

      Theo De Raadt is a noteworthy personality, same as Linus Torvalds or Bill Gates. Their comments on important matters are important.

    2. Re:When comments... by Anon,+Not+Coward+D · · Score: 1

      Come on... sometimes a comment is so informative that everyone should read it :)

      --
      Sometimes it's better not having signature
    3. Re:When comments... by MightyMartian · · Score: 5, Insightful

      As much as Theo can be an utter and insufferable prick, on this score he's right. This was an insanely trivial error which has exposed who knows how many systems to potential breaches. Right now I'm starting up a full audit of our systems. We use OpenVPN for our interoffice WAN, as well as for clients; many of them Windows, iOS and Android clients, not to mention reviewing all our *nix clients running SSH daemons. We're only a relatively small operation, and it's still a monumental pain in the ass.

      --
      The world's burning. Moped Jesus spotted on I50. Details at 11.
    4. Re:When comments... by Anonymous Coward · · Score: 0

      When Slashdot comments become full front page stories? This was already posted a few times as comments in the last OpenSSL post.

      When the pool of quality submissions is very shallow, like it has been here for quite some time.

    5. Re:When comments... by Ukab+the+Great · · Score: 1

      Theo De Raadt makes comments about Linux being for losers. Linus Torvalds makes comments about OpenBSD users being masturbating monkeys. You usually have to take some of their comments with a grain of salt.

    6. Re:When comments... by Bill,+Shooter+of+Bul · · Score: 1

      Because its Theo. HIs rants are usually pretty funny and thought provoking.

      This one, I have to admit, is pretty tame/lame. Good security design thoughts as always, but not much of a rant.

      --
      Well.. maybe. Or Maybe not. But Definitely not sort of.
    7. Re:When comments... by Anonymous Coward · · Score: 4, Interesting

      As much as Theo can be an utter and insufferable prick, on this score he's right.

      Actually, most of the time he's right; it's just the prick bit that is the problem. What's worrying about this is that he sounds completely reasonable. He didn't even call the Open SSL developers any names or anything and everything in the post was reasonable. I hope he isn't sick or something.

    8. Re:When comments... by MightyMartian · · Score: 3, Funny

      Don't worry. I've more than made up for Theo's unusual calmness this morning, and have called them names that would make the gods blanch.

      --
      The world's burning. Moped Jesus spotted on I50. Details at 11.
    9. Re:When comments... by Anonymous Coward · · Score: 1

      OpenSSH is fine. Unless you've exposed login credentials for those systems through something using TLS (via OpenSSL), there's no need to worry about SSH keys. OpenVPN, on the other hand, yes, does use OpenSSL for TLS, so you should likely revoke and reissue all certificates and change all credentials (unless your instances of OpenVPN servers were linked to OpenSSL <1.0.1).

    10. Re:When comments... by LWATCDR · · Score: 0

      It also kind of weakens the argument that OpenSource is more secure because of the idea of many eyes. I guess no one started to look for about two years.

      --
      See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
    11. Re:When comments... by lgw · · Score: 2

      He didn't even call the Open SSL developers any names or anything and everything in the post was reasonable. I hope he isn't sick or something.

      "OpenSSL is not developed by a responsible team."

      Theo the Raanter is fine. Personal abuse and profanity is just noise. This was a truly cutting comment.

      --
      Socialism: a lie told by totalitarians and believed by fools.
    12. Re:When comments... by Anonymous Coward · · Score: 0

      As much as Theo can be an utter and insufferable prick.

      He can't hold a candle to you, buddy. On the contrary, I've met Theo in person. He's quite a decent, mature fellow and no where near the juvenile asshole that Linus is.

    13. Re:When comments... by MightyMartian · · Score: 1

      All my servers are sitting at 0.9.8, but at least some of the Windows clients are not. Do I need to worry about?

      --
      The world's burning. Moped Jesus spotted on I50. Details at 11.
    14. Re:When comments... by Anonymous Coward · · Score: 0

      You don't need to worry about SSHD. It does not use TLS or heartbeats. Goddamn, read the fucking exploit before you blow your top.

    15. Re:When comments... by fahrbot-bot · · Score: 5, Funny

      Theo De Raadt makes comments about Linux being for losers. Linus Torvalds makes comments about OpenBSD users being masturbating monkeys. You usually have to take some of their comments with a grain of salt.

      He said, prophesying a future Ubuntu release name.

      --
      It must have been something you assimilated. . . .
    16. Re:When comments... by Mysticalfruit · · Score: 1

      Honestly, his insufferable personality is tolerable because on most accounts he's right. Being insufferable and consistently wrong is truly obnoxious.

      --
      Yes Francis, the world has gone crazy.
    17. Re:When comments... by Darinbob · · Score: 1

      I think the other thing he pointed out is very interesting too, that the alternative code when you disable the optimization is buggy because no one's tested it in ages. This sort of thing happens a lot.

      There are #ifdefs for valid build time alternatives (very common in third party embedded libraries, so you can turn off features you don't need). But then there are the #ifdefs that seem to be for experimental code or maybe controversial code (you tell the boss "don't worry, I'll have an ifdef so we can revert it quickly if it doesn't work out"). That's the sort of thing that ends up staying in the code for years. Almost always these experimental features aren't easily configurable, which is why they're named something like "#ifndef DISABLE_FEATURE_X" (double negative) so that they don't have to document the option or even define it in a header file where others might learn about it. So even if the QA team is smart and on the ball and is doing regression tests on all possible build configurations, they'll miss this one because it's a stealth option.

    18. Re:When comments... by psyclone · · Score: 1

      Cutting through to the truth? Seems like OpenSSL's primary technical goal should be security, secondly performance.

    19. Re:When comments... by lgw · · Score: 2

      Indeed. The harshest insults don't need any emotion or colorful language, they're simply, obviously, true.

      --
      Socialism: a lie told by totalitarians and believed by fools.
    20. Re:When comments... by Anonymous Coward · · Score: 0

      You joke, but I kept reading the release name for version 11.10 as "onanistic ocelot".

    21. Re:When comments... by rvw · · Score: 1

      OpenSSH is fine. Unless you've exposed login credentials for those systems through something using TLS (via OpenSSL), there's no need to worry about SSH keys.

      If openssh keys were at some point in RAM, they could be out in the open. Small chance, but do you want to take it? Renew your openssh keys and passwords.

    22. Re:When comments... by rvw · · Score: 1

      It also kind of weakens the argument that OpenSource is more secure because of the idea of many eyes. I guess no one started to look for about two years.

      It's not more secure. If a flaw is found - it's much faster fixed, and with a flaw like this you can bet many people are going to check it. If this was MS, we would have to trust them, and we would have to wait at least a week for a fix.

    23. Re:When comments... by Anonymous Coward · · Score: 0

      Oh, it's funny because it's true

    24. Re:When comments... by Anonymous Coward · · Score: 0

      > It also kind of weakens the argument that OpenSource is more secure because of the idea of many eyes.

      If you can't see the flaw it's a preferable technology choice? It's obvious you misunderstand what security means.

    25. Re:When comments... by Anonymous Coward · · Score: 0

      Usually, but not always. This time he's right (and I'm no great fan of the guy).

      Heartbleed is the result of a rookie coding mistake. No problem, we all make them occasionally (I know I do). But OpenSSL uses custom allocators that actively defeat attempts to detect those kinds of mistake. That's what he's ranting about, and he's absolutely correct. The bug that caused Heartbleed was a mistake, but those allocators were deliberate (and misguided, although probably not malicious).

  3. 5 second read? by Anonymous Coward · · Score: 1

    How the hell can you read 298 words in 5 seconds?

    1. Re:5 second read? by ObsessiveMathsFreak · · Score: 5, Funny

      How the hell can you read 298 words in 5 seconds?

      The trick is not to read TFA in the first place. Also, you must be new here.

      --
      May the Maths Be with you!
    2. Re:5 second read? by gweihir · · Score: 1

      You must be new to this "Internet" thing: It is easy! Just do not try to understand anything!

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    3. Re:5 second read? by gweihir · · Score: 1

      Ah, sorry. My comment is redundant, did not see yours.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    4. Re:5 second read? by Anonymous Coward · · Score: 0

      I don't see anything funny about this. The articles are not what slashdot is about.

    5. Re:5 second read? by unixisc · · Score: 1

      Playboy is where you go if you want to read articles ;-)

  4. Summary for the lazy ones by Anonymous Coward · · Score: 5, Informative

    Years ago the BSD guys added safeguards to malloc and mmap, but they were disabled for all platforms in OpenSSL only because they caused performance problems on some platforms. He finishes by saying that OpenSSL is not developed by a responsible team.

    1. Re:Summary for the lazy ones by Anonymous Coward · · Score: 0

      Years ago the BSD guys added safeguards to malloc and mmap, but they were disabled for all platforms in OpenSSL only because they caused performance problems on some platforms. He finishes by saying that OpenSSL is not developed by a responsible team.

      And yet, OpenSSL is included in OpenBSD...

  5. More commentary from OpenBSD's Ted Unangst by badger.foo · · Score: 5, Informative
    OpenBSD developer Ted Unangst (mentioned in the article) has gone into the code a bit more in two articles, both very well worth reading:

    heartbleed vs malloc.conf

    and

    analysis of openssl freelist reuse. Short articles with a lot of good information.

    --
    -- That grumpy BSD guy - http://bsdly.blogspot.com/
  6. Summary. by jythie · · Score: 4, Insightful

    So as far as I can tell, his rant is essentially that people should not use custom allocators and instead rely on the general purpose one built into libc because they can add system wide tools there.

    I can see the argument for most cases, that is kinda the point of a general purpose allocator, but encryption (esp if you are doing lots of it) really strikes me as a case where you can really benefit from having explicit control over the behavior. I have worked on a number of applications where custom allocators had significant (user facing, not just benchmarks) impacts on performance. Ironically it also meant we were able to do better checking then the general exploit detection kits since we could bake more specific knowledge into the validator.

    1. Re:Summary. by mr_mischief · · Score: 5, Insightful

      That's all true and correct. When you do that, though, you need to do at least as good a job as what you're circumventing. In this case OpenSSL didn't.

    2. Re:Summary. by bluefoxlucid · · Score: 0, Troll

      I can see his point; however he's still wrong. Mostly because he's Theo, though I'll credit he seems to have quietly accepted some of the things he was wrong about in the past.

      So then a bug shows up which leaks the content of memory mishandled by that layer. If the memoory had been properly returned via free, it would likely have been handed to munmap, and triggered a daemon crash instead of leaking your keys.

      And then the five servers in the world running OpenBSD would be safe.

      The moment the bug was discovered, we were all fucked. The bug would have gone out readily. This particular abuse wasn't tested, obviously, or someone would have gone, "Wow look at that! It gave back a chunk of data LOL! Let's fix that before relase!" So if they would have satisfied Theo, OpenSSL still wouldn't have been tested and wouldn't have crashed, and would be released vulnerable.

      That release gets on every SSL-enabled server, including into products like SonicWall VPN and firewalls. These are vulnerable--I've tested them here, the exploit actually works. FortiNet's products--FortiMail, FortiGate--these are vulnerable. CentOS 6 is vulnerable. Debian is vulnerable. Ubuntu since forever is vulnerable. Fedora is vulnerable. SuSE is vulnerable. A bunch of shit on Windows is vulnerable.

      Then we realize there's a bug.

      Let's talk about someone else who's wrong: the OpenSSL team. The OpenSSL developers are pissed off because of the full disclosure practice that put this bug out there before a patch was released. They think responsible disclosure would be better. So you disclose responsibly, and the patches come out. Some hackers see new OpenSSL release, and look into it. They see malloc() fixes in networking code, or you put up a big "THIS IS A SECURITY PATCH" notice.

      Start your engines, motherfuckers.

      Distros start rebuilding immediately because there's an automatic trigger, and because they know already: there's a huge network nobody knows about made up of distro maintainers and upstream programmers all sharing security and bugfix information. Ubuntu's devs were ready for this shit before it was released, and probably would have had packages built before the source went out. Debian, RedHat, SuSE.

      Twelve hours later, sysadmins start noticing.

      By then, someone has reverse-engineered the patches. The moment packages start propagating, the source files are up--SRPMs are out. By the time you can see and react, someone sees "SECURITY FIX", reads the diff, the best ones can do an exploit in half an hour for something like this. You might be hacked before you notice the patch.

      So it would be better with responsible disclosure. The problem is this isn't a hack you'd see: you don't know if you're hacked. You wake up and your security is now handled by Schrodinger's Cat: your SSL keys are secure or not secure, and you don't know until you shake down the blackhats who have your keys, and maybe they bury the information when you do that so you still don't know. Not only that, but maybe blackhats secretly had the hack before it was noticed by a security researcher--that's rather common.

      The moment this thing went out, it was too late. Your security is broken. And most people aren't on OpenBSD, so they'd get hacked even with OpenSSL using the system malloc(). They'd get hacked even with OpenBSD's protections, because OpenSSL is running almost always on Linux and OpenBSD's protections aren't on Linux. They'd get hacked anyway. That means everyone.

      So no, Theo. You're not helping. You didn't discover the keystone that would save us all. Maybe god damn EVERYONE should implement these protections everywhere; glibc has some, but long reads won't crash glibc unless you hit unmapped.

      Congrats Theo, your reputation continues.

    3. Re:Summary. by jythie · · Score: 2, Insightful

      True, they did not, but I would put that at the level of mistake rather then being unreasonable.

    4. Re:Summary. by akpoff · · Score: 5, Insightful

      Theo's point isn't that OpenBSD users would have been safe. It's that had OpenSSL crashed on OpenBSD (or any OS with similar mitigation in place) it would have surfaced the bug much sooner...perhaps before a worldwide release. Once found it would have been fixed and merged upstream to benefit all users.

      This is really a specific case of the larger point behind avoiding monoculture, whether OS or hardware. OpenBSD continues to support older architectures in part because it forces them to work through issues that benefit other platforms, especially the one most of us use daily: x86.

    5. Re:Summary. by bluefoxlucid · · Score: 1, Troll

      The problem is it would have crashed on OpenBSD if someone would have tested this exploit on OpenBSD, meaning someone would have to be looking for the exploit, meaning someone would have found a bunch of data coming back anyway and gone "oh lol wtf?".

      If someone crashed your server trying to exploit it, you would probably not notice; since there aren't many OpenBSD servers, probably nobody would notice that these attacks were happening and gone, "Whoa! A wild 0-day exploit!" And even if they had, there's all these non-OpenBSD servers that are getting hacked and nobody can say if they're hacked or not, so we just get ourselves into this exact situation sooner. We don't come away with smaller collateral damage; EVERY SSL CERTIFICATE EVER ISSUED IS NOW INVALID.

      Nothing Theo suggested changes the situation. Implementing malloc() protection everywhere might; but if you can show any ability to beat that protection a percentage of the time, then we're also in the same situation. We're talking about reads, so canaries aren't it. If you're crashing out on reads, then every malloc(1) that crashes if you read 2 requires 4096 bytes of real RAM to store 1 byte of data--we get into costs.

    6. Re:Summary. by Anonymous Coward · · Score: 1

      Furthermore, as Ted explains on his blog (already linked to in the comments), in this particular case OpenSSL would have aborted the connection on any system with a "standard" malloc - if only it were possible to reliably use that malloc, that is.

    7. Re:Summary. by BasilBrush · · Score: 1

      Distros start rebuilding immediately because there's an automatic trigger, and because they know already: there's a huge network nobody knows about made up of distro maintainers and upstream programmers all sharing security and bugfix information.

      "A huge network nobody knows about". The open source concept is fundamentally fucked, isn't it. Security issues happen with Windows or OS X, and it's clear - you're vulnerable, until you get a certain version of the OS, then it's patched.

    8. Re:Summary. by JDG1980 · · Score: 1

      If you're crashing out on reads, then every malloc(1) that crashes if you read 2 requires 4096 bytes of real RAM to store 1 byte of data--we get into costs.

      4096 bytes of RAM as an unacceptable cost? Seriously? Besides, how often are you repeatedly allocating really tiny buffers like this? If you have to do that, then maybe there's a more fundamental problem with the way your code flow is designed.

    9. Re:Summary. by Anonymous Coward · · Score: 5, Insightful

      but if your explicit reason for overriding the kernel is to do something as-good-or-better than the kernel, and you fail to achieve that (let alone validate that you've achieved it), that's a rather spectacular failure. mistakes are common, sure. but this is sort of like saying the seatbelts that came with your car suck (they very well might), so you replace them with something "better", that ends up being little more than a piece of string that you daintily lay across your lap. kind of "unreasonable", no...?

    10. Re:Summary. by KingOfBLASH · · Score: 1

      I thought Theo's comments were more geared to the point that malloc() was implemented to be a sort of seat belt. In the event of a crash, if you're wearing your seat belt, you might still get killed, but you have a better chance of living. Same thing with malloc(). Sure maybe it wouldn't have helped, maybe even if there were good regression tests out there someone would also have missed it, but we'll never know if someone would have caught it, and in something that is definitely going to be a target for the black hat crowd, you should have some sort of security mindedness.

      Theo is normally a bit of a wonk sometimes but on this issue, he's spot on.

    11. Re:Summary. by Kremmy · · Score: 1

      The point is that they found OpenSSL to break when compiled without the malloc wrapper. OpenSSL relies on incorrect behavior in a malloc wrapper. Relies on it. Doesn't work without it. The result is the heartbleed bug. The result is that OpenSSL must be audited from the ground up and fixed so it doesn't require a broken memory allocation routine. Theo is still right.

    12. Re:Summary. by mrchaotica · · Score: 1

      I think this Ted fellow linked from the summary makes a good case against it being "reasonable" at the end of his second post:

      This bug would have been utterly trivial to detect when introduced had the OpenSSL developers bothered testing with a normal malloc (not even a security focused malloc, just one that frees memory every now and again). Instead, it lay dormant for years until I went looking for a way to disable their Heartbleed accelerating custom allocator.

      Building exploit mitigations isn't easy. It's difficult because the attackers are relentlessly clever. And it's aggravating because there's so much shitty software that doesn't run properly even when it's not under attack, meaning that many mitigations cannot be fully enabled. But it's absolutely infuriating when developers of security sensitive software are actively thwarting those efforts by using the world's most exploitable allocation policy and then not even testing that one can disable it.

      --

      "[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz

    13. Re:Summary. by jythie · · Score: 1

      Keep in mind, we are talking about overriding libc, not the kernel. For the most part it does do things better then the default allocator. If you want to use the car analogy, this would be like replacing your standard seat-belts with a 5 point cross belt but a manufacturing defect slipped through on one of the rollers. The defect is not a little deal, but it does not mean the 5 point belt is inherently worse then the standard on, there was simply a defect that passed through quality control, which happens.

    14. Re:Summary. by bluefoxlucid · · Score: 1

      If you make 1000 allocations, that's 4MB.

      The allocator, as a single segment, already has a 50% or higher overhead. We're talking about changes that could turn into hundreds of megabytes of waste. There are literally hundreds of thousands of allocations actively in use in some workloads such as Web browsers or Word processors. Think about every pointer being 4KB at the very least--an array of 10 things of 100 bytes each is 1KB, but as a linked list it's 40KB. Expand 1KB = 40KB to the 1800MB RAM usage crap of Web browsers and OpenOffice.org and modern desktop environments.

    15. Re:Summary. by bluefoxlucid · · Score: 2

      How is he spot on?

      A specific security feature of malloc() available as an optional security enhancement on OpenBSD has the ability to crash the program in the event of an exploit of this particular bug.

      This feature isn't available on most of the installations of software using SSL, and wouldn't be triggered by OpenSSL unless it was exploited.

      Therefor, if OpenSSL had correctly used malloc() to take advantage of this feature, the situation would be--in reality--no different, except we could say that theoretically the attacks on servers running OpenBSD wouldn't work.

      The reason nobody would have caught it is simple: you'd have to intentionally fuzz the TLS heartbeat feature looking for exploits. You'd only do that when searching for exploits, you'd be analyzing responses, and you'd catch it anyway. It almost certainly would have been caught exactly as fast; and even if it was caught sooner, the impact would be "oh shit everybody in the world change your SSL keys"--i.e. the current situation.

    16. Re:Summary. by bluefoxlucid · · Score: 1

      Uh, no. The heartbleed bug isn't a result of OpenSSL breakage in the way it allocates memory; it's actually a result of incorrectly handling memory allocated properly. That OpenSSL can't work when compiled against normal malloc() implies other issues unrelated to this one.

      If OpenSSL had been compiled to properly work with glibc malloc() or freebsd malloc(), this bug would still stand.

    17. Re:Summary. by Anonymous Coward · · Score: 0

      Actually, if I read the analysis correctly, the exploit was irrelevant. Had they tested OpenSSL on OpenBSD, the crash would have happened and someone would have looked into the cause. Since OpenBSD is not a total niche distro, I have to assume that they do at least some basic testing on that platform, but even if they didn't the bug would have surfaced the moment users upgraded to the new buggy code.

      This isn't an OpenBSD issue, it's a poor architechtural choice issue in OpenSSL - somewhere, someone chose to do things in a manner that was less secure because the more secure method was slower, and then it became the default because switching to the higher security model was too hard even if the speed wasn't an issue anymore. Knowing that this happened, I would hope that the OpenSSL maintainers would turn off that compiler flag and start testing the hell out of it to find and fix whatever problems exist in the code that are being masked by their internal malloc/free system. Even if they leave the flag on for "production", bugs that aren't readily apparent at runtime are still bugs...

    18. Re:Summary. by bluefoxlucid · · Score: 1

      In the Open Source concept, when any random joe sees a security bug, they file it with upstream quietly. It gets repeated behind closed doors to all the distro maintainers, and then they all have packages ready to go when the patch is published.

      In the closed model, it takes binary analysis or other stuff to see it. There are far fewer people just stumbling around in the code going, "Whoa, this looks really weird." A lot of security holes are found and fixed by programmers who were doing something in that block and realized what they were looking at was buggy to horrendous results.

    19. Re:Summary. by Kremmy · · Score: 1

      It implies that the already uncomfortably deep issue is even deeper than it appears. It's still directly related to memory management issues. I'm not comfortable shrugging this off as unrelated, because industry standard security related software is Not handling memory correctly.

    20. Re:Summary. by davecb · · Score: 1

      If you're doing a workaround, you need to have a regression test for the thing it worked around, so when the bug get's fixed your test fails, someone reads the description and you can turn off the workaround for that platform. That's pretty-ordinary practice from an anal QA person. And yes, I do TDD and still have an QA person siting across the aisle from me.

      --
      davecb@spamcop.net
    21. Re:Summary. by akpoff · · Score: 1

      Read Ted Unangst's analysis. You don't have to do anything special to trigger the bug when using a normal malloc rather than OpenSSL's broken approach. OpenBSD's approach protects you more but any malloc would have surfaced the error:

      This bug would have been utterly trivial to detect when introduced had the OpenSSL developers bothered testing with a normal malloc (not even a security focused malloc, just one that frees memory every now and again).

      The problem is the OpenSSL code was freeing a buffer and then immediately re-allocating it to read data from. The OpenSSL team got lucky and it worked when the buffer was the right size.

    22. Re:Summary. by whois · · Score: 1

      I would like to see performance numbers on who's malloc is terrible anyway. This was true in ancient times on some systems, to the point where people specifically did what they're doing, but I haven't profiled it in forever and I always assumed it had gone away.

      I'm wondering if they cargo-culted their allocator, or if their software just happens to be old enough to fall in that range of things that ran on junky hardware from long ago. It's one of those things where if you're going to leave it in then it should be extensively documented. Not "some systems are slow" but "these exact systems have problems"

    23. Re:Summary. by bluefoxlucid · · Score: 1

      The actual heartbleed bug is an issue of reading. It's validation: it allocates 10 bytes because the payload is that big; then it reads from that buffer 10000 bytes because the payload says it's that big. It's not a double-free or an out-of-bounds write, both of which we can readily detect.

      To detect out-of-bounds read, you have to allocate in 4k blocks with malloc(), then right-align so that the buffer ends with the block, and keep the block after that unallocated. If some monkey mmap()s the following block that breaks this scheme, but that's extremely unlikely. So in normal operation, you read past the end and there is no physical RAM for that address and the system faults.

      This scheme is highly expensive. Every allocation is rounded up to 4K of memory usage. That means the millions of tiny things allocated are multiplied by their average proportion to 4k. Allocated a binary tree of hundreds of thousands of vectors in a 3D rendering engine? Congrats: now all of these 80 byte elements each require 4096 bytes, about 500x as much RAM. Because of this kind of inflation (which on average isn't 500x), most allocators don't do this. I would not be surprised to see most workloads using only 10%-30% more RAM under this scheme, but a 10% increase is considered insane and a 1% decrease is considered an excellent and important optimization to most allocator designers.

      The point is that this code would feed you broken output if you ever fed it broken user input, and it would feed you correct output and have no behavioral flaws if you just gave it correct input. The protections above would trigger if you fed this code broken input; but if you tested for that, you'd already be looking for this kind of bounds issue, and you wouldn't need a system crash to tell you that you just sent a 4 byte payload and got a 10 byte payload back.

      This bug was missed because it works normally when not fed broken data, and nobody thought to test it with broken data.

    24. Re:Summary. by neaorin · · Score: 1

      Calling malloc() and getting back the exact same contents as the previously freed buffer is not properly allocating memory. Why not? I mean the memory was "free" at that point right? Well, it's because you come to rely on that assumption. Assuming you got the same content back is also not properly handling said memory. Notice how one bad behavior encourages another? This particular bug may still stand, but how many others may still be out there because of the above assumption?

    25. Re:Summary. by bluefoxlucid · · Score: 1

      Try this analysis:

      Remember, payload is controlled by the attacker, and it's quite large at 64KB. If the actual HeartbeatMessage sent by the attacker only has a payload of, say, one byte, and its payload_length is a lie, then the above memcpy() will read beyond the end of the received HeartbeatMessage and start reading from the victim process's memory.

      It's extremely predictable behavior: send a 1 byte payload, tell the server it's 64k, server allocates 1 byte and then copies 64k starting at the address of that 1 byte into a (separately allocated) 64KB buffer that's sent to the socket. The idea that using regular malloc() would detect non-malloc() things without doing fancy things to make non-malloc() things break is laughable. With canaries, malloc() can detect that you wrote off the edge when you free() it; but it can't detect that you read off the edge, unless it mmap()s anonymous memory boundaried by an unmapped page and right-aligns your allocation.

      The buddy allocator in glibc has canaries (I think) and double-free() detection. It doesn't detect read-past-buffer. OpenBSD's probably does, at least I assume that's what they're saying.

    26. Re:Summary. by bluefoxlucid · · Score: 2

      The heartbleed bug works by the user sending a payload which OpenSSL analyzes and determines to be of a length, like strlen(). Then it calls malloc() to allocate a buffer for that length, and copies that length in. 1 byte payload, malloc(1). Then it looks at the heartbeat handshake, which says, "Oh yeah, my payload is 64KB". Then it allocates a 64KB buffer and does a memcpy() of 64KB from that 1 byte buffer into the 64KB buffer.

      Do you see the problem here? The server looks at the actual data and malloc()s something the size of the data. The client says, "The data is this big," and the server copies something the size the client dictates from where it allocated the data. Problem: what if the data is actually smaller?

      Also, for good measure: The entire bug occurs between an allocation and a free. There actually isn't a free/realloc bug involved. Here, I'll show you.

      This is the vulnerable code. Start on line 3963.

      Line 3974 starts the problem. Variable (p) is an array of bytes; the first 2 bytes are an unsigned int. n2s(p, payload) is essentially (payload = (unsigned int)(*p);). The (payload) value is an unsigned int carrying the length of the payload.

      Line 3991 is your buffer allocation. And it looks like I'm wrong: it allocates a short buffer, which it writes outside of. This was not mentioned on the analysis I read. So a canary should catch this on free() when actively exploited, but not in non-exploit operation.

      Line 3997 is the good part: it copies however many bytes you asked for from the SSL Heartbeat request into the payload buffer. That could be 64kb of shit, and the payload is 1 byte long.

      Line 4002 sends the copied bytes back to the client. This is where the exploit succeeds.

      Line 4009 is where the free() happens. If your glibc has canaries, it crashes here on exploit; this call works normally if you're not being exploited. It also works normally if your payload is short (although other errors may occur elsewhere).

      Even so, it crashes IF AND ONLY IF it's being exploited, and only AFTER exploit returns data, and ONLY on write-bounds-checking allocators that look for a dead canary (which you'll actually know after the first exploit--on a forking process like Apache--and can supply with further exploits, so you can rapidly mine with one crash). If you're not testing it by asking for a bigger payload than you supply, you won't ever get this crash.

      This isn't a double-free() issue. It's a bounds check issue.

    27. Re:Summary. by udippel · · Score: 1

      (Having no mods points as of now)
      Your first post is currently 'Troll' (OMG, where have all the competent mods gone!!); and it isn't, this one rather is. Or flamebait. or ...
      You know (and explain) the shit that is currently happening. Good. In your first post. Plus a bit of rant, and that's what we expect in here!
      The parent, however, is way out of reason. We know what is going on, we know that a proper malloc/free would
        - protect the owner from becoming a pwner
        - reasonably likely result in a crash

      Now, what's your fight with Theo based on the fact that Theo has no actual solution to this s**t (I mean, we all agree, there isn't); and then you conclude trying to rubbish a perceived wastage of RAM in exchange for security.

    28. Re:Summary. by EvanED · · Score: 1

      Nothing Theo suggested changes the situation. Implementing malloc() protection everywhere might; but if you can show any ability to beat that protection a percentage of the time, then we're also in the same situation. We're talking about reads, so canaries aren't it. If you're crashing out on reads, then every malloc(1) that crashes if you read 2 requires 4096 bytes of real RAM to store 1 byte of data--we get into costs.

      You lack imagination. The choice isn't between "traditional dense allocation" and "one page per allocation." I don't know what OpenBSD's allocate does, but it would be entirely possible to make an allocator that is comparably dense to a traditional one but still uses discontinuous pages. Such an allocator would still dramatically limit the amount of data that could be read on a system with such an allocator, because you would be unlikely to be able to overread past the end of the page. You could still get some data, but there'd be a "hard" limit of under 4K (16x less that what's possible) and even under that limit the more you request the more you run the risk of running into the neighboring unmapped page.

      ... if you can show any ability to beat that protection a percentage of the time, then we're also in the same situation

      I also reject this argument. If that were true, we might as well give up on security. Probably nothing is ever going to be 100% perfect (at least before the singularity). Using a hardened allocator, in many cases, will make attacks more difficult in the sense that you'll be relying on luck in terms of not overrunning the page. (Sometimes you'll hit something that can be done determistically, but I suspect this is not the general case except perhaps where you can leak the seed used for randomization.) Harder attacks means they'll fail more often, and each failure gives a chance of detection and discovery.

      We don't come away with smaller collateral damage; EVERY SSL CERTIFICATE EVER ISSUED IS NOW INVALID.

      I conclude from this you think no SSL certificates were issued between, say, in 2013 or 2014. I'm curious what evidence you have for this. *eyeroll*

      You're also ignoring software versions. Suppose that this had been discovered 6mths after the original version was released. Anyone who didn't upgrade during that 6 mths would have been safe. If the discovery of the bug was delayed by a significant margin, that means that many more people became vulnerable.

    29. Re:Summary. by neaorin · · Score: 1

      Thanks for going through the trouble of explaining the bug but as I already stated I know that it is unrelated to the way they manage memory in openssl. I was replying to your assertion that openssl was allocating memory properly, but mishandling it in this case. They are NOT allocating it properly. They may decide to borrow some of OpenBSD's stuff - it is a security library, after all - or they may not. But what they are doing currently with respect to memory management is not a-ok.

    30. Re:Summary. by EvanED · · Score: 1

      Had they tested OpenSSL on OpenBSD, the crash would have happened and someone would have looked into the cause.

      No, for two reasons:

      1) Theo's main point, about malloc, is that if you disable their custom allocator on any system (or any system that doesn't return a just-freed block in malloc, anyway) then it won't work -- nothing OpenBSD specific at all.

      2) Even if you could disable the custom allocator, the crash would only happen in the face of the exploit. So if you don't have a test for the exploit then you wouldn't discover the crash during testing anyway.

      The custom allocator matters for two reasons. First, it provides a last layer of defense for OpenBSD users. Second, if the attack was in use in the wild and someone with OpenBSD discovered it, then they could have found it sooner.

      Knowing that this happened, I would hope that the OpenSSL maintainers would turn off that compiler flag...

      It's not a compiler flag, it's code. Also they have some bugs to fix before they can do it. They almost certainly will, and perhaps already have.

    31. Re:Summary. by EvanED · · Score: 1

      Uh, no. The heartbleed bug isn't a result of OpenSSL breakage in the way it allocates memory; it's actually a result of incorrectly handling memory allocated properly. That OpenSSL can't work when compiled against normal malloc() implies other issues unrelated to this one.

      It's not unrelated in the sense that a hardened malloc is specifically designed to help limit damage in the face of vulnerabilities such as the heartbeat one.

    32. Re:Summary. by bluefoxlucid · · Score: 1

      Detecting reads that aren't supposed to happen in some situations is really fucking hard. We only have page-granularity CPU facilities.

    33. Re:Summary. by bluefoxlucid · · Score: 1

      I conclude from this you think no SSL certificates were issued between, say, in 2013 or 2014. I'm curious what evidence you have for this. *eyeroll*

      Why? Last week's live version of OpenSSL, latest in RHEL 6.5, latest in Ubuntu, latest in Fedora, were vulnerable. Anything currently in production on OpenSSL 1.0.1 anywhere is now suspect. That means if it was issued in 1995, has a 25 year expiration life, and is served from a machine using OpenSSL 1.0.1e, it's suspect. If it was issued two weeks ago and slapped on a new RHEL 6.5 box, it's suspect.

    34. Re:Summary. by B'Trey · · Score: 1

      True, they did not, but I would put that at the level of mistake rather then being unreasonable.

      I'm reasonably certain that the OpenSSL team did not do this on purpose. It likely wasn't a sabotage by a malicious developer. I seriously doubt someone paid the team to intentionally install the bug. You're almost certainly right that it was a mistake. But arrogance, ignorance and other weaknesses lead to mistakes which should not be made, and when they do, it's jake to point the finger. Just because it was a mistake doesn't mean it was out of their control.

      --

      "The legitimate powers of government extend only to such acts as are injurious to others." Thomas Jefferson.

    35. Re:Summary. by david_thornley · · Score: 1

      In what way is that different from proprietary software? Look at your Linux kernel version, and you'll see if you're vulnerable or not.

      --
      "When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
    36. Re:Summary. by bluefoxlucid · · Score: 1

      It reads past the end of a buffer. Otherwise, it appears to handle allocations properly in this code. Other code in OpenSSL doesn't handle allocations properly--it's broken elsewhere.

      Limited damage in the case of heartbleed are "you still don't know if your keys have been stolen, BUT it's less likely that they have." Schrodinger's cat with a longer radioactive half life is still unknown if alive or dead.

    37. Re:Summary. by EvanED · · Score: 1

      Yes, that's under reality, where OpenSSL made a bad choice to use an awful custom allocator. But that's not what you were talking about:

      And even if they had, there's all these non-OpenBSD servers that are getting hacked and nobody can say if they're hacked or not, so we just get ourselves into this exact situation sooner. We don't come away with smaller collateral damage; EVERY SSL CERTIFICATE EVER ISSUED IS NOW INVALID.

      Getting into the same situation and discovering it sooner leads to smaller collateral damage! If this problem had been discovered in 2012 instead of now, it means that any certificates issued in 2013 or 2014 (except from versions left vulnerable) would have been fine in that scenario. Currently, now they're suspect.

    38. Re:Summary. by EvanED · · Score: 1

      Limited damage in the case of heartbleed are "you still don't know if your keys have been stolen, BUT it's less likely that they have." Schrodinger's cat with a longer radioactive half life is still unknown if alive or dead.

      So? I'll take a 90% survival cancer over a 10% one.

      Everything in security is about constantly/benefit. If it were absolute, you'd be continuously issuing new keys. "Oops, maybe there's an unknown 0-day and our keys are compromised. Better issue new ones. Oops, maybe there's an unknown 0-day and our keys are compromised. Better issue new ones.Oops, maybe there's an unknown 0-day and our keys are compromised. Better issue new ones."

      The fact that you're not doing that means you have, at least implicitly, evaluated the cost of continually issuing new keys to be too high given the minimal benefit of doing that. If you choose to reissue new keys in the face of the heartbeat vulnerability you have, at least implicitly, evaluated the benefit of issuing new keys as being worth the cost.

      Lowering the risk of compromise because "we run OpenBSD and haven't seen any unexplained OpenSSL crashes (that would be possible indications of heartbeat exploit attempts that were stopped because of the hardened allocator)" means that the benefit of issuing new keys is reduced. Maybe you still decide that the benefit outweighs the cost; in this case, you're in the same situation. But maybe you decide that, given OpenBSD's allocator, the benefit is now below the cost because you judge your risk to be low enough, so you don't reissue keys. In this case, at least in your estimation, you have improved your situation over the first case: you have saved the difference between your evaluation of the benefit and your evaluation of the cost.

      In other words, the hardened allocator never makes things worse and it can make things better. That's an improvement.

    39. Re:Summary. by EvanED · · Score: 1

      Everything in security is about constantly/benefit.

      "Constantly"? "cost/benefit".

    40. Re:Summary. by bluefoxlucid · · Score: 1

      Look at the code, man. The code behaves exactly perfectly under non-exploit condition. If the payload is not smaller than the payload value, you don't read past the buffer.

      It would be caught when someone figured out about the exploit. It would be caught when someone decided to test the TLS Heartbeat extension for invalid behavior when given a Payload value larger than the actual payload.

      Coincidentally, it got caught when someone decided to give the TLS heartbeat extension a payload value larger than the actual payload.

    41. Re:Summary. by bluefoxlucid · · Score: 1

      So? I'll take a 90% survival cancer over a 10% one.

      You're not listening.

      Look in your logs. There is, right now, a 90% chance that you probably haven't had your private keys stolen.

      Of course, you can't verify that. You can't say for certain your private keys haven't been compromised.

      My point was, and is, that the "Limited damage" case of Heartbleed is that there's a 99% chance that your keys weren't stolen. Of course, you can't verify that.

      Let's say you posted your SSL keys on reddit, in public. Then you go back 1 second later and delete that post. It's deleted out of the reddit database. Probably, PROBABLY nobody saw that. The Reddit admins probably won't do crazy forensics to get them back. Probably nobody was packet sniffing Reddit and saw your keys.

      That's what we're talking about here. The mitigated situation with Heartbleed is identical in nature to logging into your bank over plain HTTP using CAT5: nobody can sniff it in the air, and there are probably no MITM attacks happening on the backbone of the internet, so it's safe to run credit card transactions and online banking without HTTPS.

    42. Re:Summary. by bugs2squash · · Score: 1

      I have to wonder why writepast, readpast and double free checks, along with memory zeroing are not directly supported for fine-grained allocations in MMU hardware, I know the MMU can handle memory in chunks, but why not extend that protection to any arbitrary length of memory allocation. I would have though that such functionality could be beneficial, effectively directly handle the basic malloc, calloc and free functionality in hardware with real GPFs thrown every time something amiss happens.

      --
      Nullius in verba
    43. Re:Summary. by complete+loony · · Score: 1

      Line 3991, that's the user supplied payload length, so it could be 64K even though the user only supplied a 1 byte message, and it doesn't cause a write outside allocated data..

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
    44. Re:Summary. by complete+loony · · Score: 1

      Also Line 3997, 64k is a large request. If you're using a hardened form of malloc, the memory you allocated for the incoming message would initially be blank, and the next page of memory wouldn't exist. So the 64k read from the incoming buffer is likely to trap and crash the process.

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
    45. Re:Summary. by Anonymous Coward · · Score: 0

      Building linked lists shorter than around 100k elements is fucked.

      you might have learnt than inserting into an array is O(n) and inserting into a linked list is O(1). This is vaguely true, when you have a direct pointer into the linked list. It turns out you usually don't, and walking the linked list to find the insertion point is more expensive than memmove() from the insertion point to the end of an array., Caches, predictors and all that.

      So yea, actually in a lot/most of cases that people are creating linked lists, they are doing it wrong.

    46. Re:Summary. by carou · · Score: 1

      I think the reason he called them irresponsible, is that the code is formatted as though their reallocater were an optional feature, but in fact they've only tested with it switched on. If you switch it off, you're running a bitrotted code path which doesn't work.

    47. Re:Summary. by EvanED · · Score: 1

      It would be caught when someone figured out about the exploit. It would be caught when someone decided to test the TLS Heartbeat extension for invalid behavior when given a Payload value larger than the actual payload.

      There are two reasons why I stand by my point.

      1) Black hats
      2) Fuzz testing or accidentally-ill-formed heartbeat requests

      Had either of these hit an OpenSSL deployment that used a hardened allocator, it would have given an opportunity for discovery before the recent discovery by white hats. If either of those hit an OpenSSL deployment with OpenSSL's bad wrapper, there is much less opportunity for discovery.

      There's of course no guarantee that it would have happened, for a number of reasons, but I never said it, just make arguments re. probabilities. And I honestly fail to see how you think that a hardened allocator isn't valuable.

    48. Re:Summary. by EvanED · · Score: 1

      You're not listening.

      From my perspective, I'm not the one not listening.

      My point was, and is, that the "Limited damage" case of Heartbleed is that there's a 99% chance that your keys weren't stolen. Of course, you can't verify that.

      So your position is that because you can't be 100% absolutely positive that your keys weren't compromised, you should regenerate them? Because that's what I understand you to be saying.

      And if it is, you better go regenerate your keys again. After all, you can't be 100% positive that someone didn't 0-day your system while you were reading my post. Hmm, you need new keys again. How can you be positive that the ones you just made weren't compromised to? Better make some new keys a third time.

      Obviously I'm being facetious here, but there's more to it than that: if you're really really paranoid about security, it actually makes sense to change keys occasionally even if you don't change the algorithm or implementation. Why? Contingencies: if your key is compromised, it limits exposure. Even if you assume that once an attacker is able to capture your key they'll be able to capture all future keys, it still limits the amount back in time the attacker will be able to retroactively do it.

      Better make another new key.

      Because you're not making new keys, you're doing a cost-benefit analysis that generating new keys each time I tell you to is not in favor. Just as most people hopefully do the cost-benefit analysis related to the heartbeat vulnerability and determine it is.

      The "benefit" is really more of a "anti-loss" here, but it's related to the risk you assess if you stick with the old keys. If you judge that your risk of exposure is higher, than the benefit of changing keys is higher, and so you're more weighted to change keys.

      Suppose you're a relatively low-importance site, and based on the importance (or lack thereof) of the information protected you decide that if you're 95% sure or better that you weren't compromised you're not going to bother to change keys because the cost-benefit ratio isn't in your favor. (Remember, if you needed to be positive you're not compromised, you'll be continuously making keys. Have you made a new key yet?) Now if you have an OpenSSL deployment without a hardened allocator, you say "I'm 90% sure that I wasn't compromised." That's below your threshold, so you make new keys. But if you have a deployment with a hardened allocator, mayde you say "I'm 99% sure that I wasn't compromised." That's above your threshold, so you don't.

      I made those numbers up of course, but that's the general idea. A hardened allocator provides a potential benefit. As I said in another reply, I don't see any cogent argument against that. The only objection I see is if you think that people are entirely incapable of making the kind of 90%/99% guesses I said, and thus removing the option for them to use a tighter threshold will artificially push people toward the failsafe option. (Actually that's not a bad argument, to be honest.)

    49. Re:Summary. by Anonymous Coward · · Score: 0

      So shutdown the operation of a custom alloc for the US GOV...

    50. Re:Summary. by Lisandro · · Score: 2

      Think it this way. Using the current memory management implementation on OpenSSL adding a test case for the exploit would not have triggered any issues. At all. Using a regular malloc() on any half-decent *nix system would've immediately triggered an alert when run through Valgrind or similar.

      Reusing memory with a freelist on a sensitive library like OpenSSL is a problem waiting to happen.

    51. Re:Summary. by bluefoxlucid · · Score: 1

      It would require a hell of a lot of overhead. Faults into memory are extremely slow, and you can't just cache that there are millions of allocations. Further, a program may legitimately read all over RAM, potentially interleaving from two arrays (sort algorithms) or jumping back and forth in linked lists; analyzing this behavior would be nigh-impossible.

    52. Re:Summary. by bluefoxlucid · · Score: 1

      Yes I misread the allocation. It's correct.

    53. Re:Summary. by bluefoxlucid · · Score: 1

      In this case, "a hardened malloc" indicates not using brk(). A long, long time ago, I started designing one based on mmap() and size classes. My inspiration was glibc's tendency to allocate large anonymous requests as mmap() anonymous segments; Hoard's non-locking multi-thread allocator (different per-thread allocation regions); and the theory that stuff allocated close together in time and of the same size class often was freed at the same time. realloc() became a compacting call as well.

      I gave up because testing malloc() on Linux was hard. I hadn't figured out how to get away from the existing allocator.

    54. Re:Summary. by bluefoxlucid · · Score: 1

      If either of those happened, we would have to assume that anything deployed in the vulnerable window was exploited. Same situation as today.

    55. Re:Summary. by bluefoxlucid · · Score: 1

      So your position is that because you can't be 100% absolutely positive that your keys weren't compromised, you should regenerate them? Because that's what I understand you to be saying.

      That's the advice being given in security circles. There's an exploit, it's mystical, it was released before patches, and someone could have silently stolen your keys. Please regenerate and reissue passwords, which may have been stolen.

      You're talking about risk analysis--which is good, although you either don't know how to do it or you're abbreviating out the use of all the good analysis techniques--and I'm talking about the current risk behavior and how it would change. The prescribed risk behavior wouldn't change, because the keys are fundamentally broken: the risk of an RSA break is ungodly small, and the risk that your keys were vacuumed away at a whim is like comparing Jupiter to an ant. Jupiter is still tiny compared to Betelgeuse, I get that. We're talking 2^127 versus 2^16 here. 1/2^16 is really fucking small (99.998% chance you weren't hacked), but uh.

      As for artificially pulling people toward a failsafe method: this is Schrodinger's cat. Your security is now imaginary, as the attacks don't show up in logs (unless you have Netwitness or something, or updated your NIDS). I can conjecture that our keys were probably not stolen here; the probability is low, maybe negligible, while the severity is ABSOLUTE. Mind you from a whole business perspective, using broken SSL keys for us is hilariously pointless--it's a "yeah, you should fix that" issue. From a perspective of "is our security broken?", we're talking about relying on something that may or may not be imaginary for this part of our security model.

      10%/90%/99%/99.99% still is too much. It's not even mathematical; it's a statistical guess, and it's based on the idea that nobody bothered to try. You hope nobody cared, you're "low profile". Mostly the "probability" here is major conjecture about what kinds of people you're dealing with (not-Mossad) and how alert they are.

    56. Re:Summary. by bluefoxlucid · · Score: 1

      If you thought up using a test case, you might look at the code and go, "o shit!" Or you might run your test case, look at the packet (does it return working data, even though it doesn't crash?) and go, "WTF IS THIS???" If your test case is "I ran it and didn't get SIGSEGV", your test case sucks.

    57. Re:Summary. by jythie · · Score: 1

      Unless they really screwed it up, a custom allocator when you know the size of your data blocks (and they are all the same) is going to out preform the general one on any system, it is just a matter of how much performance gain you get vs the risk inherent in adding the allocator. The general allocators have gotten better over the years, but there will always be additional overhead involved in maintaining pools capable of handling any arbitrary sized allocation request.

    58. Re:Summary. by Anonymous Coward · · Score: 0

      The thing is, there were no Random Joes combing OpenSSL, at least ones qualified enough to detect the bug AND ethical enough to not keep it for themselves or their black hatted friends.

      Open source claims to be more secure because it's laid bare for everyone to see, but that promise only works when people are actually auditing code. It's dreadful, boring, non-sexy work, so no one is doing it. Here we are two years later after the bug is introduced to the code base.

      Was it accidental? Likely. But there are a lot of very common packages out there that could just as easily had intentional bugs introduced to weaken them. Even then, a rogue compiler can inject whatever it wants into the binary.

      Anyone that has a useful computer in 2014 and claims they are safe because they are using open source software is either ignorant or lying. No one can possible read, let alone understand, the hundreds of millions of lines of code that can be instantiated on their machine.

      At least with a closed model, you have someone to sue.

    59. Re:Summary. by Applekid · · Score: 1

      Heartbleed the BUG would still exist, but Hearbleed the EXPLOIT would not exist, had they not been overcome with NIH syndrome for dynamic memory handling.

      --
      More Twoson than Cupertino
    60. Re:Summary. by bluefoxlucid · · Score: 1

      How would it not exist? You'd still copy 64KB of RAM to a 64KB validly-allocated buffer. If your memory allocator uses brk() (i.e. pmalloc on glibc, kmalloc on freebsd), you're vulnerable.

    61. Re:Summary. by BasilBrush · · Score: 1

      AFAIK, OpenSSL is userland, not in the kernel. But I'm not a Linux user so I could be wrong on that. OK, so which kernel version are you saying that's needed to fix this bug? And how does one tell whether one has it or not?

      If these aren't very easy answers, then that's the difference.

    62. Re:Summary. by jmcvetta · · Score: 1

      since there aren't many OpenBSD servers, probably nobody would notice that these attacks were happening

      There aren't many OpenBSD hosts relative to other OSes, but there are quite a few of them out on the net. And the people who run them tend to be the kind of folks who would notice if OpenSSL crashed.

    63. Re:Summary. by Anonymous Coward · · Score: 0

      Yup. Just the other week I *accidentally* found a critical security hole in a popular Python library. Was poking around to see how another part of the library worked, when I happened to notice it. Fixed it, submitted PR, accepted within an hour, new version pushed within 2 hours. Let's see proprietary software match that!

      (Posting anon because I at least try not to be a self-promoting git.)

    64. Re:Summary. by bluefoxlucid · · Score: 1

      "Aren't many relative to total" is "aren't many". It's like saying there "aren't many people" in your town, and you say, "But there's 100 people!" and I'm like, dude, my town has 667,000 and it's 15 miles across. There "aren't many people" in your house--15--except you have THIRTEEN KIDS, which is a lot.

      No, several thousand servers isn't a lot, because there are several hundred million servers. And Marcus Ranum didn't notice when his OpenBSD box got hacked, until he saw penis pill ads on his home page.

  7. audit by Anonymous Coward · · Score: 0

    maybe its time to re-audit all the openssl code

  8. Why OpenSSL is so popular? by sinij · · Score: 5, Interesting

    Why OpenSSL is so popular? It has FIPS-certified module, and this becomes important for selling your product to the government.

    So what could be done to prevent something like this from happening in the future? People will keep writing bad code, this is unavoidable, but what automated tests could be run to make sure to avoid the worst of it? Someone with direct development experience please educate the rest of us.

    1. Re:Why OpenSSL is so popular? by sinij · · Score: 1

      OpenSSL is still better than the alternative - home-brewing your own crypto.

    2. Re:Why OpenSSL is so popular? by swillden · · Score: 4, Insightful

      People will keep writing bad code, this is unavoidable, but what automated tests could be run to make sure to avoid the worst of it?

      Automated testing for security problems doesn't really work. Oh, you can do fuzzing, but that's hit and miss, and general unit testing can catch a few things, but not much. Mostly, security code just requires very careful implementation and code review. Eyeballs -- smart, experienced eyeballs.

      OpenSSL has terrified me for years. The code is very hard to read and understand, which is exactly the opposite of what's desired for easy review and validation of its security properties. It needs to be cleaned up and made as simple, straightforward and accessible as possible, or replaced with something else that is simple, straightforward and accessible. My theory on why it is the way it is -- and it's far from unusual in the crypto library world -- is that cryptographers tend not to be good software engineers, and software engineers tend not to have the cryptography skills needed.

      I spend some (not very much, lately) of my time working on an open source crypto library called Keyczar that tries to address one part of this problem, by providing well-engineered crypto APIs that are easy to use and hard to misuse. That effort focuses on applying good engineering to the boundary between library and application code, which is another source of rampant problems, but Keyczar uses existing crypto libs to provide the actual implementations of the primitives (the C++ implementation uses openssl, actually). I've long wished we could find crypto libs that were well-engineered, both internally and in their APIs.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    3. Re:Why OpenSSL is so popular? by Bill_the_Engineer · · Score: 2

      Why OpenSSL is so popular? It has FIPS-certified module, and this becomes important for selling your product to the government.

      FIPS certification is for a specific compiled version of OpenSSL and is not a blanket certification. The only way you are FIPS compliant is if you document that your product uses the exact same compiled version of OpenSSL or you submit your version of OpenSSL to be certified.

      --
      These comments are my own and do not necessarily reflect the views or opinions of my employer or colleagues...
    4. Re:Why OpenSSL is so popular? by jythie · · Score: 1

      Well, it is popular because it is a generally well regarded and vetted package that supports a fairly rich set of cryptography tasks out of the box.

      As for what could be done in the future? Well, automated tests really only cover cases you think about, and stress tests may or may not actually notice something. To a degree, there will always be things that slip through, and most of the time things are fixed and patched. In this case something unusually bad slipped through.

    5. Re:Why OpenSSL is so popular? by Anonymous Coward · · Score: 0

      I hear this all the time, not to say it is not right, but are there examples of small companies that got burned for home-brewing their own crypto?

    6. Re:Why OpenSSL is so popular? by frank_adrian314159 · · Score: 4, Insightful

      First, make sure that code that must be secure is transparent. That means little (or no) optimizations, standard calls to OS functions, and clearly structured. It's clear that the OpenSSL developers made their code more opaque than was prudent and the many eyes of open source land could not see through the murk. Yes, clearer code would mean that it ran more slowly and some folks would need to run a few more servers, but the security problem might have been uncovered sooner (or not have happened) if someone hadn't thought that performance was a reason to make the code more complex.

      Second, formal independent review would have helped. Most code (especially in volunteer-based open source projects) is only vetted by people directly on the development team. Any piece of software as ubiquitous and critical to the operation of today's internet as OpenSSL cannot have verification and validation mainly by its own developers. For software like this, where security is critical, you should have external review. Start an independent project that vets these things, folks.

      Third, understand the limits of testing vs. design. More unit tests would not have caught this. Simple designs lead to simple and correct implementations. Complex designs (or no designs) lead to seas of unit tests that simply tells you the ways that the code happens not to be broken at the moment. Code like that in OpenSSL ideally should be simple enough to be formally proved correct.

      I think we've known about why these sorts of things happen ever since I entered he field thirty years ago. We have ways to prevent them, but they usually take time, money, or lowered performance. That they are still happening because of performance zealotry, bad process, and "teh web-speed is everything" mentality is a black mark on our profession.

      --
      That is all.
    7. Re:Why OpenSSL is so popular? by MarcoAtWork · · Score: 4, Insightful

      it is a generally well regarded and vetted package that supports a fairly rich set of cryptography tasks out of the box.

      I would see that as a drawback for using it in webservers: if I am writing something internet-facing I want to use the smallest and simplest possible library that does the job, maybe it would be time to fork openssl into openssl-core / openssl-extras and have openssl-core have only the most minimal set of functionality related to securing connections and that's it? I would honestly also only support a few platforms for -core to simplify the code analysis even more (the more ifdefs, the more possible issues)

      --
      -- the cake is a lie
    8. Re:Why OpenSSL is so popular? by Daniel_Staal · · Score: 3, Insightful

      In this case though, general unit testing should have caught the bug: There's an option at compile time which, if used, caused the affected versions of OpenSSL to crash. (Because it disables the bug, and OpenSSL was relying on it in one location...) So, good unit testing would have helped.

      Basically, unit testing should be able to tell you if you've implemented the algorithm competently. It doesn't say if the algorithm is any good, just that your version of it works to the spec.

      --
      'Sensible' is a curse word.
    9. Re:Why OpenSSL is so popular? by sinij · · Score: 1

      Tons, my favorite example is encrypting ransomware that messed up key length and as a result could be brutforced.

      http://blog.cassidiancybersecu...

    10. Re:Why OpenSSL is so popular? by Anonymous Coward · · Score: 0

      Even if you can make a secure stack, it would be a significant effort to implement somthing like TLS, and it requires detailed understanding about the cryptographic primitives it uses and how to make secure code.

    11. Re:Why OpenSSL is so popular? by jythie · · Score: 1

      Thing is, subdividing things does not lead to as much simplicity as people tend to hope. Splitting up OpenSSL because you are only using parts of it would be a bit like taking red and green out of a graphics library because you are only using blue.

    12. Re:Why OpenSSL is so popular? by Anonymous Coward · · Score: 0

      You're free to fork the code and organize it however you wish. Yeah, they probably do not have the time to waste on that either.

    13. Re:Why OpenSSL is so popular? by gnasher719 · · Score: 1

      First, make sure that code that must be secure is transparent. That means little (or no) optimizations, standard calls to OS functions, and clearly structured. It's clear that the OpenSSL developers made their code more opaque than was prudent and the many eyes of open source land could not see through the murk. Yes, clearer code would mean that it ran more slowly and some folks would need to run a few more servers, but the security problem might have been uncovered sooner (or not have happened) if someone hadn't thought that performance was a reason to make the code more complex.

      I think the developers thought their code runs slower if all variable names are single characters. Which isn't actually true. I had the joy to look through bits of openssl to figure out how some stuff worked, and it is just generally unreadable. Just writing down in the interface files what each function does would have helped as well.

      Theo can complain as much as he likes about attack mitigation not working, but the fact is that this was a stupid bug which quite likely wouldn't have happened if struct members had had meaningful names that would have made it obvious that openssl sends more bytes back than it received.

      I don't think that better code costs performance at all. Better code is so much easier to maintain, you have some spare time to make things quicker.

    14. Re:Why OpenSSL is so popular? by Anonymous Coward · · Score: 0

      1) the "many eyes" of open source *has* worked: here we are dsscussing the code. Could you say the same from a proprietary codebase? It's only it has worked later than desired.
      2) It being open source doesn't avoid for third parties to audit the code. In fact, the case of it being open source is what allows for it. Why IBM, Oracle or CA haven't done so you should ask them.

    15. Re:Why OpenSSL is so popular? by Anonymous Coward · · Score: 0

      Splitting up OpenSSL because you are only using parts of it would be a bit like taking red and green out of a graphics library because you are only using blue.

      From the code snippets I have seen it would be more like reducing english to simple english - the code is bloated with generic defines, ifdefs and special case handling to support many different protocols/encryption algorithms in a horrible way. I really hope the people behind OpenSSL are great crypto experts, because they are horrible programmers.

    16. Re:Why OpenSSL is so popular? by EvanED · · Score: 2

      Not exactly crypto, but an amusing article on producing a way to compute the entire shuffle of a deck of cards in online poker based on those that were visible: How We Learned To Cheat At Online Poker

      Crypto is harder than card shuffling. :-)

    17. Re:Why OpenSSL is so popular? by swillden · · Score: 1

      Basically, unit testing should be able to tell you if you've implemented the algorithm competently. It doesn't say if the algorithm is any good, just that your version of it works to the spec.

      It doesn't even tell you that much, since it can only check functional aspects, not non-functional aspects. For example, unit tests can't prove that the code isn't leaking data through side channel attacks. That's a simple and obvious one, but there are more direct security failures which also won't be identified by any unit tests.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    18. Re:Why OpenSSL is so popular? by jafac · · Score: 1

      Same with as3crypto, (actionscript/flash/flex) unfortunately.

      --

      These are my friends, See how they glisten. See this one shine, how he smiles in the light.
    19. Re:Why OpenSSL is so popular? by Anonymous Coward · · Score: 0

      Unit tests don't do anything except validate behavior under certain stimuli.

      There are tools, for example Coverity, Valgrind, Ada-Ravenscar (ok, not for C in this case), which provide formal validation of all input.

      People repeatedly say code just needs to be eyeballed, but that is not enough. There are very sophisticated tools for static and dynamic analysis, and while they are not always easy to use, they are very powerful, and in fact could have detected this bug (using per-client memory coloring, and enforced annotated color mixing).

      People are still saying "oh the only way to be sure is to eyeball". No, the only way to be sure is to validate with a formal algebra. In many cases this does not even require eyeballing of the code in question, but it does require a complete understanding of the invariants and state diagrams of your program/library.

    20. Re:Why OpenSSL is so popular? by Anonymous Coward · · Score: 0

      OpenSSL is split up, into libssl and libcrypto. But SSL/TLS requires almost all the features available. There are simpler SSL libraries out there, but they don't support all the various features (like newer, safer keying schemes) that OpenSSL does.

      OpenSSL does need to be heavily overhauled, but the API should stay mostly the same. It's a de facto industry standard at this point. And while OpenSSL is messy in some places, it's also hardened in many other places, if only because it's been pounded on for nearly two decades. This most recent bug was introduced only two years ago.

      The other option is to ditch SSL entirely. The old cryptographic best practice was for protocols to allow swapping in and out various primitives as the state-of-the-art developed. But decades later we've reached a point where we can select some well known primitives and be confident that they'll remain secure for the near future--more confident than we can be in the implementation security of a more flexible protocol.

    21. Re:Why OpenSSL is so popular? by Anonymous Coward · · Score: 0

      "Son, we live in a world that has walls and those walls need to be guarded by men with Servers. Who's gonna do it? You? You, Admin Weinberg? I have a greater responsibility than you can possibly fathom. You weep for OpenSSL and curse the Developers; you have that luxury. You have the luxury of not knowing what I know: that OpenSSL's flaws, while tragic, probably saved lives and that my existence, while grotesque and incomprehensible to you, saves lives. You don't want the truth because deep down in places you don't talk about at parties you want me on that wall, you need me on that wall. We use words like honor, code, loyalty. We use then as the backbone of a life trying to defend something. You use them as a punchline. I have neither the time nor the inclination to explain myself to a man who rises and sleeps under the blanket of the very freedom I provide and then questions the manner in which I provide it. I would rather you just said "thank you," and went on your way. Otherwise, I suggest that you pick up a IDE and stand a post. Either way, I don't give a damn what you think you are entitled to." /s

    22. Re:Why OpenSSL is so popular? by Daniel_Staal · · Score: 2

      Point taken. Still, not having unit tests doesn't help...

      --
      'Sensible' is a curse word.
    23. Re:Why OpenSSL is so popular? by swillden · · Score: 1

      +1

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    24. Re:Why OpenSSL is so popular? by Poisonous+Drool · · Score: 2

      OpenSSL has terrified me for years. The code is very hard to read and understand

      I would like to second this thought. For such an important piece of code, OpenSSL is overly complex. That, by itself, is a bug.

    25. Re:Why OpenSSL is so popular? by jmcvetta · · Score: 1

      I've long wished we could find crypto libs that were well-engineered, both internally and in their APIs.

      Have you looked at the Go standard library's crypto packages? I have not done any thorough examination of them and could NOT vouch for their quality - but they were written by security conscious folks (including Ken Thompson) at GOOG, and the code is fairly easy to read.

  9. Hindsight is 20/20 by slyborg · · Score: 4, Insightful

    So, it's always great fun bashing "obvious" bugs, especially when they have such an impact, but let it be noted that thousands of implementers used openssl to build systems taking the package at face value despite these now "obvious" deficiencies in development process. If you were that concerned about security, they would have done what Google did, and audit the code. There are of course many practical reasons why people can't do that, but regardless, the blame arrow here points both ways.

    1. Re:Hindsight is 20/20 by sinij · · Score: 1

      Google can afford to audit the code, and has reasonable expectation of meaningful results. The rest of us? Not so much.
       
      Are there any decent automated code auditing tools that could be used by smaller shops?

    2. Re:Hindsight is 20/20 by DMUTPeregrine · · Score: 4, Insightful

      OpenSSL's code is a mess. Go, read it.

      Now that you're back from your stay in the sanitarium, would you like to consider that rewriting it might be a better choice than auditing? Yes?

      Let's just make sure Nyarlathotep isn't on the dev team this time...

      --
      Not a sentence!
    3. Re:Hindsight is 20/20 by imikem · · Score: 1

      Great power, great responsibility. Adding features to a piece of software as critical to so much infrastructure needs to be taken ever so seriously. Multiple levels of code audits for starters. Testing/fuzzing. I'm not qualified to do those things, but I still get to clean up the mess left by the whole fiasco.

      --
      Perscriptio in manibus tabellariorum est.
    4. Re:Hindsight is 20/20 by gweihir · · Score: 1

      While generally true, this does not apply in the current disaster: The OpenSSL code, coding guidelines, coding style and project management style actually foster this kind of thing, like they wanted critical bugs. For example to test assumptions such as boundaries always at time-of-use is a fundamental secure coding principle. Apparently, they have never heard of it. To clear memory that may contain secrets when freed is another one. And so on.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    5. Re:Hindsight is 20/20 by gweihir · · Score: 1

      Indeed.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
  10. So what is an alternative to OpenSSL? by Anonymous Coward · · Score: 0

    Forgive my ignorance, but what could someone use as an alternative?

    1. Re:So what is an alternative to OpenSSL? by mr_mischief · · Score: 4, Interesting

      GnuTLS, which recently people were being told to avoid in favor of OpenSSL. You see, there was this bug...

    2. Re:So what is an alternative to OpenSSL? by Ignacio · · Score: 1

      GnuTLS, NSS, Botan.

    3. Re:So what is an alternative to OpenSSL? by Aethedor · · Score: 1

      Definitely PolarSSL.

      --
      It doesn't have to be like this. All we need to do is make sure we keep talking.
    4. Re:So what is an alternative to OpenSSL? by IamTheRealMike · · Score: 1

      Unpopular though it is, if you can take a small(ish) performance hit, you could use a Java HTTPS server that proxies to your app. The Sun/Oracle JSSE SSL stack (in the Oracle VM, not Android) is pure Java and thus immune to these sorts of errors. In JDK8 it supports TLS 1.2, ECDSA, perfect forward secrecy and the use of AES-NI for hardware accelerated constant time stream ciphering.

    5. Re:So what is an alternative to OpenSSL? by Anonymous Coward · · Score: 0

      The Sun/Oracle JSSE SSL stack (in the Oracle VM, not Android) is pure Java and thus immune to these sorts of errors.

      That is great, if you're not concerned about all the other sorts of errors.

    6. Re:So what is an alternative to OpenSSL? by EvanED · · Score: 1

      That is great, if you're not concerned about all the other sorts of errors.

      Yeah, it's way better to have the potential for both memory and logic errors. Logic errors alone are boring. *eyeroll*

  11. Still don't get it by Anonymous Coward · · Score: 0

    Where can I find Theo De Raadt's "better roll"? I can imagine him being "on a roll", but on a better roll? That must really be something... better.

  12. How about ... by Anonymous Coward · · Score: 0

    How about he replies to the questions that the Slashdot community asked him some time ago?

    He seems to go out of his way to piss people off ...

    1. Re:How about ... by EmagGeek · · Score: 2

      He's just trying to be a bigger dick than Linus..

    2. Re:How about ... by Marginal+Coward · · Score: 1

      Size matters?

  13. not developed by a responsible team? by G3ckoG33k · · Score: 1, Interesting

    De Raadt wrote "OpenSSL is not developed by a responsible team".

    On the contrary, I believe it was developed by a responsible team, that unfortunately made an error.

    Most everyone have made errors, even if most go unnoticed and are essentially harmless. This one appears different, but I don't think it justifies De Raadt's moronic comment.

    1. Re:not developed by a responsible team? by tsalmark · · Score: 5, Insightful

      My understanding is Theo said: Developers on a security product made a conscious decision to turn off last line of defence security for all platforms in the interest of performance on some systems. That does not sound like and unfortunate error to me, it sounds outright irresponsible.

    2. Re:not developed by a responsible team? by Anonymous Coward · · Score: 0

      Responsible or not, this error happened on their watch, and it is a big one, potentially one that gets UNIX in general tossed for "more secure" Microsoft machines at every juncture. This error is going to cost millions in dollars and hundreds of thousands of man-hours to rectify... and this is just closing the hole. Bog knows what damage that exploiters have done with this since SSL is critical to virtually -every- fscking thing on the Internet.

    3. Re:not developed by a responsible team? by WaywardGeek · · Score: 1

      Sometimes the individuals involved can be responsible while the team acts irresponsibly. For example, why is my passphrase of my id_rsa key protected by only one round of hashing with no option for increased rounds? I hear there are good things coming, like being able to use bcrypt, but this is a scandal. Only a security ignorant fool would want his passphrase attached to an id_rsa key with no password stretching at all. So... how many fools do we have out there? I surely hope you weren't counting on your passphrase being secure just because the OpenSSL team was involved.

      --
      Celebrate failure, and then learn from it - Nolan Bushnell
    4. Re:not developed by a responsible team? by squiggleslash · · Score: 4, Insightful

      He said that, but is that what happened? Were OpenSSL's developers aware that malloc()/free() have special security concerns that OpenBSD's developers had specifically addressed (I assume that's what meant by "a conscious decision to turn off last-line-of-defense-security")

      I understand Theo's point, to a certain degree I kinda understand it, but I'm more inclined to feel the problem is with OpenSSL's developers clearly not understanding the security concerns about malloc(). That is, if they were aware that OpenBSD's malloc() contained code to ensure against data leakage, it would seem to me to be highly probable they would have implemented the same deal in OpenSSL given, you know, their entire point is security. The fact they didn't makes me think they didn't know OpenBSD's malloc() had these measures in the first place.

      Should they have done? And how should they have known? Genuine question, and finger pointing would be inappropriate right now: how do we make sure that certain security strategies and issues are as well known as, say, stack pointer issues are today.

      --
      You are not alone. This is not normal. None of this is normal.
    5. Re:not developed by a responsible team? by Anonymous Coward · · Score: 1

      er, a foundational security component of most 'secure' sites on the web has been leaking who-knows-what information for who-knows-how-long over this. De Raadt can be a bit over-the-top at times, but his comment is neither moronic nor does it need you to consider it justified.

    6. Re:not developed by a responsible team? by dsparil · · Score: 1

      The irresponsible part is that OpenSSL does not even compile if you decide your system malloc is fine to use. It is impossible to avoid using OpenSSL's buggy allocator.

    7. Re:not developed by a responsible team? by EvanED · · Score: 3, Informative

      Were OpenSSL's developers aware that malloc()/free() have special security concerns that OpenBSD's developers had specifically addressed (I assume that's what meant by "a conscious decision to turn off last-line-of-defense-security")

      My impression is OpenBSD's hardened allocator is relatively common knowledge and definitely should be among people writing security software. And that's not even remotely the only such allocator out there that does that sort of thing too, though it's probably the most well-known from the industrial side.

    8. Re:not developed by a responsible team? by Eunuchswear · · Score: 4, Informative

      That is, if they were aware that OpenBSD's malloc() contained code to ensure against data leakage, it would seem to me to be highly probable they would have implemented the same deal in OpenSSL given, you know, their entire point is security. The fact they didn't makes me think they didn't know OpenBSD's malloc() had these measures in the first place.

      Not just OpenBSD's malloc(). glibc can do the same thing if you set MALLOC_PERTURB.

      --
      Watch this Heartland Institute video
    9. Re:not developed by a responsible team? by Anonymous Coward · · Score: 3, Informative

      De Raadt wrote "OpenSSL is not developed by a responsible team".

      On the contrary, I believe it was developed by a responsible team, that unfortunately made an error.

      Most everyone have made errors, even if most go unnoticed and are essentially harmless. This one appears different, but I don't think it justifies De Raadt's moronic comment.

      Not so sure they're responsible.

      Did you read this?

      This bug would have been utterly trivial to detect when introduced had the OpenSSL developers bothered testing with a normal malloc (not even a security focused malloc, just one that frees memory every now and again). Instead, it lay dormant for years until I went looking for a way to disable their Heartbleed accelerating custom allocator.

      Building exploit mitigations isn’t easy. It’s difficult because the attackers are relentlessly clever. And it’s aggravating because there’s so much shitty software that doesn’t run properly even when it’s not under attack, meaning that many mitigations cannot be fully enabled. But it’s absolutely infuriating when developers of security sensitive software are actively thwarting those efforts by using the world’s most exploitable allocation policy and then not even testing that one can disable it.

      The OpenSSL team doesn't fully test their product.

      That's pretty much as good an example of incompetence that you can probably find.

    10. Re:not developed by a responsible team? by Eunuchswear · · Score: 5, Informative

      Oh, and read this: http://www.tedunangst.com/flak/post/analysis-of-openssl-freelist-reuse

      In effect at some points OpenSSL does:


              free (rec); ...
              rec = malloc (...);

      and assumes that rec is the same.

      Eeew,

      --
      Watch this Heartland Institute video
    11. Re:not developed by a responsible team? by Anonymous Coward · · Score: 1

      in light of this analysis: http://www.tedunangst.com/flak... i'm inclined to agree with Theo on this one...

    12. Re:not developed by a responsible team? by squiggleslash · · Score: 4, Interesting

      Ouch. Serious ouch. Thank you. That suggests that the situation is considerably worse than De Raadt said.

      --
      You are not alone. This is not normal. None of this is normal.
    13. Re:not developed by a responsible team? by EvanED · · Score: 1

      The OpenSSL team doesn't fully test their product.

      I agree with Theo on the broader point, but disagree here: "their product" is the code they wrote with their custom memory allocator. It's not the code they wrote with some changes someone outside the product made. Their custom allocator isn't disabled or enabled depending on configuration checks or something like that, it's always part of the product.

      It was still a pretty stupid decision.

    14. Re:not developed by a responsible team? by mrchaotica · · Score: 4, Insightful

      Should they have done? And how should they have known? Genuine question, and finger pointing would be inappropriate right now: how do we make sure that certain security strategies and issues are as well known as, say, stack pointer issues are today.

      Hell yes they should have known, because the people responsible for one of the most important security applications in the entire world damn well ought to be experts!

      --

      "[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz

    15. Re:not developed by a responsible team? by Anonymuous+Coward · · Score: 3, Interesting

      This bug would have been utterly trivial to detect when introduced had the OpenSSL developers bothered testing with a normal malloc (not even a security

      This is simply not true, stop spinning it.

      Even if OpenSSL is using system's malloc, with all its mitigation features, the bug still works. The attacker just has to be more careful, lest he should read freed() and unmapped memory, and so cause a crash and (supposedly) leave some kind of meaningful trail.

    16. Re:not developed by a responsible team? by Anonymuous+Coward · · Score: 1

      nb: he's not talking about the heartbleed bug, but about another bug he found in OpenSSL's custom allocator -- a use after free.

    17. Re:not developed by a responsible team? by Anonymous Coward · · Score: 2, Informative

      The OpenSSL team doesn't fully test their product.

      I agree with Theo on the broader point, but disagree here: "their product" is the code they wrote with their custom memory allocator. It's not the code they wrote with some changes someone outside the product made. Their custom allocator isn't disabled or enabled depending on configuration checks or something like that, it's always part of the product.

      It was still a pretty stupid decision.

      Ted Unangst merely used a different configuration option - one provided in the standard OpenSSL software.

      Given some of the other bugs he found, we know for a fact that the OpenSSL team does not fully test their product.

      Period.

      Unangst even found OpenSSL basically assumes that "free(prt); ptr=malloc()" will return the same pointer. Holy fucking shit, you gotta be kidding me. What kind of incompetent boob doesn't test a C/C++ application for that?

      Maybe the OpenSSL folks need to begin a Kickstart campaign to pay for a Purify license or two.

    18. Re:not developed by a responsible team? by Anonymous Coward · · Score: 0

      He said that, but is that what happened? Were OpenSSL's developers aware that malloc()/free() have special security concerns that OpenBSD's developers had specifically addressed (I assume that's what meant by "a conscious decision to turn off last-line-of-defense-security")

      Should they have done? And how should they have known? Genuine question, and finger pointing would be inappropriate right now: how do we make sure that certain security strategies and issues are as well known as, say, stack pointer issues are today.

      The problem was that they sacrificed speed for security. Look at the comment around the code:

      #ifndef OPENSSL_NO_BUF_FREELISTS
      /* On some platforms, malloc() performance is bad enough that you can't just

      At the very least they should have only overrided malloc() on the platforms where performance was a problem and leave the rest.

    19. Re:not developed by a responsible team? by udippel · · Score: 1

      Not being exactly a friend of Theo (who is??!), I can vouch that I ran 'his' software as long as I was sysadmin on all my servers, and neither ever failed me - except of hardware failure.
      Now to the issue that you mention: No, you are alas very wrong; in an engineering sense.
      Any complex system, when a responsible for a subsystem is unhappy or lacks confidence in the working of another subsystem, cannot be improved by that person or team circumventing the other subsystem. Out and over.
      Example: When I construct the wheel base of a car, I can't add my own brakes or my own electric motors simply because I don't trust the braking system or the engine. Any approach like that is doomed to fail. In software, garbage collection is done, or just not. But it would be ridiculous if everyone wrote her own garbage collection in Java for not trusting what is provided. And so forth ... .

      Now, when a team works on OpenSSL, it makes a lot of sense to pass malloc down to the host system. Good, if you really feel that is inadequate, and you overrule it by not free-ing what is free, and your application is all on security (like OpenSSL), then you better confirm 800% that your own solution is totally waterproof.
      If we look at what happened, the OpenSSL team contravened both suggestions. This deserves pulling the red card.

    20. Re:not developed by a responsible team? by OneAhead · · Score: 3, Interesting

      That would have been an appropriate boilerplate response to your typical Theo De Raadt comment, but... did you actually read that "analysis of openssl freelist reuse" link? It's hard for me to imagine that having been done by mistake; it's more of an instance of a coder cleverly cutting a corner. And some might argue "don't we all?", but if one is writing such security-critical software, one should at least have some notion that clever corner-cutting is a gigantic no-no.

      Now, the freelist reuse is not the cause of the heartbleed bug; it merely frustrates what would otherwise have been a relatively straightforward mitigation strategy. But it's a symptom of an attitude that is, well, irresponsible.

    21. Re:not developed by a responsible team? by Anonymous Coward · · Score: 0

      De Raadt's record is as unclean as his wordings.

      2010: arstechnica.com/information-technology/2010/12/openbsd-code-audit-uncovers-bugs-but-no-evidence-of-backdoor

      OpenBSD project leader Theo de Raadt disclosed an e-mail earlier this month in which former NETSEC CTO Gregory Perry claimed that his company was paid by the FBI to plant a "backdoor" in the OpenBSD IPSEC stack. The allegations led to a thorough code review and historical analysis of the relevant code. In a follow-up e-mail published this week, de Raadt outlined his current perspective on the controversy and his interpretation of the findings that have emerged from the ongoing code audit. Reviews are being conducted on the history and provenance of code in the IPSEC stack as well as the current implementation. Reviewers have uncovered several bugs that could have security implications, but the nature of the bugs suggests that they were not intentional, nor were they intended to facilitate a backdoor. The most serious revelation so far is the discovery of a bad conditional expression in older versions of the Encapsulating Security Payload (ESP) code. This hole was quietly closed in 2002 without the usual vulnerability disclosure process. As such, the bug is not present in modern-day OpenBSD, but has remained unknown to users due to the lack of a public advisory.

      2006: kindsoftware.com/documents/proposals/bugs_in_openbsd

      The OpenBSD source code has been and continues to be relentlessly trawled for bugs. The cause of a bug could be for example due to a misunderstanding of an API, an incorrect assumption on the part of the programmer, or a simple typo. From time to time, a bug is discovered which really represents a whole new class of mistake. Once such a class is discovered, the entire source tree is scoured for further instances of this error - thus one bug fix may lead to many more, often solving problems in other unrelated places before they are even noticed. [...] A relatively crude proof-of-concept implementation was successfully employed during the course of this project, revealing many bugs in portions of the OpenBSD source tree.

      Why does he bark at other developers? Money?

    22. Re:not developed by a responsible team? by gnasher719 · · Score: 2

      I understand Theo's point, to a certain degree I kinda understand it, but I'm more inclined to feel the problem is with OpenSSL's developers clearly not understanding the security concerns about malloc(). That is, if they were aware that OpenBSD's malloc() contained code to ensure against data leakage, it would seem to me to be highly probable they would have implemented the same deal in OpenSSL given, you know, their entire point is security. The fact they didn't makes me think they didn't know OpenBSD's malloc() had these measures in the first place.

      Here is what happens, as far as I understand: a client sends two bytes of data to the server and asks the server to send the identical two bytes back to the client, to check that the server is still alive. That's how it works normally. A client could send 65,000 bytes and ask for the 65,000 bytes to be sent back, except that would be inefficient.

      Instead, an attacker sends two bytes of data to the server and asks for the same 65,000 bytes back. The server stores the two bytes with a bit of overhead into a malloc block, creates a huge malloc block for the results, and memcpy's 65,000 bytes from the small malloc block to the huge one. 64,998 bytes that are copied are just whatever was in memory after that malloc block.

      All the usual measures against buffer overwrites don't help, because there is no buffer overwrite. Nothing is destroyed on the server, instead it is tricked into giving information it didn't want to give. What could malloc do about that? A "free" and "realloc" implementation that sets memory to zero wouldn't hurt. Of course that doesn't help if the memory after the small malloc block is actually currently used. You'd need a malloc that will crash if you read past the end of a malloc buffer. That's hard to do efficiently.

    23. Re:not developed by a responsible team? by EvanED · · Score: 5, Informative

      Theo's "rant" isn't brought about by a typical bug in the sense of a mistake in the code. His rant is brought about by the fact that OpenSSL deliberately introduced wrapper functions for malloc/free, making it impossible for a system to provide hardened ones, then made those wrappers default. He further rants about the fact that, because it's default, they later introduced a bug that means you can't turn the wrappers off.

      The heartbeat bug itself was a mistake and a bug in the traditional sense. The "hey let's replace malloc/free" is much closer to "bad decision" than "mistake."

    24. Re:not developed by a responsible team? by EvanED · · Score: 2

      All the usual measures against buffer overwrites don't help, because there is no buffer overwrite. ...You'd need a malloc that will crash if you read past the end of a malloc buffer. That's hard to do efficiently.

      OpenBSD's "usual measures" would have helped, because the OpenBSD allocator will often put allocated memory in isolated pages. So it's quite likely that 64K read will cross into an unmapped page, and boom, you have your crash.

      This level of overread protection is close to free.

    25. Re:not developed by a responsible team? by EvanED · · Score: 1

      It's related though, because the use-after-free bug occurs if you enable a configuration that would have somewhat mitigated the heartbeat bug.

    26. Re:not developed by a responsible team? by Darinbob · · Score: 1

      This depends on what you think "responsible" means. Honest mistakes are one thing, but a responsible team for a product of this sensitivity and widespread use needs to be far above average on the responsibility meter. Even code reviews for features that improve performance should be vetted by the security experts who will question whether the performance gains are necessary.

    27. Re:not developed by a responsible team? by chis101 · · Score: 4, Informative

      Even if OpenSSL is using system's malloc, with all its mitigation features, the bug still works. The attacker just has to be more careful, lest he should read freed() and unmapped memory, and so cause a crash and (supposedly) leave some kind of meaningful trail.

      You got it exactly right. He's complaining that because they provided their own malloc() wrapper that the read of freed() memory is NOT causing a crash. If they had used the system malloc() then there would be crashes, the issues would be detected, and they would have been fixed.

    28. Re:not developed by a responsible team? by dnavid · · Score: 1

      This bug would have been utterly trivial to detect when introduced had the OpenSSL developers bothered testing with a normal malloc (not even a security

      This is simply not true, stop spinning it.

      Even if OpenSSL is using system's malloc, with all its mitigation features, the bug still works. The attacker just has to be more careful, lest he should read freed() and unmapped memory, and so cause a crash and (supposedly) leave some kind of meaningful trail.

      The bug would have been easily detected had proper testing been done with the LIFO allocator turned off, because in the absence of OpenSSL's own allocator a normal malloc would have been releasing and reusing memory rather than preserving their data in a stack structure, almost guaranteeing that OpenSSL connections would be at least occasionally crashing out. That would have alerted the developers to the fact that sometimes a freed block pointer is reused after freeing, which would have compelled them to fix that error. You can reuse that freed pointer in OpenSSLs allocator because even when "freed" that memory is not available to be reused by anything else, so its data contents continue to be preserved.

    29. Re:not developed by a responsible team? by complete+loony · · Score: 1

      With default compile options, that's effectively Push(), Pop(). So obviously they aren't testing with all permutations of "supported" options.

      --
      09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
    30. Re:not developed by a responsible team? by jafac · · Score: 1

      Considering that the first advice given to any developer looking to add cryptographic functionality to their software is: "Don't try to roll your own cryptography. Instead, try to use a well-established package written by experts." . . . pretty much tells you something.

      --

      These are my friends, See how they glisten. See this one shine, how he smiles in the light.
    31. Re:not developed by a responsible team? by hobarrera · · Score: 1

      Did you read the entire email? Several of the issues he's mentioning here are not mere "errors". Skiping exploit mitigations for the sake of performance (on some platforms!) on a security library/program isn't an error, it's awfully irresponsable!

    32. Re:not developed by a responsible team? by Lisandro · · Score: 1

      Writing a custom malloc on a security sensitive product like this is a problem waiting to happen, specially when you do so carelessly. Freelists? No memory cleanup? Theo's right here.

  14. Bug Looks Deliberate by Anonymous Coward · · Score: 5, Interesting

    That code is almost a text book example of material that is submitted to the Underhanded C contest...

    http://en.wikipedia.org/wiki/Underhanded_C_Contest

  15. Rev Numbers by ISoldat53 · · Score: 1

    I'm not a developer and don't know the convention for version numbers but shouldn't they make the rev number of the repaired OpenSSL something more distinctive than adding "g" to the third digit of the new code? Maybe change it to version 2.0.0 or something more obvious.

    1. Re:Rev Numbers by ZombieBraintrust · · Score: 2

      I think that might scare people from upgrading. Going from 7.10322 to to 8 is scarier than going from 7.10322 to 7.10322g It suggests that upgrading will break things.

    2. Re:Rev Numbers by ISoldat53 · · Score: 1

      The current fright isn't enough to scare people?

  16. "Open SORES" still beats the snot of MS by walterbyrd · · Score: 1

    Linux is far more secure than Windows. Also more reliable, boots faster. Does not spend ten minutes updated when I turn on, or turn off, PC.

    Maybe you should take you childish troll somewhere else?

    1. Re:"Open SORES" still beats the snot of MS by Anonymous Coward · · Score: 0

      Linux is far more secure than Windows. Also more reliable, boots faster. Does not spend ten minutes updated when I turn on, or turn off, PC.

      Maybe you should take you childish troll somewhere else?

      Really? More reliable? Boots faster? My Windows 8.1 boots in less than 2 seconds. What about yours?

    2. Re:"Open SORES" still beats the snot of MS by jones_supa · · Score: 1

      Linux is far more secure than Windows.

      Really? Then why do we keep hearing about these serious vulnerabilities in OSS software? I just hope that the OSS people do not arrogantly laugh at these issues and realize much later that they have lost the game.

      Also more reliable

      In my experience Linux is far more unreliable than Windows these days. Sure, show-stopping kernel panics are rare, but little glitches here and there are very common.

      boots faster

      Win and Lin both boot very quickly, so it's not much of an issue anyway. However Windows 8 knows this extra trick to hibernate part of the system before shutting down, so boots can be extremely quick.

      Does not spend ten minutes updated when I turn on, or turn off, PC.

      It takes much more time to install all the updates provided by a Linux OS Update Manager than it takes to install the handful of updates that are offered once per month via Windows Update.

    3. Re:"Open SORES" still beats the snot of MS by Anonymous Coward · · Score: 0

      It takes much more time to install all the updates provided by a Linux OS Update Manager than it takes to install the handful of updates that are offered once per month via Windows Update.

      Because Windows Update does not allow anyone to update any other installed software than Microsoft software. Haven't you noticed the trillion of different updaters/downloaders that are run on Windows and each one bitching about permissions? Or how Steam circumvents permissions by just spawing SYSTEM level "service"?

      Nor Siree, no exploits on Windows, ever.

      http://en.wikipedia.org/wiki/N...

    4. Re:"Open SORES" still beats the snot of MS by jones_supa · · Score: 1

      Because Windows Update does not allow anyone to update any other installed software than Microsoft software.

      That is certainly a good point.

    5. Re:"Open SORES" still beats the snot of MS by exomondo · · Score: 1

      In my experience Linux is far more unreliable than Windows these days. Sure, show-stopping kernel panics are rare, but little glitches here and there are very common.

      It all depends on your distro and what drivers and other software you have installed. From a reliability and stability perspective the stock Windows 8.1 install and an up-to-date RHEL are both really as stable as eachother, but in both cases graphics drivers are often a quick way to compromise stability. Using "Linux" in this context is always a misnomer because nobody just runs Linux and certainly there is wide variance in the stability of Linux distros.

  17. that's pretty standard by Chirs · · Score: 1

    Major numbers are often reserved for things that break backwards compatibility, minor numbers often denote new features, and the patch revision denotes patches.

    Personally I would have bumped the patch rev (the third number).

  18. Re:Who is Theo De Raadt? by dingen · · Score: 2

    Because who in the world has ever heard of OpenSSH, right?

    --
    Pretty good is actually pretty bad.
  19. Really? by oneandoneis2 · · Score: 2, Interesting

    "it is how a few thousand of us feel about the whole thing"

    Then maybe you thousands should stop complaining and start contributing to the project, which is so under-resourced problems like this are pretty much inevitable.

    --
    So.. it has come to this
    1. Re:Really? by Anonymous Coward · · Score: 1

      All the people I know who looked at OpenSSL's source code said it was the worst piece of ugly hacks they saw in their life and left with disgust. Maybe that's why there are so few volunteers.

    2. Re:Really? by mvdwege · · Score: 1

      Maybe it wouldn't be so underresourced if the developers weren't such arrogant jackasses. This and the Debian disaster a few years back show that OpenSSL is hopelessly broken at the social level and should be ditched by all parties.

      --
      "I know I will be modded down for this": where's the option '-1, Asking for it'?
    3. Re:Really? by Anonymous Coward · · Score: 0

      Hmm, no. OpenSSL is unsalvageable (well, it is salvageable if you refactor it entirely, same as writing from scratch). Not even the API is salvagable, it is dangerous to use and really bad.

      Too bad the alternatives are not that much better (GnuTLS, NSS), but at least they have a less hideous API.

  20. Re:Who is Theo De Raadt? by TCM · · Score: 4, Insightful

    If you've never heard of him, you're not part of any important "tech community". Period.

    --
    Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
  21. Wow by Kazoo+the+Clown · · Score: 2

    That's pretty scathing. I'd hate to be THOSE guys...

  22. Unfortunately, this analysis seems to be spot-on by gweihir · · Score: 5, Insightful

    In addition, the mitigation countermeasures also prevent memory debuggers like Valgrind from finding the problem (Valgrind find use-before-init for malloc'ed blocks, but not if there is a wrapper in between that re-uses blocks), and may also neutralize code-security scanners like Fortify.

    I have to admit that while my original intuition was "screwup", this looks more and more like some parts of the OpenSSL team have been compromised and did things that make this kind of bug far more likely. Like their own insecure memory allocation. Like not requiring time-of-use boundary checks or having any secure coding guidelines in the first place. Like documenting everything badly so potential reviewers get turned away. Like not having working review for patched or a working fuzz-testing set-up (which would have found bug this easily).

    After all, the NSA does not have to sabotage FOSS crypto software. They just have to make sure the quality is low enough. The bugs they can exploit will follow. And the current mess is just a plain classic. Do enough things wrong and eventually stuff breaks spectacularly.

    --
    Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
  23. His rant could apply to almost any large project by PhrostyMcByte · · Score: 3, Insightful

    A lot of large performance-sensitive projects implement custom allocators in the form of arenas and freelists. Lots of platforms have a fast malloc implementation these days, but none of them will be as fast as this for the simple reason that the program knows more about its memory usage patterns than any general-purpose allocator ever could.

    Not to say I can't understand Theo's point of view -- if he wants maximum security, then a program which bypasses one of his layers in the name of performance might not be the best for him.

    On the flip side, the standards have no notion of such security layers and I feel it is perfectly reasonable for a team to not throw away performance in the interests of some platform-specific behavior. This was a bug, pure and simple. There's nothing wrong with using custom allocators. To say that "OpenSSL is not developed by a responsible team" is simply nonsense.

  24. De Raadt is wrong by stephc · · Score: 2, Interesting

    This is not a problem with OpenSSL, or the C Language or the Malloc implementation, this is a problem because everyone is relying on the same black box they do not understand. Because this is "standard" and common practice to use it. The only long term defense against this kind of vulnerability is software (and hardware?) diversity. Software built on custom SSL implementations may have even worse vulnerabilities, but nobody will discover them, and even if they do, it won't affect everyone on this planet. When I read Theo De Raadt, I fear his "solution" may only worsen the problem. We can't have all our secrets protected by the exact same door, no matter how strong the door is, once it's broken...

    1. Re:De Raadt is wrong by JDG1980 · · Score: 2

      This is not a problem with OpenSSL, or the C Language or the Malloc implementation, this is a problem because everyone is relying on the same black box they do not understand.

      That's a cop-out. Any kind of advanced economy needs division of labor. This is no less true of the IT industry than anywhere else. The people building the "black box" need to know what they're doing and it needs to work. Period.

    2. Re:De Raadt is wrong by tomhath · · Score: 1

      I doubt "diversity" in this case is a good idea. Instead of one bug in one package you would end up playing bop-a-mole with many bugs in a few packages.

    3. Re:De Raadt is wrong by stephc · · Score: 1

      The people building the "black box" need to know what they're doing and it needs to work. Period.

      But human nature prevent it, we know for quite a long time that software is never perfect and that security is never absolute. Diversity is the solution mother nature is using. I've wrote quite a lot of backend/server code, but I tend to use non-standard code to avoid vulnerability. Interoperability/Common Standards is a very good thing, but we don't have to all use the same implementation. Also, never trust something you don't understand.

    4. Re:De Raadt is wrong by stephc · · Score: 1

      The problem is not the cost of fixing a bug. The problem is the cost of the bug being exploited on a very large scale. It seems that we currently have some diversity, not every SSL transactions on the earth are handled by OpenSSL, and I hope it will never happen.

  25. Whinging & Complaining by Big+Hairy+Ian · · Score: 2

    is not helping!! What would be useful is a list of popular services affected by this issue the BBC is at least making a start here http://www.bbc.co.uk/news/tech...

    --

    Build a Man a Fire, and He'll Be Warm for a Day. Set a Man on Fire, and He'll Be Warm for the Rest of His Life.

  26. Re:His rant could apply to almost any large projec by JDG1980 · · Score: 3, Insightful

    A lot of large performance-sensitive projects implement custom allocators in the form of arenas and freelists. Lots of platforms have a fast malloc implementation these days, but none of them will be as fast as this for the simple reason that the program knows more about its memory usage patterns than any general-purpose allocator ever could.

    This is security software. You don't sacrifice the library's core functionality to make it run a bit faster on the old Celeron 300 running Windows 98.

  27. Natural Born Cancer by VortexCortex · · Score: 5, Insightful

    Well, what you are pointing out is that a CA is a single point of failure -- Something actual security conscious engineers avoid like the plague. What you may not realize is that collectively the entire CA system is compromised by ANY ONE of those single points of failure because any CA can create a cert for ANY domain without the domain owner's permission. See also: The Diginotar Debacle.

    The thing is, nobody actually checks the the cert chain, AND there's really no way to do so. How do I know if my email provider switched from Verisign to DigiCert? I don't, and there's no way to find out that's not susceptible to the same MITM attack.

    So, let's take a step back for a second. Symmetric stream ciphers need a key. If you have a password as the key then you need to transmit that key back and forth without anyone knowing what it is. You have to transmit the secret, and that's where Public Key Crypto comes in, however it doesn't authenticate the identity of the endpoints, that's what the CA system is supposed to do. Don't you see? All this CA PKI system is just moving the problem of sharing a secret from being the password, to being which cert the endpoint is using -- That becomes the essential "secret" you need to know, and it's far less entropy than a passphrase!

    At this time I would like to point out that if we ONLY used public key crypto between an client and server to establish a shared secret upon account creation, then we could use a minor tweak to the existing HTTP Auth Hashed Message Authentication Code (HMAC) proof of knowledge protocol (whereby one endpoint provides a nonce, then the nonce is HMAC'd with the passphrase and the unique-per-session resultant hash provides proof that the endpoints know the same secret without revealing it) to secure all the connections quite simply: Server and client exchange Nonces & available protocols for negotiation, the nonces are concatenated and HMAC'd with the shared secret stored at both ends, then fed to your key-stretching / key expansion system AND THAT KEYS THE SYMMETRIC STREAM CIPHER SIMULTANEOUSLY AT BOTH ENDS so the connection proceeds immediately with the efficient symmetric encryption without any PKI CA system required.

    PKI doesn't really authenticate the endpoint, it just obfuscates the fact that it doesn't by going through the motions and pretending to do so. It's a security theater. SSL/TLS and PKI are essentially the Emperor's New Secure Clothes. At least with the shared secret model I mention above, there's just that one-time small window of PK crypto for secret exchange at worst (failing to intercept account creation means no MITM) and at best you would actually have the CHANCE to go exchange your secret key out of band -- Visit your bank in person and exchange the passphrase, etc. then NO MITM could intercept the data. HTTP Auth asks for the password in a native browser dialog BEFORE showing you any page to login (and it could remember the PW in a list, or even generate them via hashing the domain name with a master PW and some salt so you could have one password for the entire Internet). That's how ALL security should work, it ALL relies on a shared secret, so you want the MOST entropic keyspace not the least entropic selection (which CA did they use). If you're typing a password into a form field on a web page, it's ALREADY game over.

    Do this: Check the root certs in your browser. For Firefox > Preferences > Advanced > Certificates > View. See that CNNIC one? What about the Hong Kong Post? Those are Known bad actors that your country is probably at cyber war with, and THEY ARE TRUSTED ROOTS IN YOUR FUCKING BROWSER?! Not to mention all the other Russian ones or Turkish, etc. ones that are on the USA's official "enemy" list. Now, ANY of those can pretend to be whatever domain's CA they want, and if your traffic bounces through their neck of the woods they can MITM you and you'll be n

    1. Re:Natural Born Cancer by Anonymous Coward · · Score: 1

      The fact that PFS wasn't put in at version 1.0 is just insane. The transaction should be completely ephemeral.

      Ideally, each root CA shouldn't be 100% trusted, but should have a trust value from 0 to 1, and a threshold of trust (where one might feel they are confident bank.com's website is really that at a threshold of 1, others require more than 2-3 root CAs agreeing on that.)

      However, just getting to the CAs and editing trust can be a problem. In iOS, there is no way to remove existing CAs, so if a root cert holder gets compromised, there is absolutely no way to rectify the issue until Apple issues an iOS patch. All other operating systems offer a way to disable this.

      So, SSL/TLS in itself has major weaknesses. The issue with Heartbleed is that it changes the attack surface from someone with the resources to pwn a CA to virtually anyone running a program.

    2. Re:Natural Born Cancer by Anonymous Coward · · Score: 0

      My custom VPN uses the HMAC key expansion protocol mentioned above.

      Any pointer to good solid open source implementation examples of what you describe?

  28. Re:His rant could apply to almost any large projec by PhrostyMcByte · · Score: 1

    This is security software. You don't sacrifice the library's core functionality to make it run a bit faster on the old Celeron 300 running Windows 98.

    malloc's core functionality is to allocate memory. Any security additions are platform-specific and irrelevant.

  29. Theo who? by Aethedor · · Score: 1

    Wasn't that the guy of the lamest vendor response in 2007? A little less harsh on your comment would be appropriate, mr. Theo.

    --
    It doesn't have to be like this. All we need to do is make sure we keep talking.
    1. Re:Theo who? by Anonymous Coward · · Score: 0

      Yes, you obviously don't know who Theo is if you're seeking a less harsh comment.
      Also, in order to bash Theo, you need to go back to a 2007 post that essentially criticizes the OpenBSD team for mis-categorizing their best guess about the effects of a bug that they already fixed? (Since this is related to how the patch was categorized, that means the fix was available. Personally, when I've fixed a bug, I'm happy to put it behind me. I don't usually get much of a kick out of playing around with previously broken code to see if I could develop an attack to justify more dire terms in my description.) That seems to be a pretty lame poke at Theo.

      You obviously know who Theo is (because the website you quote doesn't even mention Theo; it only mentions OpenBSD). So, I hereby say that you've just done a stupendously incompetent job at bashing Theo. Even I, a person who actually generally agrees with Theo, know that there's plenty of low-hanging weapons available to you. You wished to enjoy doling out criticism so much that you didn't even bother to do a decent job of it.

    2. Re:Theo who? by Aethedor · · Score: 1

      That escalated quickly...

      --
      It doesn't have to be like this. All we need to do is make sure we keep talking.
  30. Re:Who is Theo De Raadt? by KingOfBLASH · · Score: 3, Informative

    Theo De Raadt is the king of tinfoil hats, and behind OpenBSD -- a version of BSD designed to be as secure as possible.

  31. Re:wrong by Anonymous Coward · · Score: 0

    wrong

  32. Feeding Allo(g)ators by Anonymous Coward · · Score: 3, Interesting

    Allocators in this case make no significant difference with regards to severity of the problem.

    What is or is not in process free list makes no difference when you can arbitrarily request any block of memory you please.. only slightly effects chance of success when it becomes necessary to shoot in the dark. Lets not forget most OS provided optimized allocators keep freed memory in their heaps for some time as well and may still not throw anything when referenced.

    Looking at code for this bug I am amazed any of this garbage was accepted in the first place. There is no effort at all to minimize chance of error with redundant + 3's and 1 + 2's sprinkled everywhere complete with unchecked allocation for good measure.

    buffer = OPENSSL_malloc(1 + 2 + payload + padding);
     
    r = dtls1_write_bytes(s, TLS1_RT_HEARTBEAT, buffer, 3 + payload + padding);

    Suppose I should be glad 1 + 2 = 3 today and they have not used signed integers when dealing with lengths.

    unsigned int payload;
    unsigned int padding = 16; /* Use minimum padding */

    ... oh dear god ...

    int dtls1_write_bytes(SSL *s, int type, const void *buf, int len)

    Well at least they learned their lesson and have stopped sprinkling redundant and error prone type + length + padding garbage everywhere... see..

    + buffer = OPENSSL_malloc(write_length);
     
    - buffer, 3 + payload + padding,
    + buffer, write_length,

    and here ..

    + if (1 + 2 + 16 > s->s3->rrec.length)+ return 0; /* silently discard */
     
    + if (1 + 2 + payload + 16 > s->s3->rrec.length)+ return 0; /* silently discard per RFC 6520 sec. 4 */
     
    + if (1 + 2 + 16 > s->s3->rrec.length)+ return 0; /* silently discard */
     
    + if (1 + 2 + payload + 16 > s->s3->rrec.length)+ return 0; /* silently discard per RFC 6520 sec. 4 */

    ... oh well .. Looks like plenty of low hanging fruit to be had for anyone with a little spare time.

    1. Re:Feeding Allo(g)ators by phantomfive · · Score: 1

      they have not used signed integers when dealing with lengths.

      What's wrong with that?

      --
      "First they came for the slanderers and i said nothing."
  33. Re:His rant could apply to almost any large projec by phantomfive · · Score: 3, Informative

    To say that "OpenSSL is not developed by a responsible team" is simply nonsense.

    Unless you look at the code, and notice they are using unvalidated data for......anything. That's a rookie mistake.

    --
    "First they came for the slanderers and i said nothing."
  34. The chilling thing about Ted Unangst's analysis by OmniGeek · · Score: 5, Interesting

    As I read his analysis, OpenSSL relies on releasing a buffer, reallocating it, and getting the PREVIOUS contents of that buffer back -- or else it will abort the connection. (Search for the string "On line 1059, we find a call to ssl3_release_read_buffer after we have read the header, which will free the current buffer." in his article referenced by the parent post).

    Now, IMO, this goes way beyond sloppy. Releasing a buffer before you're done with it, and relying on a wacky LIFO reallocation scheme giving you back that very same buffer so you can process it, is either 1) an utterly incompetent coding blunder that just happened to work when combined with an utterly terrible, insecure custom allocation scheme, or 2) specifically designed to ensure that this insecure combination is widely deployed to provide a custom-made back door, as it works only with the leaky custom allocator.

    If 1), then I must agree with Theo that the OpenSSL team were indeed irresponsible, since at least one of these two cooperating blunders ought to have shown up in a decent security audit of the code, and any decent set of security-oriented coding standards would forbid them both.

    If 2), then it was deliberate, and the tinfoil-hat crowd is right for once.

    --

    "My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
    1. Re:The chilling thing about Ted Unangst's analysis by Anonymous Coward · · Score: 1

      If 2), then it was deliberate, and the tinfoil-hat crowd is right for once.

      At this point, it seems with enemies of security both in the criminal ranks trying to make millions off the money they can get from hacking millions of people and in the government/NSA ranks of paranoid techocrats looking to snoop on everyone I am surprised when software, both commercial and free, doesn't have a backdoor written into it.

    2. Re:The chilling thing about Ted Unangst's analysis by semi-extrinsic · · Score: 1

      If 2), then it was deliberate, and the tinfoil-hat crowd is right for once.

      Well, it would be one very inefficient way of introducing a backdoor. The custom allocator stuff in OpenSSL is very old, but the heartbleed bug is quite recent, so someone would have had to plant a backdoor-seed if you will and then wait for many years for that tree to grow fruits. Could be done, but it doesn't sound likely.

      --
      for i in `facebook friends "=bday" 2>/dev/null | cut -d " " -f 3-`; do facebook wallpost $i "Happy birthday!"; done
  35. Re:wrong by Anonymous Coward · · Score: 1

    No, they're correct.

    A compromised flash drive can infect an air-gapped computer (as could a usb keyboard that includes a hidden flash-drive).

    I suppose you could argue that connecting a flash drive is breeching the air-gap, but at that point you're coming very close to re-inventing the "if you never turn it on you can't get a virus" level of pointless pedantry.

  36. Why is important software still in C ? by presidenteloco · · Score: 0

    It seems to me there would be fewer of these reading wrong memory security violations if core OS utilities were re-written in, say, D, or Go, which have GASP! bounds checking, data types, and even memory management built in.

    Sure, there might be a bug in the language environment's implementation of one of these things, but if there is, it only has to be found and fixed in one place, and all derivative software will eventually pick up the fix. Oh and the bug will be instantly and universally famous, because a language is so widely used.

    Relying on old C code today seems like flying in a 1955 helicopter. Still works like a charm, til it doesn't.

    --

    Where are we going and why are we in a handbasket?
  37. Re:His rant could apply to almost any large projec by neaorin · · Score: 1

    If you absolutely need (and can afford) to roll your own, sure, no problem. But I'd say you need to make sure that you do "at least as good" of a job as the standard stuff, in the areas that are relevant to your software. In this case since you are writing a security library, you can't really ignore security additions to malloc etc. as irrelevant. You may want to either test on those specific platforms, or incorporate those additions into your custom built stuff. Because, again, you are writing a security library that's used by half the web. If you don't have the time and / or manpower for that, then essentially you are choosing between performance and security - and you are doing security so the choice should be obvious. The second argument is the rather obvious fact (by now) that memory management has had a significant impact in the general security (or lack thereof) of software. So you can't just dismiss it on the grounds that "it's just memory allocation" because it's not.

  38. Re:wrong by Anonymous Coward · · Score: 1

    When I was dealing with AD Certificate Services, I was told that the root should be disconnected and offline whenever not in use, but WTFDIK.

  39. Re:His rant could apply to almost any large projec by neaorin · · Score: 2

    Also, they didn't need to use OpenBSD's stuff - any standard malloc would have caused OpenSSL to abort the connection in this case, as Ted shows in his blog post.

  40. car analogy by spatley · · Score: 2

    mod parent up

  41. the RFC is horrible by DrProton · · Score: 4, Interesting

    Let's not miss the opportunity to point a finger of blame at the RFC, which says " to make the extension as versatile as possible, an arbitrary payload and a random padding is preferred, ". https://tools.ietf.org/html/rf... Arbitrary payload and a random padding for a heartbeat instead of a specified sequence of bits? This is very suspicious.

    --
    "Mit der Dummheit kaempfen Goetter selbst vergebens." - Schiller
    1. Re:the RFC is horrible by Anonymous Coward · · Score: 0

      Heck, my suggestion to minimize attacks (including DoS, known plaintext and would also have nipped this in the bud) would have the client send random bits of the amount they choose and then the server would respond with an equal amount of random bits.

      CAPTCHA is, fittingly, "mitigate."

    2. Re:the RFC is horrible by Anonymous Coward · · Score: 0

      It's even more suspicious when you consider the guy who wrote the buggy code was also one of the guys who wrote the RFC.

    3. Re:the RFC is horrible by Anonymous Coward · · Score: 0

      Unnecessarily passing around "random" bits is only going to leak information about the systems involved and deplete their available entropy. Why waste system entropy on something as mundane as a heartbeat?

    4. Re:the RFC is horrible by Anonymous Coward · · Score: 0

      Cryptologically arbitrary length with very high entropy is desirable to prevent analysis based on the heartbeat transaction.

  42. thanks by Anonymous Coward · · Score: 0

    THAT was a 5 second read.

  43. Yep by ledow · · Score: 2

    Can't say I'm surprised. OpenSSL is a pile of dung. It's nothing to do with being written in any language, it's just horrible.

    There's not even any documentation. I mean, literally, none. Nothing vaguely useful. How do I programmatically load a certificate into the store, along with a chain of related trusted certificates, and then set my requirements (must be in-date, must be validly signed, etc.) and get out a "It's fine" / "Something's not right" response? The only answers I could ever find were to follow published examples and tweak.

    And when it comes to working out where in the published examples structure X comes from, or how to convert it to structure Y, you're on your own unless you happen to have picked a comprehensive (and almost certainly not OpenSSL-supplied) example.

    It's just that bad. I was writing a pseudo-DRM for a game / Steam-like distribution platform as a hobbyist project. It was literally horrible to even try to self-sign some certificate and then see if it all panned out later from another computer to guarantee integrity. In the end, I had to "imagine" every possible case and find a way to counter it (i.e. client cert expired, client cert invalid, server cert not signed client cert, server cert has bad chain of trust, client cert not signable for that purpose, etc.) - and almost always there was NOTHING to indicate what the recommended way to do it was.

    There is no decent OpenSSL documentation at all. Not even a decent overview of the process of checking certificates. It scared me at the time, knowing how important the library is, and it can only lead to bad code.

    In the end, I'm quite glad I don't have to program against it for a living. If I did, I'd be seriously looking for something else.

    1. Re:Yep by GodfatherofSoul · · Score: 1

      The code snippet you see in the summary is the same kind of thing you see in libraries all over the internet in all kinds of projects. Programmers simply don't consistently follow safe memory management conventions yet will chest-thump over how beneath them managed languages are. Poor documentation also accompanies the same "just get it working" culture.

      --
      I swear to God...I swear to God! That is NOT how you treat your human!
    2. Re:Yep by dfsmith · · Score: 1

      When I programmed against OpenSSL, first I made my own API wrapper so I'd never have to read the OpenSSL doc(s) again....

    3. Re:Yep by Anonymous Coward · · Score: 0

      There's not even any documentation. I mean, literally, none.

      Lies http://www.openssl.org/docs

      Nothing vaguely useful. How do I programmatically load a certificate into the store

      You copy them to the same directory as all other trusted certificates. Loaded by http://www.openssl.org/docs/ss...

      Cert filename you will use is keyed to hash of cert (e.g. openssl x509 -hash ...)

      and then set my requirements (must be in-date, must be validly signed, etc.) and get out a "It's fine" / "Something's not right" response?

      If you want custom behavior during chain validation you need to create your own verify callback.
      https://www.openssl.org/docs/s...

      Otherwise see here to set validation parameters:
      https://www.openssl.org/docs/s...

      It's just that bad. I was writing a pseudo-DRM for a game / Steam-like distribution platform as a hobbyist project. It was literally horrible to even try to self-sign some certificate and then see if it all panned out later from another computer to guarantee integrity. In the end, I had to "imagine" every possible case and find a way to counter it (i.e. client cert expired, client cert invalid, server cert not signed client cert, server cert has bad chain of trust, client cert not signable for that purpose, etc.) - and almost always there was NOTHING to indicate what the recommended way to do it was.

      The above API calls handle trust chain validation. If you also need to query cert parameters (name validation) see https://www.openssl.org/docs/s...

      There is no decent OpenSSL documentation at all. Not even a decent overview of the process of checking certificates. It scared me at the time, knowing how important the library is, and it can only lead to bad code.

      Having visited several areas of OpenSSL API where there is no documentation I agree it is not complete or the best but hardly terrible or non-existent.

      Those places most lacking documentation are almost always very specialized where anyone going there should expect to have to read example/test code. Examples you have provided for lack of documentation appear to be relatively trivial, well traveled and relatively well documented.

      It scared me at the time, knowing how important the library is, and it can only lead to bad code.

      I think there are dangerous and stupid things in OpenSSL like this business of having to register locking primitives to protect the session cache which I have seen take a number of developers both in commercial and open source projects by surprise.

      Having looked at the "heartbleed" patch and surrounding code portions very much seem to be full of questionable practices. But hey its open source .. we can all work to fix the problems if we feel they are worth our time to fix. From my perspective everyone is better off if those of us able to step up and fix what is broken rather than or at least in addition to throwing stones.

  44. "under-resourced": I call bullshit by Anonymous Coward · · Score: 0

    Then maybe you thousands should stop complaining and start contributing to the project, which is so under-resourced problems like this are pretty much inevitable.

    Hey, some of us really are moved by this, but not quite in the way you might expect. It made me realize it's been a long, long time since I gave money to OpenBSD, so I just set up a new recurring donation. Ted Unangst's writeup proves that this was no mere bug; it was sort of like the Challenger investigation, where you see that a major fuckup was pretty much inevitable, because somebody didn't give a fuck. We all make mistakes (mine are horrible too), but the failure to even do basic testing with the OPENSSL_NO_BUF_FREELIST option flipped either way, is not a mere mistake.

    I can't trust OpenSSL after this, because I know that beyond the topic of bugs and mistakes, that team gives less than 1.0 fucks.

    Yet there are teams out there who do give two fucks, such as the OpenBSD team.

    So maybe if I keep throwing money at them, they will keep on generating great code (like openssh, for example) that I'll eventually get to use in my own Linux (not even OpenBSD!) systems. And then some day I won't have to have OpenSSL installed at all. And resources are not OpenSSL's problem, any more than NASA's resources were a problem on January 28, 1986.

  45. Re:Unfortunately, this analysis seems to be spot-o by IamTheRealMike · · Score: 1

    Much though I love NSA related conspiracy theories, especially lately, I think "the NSA writes a pile of crap and gives it away for free in the hope it becomes inexplicably popular" is perhaps not the best one available. OpenSSL has been around for a loooong time with virtually no resources put into it, which is one reason it sucks. The other reason being that the original author wrote OpenSSL in order to teach himself C (and it shows).

    Recall that SSL was not very widely used up until a few years ago, and it's only in the last 18 months that suddenly every man and his dog wants a secure website. It's not surprising that core libraries that do it are subpar. Even very large companies like Google or Microsoft have typically only had one or two people who really understood and cared about SSL.

  46. Re:Unfortunately, this analysis seems to be spot-o by gweihir · · Score: 1

    Well, true, it could still be plain old incompetence. I am not disputing that. It is pretty gross incompetence in that case though.

    --
    Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
  47. Re:His rant could apply to almost any large projec by Anonymous Coward · · Score: 0

    A lot of large performance-sensitive projects implement custom allocators in the form of arenas and freelists. Lots of platforms have a fast malloc implementation these days, but none of them will be as fast as this for the simple reason that the program knows more about its memory usage patterns than any general-purpose allocator ever could.

    ...

    And good lord the results are generally execrable. Doing something stupid like, for example, writing and using OPENSSL_malloc() instead of just malloc() means you can't use plug-in memory debugging/problem-finding tools like Valgrind and Purify. Eliminating the ability to use a whole class of tools that provide a huge amount of testing and validation capability is not what I'd characterize as a smart choice. Just use the normal calls malloc()/free()/etc and change the libraries you link to.

    And get over yourself - you're not going to write a memory allocator better than what comes with your OS. If your implementation is faster, it's likely only because you've eliminated the locking implicit in most standard library malloc() implementations.

    And even then no memory allocator is faster than using local variables on the stack. Zero performance cost - the stack pointer is incremented anyway. No locking - each thread owns its own stack.

    If you know your application's memory usage patterns well enough to be able to write your own memory allocator, you'd probably be better served just using local variables on the stack. Need a big stack? See "man pthread_create". Have fun; learn something.

    And if speed is so damn important that system-supplied malloc() and free() aren't fast enough for you and you can't run thread(s) on custom-sized stacks, rethink your damn design. Don't call malloc()/free() or new/delete so many damn times. And remember, it's gonna be a helluva lot cheaper to license something like Hoard than it would be to write your own. Not only cheaper, you won't do it as well, your implementation won't be as fast, and your implementation will definitely be buggier.

    But hey, you're a DEVELOPER, and for every nail that's a problem your hammer is to WRITE MORE CODE!!!!

  48. OpenSSL is a POS by Anonymous Coward · · Score: 1

    As someone who had to program communication over SSL connection using just C++ and OpenSSL, I'd like to chime in and say -- OpenSSL is a piece of shit. It's API is horrible, error messages/codes are useless and there is no decent instrumentation to investigate (and understand) problems. Not surprised at all that it sucks on implementation level too. The sooner word replace it with a decent library -- the better.

  49. Theo .. Rant by nurb432 · · Score: 1

    So what is new?

    --
    ---- Booth was a patriot ----
  50. Is OpenBSD safe? by metrix007 · · Score: 0

    Is the alleged most secure operating system safe, or just as vulnerable as every other OS?

    --
    If you ignore ACs because they are anonymous - you're an idiot.
  51. Reasons to avoid openssl by Anonymous Coward · · Score: 0

    1. No documentation. None. Nicht. Zero. Nada.

    2. Valgrind goes crazy unless you compile OpenSSL with special flags.

    3. (subjective) APIs suck

    And one reason to use it: Nothing better available.

  52. Welcome to 2014 by Anonymous Coward · · Score: 0

    Both Verisign and Thawte are brands own by Symantec

  53. Test servers your self with PoC by psyclone · · Score: 1

    Test your servers yourself: https://gist.github.com/takesh... This was a server-side attack so clients are unaffected.

    1. Re:Test servers your self with PoC by MightyMartian · · Score: 1

      I know my webservers are all good, because they're linking against openssl 0.9.8. I just managed to confirm that Debian Squeeze's stock OpenVPN package links to the 0.9.8 library as well, and isn't statically linked, so, so far as I understand the vulnerability, there's no chance I was compromised.

      It does indeed pay, on occasion, to stick with older versions. I had actually been looking to upgrade my VPN gateways to Wheezy a few months ago, and am rather glad I didn't.

      --
      The world's burning. Moped Jesus spotted on I50. Details at 11.
    2. Re:Test servers your self with PoC by _merlin · · Score: 2

      That's not true. A malicious server can exploit a vulnerable client and read the contents of its memory.

    3. Re:Test servers your self with PoC by psyclone · · Score: 1

      How? The client also responds to the heartbeat?

    4. Re:Test servers your self with PoC by _merlin · · Score: 1

      Yes, heartbeats are symmetrical. The malicious server can only read the contents of the browser's memory, but that could have your banking details, your Facebook login cookies, your wife's nude selfies, or God knows what else considering everything done through web web browsers these days.

  54. Pay The OSSL Dev's by Anonymous Coward · · Score: 0

    In Bitcoins! Then they can hire more dev's and pay them with Bitcoins, new dev's that can help tidy up code that go where it should when there's a problem.

  55. Re:His rant could apply to almost any large projec by udippel · · Score: 1

    What's your intention with your statement? Or didn't you read the whole lot?
    There are a number of angles to it; Theo mentioned one only.
    Someone who releases a buffer, and then rereads it, hoping to get the same data commits definitively a very major and unforgivable error. And then plug it into one of the basic security-relevant software without testing is irresponsible.

  56. Re:His rant could apply to almost any large projec by Anonymous Coward · · Score: 0

    Any security additions are platform-specific and irrelevant.

    Funny, considering that the developers of every major OS would disagree with you, especially after countless bugs and security holes have shown that letting a process have free reign over memory that it does not own (page management, virtual address space), that it never marked as containing executable code (data execution prevention) or that it just told the library that it will no longer touch (undefined behavior) is a really bad idea.

    Each of these security features became a necessity after countless C developers repeatedly showed the world that just managing memory was not enough to handle the huge amount of security and trust problems an OS and the programs that run on it have to deal with. Having a security library bypass these necessary extensions under the guise of "lawl we are the uber 1337 C haxx0rs, fuck all those silly access and overflow checks we wanna have speed 111!!!!111!!".

  57. Re:Who is Theo De Raadt? by Anonymous Coward · · Score: 0, Informative

    Doesn't affect OpenSSH, because it's openSSL is used for key generation, not SSL in the context of openSSH.

  58. That's not how it works by Shuntros · · Score: 1

    Certificate Authorities who operate on the scale absolutely do NOT keep private keys of the issuing intermediate available for harvest. That's what HSMs are for; devices which hold the private key material and perform signing operations on behalf of the CA. The CA can never retrieve the private key(s) so compromising the CA in that scenario should never result in private key disclosure.

    1. Re:That's not how it works by Antique+Geekmeister · · Score: 1

      > Certificate Authorities who operate on the scale absolutely do NOT keep private keys of the issuing intermediate available for harvest.

      And they don't put passwords on post-it notes or send them in plain text email, because they all know better, right? Or leave them on backup tabpes or recorded in Wikis?

      I'm afraid that I find your confidence in developer and corporate IT to be unfounded.

  59. Re:His rant could apply to almost any large projec by pnutjam · · Score: 1

    if 2+2=5, then 5=4

  60. Re:Who is Theo De Raadt? by ppanon · · Score: 3, Insightful

    Theo has been a strong proponent of hard hats/steel helmets, not tinfoil hats. Most people thought that approach was overkill for general walking around, at least up until Snowden showed there were lots of national actors firing lots of rocks into the air.

    --
    Laissez lire, et laissez danser; ces deux amusements ne feront jamais de mal au monde. - Voltaire
  61. Re:wrong by Anonymous Coward · · Score: 0

    (And there are proof-of-concept worms that can jump the air gap via audio once the initial infection is accomplished.)

  62. And are these any better - or just different? by slincolne · · Score: 1

    Can anyone offer advice on these? Are they just something diferent, or are they significantly better than OpenSSL?

  63. SSL in OpenBSD by Thiarna · · Score: 1

    What does OpenBSD use for SSL by default - does it have it's own implementation or is it affected by the OpenSSL bug?

  64. Bingo by Anonymous Coward · · Score: 0

    Computers cannot be properly secured.

  65. All by design by Anonymous Coward · · Score: 0

    Ciphers are supposed to be so fucking complex that all implementations will be impossible to do securely.

    Our minds have been pwned by some really smart admirals and generals. They often convene in Maryland.

  66. Here's a protip by Anonymous Coward · · Score: 0

    Rip out all the SSL crapola and roll your own minimalist 3DES-based block cipher crypto. It will definitely be more secure against NSA and the Russkie mafia. And the Nork long-range recon forces.

  67. Waking up is hardest by Anonymous Coward · · Score: 0

    Computors are not supposed to be secure from inspection by the goons of 1%.

  68. If you actually try to do this by Anonymous Coward · · Score: 0

    ...please do all the development and the proofs clandestinely. Post on usenet signed using gpg. Otherwise your great code will soon be "improved" by some people who "join your team" and "make mistakes".

  69. If he were by Anonymous Coward · · Score: 0

    ...he would not use C. He merely has some stricter standards in some aspects.

  70. It was quite obvious a long time ago by Anonymous Coward · · Score: 0

    There was this random-source bug with OpenSSL not long ago. I looked at the code then and was horrified.

    Now, I do think that TPTB really don't want this kind of thing to be secure. It is obvious that defensively coded C++ would be way more robust and would have a lesser number of bugs.

    1. Re:It was quite obvious a long time ago by gweihir · · Score: 1

      C++ allows you do make an even less transparent mess. That language is a violation of almost any principle of language design in itself. Defensively coded C would be perfectly fine. As in using all the things on how to do secure software engineering that have been known for decades.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
  71. Some other people by Anonymous Coward · · Score: 0

    ...have more dollars and more bitcoins.

  72. If you think about this by Anonymous Coward · · Score: 0

    ...the thing opens a capability to folks who have the best minds, lots of money and lots of time at their hands. Say, computer scientists and mathematicians. Not Kayleen F Hacktress of 17 years living in her bedroom in the same house as her partents.

    Easy enough for pros, too difficult for the amateurs. Malice or incompetence ?

  73. Let me say this by Anonymous Coward · · Score: 0

    A) I was blasting M$ in the past; they deserved it. Monopoly, lazy coding, greed,....
    B) I am aware of the HPUX ping of death thing
    C) By growing older, I also grew much more cynical. FOSS and Unix have serious problems.

  74. I think he's on to something by Anonymous Coward · · Score: 0

    Many people have forgotten a cardinal rule of software development:

          First you make it right. Then you make it fast.

    Implied is, you never sacrifice the former for the latter. Performance is always good but you don't want to start making significant sacrifices on functionality, correctness, code readability, documentation, etc. And, as others here have pointed out, when you subvert the primary purpose and mandate of the software you have created, no matter how that happened, time for a rethink.

  75. Who is responsible? by Anonymous Coward · · Score: 0

    OK, so an open-source program that is used by almost EVERYONE had a bug. Who is accountable for this problem? The last programmer that touched the code? The lead programmer? Who?

  76. Re:wrong by Anonymous Coward · · Score: 0

    go back to massively parallel abaci; it's the only solution.

  77. A Boring Article About a Check of the OpenSSL Proj by Andrey_Karpov · · Score: 1

    Some time ago, a vulnerability was revealed in OpenSSL, and I guess there's no programmer who hasn't been talking about it since then. I knew that PVS-Studio could not catch the bug leading to this particular vulnerability, so I saw no reason for writing about OpenSSL. Besides, quite a lot of articles have been published on the subject recently. However, I received a pile of e-mails, people wanting to know if PVS-Studio could detect that bug. So I had to give in and write this article: http://www.viva64.com/en/b/025...