Slashdot Mirror


2014: The Year We Learned How Vulnerable Third-Party Code Libraries Are

jfruh writes Heartbleed, Shellshock, Poodle — all high-profile vulnerabilities in widely used libraries that rocked the software industry in 2014. Sadly, experts are now beginning to believe that these aren't the only bugs lurking out there in widely used open source code, just the ones that grabbed the most attention. It's beginning to look like one of the foundation concepts of open source — that with enough eyes, all bugs are shallow — is a myth. Of course, probably no one believes that all bugs are instantly shallow, no matter how open is the source, or that open source software is immune from bugs -- particularly ESR, coiner of the phrase.

25 of 255 comments (clear)

  1. But *are* there enough eyes? by tj2 · · Score: 5, Insightful

    The phrase might be true, but we're seeing the effects of insufficient eyes. In reality, how many sets of eyes are actually reviewing these libraries at a source code level? I rather strongly suspect that in most cases they are simply used under the assumption that "well, everyone uses it, it must be okay".

    1. Re:But *are* there enough eyes? by TheRaven64 · · Score: 4, Insightful

      Exactly. The problem with OpenSSL was that everyone used it, no one looked inside it. Those of us who did look inside it recoiled in horror and didn't do proper code review because 'kill it with fire' was the immediate reaction for anyone looking at the code. Open source isn't magic. People who build their business on a third party library have a lot of choice in terms of who provides the support (bug fixing, code auditing, and so on) for that library. Far more than with a proprietary library, where you're locked into a single source. If everyone choses to get support for the product using the power of wishful thinking, then don't be surprised if the code is crap.

      --
      I am TheRaven on Soylent News
    2. Re:But *are* there enough eyes? by tj2 · · Score: 5, Insightful

      This is just FUD. Whatever the number of the eyes, they are certainly far more than open source. I have already contributed many bug reports and often fixes, would you care to elaborate how I would do that in a closed source model? Because I am *very* curious.

      FUD? Sure, there are *more* eyes in open source than closed source: that's not the point. Are there *enough* eyes to prevent potentially catastrophic bugs from being exploited? I'd submit that we're seeing that there isn't. I'm not suggesting that closed source is superior, but let's not confuse some sort of moral superiority being attributed to open source as being equivalent to automatic technical superiority. In most cases, I'd agree that open source has technical superiority, but it's not automatic.

    3. Re:But *are* there enough eyes? by antientropic · · Score: 4, Interesting

      A long time ago, I saw Bertrand Meyer (the Eiffel guy) give a keynote at ICSE, where he pointed out that the "given enough eyeballs, all bugs are shallow" claim is unscientific, because it can't be falsified: if a bug is not found, people can always say that there were not enough eyeballs, so "Linus' Law" still holds.

    4. Re:But *are* there enough eyes? by ledow · · Score: 4, Informative

      Amen.

      I touched OpenSSL once. It was a nightmare. No idea where or how it passed anything as it wasn't at all clear the path that simple things, like certificate checking, were supposed to take.

      In the end, I hacked onto it rather than play with it. The documentation was non-existent. The code samples were incomplete and with almost zero explanation of what you were supposed to be checking for and where things COULD go wrong. Hence 90% of the code I see that touches OpenSSL looks exactly like the samples and nothing more.

      All I wanted to do was have two x509 certificates, and check that both were valid and one properly signed the other, as part of a primitive DRM scheme I was toying with. It turned into a nightmare scenario of IMAGINING every possible outcome and specifically coding for each one, rather than anything sensible.

      I don't think I'd ever touch it again, and was not at all surprised that there were problems with it. I was more surprised that others had had the same problems, yet OpenSSL was still regarded as the "gold standard" library to integrate with.

    5. Re:But *are* there enough eyes? by rapiddescent · · Score: 5, Interesting

      one of the issues is that there are indeed *more eyes* but they are incentivised to look for exploits and sell them to the bug-buyers rather than report or fix them. I did a hands up poll (buyer beware) at our local OWASP chapter and over half had sold a bug to such an organisation. pretty shocking.

      certainly, one of the first moderately important bugs I found, I was daft and got in touch with the software vendor and then faced legal action from them which luckily they saw sense and dropped. So many people nowadays just can't be bothered with that problem and can make a fast and low risk buck by selling the 0-day.

    6. Re:But *are* there enough eyes? by ledow · · Score: 4, Informative

      "once" = several weeks of fighting with the damn thing to do one simple task, clearly specified, that's way within it's scope.

      There is no serious documentation. The examples are given as documentation and are vastly incomplete.

      It's not a question of "glanced at it, it was horrible", but for anything serious even a quick glance will show whether or not it's a nicely-produced library or not. One quick glance at a library is normally all I do in order to get a handle on whether it's good enough and clean enough for me to program against.

      The problem with OpenSSL was that the only people who knew how it worked never bothered to simplify that or document that enough. There is no "is this certificate valid" function, that returns a enum from a list of potential problems (CERT_EXPIRED, CERT_NOT_YET_VALID, CERT_CORRUPT, CERT_UNTRUSTED, CERT_INSECURE, etc.), for instance. There are lots of things that LOOK like that, but none actually do it in OpenSSL.

      Given that it's a library who's primary purpose is - given a configuration of particular algorithms and keys - to produce a encrypted bitstream from an unencrypted one (or vice versa), it's suprisingly complex to do anything simple with any guarantee that you're doing it right.

    7. Re:But *are* there enough eyes? by BasilBrush · · Score: 3

      That entirely depends on the company and the seriousness of the defect.

      You're even more likely to get ignored by everyone if reporting a defect on an open source project. Mostly they'll expect you to fix it yourself or wait until someone with the appropriate skills takes an interest. And of course usually almost no-one that experiences the bug has the skills and knowledge of the particular project's internals to be able to fix it.

    8. Re:But *are* there enough eyes? by BasilBrush · · Score: 3

      The only positive there for open source is the visibility. With the defect report as with the source. It's certainly not any more likely to be dealt with in a timely way, or at all with open source. People being paid is the best way to get uninteresting bugs fixed in a timely way, and that happens a lot more often with commercial software.

  2. Libraries? by fnj · · Score: 3, Interesting

    Shellshock did not affect a "library", but an executable.

  3. So if I'm understanding this correctly by Ukab+the+Great · · Score: 3

    the big news is that people are now thinking that bugs in software is big news.

  4. Magic ball prediction - 2015 by sinij · · Score: 5, Informative

    My magic 8 ball tells me that in 2015 we will learn that proprietary and embedded software is even more vulnerable. My Tarot Card deck tell me that we will see a lot of hacked car wrecks in 2015, now that Volvo released the demon by putting a web browser into in-dash system. Rest of the lemmings are sure to follow. Not that you really need a browser to pwn a car, with Bluetooth-to-CAN-BUS exploits shutting down cars demonstrated as early as 2012.

    1. Re:Magic ball prediction - 2015 by Qzukk · · Score: 3, Funny

      requires physical access to the car.

      Good thing my car stays in a locked server rack in a room with a security guard posted at the door requiring a finger print and an RFID card to access.

      --
      If I have been able to see further than others, it is because I bought a pair of binoculars.
  5. Re:um what? by meta-monkey · · Score: 3, Insightful

    Not only can they fix them, but they do fix them. As soon as these vulnerabilities were discovered (by non-malicious actors) patches were available within days or even hours. Commercial vendors take their good sweet time. Or in the case of Microsoft, discontinue support for the still widely-used Windows XP. Find a vulnerability in that? Too damn bad. It'll never get fixed.

    Nobody's saying open source software is bug-free or even necessarily has fewer bugs than closed source software. It's what happens when a bug is discovered that makes the difference.

    --
    We don't have a state-run media we have a media-run state.
  6. Re:not just many eyes by plover · · Score: 4, Interesting

    The security of the open source model isn't really the problem or the answer here. The problem is homogeneity. A million different sites and applications rely on just a few libraries, so that when a bug hits one, it has massive impact on the entire internet.

    We also know that the answer isn't in rolling your own security. Very few people or organizations are likely to be able to securely implement their own version of TLS. Even the best packages of today didn't start out perfect, they had to iterate through several flaws to get to where they are today.

    So perhaps the better answer is in having more packages to choose from? Instead of picking just openssl by default, it would be better to have a broad array of choices. With a dozen packages on the market, that might mean 11 times out of 12 the bad guys wouldn't exploit our site. If the packages are interchangeable, we'd be better positioned to switch them quickly in case of emergency.

    --
    John
  7. Misunderstood by TopSpin · · Score: 3, Interesting

    ESR's claim has nothing to do with the frequency or discovery of bugs. All he says is that given enough observers, bugs are quickly characterized. It is implied that any given bug has already been discovered. There is no benevolent cohort of experts continuously auditing code bases and his statement doesn't claim there is.

    --
    Lurking at the bottom of the gravity well, getting old
  8. Re: not just many eyes by allan572 · · Score: 4, Informative

    So all we need are 11 more sets of programmers to program free version of SSL 2-12?

  9. No, that is false by Sycraft-fu · · Score: 3, Insightful

    An open source project can have as few as just one set. There are some projects that nobody other than the developer ever contributes to. Just one guy occasionally working on some little project that some people use. Those all those people COULD look at the code, they don't.

    Likewise commercial firms can, and sometimes do, pay many people to look at the code. In addition to having a big development staff they can have dedicated QA staffs. They can have a person, or many people, who's job it is literally to sit and look over the code for security issues every single day.

    You are buying in to the same fallacy that the article is talking about: That because something is open is just means that more people MUST be looking at it. No it means people can look at it, but they may choose not to.

  10. Re:not just many eyes by binarylarry · · Score: 3, Insightful

    So "Security through complexity" then?

    Sounds complex.

    --
    Mod me down, my New Earth Global Warmingist friends!
  11. Re: not just many eyes by plover · · Score: 3, Interesting

    So all we need are 11 more sets of programmers to program free version of SSL 2-12?

    Yes, and demand for them. But the big problem you're correctly implying is there's no economic justification that will drive this behavior. Maybe it will take a dozen big companies and foundations to drive this. Imagine if IBM, Microsoft, Google, RedHat, Yahoo, HP, Dell, Apache, Wikimedia, Mozilla, FSF, Apple, Intel, AMD, nVidia, Bungiesoft, and others each contributed their own versions of openSSL; each written in their own choice of language, using their own code, and building their own implementations of everything from the crypto through the command line interpreter logic. My company may decide we do more business with Intel, so we choose theirs. Or your company may be more Apple focused, so you'd choose theirs. In every case, we'd all nervously watch each other looking for signs of intrusions, hoping we won't be the victims, but knowing that alternatives exist if we are.

    While a 1/12th scale incident of Heartbleed is still a huge problem for a lot of companies, it's no longer the catastrophe-sized disaster that Heartbleed actually was.

    --
    John
  12. Make that THREE other things by calidoscope · · Score: 4, Interesting

    3) Port to to multiple architectures (and OS's) to catch bugs not reported by the original build environment. This is one of the approaches OpenBSD uses to improve security and was quite common in the open source software world when ESR coined the phrase.

    The OpenBSD team found one very long lasting (30+ years) bug in the legacy BSD code when the Sparc64 build barfed.

    --
    A Shadeless room is a brighter room.
  13. Re: not just many eyes by dkman · · Score: 4, Insightful

    Sadly we humans only seem to be able to handle 2 or 3 options. If 12 existed we'd hone in on 3 favorites and 9 would be outliers.

    It's not that just "being open source" automatically means code is being validated by lots of eyes. It means that you can look at the code. All we need is more people interested in doing that, or paid to do so. They also need to have the knowledge/skill necessary to do that.

    And as always, being closed source would not have made the issues easier to find. And then you'd be at their mercy waiting for a fix. These were all found and all fixed relatively quickly, so let's focus on that.

    SSL certainly isn't a simple library. Increased complexity makes it easier to make a mistake and harder to find it.

    --
    I refuse to sign
  14. Re:not just many eyes by TechyImmigrant · · Score: 4, Insightful

    >We also know that the answer isn't in rolling your own security. Very few people or organizations are likely to be able to securely implement their own version of TLS.

    TLS is the issue. It isn't simple. It isn't even secure in many compliant configurations. It invites implementation errors.
    A good spec of a secure protocol would make secure implementations easy. If you don't think about the implementability while you're writing a spec, you're doomed, like TLS is.

    "Don't roll your own security" is advice aimed at people who don't know about security. Some of us have to implement and 'roll' the specs. The world looks different when your reputation is tied to your stuff not get broken before senility sets in. You can do it right, but you need all the elements in place including a well thought out spec.

    --
    I should use this sig to advertise my book ISBN-13 : 978-1501515132.
  15. Re: not just many eyes by drinkypoo · · Score: 3, Insightful

    Microsoft, Google, RedHat, Yahoo, HP, Dell, Apache, Wikimedia, Mozilla, FSF, Apple, Intel, AMD, nVidia, Bungiesoft, and others each contributed their own versions of openSSL; each written in their own choice of language, using their own code, and building their own implementations of everything from the crypto through the command line interpreter logic.

    ...and then, due to the nature of software today, you wind up with all of these in your organization, and now if any of them are compromised you have a problem. I'd rather engineers from all of those organizations worked on OpenSSL (or whatever) to make it as secure as possible, perhaps forking it occasionally to solve some particular problem and then merging the code later.

    --
    "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
  16. Mo Code, Mo Problems by TiggertheMad · · Score: 4, Insightful

    I agree that more diversity in the software ecosystem will cause critical bugs to have less impact to the world overall, and will hopefully drive competition to make the offerings more efficient and stable. However I think that this is a straw-man and the real conclusion we should draw is this:

    When you write code, you are going to screw up. If you aren't writing bugs that people notice, you aren't working on anything worthwhile. While the bugs that were found were costly and dangerous, the question is were these found quicker than a closed source solution? Were they fixed faster than a closed source solution? Is there anything that can be done to allow quicker roll back or disabling of vulnerable features? When you write code, you need to design for failure, because it will happen and plan so that the recovery will be as quick as possible.

    Adding additional software library offerings will only add stability in the sense that one particular vector wont affect as much of the Internet, but you introduce more surface area for attackers to poke at, and more vulnerabilities overall. Given the challenges to write really solid code, I think I'd like to have fewer, but really well vetted open source software solutions. Of course, I am not correct in this opinion, as there are no 'right' decisions here.

    --

    HA! I just wasted some of your bandwidth with a frivolous sig!