2014: The Year We Learned How Vulnerable Third-Party Code Libraries Are

jfruh writes Heartbleed, Shellshock, Poodle — all high-profile vulnerabilities in widely used libraries that rocked the software industry in 2014. Sadly, experts are now beginning to believe that these aren't the only bugs lurking out there in widely used open source code, just the ones that grabbed the most attention. It's beginning to look like one of the foundation concepts of open source — that with enough eyes, all bugs are shallow — is a myth. Of course, probably no one believes that all bugs are instantly shallow, no matter how open is the source, or that open source software is immune from bugs -- particularly ESR, coiner of the phrase.

Comments

But *are* there enough eyes?

[tj2]

The phrase might be true, but we're seeing the effects of insufficient eyes. In reality, how many sets of eyes are actually reviewing these libraries at a source code level? I rather strongly suspect that in most cases they are simply used under the assumption that "well, everyone uses it, it must be okay".

Re:But *are* there enough eyes?

[TheRaven64]

Exactly. The problem with OpenSSL was that everyone used it, no one looked inside it. Those of us who did look inside it recoiled in horror and didn't do proper code review because 'kill it with fire' was the immediate reaction for anyone looking at the code. Open source isn't magic. People who build their business on a third party library have a lot of choice in terms of who provides the support (bug fixing, code auditing, and so on) for that library. Far more than with a proprietary library, where you're locked into a single source. If everyone choses to get support for the product using the power of wishful thinking, then don't be surprised if the code is crap.

Re:But *are* there enough eyes?

[tj2]

This is just FUD. Whatever the number of the eyes, they are certainly far more than open source. I have already contributed many bug reports and often fixes, would you care to elaborate how I would do that in a closed source model? Because I am *very* curious.

FUD? Sure, there are *more* eyes in open source than closed source: that's not the point. Are there *enough* eyes to prevent potentially catastrophic bugs from being exploited? I'd submit that we're seeing that there isn't. I'm not suggesting that closed source is superior, but let's not confuse some sort of moral superiority being attributed to open source as being equivalent to automatic technical superiority. In most cases, I'd agree that open source has technical superiority, but it's not automatic.

Re:But *are* there enough eyes?

[antientropic]

A long time ago, I saw Bertrand Meyer (the Eiffel guy) give a keynote at ICSE, where he pointed out that the "given enough eyeballs, all bugs are shallow" claim is unscientific, because it can't be falsified: if a bug is not found, people can always say that there were not enough eyeballs, so "Linus' Law" still holds.

Re:But *are* there enough eyes?

[ledow]

Amen.

I touched OpenSSL once. It was a nightmare. No idea where or how it passed anything as it wasn't at all clear the path that simple things, like certificate checking, were supposed to take.

In the end, I hacked onto it rather than play with it. The documentation was non-existent. The code samples were incomplete and with almost zero explanation of what you were supposed to be checking for and where things COULD go wrong. Hence 90% of the code I see that touches OpenSSL looks exactly like the samples and nothing more.

All I wanted to do was have two x509 certificates, and check that both were valid and one properly signed the other, as part of a primitive DRM scheme I was toying with. It turned into a nightmare scenario of IMAGINING every possible outcome and specifically coding for each one, rather than anything sensible.

I don't think I'd ever touch it again, and was not at all surprised that there were problems with it. I was more surprised that others had had the same problems, yet OpenSSL was still regarded as the "gold standard" library to integrate with.

Re:But *are* there enough eyes?

[rapiddescent]

one of the issues is that there are indeed *more eyes* but they are incentivised to look for exploits and sell them to the bug-buyers rather than report or fix them. I did a hands up poll (buyer beware) at our local OWASP chapter and over half had sold a bug to such an organisation. pretty shocking.

certainly, one of the first moderately important bugs I found, I was daft and got in touch with the software vendor and then faced legal action from them which luckily they saw sense and dropped. So many people nowadays just can't be bothered with that problem and can make a fast and low risk buck by selling the 0-day.

Re:But *are* there enough eyes?

[ledow]

"once" = several weeks of fighting with the damn thing to do one simple task, clearly specified, that's way within it's scope.

There is no serious documentation. The examples are given as documentation and are vastly incomplete.

It's not a question of "glanced at it, it was horrible", but for anything serious even a quick glance will show whether or not it's a nicely-produced library or not. One quick glance at a library is normally all I do in order to get a handle on whether it's good enough and clean enough for me to program against.

The problem with OpenSSL was that the only people who knew how it worked never bothered to simplify that or document that enough. There is no "is this certificate valid" function, that returns a enum from a list of potential problems (CERT_EXPIRED, CERT_NOT_YET_VALID, CERT_CORRUPT, CERT_UNTRUSTED, CERT_INSECURE, etc.), for instance. There are lots of things that LOOK like that, but none actually do it in OpenSSL.

Given that it's a library who's primary purpose is - given a configuration of particular algorithms and keys - to produce a encrypted bitstream from an unencrypted one (or vice versa), it's suprisingly complex to do anything simple with any guarantee that you're doing it right.

Magic ball prediction - 2015

[sinij]

My magic 8 ball tells me that in 2015 we will learn that proprietary and embedded software is even more vulnerable. My Tarot Card deck tell me that we will see a lot of hacked car wrecks in 2015, now that Volvo released the demon by putting a web browser into in-dash system. Rest of the lemmings are sure to follow. Not that you really need a browser to pwn a car, with Bluetooth-to-CAN-BUS exploits shutting down cars demonstrated as early as 2012.

Re:not just many eyes

[plover]

The security of the open source model isn't really the problem or the answer here. The problem is homogeneity. A million different sites and applications rely on just a few libraries, so that when a bug hits one, it has massive impact on the entire internet.

We also know that the answer isn't in rolling your own security. Very few people or organizations are likely to be able to securely implement their own version of TLS. Even the best packages of today didn't start out perfect, they had to iterate through several flaws to get to where they are today.

So perhaps the better answer is in having more packages to choose from? Instead of picking just openssl by default, it would be better to have a broad array of choices. With a dozen packages on the market, that might mean 11 times out of 12 the bad guys wouldn't exploit our site. If the packages are interchangeable, we'd be better positioned to switch them quickly in case of emergency.

Re: not just many eyes

[allan572]

So all we need are 11 more sets of programmers to program free version of SSL 2-12?

Make that THREE other things

[calidoscope]

3) Port to to multiple architectures (and OS's) to catch bugs not reported by the original build environment. This is one of the approaches OpenBSD uses to improve security and was quite common in the open source software world when ESR coined the phrase.

The OpenBSD team found one very long lasting (30+ years) bug in the legacy BSD code when the Sparc64 build barfed.

Re: not just many eyes

[dkman]

Sadly we humans only seem to be able to handle 2 or 3 options. If 12 existed we'd hone in on 3 favorites and 9 would be outliers.

It's not that just "being open source" automatically means code is being validated by lots of eyes. It means that you can look at the code. All we need is more people interested in doing that, or paid to do so. They also need to have the knowledge/skill necessary to do that.

And as always, being closed source would not have made the issues easier to find. And then you'd be at their mercy waiting for a fix. These were all found and all fixed relatively quickly, so let's focus on that.

SSL certainly isn't a simple library. Increased complexity makes it easier to make a mistake and harder to find it.

Re:not just many eyes

[TechyImmigrant]

>We also know that the answer isn't in rolling your own security. Very few people or organizations are likely to be able to securely implement their own version of TLS.

TLS is the issue. It isn't simple. It isn't even secure in many compliant configurations. It invites implementation errors.
A good spec of a secure protocol would make secure implementations easy. If you don't think about the implementability while you're writing a spec, you're doomed, like TLS is.

"Don't roll your own security" is advice aimed at people who don't know about security. Some of us have to implement and 'roll' the specs. The world looks different when your reputation is tied to your stuff not get broken before senility sets in. You can do it right, but you need all the elements in place including a well thought out spec.

Mo Code, Mo Problems

[TiggertheMad]

I agree that more diversity in the software ecosystem will cause critical bugs to have less impact to the world overall, and will hopefully drive competition to make the offerings more efficient and stable. However I think that this is a straw-man and the real conclusion we should draw is this:

When you write code, you are going to screw up. If you aren't writing bugs that people notice, you aren't working on anything worthwhile. While the bugs that were found were costly and dangerous, the question is were these found quicker than a closed source solution? Were they fixed faster than a closed source solution? Is there anything that can be done to allow quicker roll back or disabling of vulnerable features? When you write code, you need to design for failure, because it will happen and plan so that the recovery will be as quick as possible.

Adding additional software library offerings will only add stability in the sense that one particular vector wont affect as much of the Internet, but you introduce more surface area for attackers to poke at, and more vulnerabilities overall. Given the challenges to write really solid code, I think I'd like to have fewer, but really well vetted open source software solutions. Of course, I am not correct in this opinion, as there are no 'right' decisions here.