Slashdot Mirror
2014: The Year We Learned How Vulnerable Third-Party Code Libraries Are
jfruh writes Heartbleed, Shellshock, Poodle — all high-profile vulnerabilities in widely used libraries that rocked the software industry in 2014. Sadly, experts are now beginning to believe that these aren't the only bugs lurking out there in widely used open source code, just the ones that grabbed the most attention. It's beginning to look like one of the foundation concepts of open source — that with enough eyes, all bugs are shallow — is a myth. Of course, probably no one believes that all bugs are instantly shallow, no matter how open is the source, or that open source software is immune from bugs -- particularly ESR, coiner of the phrase.
The phrase might be true, but we're seeing the effects of insufficient eyes. In reality, how many sets of eyes are actually reviewing these libraries at a source code level? I rather strongly suspect that in most cases they are simply used under the assumption that "well, everyone uses it, it must be okay".
Shellshock did not affect a "library", but an executable.
the big news is that people are now thinking that bugs in software is big news.
My magic 8 ball tells me that in 2015 we will learn that proprietary and embedded software is even more vulnerable. My Tarot Card deck tell me that we will see a lot of hacked car wrecks in 2015, now that Volvo released the demon by putting a web browser into in-dash system. Rest of the lemmings are sure to follow. Not that you really need a browser to pwn a car, with Bluetooth-to-CAN-BUS exploits shutting down cars demonstrated as early as 2012.
Not only can they fix them, but they do fix them. As soon as these vulnerabilities were discovered (by non-malicious actors) patches were available within days or even hours. Commercial vendors take their good sweet time. Or in the case of Microsoft, discontinue support for the still widely-used Windows XP. Find a vulnerability in that? Too damn bad. It'll never get fixed.
Nobody's saying open source software is bug-free or even necessarily has fewer bugs than closed source software. It's what happens when a bug is discovered that makes the difference.
We don't have a state-run media we have a media-run state.
The security of the open source model isn't really the problem or the answer here. The problem is homogeneity. A million different sites and applications rely on just a few libraries, so that when a bug hits one, it has massive impact on the entire internet.
We also know that the answer isn't in rolling your own security. Very few people or organizations are likely to be able to securely implement their own version of TLS. Even the best packages of today didn't start out perfect, they had to iterate through several flaws to get to where they are today.
So perhaps the better answer is in having more packages to choose from? Instead of picking just openssl by default, it would be better to have a broad array of choices. With a dozen packages on the market, that might mean 11 times out of 12 the bad guys wouldn't exploit our site. If the packages are interchangeable, we'd be better positioned to switch them quickly in case of emergency.
John
I don't know anyone that ever thought "Open source" was bug free.
Every FOSS fanatic on Slashdot for the last 17 years has implied that bugs would be found and fixed FAST - not linger for years.
The point is that people can more easilly find and fix bugs with open source.
They could but do they? Nope.
ESR's claim has nothing to do with the frequency or discovery of bugs. All he says is that given enough observers, bugs are quickly characterized. It is implied that any given bug has already been discovered. There is no benevolent cohort of experts continuously auditing code bases and his statement doesn't claim there is.
Lurking at the bottom of the gravity well, getting old
So all we need are 11 more sets of programmers to program free version of SSL 2-12?
An open source project can have as few as just one set. There are some projects that nobody other than the developer ever contributes to. Just one guy occasionally working on some little project that some people use. Those all those people COULD look at the code, they don't.
Likewise commercial firms can, and sometimes do, pay many people to look at the code. In addition to having a big development staff they can have dedicated QA staffs. They can have a person, or many people, who's job it is literally to sit and look over the code for security issues every single day.
You are buying in to the same fallacy that the article is talking about: That because something is open is just means that more people MUST be looking at it. No it means people can look at it, but they may choose not to.
So "Security through complexity" then?
Sounds complex.
Mod me down, my New Earth Global Warmingist friends!
Unfortunately, assuming the number of developers is constant, with 12x as many libraries every one of them is virtually guaranteed to be far less secure than if all the developers were working on improving the same library.
So the root of the issue is that we need to dedicate more total resources to development, auditing, etc. the fundamental security libraries. Whether the best outcomes would then be seen by focussing all those resources on a single library, or spreading them across multiple competitors, is a seperate question. Somehow though I suspect that having 12x the resources focussed on a single library would be far more effective than having 12 competing libraries all staffed at current levels - wherin the majority of labor would be independently duplicating effort between projects. With enough resources there might be some benefit in having two or three competing implementations under development, but I suspect we're currently a long way away from "enough".
--- Most topics have many sides worth arguing, allow me to take one opposite you.
So all we need are 11 more sets of programmers to program free version of SSL 2-12?
Yes, and demand for them. But the big problem you're correctly implying is there's no economic justification that will drive this behavior. Maybe it will take a dozen big companies and foundations to drive this. Imagine if IBM, Microsoft, Google, RedHat, Yahoo, HP, Dell, Apache, Wikimedia, Mozilla, FSF, Apple, Intel, AMD, nVidia, Bungiesoft, and others each contributed their own versions of openSSL; each written in their own choice of language, using their own code, and building their own implementations of everything from the crypto through the command line interpreter logic. My company may decide we do more business with Intel, so we choose theirs. Or your company may be more Apple focused, so you'd choose theirs. In every case, we'd all nervously watch each other looking for signs of intrusions, hoping we won't be the victims, but knowing that alternatives exist if we are.
While a 1/12th scale incident of Heartbleed is still a huge problem for a lot of companies, it's no longer the catastrophe-sized disaster that Heartbleed actually was.
John
3) Port to to multiple architectures (and OS's) to catch bugs not reported by the original build environment. This is one of the approaches OpenBSD uses to improve security and was quite common in the open source software world when ESR coined the phrase.
The OpenBSD team found one very long lasting (30+ years) bug in the legacy BSD code when the Sparc64 build barfed.
A Shadeless room is a brighter room.
Hey now, I'm all for closed-source bashing, but let's be fair: Windows XP is over 13 years old. Yes, if you found a bug in one of it's contemporaries such as Red Hat Linux 6.1, but it's much more likely you'd upgrade to a newer version of the software where the bug has (hopefully) been fixed.
--- Most topics have many sides worth arguing, allow me to take one opposite you.
Actually the number of developers ( and developer effective effort ) that would work on such a library is not constant. Once you get too many developers on a single project a larger effort is required to required to just for code / development syncing. The other issue is especially with open source developers is not all developers are interested in working in the same development models among other issues. Which means single projects tend to discourage some potential developers from joining in. There also is the issue of developers wanting to feel helpful a good developer will often feel more appreciated on working on a smaller project.
Finally just because implementations are separate does not mean that their is not cross pollination. Fundamental concerns found by one project are very likely to be searched for in other projects. Likewise with good ideas introduced in one project will often be picked up by other projects. Further because of a diverse system of libraries it is more likely that good ideas can be experimented with safely in different projects.
Sadly we humans only seem to be able to handle 2 or 3 options. If 12 existed we'd hone in on 3 favorites and 9 would be outliers.
It's not that just "being open source" automatically means code is being validated by lots of eyes. It means that you can look at the code. All we need is more people interested in doing that, or paid to do so. They also need to have the knowledge/skill necessary to do that.
And as always, being closed source would not have made the issues easier to find. And then you'd be at their mercy waiting for a fix. These were all found and all fixed relatively quickly, so let's focus on that.
SSL certainly isn't a simple library. Increased complexity makes it easier to make a mistake and harder to find it.
I refuse to sign
Or in the case of Microsoft, discontinue support for the still widely-used Windows XP. Find a vulnerability in that? Too damn bad. It'll never get fixed.
Like when Ubuntu Server 13.04 didn't get a fix for Heartbleed because they discontinued support after 1 year despite the criticality of the bug and the servers seeing considerable use? All the official replies were "it's your own fault" and "change distro version immediately". Which you often can't do quickly. No users really expected 12.04, 12.10, 13.10 and 14.04 to get the fix while 13.04 in the middle was left out - except people who read the really, really fine print and took it seriously. Shipping the security fix would have been trivial and saved a lot of people a lot of work; they just refused on principle.
It was probably the first time many users found out Canonical had changed the support duration (that's why 12.10 got the fix).
OpenSSL, Netscape NSS, GnuTLS, as well as various proprietary SSL implementations have all had serious bugs over the last few years. Some of the bugs affected multiple unrelated code bases.
How much diversity do you want? How did the diversity help?
Finally! A year of moderation! Ready for 2019?
First off, let's be clear - I'm not arguing against alternatives, there are many good reasons to have them. But when the single major widely-used library is already "under-funded", starting a dozen more similar projects all competing for basically the same developer base is unlikely to improve things.
As for attracting OSS developers, I'm unconvinced. Sure, for "sexy" projects many will prefer a smaller project where their contributions are likely to be more appreciated. On the other hand, for a utility library only ever seen by a tiny percentage of developers, especially one designed to be a drop-in replacement for competitors, that appeal is likely going to be overshadowed by the fact that they're working on "just another implementation of X"
And perhaps more to the point, those motivations are primarily relevant to *volunteer* developers. But as I understand it for a big, high-visibility project like Linux the majority of code contribution now comes from developers working in the employ of major downstream users. No reason at all they shouldn't be paid to spend a bit more time on high-stakes things like security.
As for the resources needed - I suspect there are plenty of developers per se already working on the projects - major bugs are after all typically fixed within weeks. What's really needed is better auditing. Which yes, does call for a skillset with considerable overlap with developers, but is not really the same thing.
And I hear you on the cross-pollination - the point though is that when problem X is discovered in codebase Y, you now also need need enough auditor man-hours to also analyze codebases A,B, C, D, E, and F for similar problems. And if you want the libraries to be drop-in replacements for each other then god forbid you should add a feature or fix a bug that requires an interface update. Assuming the internal architecture is at least moderately different in all the implementations it may not even be possible to change the interface in a way that doesn't require a major rework of one or more implementing projects. Even if it is you're going to be looking at a lot of duplication of effort for any changes.
--- Most topics have many sides worth arguing, allow me to take one opposite you.
>We also know that the answer isn't in rolling your own security. Very few people or organizations are likely to be able to securely implement their own version of TLS.
TLS is the issue. It isn't simple. It isn't even secure in many compliant configurations. It invites implementation errors.
A good spec of a secure protocol would make secure implementations easy. If you don't think about the implementability while you're writing a spec, you're doomed, like TLS is.
"Don't roll your own security" is advice aimed at people who don't know about security. Some of us have to implement and 'roll' the specs. The world looks different when your reputation is tied to your stuff not get broken before senility sets in. You can do it right, but you need all the elements in place including a well thought out spec.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
Microsoft, Google, RedHat, Yahoo, HP, Dell, Apache, Wikimedia, Mozilla, FSF, Apple, Intel, AMD, nVidia, Bungiesoft, and others each contributed their own versions of openSSL; each written in their own choice of language, using their own code, and building their own implementations of everything from the crypto through the command line interpreter logic.
...and then, due to the nature of software today, you wind up with all of these in your organization, and now if any of them are compromised you have a problem. I'd rather engineers from all of those organizations worked on OpenSSL (or whatever) to make it as secure as possible, perhaps forking it occasionally to solve some particular problem and then merging the code later.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
I think reuse is far more secure than multiple options. In the long run. Reuse will find more bugs and make it more aware. Reuse will have a much larger impact, but that just ups the priority to get it fixed. Multiple options will only decrease the impact but will open up for duplicate bugs, finding less of them, more entropy for fragmentation, and less priority & longer time for getting them fixed. Multiple options will be less secure... you just won't think it is.
I agree that more diversity in the software ecosystem will cause critical bugs to have less impact to the world overall, and will hopefully drive competition to make the offerings more efficient and stable. However I think that this is a straw-man and the real conclusion we should draw is this:
When you write code, you are going to screw up. If you aren't writing bugs that people notice, you aren't working on anything worthwhile. While the bugs that were found were costly and dangerous, the question is were these found quicker than a closed source solution? Were they fixed faster than a closed source solution? Is there anything that can be done to allow quicker roll back or disabling of vulnerable features? When you write code, you need to design for failure, because it will happen and plan so that the recovery will be as quick as possible.
Adding additional software library offerings will only add stability in the sense that one particular vector wont affect as much of the Internet, but you introduce more surface area for attackers to poke at, and more vulnerabilities overall. Given the challenges to write really solid code, I think I'd like to have fewer, but really well vetted open source software solutions. Of course, I am not correct in this opinion, as there are no 'right' decisions here.
HA! I just wasted some of your bandwidth with a frivolous sig!
TLS is the issue. It isn't simple. It isn't even secure in many compliant configurations. It invites implementation errors.
A good spec of a secure protocol would make secure implementations easy. If you don't think about the implementability while you're writing a spec, you're doomed, like TLS is.
Well said! You deserve +5, Insightful. Get on it, mods!