Slashdot Mirror
Are All Bugs Shallow? Questioning Linus's Law
root777 writes to point out a provocative blog piece by a Microsoft program manager, questioning one of the almost unquestioned tenets of open source development: that given enough eyeballs, all bugs are shallow. Are they? Shawn Hernan looks at DARPA's Sardonix experiment and the Coverity static-analysis bug discovery program in open source projects to conclude that perhaps not enough eyeballs are in evidence. Is he wrong? Why? "Most members of the periphery [those outside the core developer group] do not have the necessary debugging skills ... the vast numbers of 'eyeballs' apparently do not exist. ... [C]ode review is hardly all that makes software more secure. Getting software right is very, very difficult. ... Code review alone is not sufficient. Testing is not sufficient. Tools are not sufficient. Features are not sufficient. None of the things we do in isolation are sufficient. To get software truly correct, especially to get it secure, you have to address all phases of the software development lifecycle, and integrate security into the day-to-day activities."
As we can all see, this has gone famously for Microsoft.
...the proof is in the pudding?
What do they say?
We should be careful not to let Microsoft deflect the conversation about software away from the ethics of using software you can't change, provide to your neighbor, or improve when you need more features. If the OPs conclusion is that free software may not have this particular leg to stand on in the arena of technical superiority, we must point out that freedom is our primary concern and that we each focus on security to the extent that we must obtain additional security for our software.
That's kinda funny.
I spent part of today working around problems with a closed source application.
The other part of the day has been working with an open source program, where I've already solved the problem, and am documenting my changes to pass back to the author for the next release.
I'm not a "core" developer for any public projects. I've never submitted a bug fix to someone like Microsoft (but have sent bug complaints that went unanswered). I have sent quite a few bug fixes for open source applications, most of which were used in future release. I'm just another guy, or as indicated, another pair of eyes.
Serious? Seriousness is well above my pay grade.
Bugs are an error in the process, not the code. If you find a bug, you need to find the process error that allowed that bug to occur.
Agreed!
I read, with interest, the referenced article. I was expecting FUD - but I didn't find much, until I reached the Conclusion.
eg.
The many eyeballs argument is neat, tidy, compelling, and wrong.
The article starts with
Eric S. Raymond wrote , “Given enough eyeballs, all bugs are shallow.” He calls this Linus’ law.
and then attempts to refute. Fair enough. Except - the link leads to The Cathedral And The Bazaar - where I cannot find the quote... Hmmm
Now this might be relevant if the "many eyes" routine was the only form of audit used in GNU/Linux - but is not the only form of review/audit used. I'm sure other, more knowledgable posters will be able to provide more evidence than I could find in a quick search.
I call FUD
There is a problem of deflection on another level. Most of Microsoft's problems when it comes
to security are design issues. Creating and then enforcing standards and policies with respect
to source code and development process is not going to help if the whole thing is broken as
designed. You will end up with a very consistent turd that looks good on paper.
Buffer overruns and such are not the most serious problem.
A Pirate and a Puritan look the same on a balance sheet.
I'm not convinced that Free Software isn't superior. If he'd let us look at his code maybe we could tell. No?, well...without proof it's just opinion.
The funny thing about this article is that he essentially never mentions (a) design flaws or (b) perverse economic incentives to sell defective software. IMO these are probably the two biggest reason why MS has such a terrible reputation on security.
As an example of a design flaw, there are lots and lots of things that MS designed for ease of use, while ignoring security. MS software is way too willing to execute code in an email or on a web page just because they wanted to do something flashy without putting any responsibility on the user to know what the heck was going on. This is a design flaw. No amount of debugging will ever fully succeed in working around it.
The economic incentives to ship buggy, insecure software are also huge. Companies gather revenue by putting out a new version of the software with a long list of features. Users who buy the new version of the software generally have no way of knowing that it's full of bugs. MS is of course infamous for this.
Of course the implication of the whole article is that MS pays people to fix bugs, while nothing like that is going on in the open source world. This is complete nonsense. Most well known open-source projects are written by paid coders. But let's not let facts get in the way of MS advertising.
Find free books.
From the article:
One cannot deny the logic. In fact, it is a tautology. If you assume that all individuals have a non-zero probability of finding and fixing a bug, then all you need is "enough" individuals.
Emphasis added by me to show where I think his argument goes off the rails. "Linus' law" does not assumed that each eyeball is a bug fixer--it simply states that bugs are made shallow. Often the hardest part of fixing a bug is knowing about it, and finding it. The open source process makes it easier to do both, even if there are only a small group of coders actually fixing things.
This is not about how many software engineers you have reviewing your code. It's about how your end users can interact with the software engineers.
Build a man a fire, he's warm for one night. Set him on fire, and he's warm for the rest of his life.
What the essay fails to capture is the nature of the functioning of the eyeballs in practice, between open source and closed source. In closed source, the eyeballs only look at what they are paid to look at, if the code is just barely good enough to sell, then out it goes and nobody looks at that code again until the complaints start rolling in and then and only the do they fix it, well, sort of fix it, they of course only fix it just barely enough to silence the noisiest of complaints and the only if there are real consequences for failing to do so. Don't think so then try this http://social.technet.microsoft.com/Search/en-GB?query=this%20is%20a%20know%20fault&ac=8 and a huge number of them have never been fixed.
Open source follows a completely different series of routes;
1) People looking for faults because they get a kick out of finding them and fixing them.
2) Tweaks to functions that indirectly remove bugs by simply replacing them with better code.
3) Discoveries in user interactions, less of a complaint because there is no force in pushing the fix.
5) Governments and government departments directly pursuing more secure code.
6) Corporations seeking to build a public reputation by demonstrating coding expertise.
So in the case of open source software there are many 'different' kinds of eyes, so those eyes all working from different perspectives do in reality make bugs very shallow. In the closed source proprietary world the bugs are buried in the depths of the code, hiding in the dark, basically because of profits versus workmanship issues, which means no light is shone on them because only one set of eyes looking from a single 'shallow' perspective looks at them.
There is of course one other set of eyes looking at code, the saboteurs both private and government, looking for faults to exploit. Hard with open source because it can rapidly turn around and bite you on the arse if you use it (if you protect against it everybody notices). Closed source (mostly but a lot of less than honourable eyes lend up looking at it), of course can be targeted as long as you, well, use open source code yourself whilst promoting closed source to everybody else (hmm, kind of reminds me of all those mainland China computer companies, odd that, isn't it).
Chaos - everything, everywhere, everywhen
I think that in Microsoft's case in particular, all the exploits out there prove the opposite of his case.
I'm not a MS dev or even anyone important, just a small business owner who fixes infected Windows machines (it's better than 3/4 of the work I do, sadly) so it seems to me that security wise at least he is way off base - the many more eyes that are looking at MS Windows without even having access to the code base are doing a pretty damned good job of finding security bugs in it.
SB
It's old. The more humans I meet, the more I like my cats. At least they are honest.
Let me rephrase this for him -
"For 25 years, we deliberately chose to ignore the bitter lessons that were learned by the big vendors, to take shortcuts
to ship shit software first and fix it later and to build up massive layers of cruft in the name of backward compatibility. Now we are caught in a nice pickle
as we've spent years trying fill the leaks in our crap - some of which is so insecure that, 8 years after the launch, we still have record numbers of bugs in
Windows XP almost every fucking Patch Tuesday -and restructure it into something rock solid.
However, until we can get this done, we need to play smoke and mirrors, convince you to toss Win XP - and your old PC, most likely, buy our latest
and greatest and spit out evermore FUD about how nobody else can get stuff done except us.
Ladies and gentlemen, I give you the M$ business plan and I'm pleased to say that it's working as well as ever and thank you all"
Pain is merely failure leaving the body
Ok, I've got some news for you. The quotation is not meant like an immutable law. There's a really good, important point there, but it's still just a meaningful aphorism. Let me help you with this -- when you see "given enough eyeballs, all bugs are shallow", read it as "given enough eyeballs, [almost all] bugs are shallow". Does that help? Can we move on now? This discussion is so stupid it's almost painful. Here are some other things to know: MS blog author wants attention; ESR is a self-important moron. Thank me later.
In product after product, Microsoft continues to ship fewer vulnerabilities than our competitors.
I wish he had cited some. It does not seem to be anyone's experience, and the only study I have ever seen that said that Windows was more secure than Linux did so by counting each Linux vulnerability several times (once per distro), and comparing just Windows against entire Linux repositories.
He also looks only at whether more eyeballs are good, neglecting the disadvantage of the uniformity of the WIndows monoculture, etc.
He also argues that the Coverity scan was not an example of many eyeballs because it was government funded. So, the government paid for it - but it still happened.
He does cite some stuff including, hilariously, a study carried out in 2002 that concluded that Linux was close to becoming unmaintainable. Eight years later I am pretty sure it is being maintained.
I am also wondering about the advantages of there beinga lot of code that is shared by multiple projects. I remember a BSD code review catching an X Windows bug. In that particular case it was not fixed upstream because the XFree86 people were being awkward, but I wonder how many cases there are of stuff getting fixed.
It is also easier to report open source bugs. I have never reported a bug in a proprietary app, but I have reported lots of Linux bugs (mostly distro level, or fixable at distro level) because I can follow what it happening, and I know what the (usually good) reaction to my individual report is.
I feel the need to explicitly call this guy a shill, rather than imply it. IF he honestly believes what he wrote, he's merely an idiot.
Shawn Hernan has deliberately misconstrued what Raymond wrote. Raymond explicitly said that the phrase "Given enough eyeballs, all bugs are shallow" was an informal phrasing of the lesson, in the very first sentence of the lesson. The actual phrasing was given as "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone." There's not even one sentence separating the two.
Trying to rip apart an informal phrasing, and ascribing hidden syllogisms to it, tells me this man is either an ideologue or an idiot. Given his position, he's a dangerous ideologue or idiot.
If opportunity came disguised as temptation, one knock would be enough.
3^2 * 67^1 * 977^1
Some of my points (IMHO, my 2 cents, works for me, etc.):
Mr Web Man: "Safari is way faster than Firefox on OS X and uses less resources."
Me: "Safari doesn't run at all on GNU/Linux or Solaris or FreeBSD. Besides, Firefox has a LOT of features that I like"
Mr Netbook Man: "The Gnome desktop is still kinda clunky, even after all these years."
Me: "I don't know what you mean by Clunky, but I prefer the functionality of Gnome over Windows or OSX any day of the week. Anyway, I like KDE and XFCE more than I like Gnome."
Mr Graphic Designer Man: "Linux still doesn't do proper color management."
Me: "I don't know what that means. You may be right."
Mr Gamer Man: "There aren't any decent games for Linux."
Me: "There are actually some decent games for GNU/Linux, but I agree that the selection could be greater. I hope the situation improves, but gaming is far from my primary concern"
You'll notice that I don't have to mention software freedom.
Lemon curry???