Slashdot Mirror
Are All Bugs Shallow? Questioning Linus's Law
root777 writes to point out a provocative blog piece by a Microsoft program manager, questioning one of the almost unquestioned tenets of open source development: that given enough eyeballs, all bugs are shallow. Are they? Shawn Hernan looks at DARPA's Sardonix experiment and the Coverity static-analysis bug discovery program in open source projects to conclude that perhaps not enough eyeballs are in evidence. Is he wrong? Why? "Most members of the periphery [those outside the core developer group] do not have the necessary debugging skills ... the vast numbers of 'eyeballs' apparently do not exist. ... [C]ode review is hardly all that makes software more secure. Getting software right is very, very difficult. ... Code review alone is not sufficient. Testing is not sufficient. Tools are not sufficient. Features are not sufficient. None of the things we do in isolation are sufficient. To get software truly correct, especially to get it secure, you have to address all phases of the software development lifecycle, and integrate security into the day-to-day activities."
As we can all see, this has gone famously for Microsoft.
...the proof is in the pudding?
What do they say?
This is precisely the kind of argument you become susceptible to if you think that an attribute of software (security) is more important than your freedom. Shawn makes some good points about the technical quality of software and it's true there may not be enough eyeballs to find bugs in free software let alone hands to fix them. What Shawn would have us take from this article is that free software may not be technically superior. It's an attempt to frame the argument and shape what's people think is important in software. Unfortunately, if you care about software freedom, Microsoft's FXCop and PreFast-clean mean nothing. Their software disrespects you as a user and keeps pushing the limits in dividing and taking power away from their user base. Don't buy this line. Choose freedom first and interested parties will take care of attributes like security, ease-of-use, and compatibility over time.
Comment removed based on user account deletion
We should be careful not to let Microsoft deflect the conversation about software away from the ethics of using software you can't change, provide to your neighbor, or improve when you need more features. If the OPs conclusion is that free software may not have this particular leg to stand on in the arena of technical superiority, we must point out that freedom is our primary concern and that we each focus on security to the extent that we must obtain additional security for our software.
Except the point he is trying to make is that his code is better then the competing individual because he follows process doctrine.
Unfortunately, to make his claims stick he took a failed project as an example to support his theories. While being quite pointed in defining what projects failed he did not cite which projects of his has succeeded. This would have been at least a good starting point for a real argument.
Is good process doctrine wrong? No, it won't hurt of course, but it's not quite a kevlar vest against root shells.
Besides more examples from both sides of the camp he really does neglect several facts. Many open source projects are often led or particpated by professionals as well. In fact a recent article suggested a great more open source projects are corporate sponsored.
It's just an awful piece when you consider he is painting his enemy as both unprofessional and only arming that foe with one failed project example.
Personally, I wanted to read something useful that I could learn from and grow with, but this is pretty standard tripe.
"You should always go to other people's funerals; otherwise, they won't come to yours." -- Yogi Berra
Since when does MS have the right to say "To get software truly correct..."? They KNOW how to make software secure?
That's kinda funny.
I spent part of today working around problems with a closed source application.
The other part of the day has been working with an open source program, where I've already solved the problem, and am documenting my changes to pass back to the author for the next release.
I'm not a "core" developer for any public projects. I've never submitted a bug fix to someone like Microsoft (but have sent bug complaints that went unanswered). I have sent quite a few bug fixes for open source applications, most of which were used in future release. I'm just another guy, or as indicated, another pair of eyes.
Serious? Seriousness is well above my pay grade.
Bugs are an error in the process, not the code. If you find a bug, you need to find the process error that allowed that bug to occur.
Agreed!
I read, with interest, the referenced article. I was expecting FUD - but I didn't find much, until I reached the Conclusion.
eg.
The many eyeballs argument is neat, tidy, compelling, and wrong.
The article starts with
Eric S. Raymond wrote , “Given enough eyeballs, all bugs are shallow.” He calls this Linus’ law.
and then attempts to refute. Fair enough. Except - the link leads to The Cathedral And The Bazaar - where I cannot find the quote... Hmmm
Now this might be relevant if the "many eyes" routine was the only form of audit used in GNU/Linux - but is not the only form of review/audit used. I'm sure other, more knowledgable posters will be able to provide more evidence than I could find in a quick search.
I call FUD
I also think a big difference is that you psychologically don't write shitty code when you think others are going to look at it.
Coders that write shitty code don't know that they write shitty code. From their perspective the code is just fine and even very good. When ever I told someone I don't like his code and challenged him to explain what he did and why, he only answered: "erm, well that is what the code should do as the requirements demand that", they had no idea what my point was and when I pointed i tout they shrugged and did not understand or value my concerns.
angel'o'sphere
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
There is a problem of deflection on another level. Most of Microsoft's problems when it comes
to security are design issues. Creating and then enforcing standards and policies with respect
to source code and development process is not going to help if the whole thing is broken as
designed. You will end up with a very consistent turd that looks good on paper.
Buffer overruns and such are not the most serious problem.
A Pirate and a Puritan look the same on a balance sheet.
The funny thing about this article is that he essentially never mentions (a) design flaws or (b) perverse economic incentives to sell defective software. IMO these are probably the two biggest reason why MS has such a terrible reputation on security.
As an example of a design flaw, there are lots and lots of things that MS designed for ease of use, while ignoring security. MS software is way too willing to execute code in an email or on a web page just because they wanted to do something flashy without putting any responsibility on the user to know what the heck was going on. This is a design flaw. No amount of debugging will ever fully succeed in working around it.
The economic incentives to ship buggy, insecure software are also huge. Companies gather revenue by putting out a new version of the software with a long list of features. Users who buy the new version of the software generally have no way of knowing that it's full of bugs. MS is of course infamous for this.
Of course the implication of the whole article is that MS pays people to fix bugs, while nothing like that is going on in the open source world. This is complete nonsense. Most well known open-source projects are written by paid coders. But let's not let facts get in the way of MS advertising.
Find free books.
From the article:
One cannot deny the logic. In fact, it is a tautology. If you assume that all individuals have a non-zero probability of finding and fixing a bug, then all you need is "enough" individuals.
Emphasis added by me to show where I think his argument goes off the rails. "Linus' law" does not assumed that each eyeball is a bug fixer--it simply states that bugs are made shallow. Often the hardest part of fixing a bug is knowing about it, and finding it. The open source process makes it easier to do both, even if there are only a small group of coders actually fixing things.
This is not about how many software engineers you have reviewing your code. It's about how your end users can interact with the software engineers.
Build a man a fire, he's warm for one night. Set him on fire, and he's warm for the rest of his life.
What the essay fails to capture is the nature of the functioning of the eyeballs in practice, between open source and closed source. In closed source, the eyeballs only look at what they are paid to look at, if the code is just barely good enough to sell, then out it goes and nobody looks at that code again until the complaints start rolling in and then and only the do they fix it, well, sort of fix it, they of course only fix it just barely enough to silence the noisiest of complaints and the only if there are real consequences for failing to do so. Don't think so then try this http://social.technet.microsoft.com/Search/en-GB?query=this%20is%20a%20know%20fault&ac=8 and a huge number of them have never been fixed.
Open source follows a completely different series of routes;
1) People looking for faults because they get a kick out of finding them and fixing them.
2) Tweaks to functions that indirectly remove bugs by simply replacing them with better code.
3) Discoveries in user interactions, less of a complaint because there is no force in pushing the fix.
5) Governments and government departments directly pursuing more secure code.
6) Corporations seeking to build a public reputation by demonstrating coding expertise.
So in the case of open source software there are many 'different' kinds of eyes, so those eyes all working from different perspectives do in reality make bugs very shallow. In the closed source proprietary world the bugs are buried in the depths of the code, hiding in the dark, basically because of profits versus workmanship issues, which means no light is shone on them because only one set of eyes looking from a single 'shallow' perspective looks at them.
There is of course one other set of eyes looking at code, the saboteurs both private and government, looking for faults to exploit. Hard with open source because it can rapidly turn around and bite you on the arse if you use it (if you protect against it everybody notices). Closed source (mostly but a lot of less than honourable eyes lend up looking at it), of course can be targeted as long as you, well, use open source code yourself whilst promoting closed source to everybody else (hmm, kind of reminds me of all those mainland China computer companies, odd that, isn't it).
Chaos - everything, everywhere, everywhen
I think that in Microsoft's case in particular, all the exploits out there prove the opposite of his case.
I'm not a MS dev or even anyone important, just a small business owner who fixes infected Windows machines (it's better than 3/4 of the work I do, sadly) so it seems to me that security wise at least he is way off base - the many more eyes that are looking at MS Windows without even having access to the code base are doing a pretty damned good job of finding security bugs in it.
SB
It's old. The more humans I meet, the more I like my cats. At least they are honest.
Not necessarily. If its a quick and dirty hack to get something done in a short period of time on a "temporary" basis, then its quite possible the programmer intentionally wrote "shitty code" - and KNEW it was shitty code.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
True, but that's not what he is questioning. Given two identical projects that are fairly complex (i.e. an OS kernel) he's saying that just being open source doesn't necessarily provide "more eyes". While I think there is a bit of merit to this, it certainly doesn't hurt to have more eyes possible - especially when you don't have to pay for them.
Agreed, of course. However, the converse is important, too:
Given two identical projects that are fairly complex (i.e. an OS kernel), being closed source virtually guarantees that there won't be 'more eyes'.
But the real question is: How many eyes are enough?
The answer is its own problem: Only one more pair. The tricky part is figuring out whose they are. (Yes, I'm in screaming agreement with what the OP is saying.)
It's a quality issue as much as it's a question of quantity. Ben Laurie, writing about the Debian OpenSSL Fiasco, states:
So yes, it does matter whose eyes are turned to a particular problem. The difference between FOSS/Open Source and Closed Source is therefore whether the Closed Source company has hired the right people and whether the FOSS project has gained the attention and interest of the right people.
Neither of those situations is guaranteed, but they are not at all equivalent. (Especially when we consider that for many of the best FOSS products, gaining the attention and interest of the right people is done by employing them.) Realistically, FOSS faces better odds of having bugs found and fixed, all else being equal.
Crumb's Corollary: Never bring a knife to a bun fight.
Let me rephrase this for him -
"For 25 years, we deliberately chose to ignore the bitter lessons that were learned by the big vendors, to take shortcuts
to ship shit software first and fix it later and to build up massive layers of cruft in the name of backward compatibility. Now we are caught in a nice pickle
as we've spent years trying fill the leaks in our crap - some of which is so insecure that, 8 years after the launch, we still have record numbers of bugs in
Windows XP almost every fucking Patch Tuesday -and restructure it into something rock solid.
However, until we can get this done, we need to play smoke and mirrors, convince you to toss Win XP - and your old PC, most likely, buy our latest
and greatest and spit out evermore FUD about how nobody else can get stuff done except us.
Ladies and gentlemen, I give you the M$ business plan and I'm pleased to say that it's working as well as ever and thank you all"
Pain is merely failure leaving the body
Ok, I've got some news for you. The quotation is not meant like an immutable law. There's a really good, important point there, but it's still just a meaningful aphorism. Let me help you with this -- when you see "given enough eyeballs, all bugs are shallow", read it as "given enough eyeballs, [almost all] bugs are shallow". Does that help? Can we move on now? This discussion is so stupid it's almost painful. Here are some other things to know: MS blog author wants attention; ESR is a self-important moron. Thank me later.
In product after product, Microsoft continues to ship fewer vulnerabilities than our competitors.
I wish he had cited some. It does not seem to be anyone's experience, and the only study I have ever seen that said that Windows was more secure than Linux did so by counting each Linux vulnerability several times (once per distro), and comparing just Windows against entire Linux repositories.
He also looks only at whether more eyeballs are good, neglecting the disadvantage of the uniformity of the WIndows monoculture, etc.
He also argues that the Coverity scan was not an example of many eyeballs because it was government funded. So, the government paid for it - but it still happened.
He does cite some stuff including, hilariously, a study carried out in 2002 that concluded that Linux was close to becoming unmaintainable. Eight years later I am pretty sure it is being maintained.
I am also wondering about the advantages of there beinga lot of code that is shared by multiple projects. I remember a BSD code review catching an X Windows bug. In that particular case it was not fixed upstream because the XFree86 people were being awkward, but I wonder how many cases there are of stuff getting fixed.
It is also easier to report open source bugs. I have never reported a bug in a proprietary app, but I have reported lots of Linux bugs (mostly distro level, or fixable at distro level) because I can follow what it happening, and I know what the (usually good) reaction to my individual report is.
Well I don't see people joining PETA and saying "Hey you know what, our views are a little extreme, lets try be a little more level headed".
I don't see people joining Greenpeace and saying "Hey now, Genetic Engineering's alright y'all". And lets not get started on Sea Shepard.
You also don't see hippies and vegans going to MacDonald's or Wallmart and working there in the hope to make it more ethical.
The point I am trying to make is that GNU started as the environment for people who cared about those Freedoms. Linux became part of that and is Licensed under the GPL. It is part of the Ecosystem that cares about those Freedoms. To turn around and say, well maybe those Freedoms aren't important, maybe we should become more mainstream so we can cater to the masses who like MacDonalds and Wallmart and don't care about Hens in cages or sweatshops, is kind of besides the point.
We all have our own reasons for using Linux but it would not exist without those freedoms... If you have a different view on freedoms you can also use *BSD, Solaris or something like Haiku (Etc. etc.). If you don't care, there is NOTHING that is stopping you from using Windows or OSX.
I certainly know that if I emigrated to a country and started saying people should follow my political views I certainly wouldn't be well received, it's no different with the F/OSS sphere. It is what it is. It is what it is because of what it is and really, most of us have bigger mouths than we should.
The Developers are free to do what ever they want and their projects can go in what ever directions they want them to. Users like me can be thankful for what they give us. Yes some are more rabid in proclaiming the Freedoms, but then again if a single project isn't free enough, a half-assed effort of replacing it is at least made.
Long post after a tired and long day tl;dr: Freedoms could be only a concern for a minority, but a large part of what exists is because of them. Even if they aren't the most important thing doesn't mean they aren't important.
A ridiculous amount of the linux kernel code is written by programmers paid by IBM, Intel, RedHat, etc.
Someone pays. I'm just glad it isn't me.
...A perfect program that is never written isn't very useful.
It is, however, bug-free!
I feel the need to explicitly call this guy a shill, rather than imply it. IF he honestly believes what he wrote, he's merely an idiot.
Shawn Hernan has deliberately misconstrued what Raymond wrote. Raymond explicitly said that the phrase "Given enough eyeballs, all bugs are shallow" was an informal phrasing of the lesson, in the very first sentence of the lesson. The actual phrasing was given as "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone." There's not even one sentence separating the two.
Trying to rip apart an informal phrasing, and ascribing hidden syllogisms to it, tells me this man is either an ideologue or an idiot. Given his position, he's a dangerous ideologue or idiot.
If opportunity came disguised as temptation, one knock would be enough.
3^2 * 67^1 * 977^1
Some of my points (IMHO, my 2 cents, works for me, etc.):
Mr Web Man: "Safari is way faster than Firefox on OS X and uses less resources."
Me: "Safari doesn't run at all on GNU/Linux or Solaris or FreeBSD. Besides, Firefox has a LOT of features that I like"
Mr Netbook Man: "The Gnome desktop is still kinda clunky, even after all these years."
Me: "I don't know what you mean by Clunky, but I prefer the functionality of Gnome over Windows or OSX any day of the week. Anyway, I like KDE and XFCE more than I like Gnome."
Mr Graphic Designer Man: "Linux still doesn't do proper color management."
Me: "I don't know what that means. You may be right."
Mr Gamer Man: "There aren't any decent games for Linux."
Me: "There are actually some decent games for GNU/Linux, but I agree that the selection could be greater. I hope the situation improves, but gaming is far from my primary concern"
You'll notice that I don't have to mention software freedom.
Lemon curry???
Sorry, that's totally wrong. The Airbus FBW systems do allow reversion back to "just do what the damn human says". However, in the situation that aircraft was in, if it were a 1972 manufactured Boeing 747 with the same fault (no airspeed indication, inside a storm, in a flight regime where stall speed and maximum mach number are very close together) it is likely that the end result would have been the same.
Incidentally, how the A320 allows human handling of exception was very well demonstrated by the United flight that ended up in the Hudson - in which no lives were lost despite a very difficult situation.
Oolite: Elite-like game. For Mac, Linux and Windows