Slashdot Mirror
Are All Bugs Shallow? Questioning Linus's Law
root777 writes to point out a provocative blog piece by a Microsoft program manager, questioning one of the almost unquestioned tenets of open source development: that given enough eyeballs, all bugs are shallow. Are they? Shawn Hernan looks at DARPA's Sardonix experiment and the Coverity static-analysis bug discovery program in open source projects to conclude that perhaps not enough eyeballs are in evidence. Is he wrong? Why? "Most members of the periphery [those outside the core developer group] do not have the necessary debugging skills ... the vast numbers of 'eyeballs' apparently do not exist. ... [C]ode review is hardly all that makes software more secure. Getting software right is very, very difficult. ... Code review alone is not sufficient. Testing is not sufficient. Tools are not sufficient. Features are not sufficient. None of the things we do in isolation are sufficient. To get software truly correct, especially to get it secure, you have to address all phases of the software development lifecycle, and integrate security into the day-to-day activities."
They become a lot shallower when you can look at the source code.
As we can all see, this has gone famously for Microsoft.
...the proof is in the pudding?
What do they say?
Unless you're writing some insanely complex application like a launcher for thermonuclear missiles, you pretty much will have user error as a major instigator of bugs.
Until you get your code into the hands of users who - for example - will repeatedly hit the ENTER key wile waiting for a response, you don't have a clue what might happen.
The Kai's Semi-Updated Website Thingy
This is precisely the kind of argument you become susceptible to if you think that an attribute of software (security) is more important than your freedom. Shawn makes some good points about the technical quality of software and it's true there may not be enough eyeballs to find bugs in free software let alone hands to fix them. What Shawn would have us take from this article is that free software may not be technically superior. It's an attempt to frame the argument and shape what's people think is important in software. Unfortunately, if you care about software freedom, Microsoft's FXCop and PreFast-clean mean nothing. Their software disrespects you as a user and keeps pushing the limits in dividing and taking power away from their user base. Don't buy this line. Choose freedom first and interested parties will take care of attributes like security, ease-of-use, and compatibility over time.
Comment removed based on user account deletion
We should be careful not to let Microsoft deflect the conversation about software away from the ethics of using software you can't change, provide to your neighbor, or improve when you need more features. If the OPs conclusion is that free software may not have this particular leg to stand on in the arena of technical superiority, we must point out that freedom is our primary concern and that we each focus on security to the extent that we must obtain additional security for our software.
Since when does MS have the right to say "To get software truly correct..."? They KNOW how to make software secure?
That's kinda funny.
I spent part of today working around problems with a closed source application.
The other part of the day has been working with an open source program, where I've already solved the problem, and am documenting my changes to pass back to the author for the next release.
I'm not a "core" developer for any public projects. I've never submitted a bug fix to someone like Microsoft (but have sent bug complaints that went unanswered). I have sent quite a few bug fixes for open source applications, most of which were used in future release. I'm just another guy, or as indicated, another pair of eyes.
Serious? Seriousness is well above my pay grade.
There is a problem of deflection on another level. Most of Microsoft's problems when it comes
to security are design issues. Creating and then enforcing standards and policies with respect
to source code and development process is not going to help if the whole thing is broken as
designed. You will end up with a very consistent turd that looks good on paper.
Buffer overruns and such are not the most serious problem.
A Pirate and a Puritan look the same on a balance sheet.
...though perhaps not in the way he intends.
Look, software is *hard*. Building an OS kernel is like assembling a thousand watch movements by hand. You're going to screw up. It's not a matter of "if". There Are Always Mistakes.
Now, when he says "truly correct", I'm assuming he doesn't mean formal proving. That would be absurd, especially for an operating system as complex as Windows or Linux (or really anything with limited resources). Anything short of the formal proof and you just have empirical evidence that it works - but if there's a billion branches and trillions of code paths, nobody will hit all of them with all data.
Fact is, stuff is going to break. You can't prevent it.
So if we can't keep code from breaking - if all significant code is buggy - what's the answer? Well, with open-source code you can find a bug in your application and debug through the kernel itself, finding out why your syscall isn't returning the right information, and fix it yourself. Then everybody benefits from your work - keep in mind, you only did it (or needed to) because your application exposed a flaw. If you're using Linux 1.8 for some unholy reason, well you can fix it anyway (just nobody else will care).
But if you're using Windows, and you get bad return data from a method, your best shot is probably going to be to just coerce the data how you want it. This happens *all the time* in closed-source software - handle a buggy OS method with a special case.
So "many eyeballs" is correct, but not because there are thousands of expert code analysts poring over every git commit. It's correct because any piddly little application developer can debug the kernel itself, following his own method calls around to make sure they do the right thing. Even if he doesn't know how to fix it, he'll be able to say "doThis(*myData) isn't returning the right value" and lead the experts (writers/kernel hackers) straight to a fix.
This is the strength of open source, at least from a code quality standpoint.
I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
The funny thing about this article is that he essentially never mentions (a) design flaws or (b) perverse economic incentives to sell defective software. IMO these are probably the two biggest reason why MS has such a terrible reputation on security.
As an example of a design flaw, there are lots and lots of things that MS designed for ease of use, while ignoring security. MS software is way too willing to execute code in an email or on a web page just because they wanted to do something flashy without putting any responsibility on the user to know what the heck was going on. This is a design flaw. No amount of debugging will ever fully succeed in working around it.
The economic incentives to ship buggy, insecure software are also huge. Companies gather revenue by putting out a new version of the software with a long list of features. Users who buy the new version of the software generally have no way of knowing that it's full of bugs. MS is of course infamous for this.
Of course the implication of the whole article is that MS pays people to fix bugs, while nothing like that is going on in the open source world. This is complete nonsense. Most well known open-source projects are written by paid coders. But let's not let facts get in the way of MS advertising.
Find free books.
From the article:
One cannot deny the logic. In fact, it is a tautology. If you assume that all individuals have a non-zero probability of finding and fixing a bug, then all you need is "enough" individuals.
Emphasis added by me to show where I think his argument goes off the rails. "Linus' law" does not assumed that each eyeball is a bug fixer--it simply states that bugs are made shallow. Often the hardest part of fixing a bug is knowing about it, and finding it. The open source process makes it easier to do both, even if there are only a small group of coders actually fixing things.
This is not about how many software engineers you have reviewing your code. It's about how your end users can interact with the software engineers.
Build a man a fire, he's warm for one night. Set him on fire, and he's warm for the rest of his life.
I think that in Microsoft's case in particular, all the exploits out there prove the opposite of his case.
I'm not a MS dev or even anyone important, just a small business owner who fixes infected Windows machines (it's better than 3/4 of the work I do, sadly) so it seems to me that security wise at least he is way off base - the many more eyes that are looking at MS Windows without even having access to the code base are doing a pretty damned good job of finding security bugs in it.
SB
It's old. The more humans I meet, the more I like my cats. At least they are honest.
Ladies and gentleman, the article author is making a strawman argument. By transforming the "Linus' Law" into a badly written syllogism, and pointing out examples where _his invented syllogism_ fails, he's implying that closed source is _better_. Unfortunately, the vulnerabilities of closed source are often worse, by comparison and from experience.
Not necessarily. If its a quick and dirty hack to get something done in a short period of time on a "temporary" basis, then its quite possible the programmer intentionally wrote "shitty code" - and KNEW it was shitty code.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Ok, I've got some news for you. The quotation is not meant like an immutable law. There's a really good, important point there, but it's still just a meaningful aphorism. Let me help you with this -- when you see "given enough eyeballs, all bugs are shallow", read it as "given enough eyeballs, [almost all] bugs are shallow". Does that help? Can we move on now? This discussion is so stupid it's almost painful. Here are some other things to know: MS blog author wants attention; ESR is a self-important moron. Thank me later.
One big piece of FUD here is the notion that Microsoft programmers are paid, while open source programmers are not. The open source projects I know of advance mostly because of paid programmers, and I suspect that that is the case in general. That gives them the usual capitalist incentives for finding and removing bugs.
In product after product, Microsoft continues to ship fewer vulnerabilities than our competitors.
I wish he had cited some. It does not seem to be anyone's experience, and the only study I have ever seen that said that Windows was more secure than Linux did so by counting each Linux vulnerability several times (once per distro), and comparing just Windows against entire Linux repositories.
He also looks only at whether more eyeballs are good, neglecting the disadvantage of the uniformity of the WIndows monoculture, etc.
He also argues that the Coverity scan was not an example of many eyeballs because it was government funded. So, the government paid for it - but it still happened.
He does cite some stuff including, hilariously, a study carried out in 2002 that concluded that Linux was close to becoming unmaintainable. Eight years later I am pretty sure it is being maintained.
I am also wondering about the advantages of there beinga lot of code that is shared by multiple projects. I remember a BSD code review catching an X Windows bug. In that particular case it was not fixed upstream because the XFree86 people were being awkward, but I wonder how many cases there are of stuff getting fixed.
It is also easier to report open source bugs. I have never reported a bug in a proprietary app, but I have reported lots of Linux bugs (mostly distro level, or fixable at distro level) because I can follow what it happening, and I know what the (usually good) reaction to my individual report is.
Well I don't see people joining PETA and saying "Hey you know what, our views are a little extreme, lets try be a little more level headed".
I don't see people joining Greenpeace and saying "Hey now, Genetic Engineering's alright y'all". And lets not get started on Sea Shepard.
You also don't see hippies and vegans going to MacDonald's or Wallmart and working there in the hope to make it more ethical.
The point I am trying to make is that GNU started as the environment for people who cared about those Freedoms. Linux became part of that and is Licensed under the GPL. It is part of the Ecosystem that cares about those Freedoms. To turn around and say, well maybe those Freedoms aren't important, maybe we should become more mainstream so we can cater to the masses who like MacDonalds and Wallmart and don't care about Hens in cages or sweatshops, is kind of besides the point.
We all have our own reasons for using Linux but it would not exist without those freedoms... If you have a different view on freedoms you can also use *BSD, Solaris or something like Haiku (Etc. etc.). If you don't care, there is NOTHING that is stopping you from using Windows or OSX.
I certainly know that if I emigrated to a country and started saying people should follow my political views I certainly wouldn't be well received, it's no different with the F/OSS sphere. It is what it is. It is what it is because of what it is and really, most of us have bigger mouths than we should.
The Developers are free to do what ever they want and their projects can go in what ever directions they want them to. Users like me can be thankful for what they give us. Yes some are more rabid in proclaiming the Freedoms, but then again if a single project isn't free enough, a half-assed effort of replacing it is at least made.
Long post after a tired and long day tl;dr: Freedoms could be only a concern for a minority, but a large part of what exists is because of them. Even if they aren't the most important thing doesn't mean they aren't important.
Absolutely right. The author seems to be making the argument a lack of pay implies a lack of skill.
From the article:
According to Cowan, who is now a Security Program Manager for Windows, “the scientific conclusion of Sardonix is that auditing is both demanding of high skill and tedious, and so karma/reputation/good will is not enough to motivate people to do it. You must pay them to do it, precisely as Microsoft does.
The author is right that the "many eyeballs" scheme needs skilled eyeballs to work, but assumes that the only way to get good people on a project is by paying them. It seems odd that an article that tries so hard to provide a compelling argument makes such a poorly backed assumption. It's certainly true that good people need to be payed, but they can be paid to work on free software or write free software in their spare time; both cases have many examples.
So if this is the future...where's my jet pack?
I feel the need to explicitly call this guy a shill, rather than imply it. IF he honestly believes what he wrote, he's merely an idiot.
Shawn Hernan has deliberately misconstrued what Raymond wrote. Raymond explicitly said that the phrase "Given enough eyeballs, all bugs are shallow" was an informal phrasing of the lesson, in the very first sentence of the lesson. The actual phrasing was given as "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone." There's not even one sentence separating the two.
Trying to rip apart an informal phrasing, and ascribing hidden syllogisms to it, tells me this man is either an ideologue or an idiot. Given his position, he's a dangerous ideologue or idiot.
If opportunity came disguised as temptation, one knock would be enough.
3^2 * 67^1 * 977^1
Some of my points (IMHO, my 2 cents, works for me, etc.):
Mr Web Man: "Safari is way faster than Firefox on OS X and uses less resources."
Me: "Safari doesn't run at all on GNU/Linux or Solaris or FreeBSD. Besides, Firefox has a LOT of features that I like"
Mr Netbook Man: "The Gnome desktop is still kinda clunky, even after all these years."
Me: "I don't know what you mean by Clunky, but I prefer the functionality of Gnome over Windows or OSX any day of the week. Anyway, I like KDE and XFCE more than I like Gnome."
Mr Graphic Designer Man: "Linux still doesn't do proper color management."
Me: "I don't know what that means. You may be right."
Mr Gamer Man: "There aren't any decent games for Linux."
Me: "There are actually some decent games for GNU/Linux, but I agree that the selection could be greater. I hope the situation improves, but gaming is far from my primary concern"
You'll notice that I don't have to mention software freedom.
Lemon curry???
I think the matter that people get paid, nor that most of those working on the same area are from the same company will help in making Linus's Law 'more true'.
Yes, in general, the more people look at an issue, the more likely it is that someone will spot a bug, if there is one.
But - I give you the following caveats to this:
* people working closely together might reduce design flaws, but not necessarily implementation flaws - knowing specifically what a piece of code is doing CAN stand in your way of spotting subtle bugs in it (because the code more or less reads like what you expect). So, it helps to have more 'independent' pairs of eyeballs looking at the code.
* people not knowing the subject matter inside out are not on par with people who do. People who know how buffer overruns come about may figure out potential buffer overruns more likely than others. On the other hand, if, say, these people were to look at encryption code, they may see a potential for a buffer overrun, but not necessarily, whether the implementation of the encryption routines has a (not totally obvious) security flaw in the way it handles its keys; or whether any s-boxes may be good or not.
So, the more 'subject-matter-aware' eyeballs, which work independently of each other, look at a given code, the more likely you are getting a better review of the code.
I don't think I'm a bad C developer, but I don't think I could spot the majority of the linux kernel flaws because I do not know enough of the design of the kernel and potential interaction of areas of code.