Slashdot Mirror
Are All Bugs Shallow? Questioning Linus's Law
root777 writes to point out a provocative blog piece by a Microsoft program manager, questioning one of the almost unquestioned tenets of open source development: that given enough eyeballs, all bugs are shallow. Are they? Shawn Hernan looks at DARPA's Sardonix experiment and the Coverity static-analysis bug discovery program in open source projects to conclude that perhaps not enough eyeballs are in evidence. Is he wrong? Why? "Most members of the periphery [those outside the core developer group] do not have the necessary debugging skills ... the vast numbers of 'eyeballs' apparently do not exist. ... [C]ode review is hardly all that makes software more secure. Getting software right is very, very difficult. ... Code review alone is not sufficient. Testing is not sufficient. Tools are not sufficient. Features are not sufficient. None of the things we do in isolation are sufficient. To get software truly correct, especially to get it secure, you have to address all phases of the software development lifecycle, and integrate security into the day-to-day activities."
They become a lot shallower when you can look at the source code.
As we can all see, this has gone famously for Microsoft.
...the proof is in the pudding?
What do they say?
Unless you're writing some insanely complex application like a launcher for thermonuclear missiles, you pretty much will have user error as a major instigator of bugs.
Until you get your code into the hands of users who - for example - will repeatedly hit the ENTER key wile waiting for a response, you don't have a clue what might happen.
The Kai's Semi-Updated Website Thingy
This is precisely the kind of argument you become susceptible to if you think that an attribute of software (security) is more important than your freedom. Shawn makes some good points about the technical quality of software and it's true there may not be enough eyeballs to find bugs in free software let alone hands to fix them. What Shawn would have us take from this article is that free software may not be technically superior. It's an attempt to frame the argument and shape what's people think is important in software. Unfortunately, if you care about software freedom, Microsoft's FXCop and PreFast-clean mean nothing. Their software disrespects you as a user and keeps pushing the limits in dividing and taking power away from their user base. Don't buy this line. Choose freedom first and interested parties will take care of attributes like security, ease-of-use, and compatibility over time.
Comment removed based on user account deletion
We should be careful not to let Microsoft deflect the conversation about software away from the ethics of using software you can't change, provide to your neighbor, or improve when you need more features. If the OPs conclusion is that free software may not have this particular leg to stand on in the arena of technical superiority, we must point out that freedom is our primary concern and that we each focus on security to the extent that we must obtain additional security for our software.
Except the point he is trying to make is that his code is better then the competing individual because he follows process doctrine.
Unfortunately, to make his claims stick he took a failed project as an example to support his theories. While being quite pointed in defining what projects failed he did not cite which projects of his has succeeded. This would have been at least a good starting point for a real argument.
Is good process doctrine wrong? No, it won't hurt of course, but it's not quite a kevlar vest against root shells.
Besides more examples from both sides of the camp he really does neglect several facts. Many open source projects are often led or particpated by professionals as well. In fact a recent article suggested a great more open source projects are corporate sponsored.
It's just an awful piece when you consider he is painting his enemy as both unprofessional and only arming that foe with one failed project example.
Personally, I wanted to read something useful that I could learn from and grow with, but this is pretty standard tripe.
"You should always go to other people's funerals; otherwise, they won't come to yours." -- Yogi Berra
Since when does MS have the right to say "To get software truly correct..."? They KNOW how to make software secure?
That's kinda funny.
I spent part of today working around problems with a closed source application.
The other part of the day has been working with an open source program, where I've already solved the problem, and am documenting my changes to pass back to the author for the next release.
I'm not a "core" developer for any public projects. I've never submitted a bug fix to someone like Microsoft (but have sent bug complaints that went unanswered). I have sent quite a few bug fixes for open source applications, most of which were used in future release. I'm just another guy, or as indicated, another pair of eyes.
Serious? Seriousness is well above my pay grade.
Bugs are an error in the process, not the code. If you find a bug, you need to find the process error that allowed that bug to occur.
Agreed!
I read, with interest, the referenced article. I was expecting FUD - but I didn't find much, until I reached the Conclusion.
eg.
The many eyeballs argument is neat, tidy, compelling, and wrong.
The article starts with
Eric S. Raymond wrote , “Given enough eyeballs, all bugs are shallow.” He calls this Linus’ law.
and then attempts to refute. Fair enough. Except - the link leads to The Cathedral And The Bazaar - where I cannot find the quote... Hmmm
Now this might be relevant if the "many eyes" routine was the only form of audit used in GNU/Linux - but is not the only form of review/audit used. I'm sure other, more knowledgable posters will be able to provide more evidence than I could find in a quick search.
I call FUD
I also think a big difference is that you psychologically don't write shitty code when you think others are going to look at it.
Coders that write shitty code don't know that they write shitty code. From their perspective the code is just fine and even very good. When ever I told someone I don't like his code and challenged him to explain what he did and why, he only answered: "erm, well that is what the code should do as the requirements demand that", they had no idea what my point was and when I pointed i tout they shrugged and did not understand or value my concerns.
angel'o'sphere
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
There is a problem of deflection on another level. Most of Microsoft's problems when it comes
to security are design issues. Creating and then enforcing standards and policies with respect
to source code and development process is not going to help if the whole thing is broken as
designed. You will end up with a very consistent turd that looks good on paper.
Buffer overruns and such are not the most serious problem.
A Pirate and a Puritan look the same on a balance sheet.
...though perhaps not in the way he intends.
Look, software is *hard*. Building an OS kernel is like assembling a thousand watch movements by hand. You're going to screw up. It's not a matter of "if". There Are Always Mistakes.
Now, when he says "truly correct", I'm assuming he doesn't mean formal proving. That would be absurd, especially for an operating system as complex as Windows or Linux (or really anything with limited resources). Anything short of the formal proof and you just have empirical evidence that it works - but if there's a billion branches and trillions of code paths, nobody will hit all of them with all data.
Fact is, stuff is going to break. You can't prevent it.
So if we can't keep code from breaking - if all significant code is buggy - what's the answer? Well, with open-source code you can find a bug in your application and debug through the kernel itself, finding out why your syscall isn't returning the right information, and fix it yourself. Then everybody benefits from your work - keep in mind, you only did it (or needed to) because your application exposed a flaw. If you're using Linux 1.8 for some unholy reason, well you can fix it anyway (just nobody else will care).
But if you're using Windows, and you get bad return data from a method, your best shot is probably going to be to just coerce the data how you want it. This happens *all the time* in closed-source software - handle a buggy OS method with a special case.
So "many eyeballs" is correct, but not because there are thousands of expert code analysts poring over every git commit. It's correct because any piddly little application developer can debug the kernel itself, following his own method calls around to make sure they do the right thing. Even if he doesn't know how to fix it, he'll be able to say "doThis(*myData) isn't returning the right value" and lead the experts (writers/kernel hackers) straight to a fix.
This is the strength of open source, at least from a code quality standpoint.
I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
The funny thing about this article is that he essentially never mentions (a) design flaws or (b) perverse economic incentives to sell defective software. IMO these are probably the two biggest reason why MS has such a terrible reputation on security.
As an example of a design flaw, there are lots and lots of things that MS designed for ease of use, while ignoring security. MS software is way too willing to execute code in an email or on a web page just because they wanted to do something flashy without putting any responsibility on the user to know what the heck was going on. This is a design flaw. No amount of debugging will ever fully succeed in working around it.
The economic incentives to ship buggy, insecure software are also huge. Companies gather revenue by putting out a new version of the software with a long list of features. Users who buy the new version of the software generally have no way of knowing that it's full of bugs. MS is of course infamous for this.
Of course the implication of the whole article is that MS pays people to fix bugs, while nothing like that is going on in the open source world. This is complete nonsense. Most well known open-source projects are written by paid coders. But let's not let facts get in the way of MS advertising.
Find free books.
From the article:
One cannot deny the logic. In fact, it is a tautology. If you assume that all individuals have a non-zero probability of finding and fixing a bug, then all you need is "enough" individuals.
Emphasis added by me to show where I think his argument goes off the rails. "Linus' law" does not assumed that each eyeball is a bug fixer--it simply states that bugs are made shallow. Often the hardest part of fixing a bug is knowing about it, and finding it. The open source process makes it easier to do both, even if there are only a small group of coders actually fixing things.
This is not about how many software engineers you have reviewing your code. It's about how your end users can interact with the software engineers.
Build a man a fire, he's warm for one night. Set him on fire, and he's warm for the rest of his life.
What the essay fails to capture is the nature of the functioning of the eyeballs in practice, between open source and closed source. In closed source, the eyeballs only look at what they are paid to look at, if the code is just barely good enough to sell, then out it goes and nobody looks at that code again until the complaints start rolling in and then and only the do they fix it, well, sort of fix it, they of course only fix it just barely enough to silence the noisiest of complaints and the only if there are real consequences for failing to do so. Don't think so then try this http://social.technet.microsoft.com/Search/en-GB?query=this%20is%20a%20know%20fault&ac=8 and a huge number of them have never been fixed.
Open source follows a completely different series of routes;
1) People looking for faults because they get a kick out of finding them and fixing them.
2) Tweaks to functions that indirectly remove bugs by simply replacing them with better code.
3) Discoveries in user interactions, less of a complaint because there is no force in pushing the fix.
5) Governments and government departments directly pursuing more secure code.
6) Corporations seeking to build a public reputation by demonstrating coding expertise.
So in the case of open source software there are many 'different' kinds of eyes, so those eyes all working from different perspectives do in reality make bugs very shallow. In the closed source proprietary world the bugs are buried in the depths of the code, hiding in the dark, basically because of profits versus workmanship issues, which means no light is shone on them because only one set of eyes looking from a single 'shallow' perspective looks at them.
There is of course one other set of eyes looking at code, the saboteurs both private and government, looking for faults to exploit. Hard with open source because it can rapidly turn around and bite you on the arse if you use it (if you protect against it everybody notices). Closed source (mostly but a lot of less than honourable eyes lend up looking at it), of course can be targeted as long as you, well, use open source code yourself whilst promoting closed source to everybody else (hmm, kind of reminds me of all those mainland China computer companies, odd that, isn't it).
Chaos - everything, everywhere, everywhen
I think that in Microsoft's case in particular, all the exploits out there prove the opposite of his case.
I'm not a MS dev or even anyone important, just a small business owner who fixes infected Windows machines (it's better than 3/4 of the work I do, sadly) so it seems to me that security wise at least he is way off base - the many more eyes that are looking at MS Windows without even having access to the code base are doing a pretty damned good job of finding security bugs in it.
SB
It's old. The more humans I meet, the more I like my cats. At least they are honest.
Ladies and gentleman, the article author is making a strawman argument. By transforming the "Linus' Law" into a badly written syllogism, and pointing out examples where _his invented syllogism_ fails, he's implying that closed source is _better_. Unfortunately, the vulnerabilities of closed source are often worse, by comparison and from experience.
Not necessarily. If its a quick and dirty hack to get something done in a short period of time on a "temporary" basis, then its quite possible the programmer intentionally wrote "shitty code" - and KNEW it was shitty code.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
True, but that's not what he is questioning. Given two identical projects that are fairly complex (i.e. an OS kernel) he's saying that just being open source doesn't necessarily provide "more eyes". While I think there is a bit of merit to this, it certainly doesn't hurt to have more eyes possible - especially when you don't have to pay for them.
Agreed, of course. However, the converse is important, too:
Given two identical projects that are fairly complex (i.e. an OS kernel), being closed source virtually guarantees that there won't be 'more eyes'.
But the real question is: How many eyes are enough?
The answer is its own problem: Only one more pair. The tricky part is figuring out whose they are. (Yes, I'm in screaming agreement with what the OP is saying.)
It's a quality issue as much as it's a question of quantity. Ben Laurie, writing about the Debian OpenSSL Fiasco, states:
So yes, it does matter whose eyes are turned to a particular problem. The difference between FOSS/Open Source and Closed Source is therefore whether the Closed Source company has hired the right people and whether the FOSS project has gained the attention and interest of the right people.
Neither of those situations is guaranteed, but they are not at all equivalent. (Especially when we consider that for many of the best FOSS products, gaining the attention and interest of the right people is done by employing them.) Realistically, FOSS faces better odds of having bugs found and fixed, all else being equal.
Crumb's Corollary: Never bring a knife to a bun fight.
Let me rephrase this for him -
"For 25 years, we deliberately chose to ignore the bitter lessons that were learned by the big vendors, to take shortcuts
to ship shit software first and fix it later and to build up massive layers of cruft in the name of backward compatibility. Now we are caught in a nice pickle
as we've spent years trying fill the leaks in our crap - some of which is so insecure that, 8 years after the launch, we still have record numbers of bugs in
Windows XP almost every fucking Patch Tuesday -and restructure it into something rock solid.
However, until we can get this done, we need to play smoke and mirrors, convince you to toss Win XP - and your old PC, most likely, buy our latest
and greatest and spit out evermore FUD about how nobody else can get stuff done except us.
Ladies and gentlemen, I give you the M$ business plan and I'm pleased to say that it's working as well as ever and thank you all"
Pain is merely failure leaving the body
Ok, I've got some news for you. The quotation is not meant like an immutable law. There's a really good, important point there, but it's still just a meaningful aphorism. Let me help you with this -- when you see "given enough eyeballs, all bugs are shallow", read it as "given enough eyeballs, [almost all] bugs are shallow". Does that help? Can we move on now? This discussion is so stupid it's almost painful. Here are some other things to know: MS blog author wants attention; ESR is a self-important moron. Thank me later.
I get it, ULSER. Good one. They cause me that sort of stress too.
* File Locked rather than writeable by administrator for upgrade purposes.
* Ring 1 or higher code being able to write to Ring 0 locations.
* Administrative users necessary to run most things (MS software or otherwise).
* Proprietary networking.
* Lack of regression testing (LAND should just never have happened).
There's 5, who wants to take up the mantle from there.
Me failed English...
FreeBSD over Linux. If my comments seem odd, this may explain...
One big piece of FUD here is the notion that Microsoft programmers are paid, while open source programmers are not. The open source projects I know of advance mostly because of paid programmers, and I suspect that that is the case in general. That gives them the usual capitalist incentives for finding and removing bugs.
In product after product, Microsoft continues to ship fewer vulnerabilities than our competitors.
I wish he had cited some. It does not seem to be anyone's experience, and the only study I have ever seen that said that Windows was more secure than Linux did so by counting each Linux vulnerability several times (once per distro), and comparing just Windows against entire Linux repositories.
He also looks only at whether more eyeballs are good, neglecting the disadvantage of the uniformity of the WIndows monoculture, etc.
He also argues that the Coverity scan was not an example of many eyeballs because it was government funded. So, the government paid for it - but it still happened.
He does cite some stuff including, hilariously, a study carried out in 2002 that concluded that Linux was close to becoming unmaintainable. Eight years later I am pretty sure it is being maintained.
I am also wondering about the advantages of there beinga lot of code that is shared by multiple projects. I remember a BSD code review catching an X Windows bug. In that particular case it was not fixed upstream because the XFree86 people were being awkward, but I wonder how many cases there are of stuff getting fixed.
It is also easier to report open source bugs. I have never reported a bug in a proprietary app, but I have reported lots of Linux bugs (mostly distro level, or fixable at distro level) because I can follow what it happening, and I know what the (usually good) reaction to my individual report is.
Well I don't see people joining PETA and saying "Hey you know what, our views are a little extreme, lets try be a little more level headed".
I don't see people joining Greenpeace and saying "Hey now, Genetic Engineering's alright y'all". And lets not get started on Sea Shepard.
You also don't see hippies and vegans going to MacDonald's or Wallmart and working there in the hope to make it more ethical.
The point I am trying to make is that GNU started as the environment for people who cared about those Freedoms. Linux became part of that and is Licensed under the GPL. It is part of the Ecosystem that cares about those Freedoms. To turn around and say, well maybe those Freedoms aren't important, maybe we should become more mainstream so we can cater to the masses who like MacDonalds and Wallmart and don't care about Hens in cages or sweatshops, is kind of besides the point.
We all have our own reasons for using Linux but it would not exist without those freedoms... If you have a different view on freedoms you can also use *BSD, Solaris or something like Haiku (Etc. etc.). If you don't care, there is NOTHING that is stopping you from using Windows or OSX.
I certainly know that if I emigrated to a country and started saying people should follow my political views I certainly wouldn't be well received, it's no different with the F/OSS sphere. It is what it is. It is what it is because of what it is and really, most of us have bigger mouths than we should.
The Developers are free to do what ever they want and their projects can go in what ever directions they want them to. Users like me can be thankful for what they give us. Yes some are more rabid in proclaiming the Freedoms, but then again if a single project isn't free enough, a half-assed effort of replacing it is at least made.
Long post after a tired and long day tl;dr: Freedoms could be only a concern for a minority, but a large part of what exists is because of them. Even if they aren't the most important thing doesn't mean they aren't important.
A ridiculous amount of the linux kernel code is written by programmers paid by IBM, Intel, RedHat, etc.
Someone pays. I'm just glad it isn't me.
Absolutely right. The author seems to be making the argument a lack of pay implies a lack of skill.
From the article:
According to Cowan, who is now a Security Program Manager for Windows, “the scientific conclusion of Sardonix is that auditing is both demanding of high skill and tedious, and so karma/reputation/good will is not enough to motivate people to do it. You must pay them to do it, precisely as Microsoft does.
The author is right that the "many eyeballs" scheme needs skilled eyeballs to work, but assumes that the only way to get good people on a project is by paying them. It seems odd that an article that tries so hard to provide a compelling argument makes such a poorly backed assumption. It's certainly true that good people need to be payed, but they can be paid to work on free software or write free software in their spare time; both cases have many examples.
So if this is the future...where's my jet pack?
...A perfect program that is never written isn't very useful.
It is, however, bug-free!
I feel the need to explicitly call this guy a shill, rather than imply it. IF he honestly believes what he wrote, he's merely an idiot.
Shawn Hernan has deliberately misconstrued what Raymond wrote. Raymond explicitly said that the phrase "Given enough eyeballs, all bugs are shallow" was an informal phrasing of the lesson, in the very first sentence of the lesson. The actual phrasing was given as "Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone." There's not even one sentence separating the two.
Trying to rip apart an informal phrasing, and ascribing hidden syllogisms to it, tells me this man is either an ideologue or an idiot. Given his position, he's a dangerous ideologue or idiot.
If opportunity came disguised as temptation, one knock would be enough.
3^2 * 67^1 * 977^1
Some of my points (IMHO, my 2 cents, works for me, etc.):
Mr Web Man: "Safari is way faster than Firefox on OS X and uses less resources."
Me: "Safari doesn't run at all on GNU/Linux or Solaris or FreeBSD. Besides, Firefox has a LOT of features that I like"
Mr Netbook Man: "The Gnome desktop is still kinda clunky, even after all these years."
Me: "I don't know what you mean by Clunky, but I prefer the functionality of Gnome over Windows or OSX any day of the week. Anyway, I like KDE and XFCE more than I like Gnome."
Mr Graphic Designer Man: "Linux still doesn't do proper color management."
Me: "I don't know what that means. You may be right."
Mr Gamer Man: "There aren't any decent games for Linux."
Me: "There are actually some decent games for GNU/Linux, but I agree that the selection could be greater. I hope the situation improves, but gaming is far from my primary concern"
You'll notice that I don't have to mention software freedom.
Lemon curry???
In my world, the software stands or falls on its own merits. There's plenty of truly excellent FOSS out there, and as I said in a neighboring message, I use some of it daily. There are also areas where the FOSS world has failed to produce anything beyond clunky second-rate knock-offs of proprietary software. And there are areas where proprietary software has built on and then surpassed the FOSS software it's riffing off.
Specifics?
Some of my favorite FOSS stuff -- things I'd pick over the commercial alternatives any day of the week, purely on their own merits: The Linux kernel and GNU command-line utilities. PostgreSQL. The Dojo toolkit. Firefox. Thunderbird. Eclipse. CUPS. Apache (web server, many of their other projects suck). Various Debian package managers. VirtualBox.
Some cheap and clunky and altogether second-rate things that attempt to duplicate functionality of commercial software that does the job much better, that I (hate to but nevertheless) use, for any of a number of reasons: GIMP. OpenOffice (especially the Word and Excel clones -- and good grief, it oughtn't be that hard to do better than *Word,* of all things!) GNOME/KDE/any other Linux desktop. Various RAW conversion utilities.
Some commercial software that does stuff better than the FOSS stuff they're riffing off or building on: Jira. Confluence. Mac OS X.
Some areas where the FOSS world has consistently failed to deliver, despite years and years of effort and constant promise, and the fact that the problems appear ideally suited to being solved the FOSS way:
Content management systems. There are a gazillion FOSS ones out there, and all of them suck in some significant way -- either they're a big ol' mess of vaguely connected utilities (Drupal), they make very big assumptions about how you want your site to work (Joomla), or they're half-finished while incorporating several internally competing ways of doing things (Lenya and its plethora of editors, none of which really work very well.)
Anything related to proper graphic design tasks. This requires a full chain of utilities from the RAW file in the camera to the finished file to be sent to the printer (or put up on the web). Most of the chain just isn't there: no system-wide color management, no RAW conversion software with accurate, consistent profiles for a wide range of cameras, no genuinely functional (and color managed) page layout software.
I could go on, but you get my drift. I don't care for ideological arguments. If FOSS is a genuinely and universally better way to make software, it would have incontrovertibly proved it by now. If it was genuinely and incontrovertibly unworkable, it would have failed by now. Instead, it's done neither -- it works brilliantly for some things, fails miserably in other things, and muddles along for lots of others. Just like any other way of making software.
Whew. I feel better now.
He reports Coverity's results on open source software
... but doesn't report Coverity's results on Microsoft's software.
He reports that Coverity scanned 280 open-source projects
...but doesn't report that only 180 of those have "active developer support".
He can't be bothered to present any data at all on the distribution of the reported or corrected defects — how many are in nethack or aalib or that long-abandoned "flash-based photo album generator"?
He doesn't, for instance, mention that Samba and several others have no defects Coverity can discover. None.
Vim has none. X.org has ... three. All of KDE, nearly five million lines of code, has ... ninety. glibc has none.
There have been MySQL and PostgreSQL and Berkeley DB versions with none. His bioblurb says he's "currently working to ensure that Microsoft SQL Server is secure". That's interesting. You mean it isn't, now? How many defects can Coverity find in SQL Server?
TFA is a nauseating pile of sneers and aspersions ("Hope is not a security strategy"?) built on a very carefully selected and very few facts. "No one is doing auditing" he quotes. "No one is doing auditing" and reporting it to some self-styled central authority almost no one ever heard of is what's true, but telling the truth isn't what he's doing here. He's a "Program Manager", and he works for Microsoft.
As always, all IMO. Insert "I think" everywhere grammatically possible.
Creating and then enforcing standards and policies with respect
to source code and development process is not going to help if the whole thing is broken as
designed.
I was thinking of the irony of an MS project manager lecturing the Linux kernel devs on "bugginess."
Put identity in the browser.
To be fair to Microsoft this is no longer true. UAC asks the user if they wish to elevates privileges when an app does something unsafe. Vista took a lot of flak when UAC appeared (including from myself) but it did force user land applications to stop abusing the registry (e.g opening HKLM with read/write permissions), writing random files to random locations on disk and other unnecessary operations. The consequence is apps written / patched in the last 3 years run pretty cleanly and if they don't, you get the UAC popup. In practice it's little different from what happens in Ubuntu or OS X in similar circumstances.
Plan9 is in the Unix family, one secuirty alert in 15 years
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
Sorry, that's totally wrong. The Airbus FBW systems do allow reversion back to "just do what the damn human says". However, in the situation that aircraft was in, if it were a 1972 manufactured Boeing 747 with the same fault (no airspeed indication, inside a storm, in a flight regime where stall speed and maximum mach number are very close together) it is likely that the end result would have been the same.
Incidentally, how the A320 allows human handling of exception was very well demonstrated by the United flight that ended up in the Hudson - in which no lives were lost despite a very difficult situation.
Oolite: Elite-like game. For Mac, Linux and Windows
I think the matter that people get paid, nor that most of those working on the same area are from the same company will help in making Linus's Law 'more true'.
Yes, in general, the more people look at an issue, the more likely it is that someone will spot a bug, if there is one.
But - I give you the following caveats to this:
* people working closely together might reduce design flaws, but not necessarily implementation flaws - knowing specifically what a piece of code is doing CAN stand in your way of spotting subtle bugs in it (because the code more or less reads like what you expect). So, it helps to have more 'independent' pairs of eyeballs looking at the code.
* people not knowing the subject matter inside out are not on par with people who do. People who know how buffer overruns come about may figure out potential buffer overruns more likely than others. On the other hand, if, say, these people were to look at encryption code, they may see a potential for a buffer overrun, but not necessarily, whether the implementation of the encryption routines has a (not totally obvious) security flaw in the way it handles its keys; or whether any s-boxes may be good or not.
So, the more 'subject-matter-aware' eyeballs, which work independently of each other, look at a given code, the more likely you are getting a better review of the code.
I don't think I'm a bad C developer, but I don't think I could spot the majority of the linux kernel flaws because I do not know enough of the design of the kernel and potential interaction of areas of code.