Slashdot Mirror
Bug Hunting Open-Source vs. Proprietary Software
PreacherTom writes "An analysis comparing the top 50 open-source software projects to proprietary software from over 100 different companies was conducted by Coverity, working in conjunction with the Department of Homeland Security and Stanford University. The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy. On the other hand, the open-source software was found to be of greater average overall quality. Not surprisingly, dissenting opinions already exist, claiming Coverity's scope was inappropriate to their conclusions."
Knuth used to have this great offer where he'd send you a check for pi or e or something if you managed to find a bug in his code.
Well, what is a bug?
I doubt he'd send me a check if I told him that TeX doesn't have an easily accessible iconic user interface. No, his concept of a bug is a deviation from the specified functionality.
But what if that functionality is wrong or sucks?
Apple does really well at creating functionality that doesn't suck. They suffer from the same problems of deviations from the spec as much as anyone, but they manage to mold their spec around what users want. Microsoft, to some extent, does the same and they release products that conform to what users want (generally) because they change the spec as necessary when customers demand change.
If you are implementing towards a standard (like most OSS projects with any traction are wont to do), then you are necessarily restricted by what that spec says. If the spec says to do something inane, the standard-follower must implement it that way.
I don't really have a point here except to say that unless they say "this is what we mean by bug", there can be no way to really examine their results.
they tested it by using a program that systemattically scans code for common errors.
I don't know if the closed source statistics are online somewhere, but these are the open source statistics.
http://scan.coverity.com/
and if you ask me the "Defect Reports / KLOC" is pretty low, and such software would normally be considered "good" software.
"Deanna Asks A Ninja: What is the circumference of a moose?!"
"It's michael pailum with his face in a pie times douglas adams squared."
This answer makes as much sense as the article.
Except "Ask A Ninja" made more sense. And was more accurate. And more entertaining.
Can I just get a Ninja hit out on this guy something so these articles will not make it slashdot anymore?
The problem is that there are different types of Bugs. things like a typo in a help file, or American spelling vs British spelling, vs a bug were the app crashes the system when installed on a system with an early version of Quicktime are clasdsified differently.
The summary just says all bugs, which is not fair if the proprietary has 5 times the number of critical or super-critical bugs.
"It is a greater offense to steal men's labor, than their clothes"
...and while it is on the list on the web page, I was happy to determine that most of the issues they found were false alarms. They found three real bugs, none of which were likely to bite, and even if they did bite it is not exploitable. Nonetheless, those bugs probably wouldn't have been found otherwise, so I was happy for the scan.
Rather than brag (I won't say who I am or the name of my project), I'm just going to sit back and read all the defensive flames from self-appointed "security experts" whose open-source project didn't do so well. After all the flames from these "security experts" that I've endured, I'm going to enjoy watching them squirm.
It's karma.
While I appreciate that PreacherTom was good enogh to bring this to us, the sentence "...no open source project had fewer software defects than proprietary code." just does not match TFA.
TFA says that no open source project is as good as the BEST of proprietary, but it also says that the AVERAGE open source is better than the AVERAGE proprietary.
No, *popular* open-source software is 5x as buggy as *safety-critical* closed software. The linked dissenting opinion is at least partly right; they're comparing apples to oranges.
Maybe they should try comparing open- and closed-source software that's actually trying to solve the same problem? That'd be a bit more valid of a comparison...
The article makes it quite clear that the proprietary software which is much better that open source is mission-critical software. A class of software where ensuring minimum bugs is a top priority, and also a class of software which mostly does just not exist in OSS. If you are an OSS developer, would you try to develop open source air traffic control software? And even if yes, how would you do it anyway?
Basically, my own conclusion from reading the article was that it IS possible to write excellent software with very few bugs, if that is a top priority. And, that the author seems to say that while mission-critical software (which happens to be proprietary) is fortunately much better than the rest, among all that other non-mission-critical software, open source tends to be better than proprietary.
Not surprising, and quite encouraging...
> wine for example only has 0.112 / 1000 lines of code as well. ;)
> and we all know it by far doesn't always do what we want it to do.
Well duh! It is an implementation of the Windows API. And when considering how often the WinAPI does what you want, I think they have made a perfect copy.
He's comparing "bugs" in a project such as Apache with "bugs" in the software controlling a jet engine on an airplane.
He refuses to accept that different projects have different requirements. When the project results in people dying if it fails, you spend a LOT more money and time finding all the "bugs".
When the worst that happens is that you don't see a web page, your money/time requirements are not so high.
Even so, from his finding, Open Source is, on average, better than the closed source projects (not counting the closed source projects that result in loss-of-life in the event of a failure).
He's an idiot for confusing the different requirements.
The selection of programs from the two populations of programs (open source, proprietary) are not going to be comparable: vendors of proprietary software have a say over which code gets scanned, and they are going to select a different population of programs than the company selected for open source projects. This isn't a fixable problem: there is no way of doing this sort of study so that you can compare the two data sets. The best they could do is compare something like OpenOffice against Microsoft Office, or Apache against IIS.
Furthermore, Coverity simply cannot accomplish what they claim to accomplish: there is no way of detecting "bugs" automatically--if there were, compilers would already be doing it. Coverity effectively does little more than compare code against a set of internal coding conventions; that can be useful if it's done right, but it's not a measure of code quality. Some completely correct code will score thousands of violations against their tool, while other code may contain thousands of bugs, none of which register. Furthermore, it is likely that a lot of their customers are Windows based and that Coverity is biased towards Windows-based coding conventions, giving more false positives on non-Windows code. Before publishing such comparisons, Coverity first would need to demonstrate that their tool does not contain such biases.
Finally, and perhaps most importantly, the company isn't publishing its data, so nobody can verify or even evaluate their claims. Not only do they fail to publish their raw data (obviously, they can't do that for proprietary software), they also fail to list their summary statistics by vendor and project (which they could, but obviously won't do). They don't even give a summary statistic by class of application, class of organization, and code size. Their results are meaningless because they're not reproducible.
These numbers tell you nothing about FOSS code quality relative to commercial code quality. What they tell you is that Coverity apparently doesn't know how to do statistics, misrepresents what their product can do, and doesn't know how to report experimental results properly. Now, do you want to put your trust in such a company?
Says who? QA and testing covers the entire gamut, from formalized unit-testing at every level, to 'throw it at the beta testers and hope nothing breaks'. it's got nothing to do with 'proprietary' (not 'propriety') vs open source.
Where on Earth did you get that? Are you completely oblivious to all the testing methodologies and systems developed by the open source community? Here's a few for you to research: JUnit, Test::Unit, and Selenium.
Again with the generalizations! Commercial software development is, by definition, proprietary, so you don't know how they do it! They might tell you they have a 'strong QA engineering component' (whatever that means) but they could be full of shit!
www.lucernesys.comHorizon: Calendar-based personal finance
SIGSEGV caught, terminating
wait... not that kind of sig.