Bug Hunting Open-Source vs. Proprietary Software

PreacherTom writes "An analysis comparing the top 50 open-source software projects to proprietary software from over 100 different companies was conducted by Coverity, working in conjunction with the Department of Homeland Security and Stanford University. The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy. On the other hand, the open-source software was found to be of greater average overall quality. Not surprisingly, dissenting opinions already exist, claiming Coverity's scope was inappropriate to their conclusions."

Comments

Re:So how did they test the proprietary software?

[msh104]

they tested it by using a program that systemattically scans code for common errors.

I don't know if the closed source statistics are online somewhere, but these are the open source statistics.
http://scan.coverity.com/

and if you ask me the "Defect Reports / KLOC" is pretty low, and such software would normally be considered "good" software.

Misquoting TFA

[Harmonious Botch]

While I appreciate that PreacherTom was good enogh to bring this to us, the sentence "...no open source project had fewer software defects than proprietary code." just does not match TFA.

TFA says that no open source project is as good as the BEST of proprietary, but it also says that the AVERAGE open source is better than the AVERAGE proprietary.

Re:just an example of how "buggy" OSS software.

[Reziac]

Quoth the poster:

linux 2.6: 3,315,274 lines of code, 0.138 / 1000 lines of code.
kde: 4,518,450 lines of code, 0.012 bugs / 1000 lines of code.

So far so good! But for contrast, I'll add this stat from TFChart:

Gnome: 31,596 lines of code, 1.931 bugs / 1000 lines of code.

Eeeep!!

(No wonder I prefer KDE :)

Re:What's a bug?

[AJWM]

Knuth used to have this great offer where he'd send you a check for pi or e or something if you managed to find a bug in his code.

I think you're conflating two things. The check was (is?) for $50 or some such. The version number of the software is pi (or e) to whatever number of decimals, where each subsequent release adds a decimal place (becomes a closer approximation to the real thing.)

No, his concept of a bug is a deviation from the specified functionality.

That's the only reasonable definition of a bug in the software.

But what if that functionality is wrong or sucks?

Then that's a bug in the specification or in the requirements. I spent the better part of six months debugging the requirements on a major project once. Part of that was getting mutual agreement from three major customers, part of that was resolving internal inconsistencies in the requirements document, and part of that was a high level design process in parallel, to be sure we had a chance of actually satisfying the requirements.

Of course the end user (especially of off-the-shelf software) generally doesn't differentiate between a bug in the software vs a bug in the specification or requirements. The end user generally never sees the spec, and only has a vague idea of the requirements. (Sometimes worse than vague -- how many people do you know who use a spreadsheet for a database?)

(And to BadAnalogyGuy -- I'm not disagreeing, just amplifying.)

Re:Why is this surprising?

[tb3]

Are you nuts? Or are you just trying to see how many vapid over-generalizations you can jam into a single comment?

Propriety software traditionally undergoes a formalized, designed testing process. It's not perfect, but it's an ordered approach to boundary testing, design level implementation of quality, and more.

Says who? QA and testing covers the entire gamut, from formalized unit-testing at every level, to 'throw it at the beta testers and hope nothing breaks'. it's got nothing to do with 'proprietary' (not 'propriety') vs open source.

Open source software must rely on after-the-fact testing in the form of "this broke when I tried to do this".

Where on Earth did you get that? Are you completely oblivious to all the testing methodologies and systems developed by the open source community? Here's a few for you to research: JUnit, Test::Unit, and Selenium.

Commercial software has a strong QA engineering component. Open Source software relies primarily on a black box testing approach.

Again with the generalizations! Commercial software development is, by definition, proprietary, so you don't know how they do it! They might tell you they have a 'strong QA engineering component' (whatever that means) but they could be full of shit!

A bug can be many things

[jchenx]

I work at MS. In my group (and I imagine it's the same in others), a bug can be many things. Here's what they typically are though:

1. A product defect
   - This is the typical meaning behind the word "bug".
   
2. DCR (Design Change Request)
   - That's where your TeX complaint would fall under. It's "by design" that it doesn't have an iconic user interface, but that doesn't mean it's something that shouldn't be addressed ever
   
3. Work item
   - This is actually a result of the bug tracking system that we use. Rather than sending e-mail, which often gets lost, we often track work items as bugs. For example, "Need to turn off switch X on the test server when we get to milestone Y"
   

To further complicate things, there is a severity and priority attached to every bug. Severity is a measure of the impact the bug has on the customer/end-product. It can range from 1 (Bug crashes system) to 4 (Just a typo). Priority is a measure of the importance of the bug. It ranges from 0 (Bug blocks team from doing any further work, must fix now), to 3 (Trivial bug, fix if there is time). (I don't know why the ranges don't match, BTW, seems silly to me)

As anyone who works on large-scale project probably knows, there are always a wide range of bugs, across all the pri/sev levels. To me, a simple count of all the bugs isn't terribly useful. A project could have a ton of bugs, but most of them being DCRs (which are knowingly going to be postponed till the next release) and/or low pri/sev bugs. Or maybe it's the beginning of the project and they're all known work items. Or a project could have only a few bugs, but with all of them being critical pri/sev ones.

So, whenever I see a report that simply talks about bug count, I take it with a huge grain of salt. If I had to guess (I skimmed the article), it seems like OSS projects have far more bugs, but perhaps lower pri/sev since the product itself has been evaluated as being higher quality. In the end, it's the quality that the customer really cares about.

Re:How much is it really true?

[angel'o'sphere]

And n00b developers are also capable of finding bugs. Aren't they?
No they are not to the extend of a experienced developt.
going through the code dow not find bugs. Either you do a formal correct approach, that is a walk through or a code inspection then you may find bugs, or you only have the chance to find occasional off by one errors in a loop or array index. Just by looking over code as you say in your n00b appoach you only find suspicious pieces of code.
What now? You change it to be less suspicious? And then? You commit it? So you don't know if somethign elsewhere is breaking now because of your change? Ah .... you have test cases for the software? So you run them after your refactoring? What now? All pass as before? Oops, if so: then you had no test case for that piece of suspicious code you just have fixed! So you still don't know if there was an error or not!

Testing means to DEFINE how individual pieces of code should behave and writing a test case exactly for that. Changing software and fixing bugs means to have tests, lots of tests, not eyeballs.

angel'o'sphere

P.S. that does not mean that formal walk throughs / inspections don't work, they do!! But informal ones are only for educational purpose intersting.

openoffice

[higuita]

openoffice code is a mess because i is very old... remember that there was a staroffice in DOS time, the same code was update over and over and over before release to the opensource community...
since then many people try to clean it, but its hard and risky to clean a such big app

most projects have a coding style that everyone should follow, and many force you to comply if they want their code to be accepted

OpenOffice is a bad example.

[Ayanami Rei]

The codebase is very old, contains a bunch of legacy stuff nobody really understands, as the codebase has passed hands from a German company to Sun to the OpenOffice.org foundation. It's also picked up a layer of java along the way (for whatever reason).

It's too bad because it actually works kinda okay, but it's a real effort to get your hands dirty with.
Blender is also like that... it seems when a codebase has 'gotten around' it tends to pick up the bad habits of all the hands its been through.

MySQL is a bad state because it's really only developed by MySQL AB -- no one else is contributing to it so they have no reason to make it any more maintainable than it is. PostgreSQL, on the other hand, had the luxury of being the fruit of some academic research projects and was rewritten once or twice, so it's a little more maintainable.