Bug Hunting Open-Source vs. Proprietary Software

PreacherTom writes "An analysis comparing the top 50 open-source software projects to proprietary software from over 100 different companies was conducted by Coverity, working in conjunction with the Department of Homeland Security and Stanford University. The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy. On the other hand, the open-source software was found to be of greater average overall quality. Not surprisingly, dissenting opinions already exist, claiming Coverity's scope was inappropriate to their conclusions."

Comments

What's a bug?

[BadAnalogyGuy]

Knuth used to have this great offer where he'd send you a check for pi or e or something if you managed to find a bug in his code.

Well, what is a bug?

I doubt he'd send me a check if I told him that TeX doesn't have an easily accessible iconic user interface. No, his concept of a bug is a deviation from the specified functionality.

But what if that functionality is wrong or sucks?

Apple does really well at creating functionality that doesn't suck. They suffer from the same problems of deviations from the spec as much as anyone, but they manage to mold their spec around what users want. Microsoft, to some extent, does the same and they release products that conform to what users want (generally) because they change the spec as necessary when customers demand change.

If you are implementing towards a standard (like most OSS projects with any traction are wont to do), then you are necessarily restricted by what that spec says. If the spec says to do something inane, the standard-follower must implement it that way.

I don't really have a point here except to say that unless they say "this is what we mean by bug", there can be no way to really examine their results.

Number of Bugs vs Bug types

[Alien54]

The problem is that there are different types of Bugs. things like a typo in a help file, or American spelling vs British spelling, vs a bug were the app crashes the system when installed on a system with an early version of Quicktime are clasdsified differently.

The summary just says all bugs, which is not fair if the proprietary has 5 times the number of critical or super-critical bugs.

Not quite...

[Timothy Brownawell]

The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy. On the other hand, the open-source software was found to be of greater average overall quality.

No, *popular* open-source software is 5x as buggy as *safety-critical* closed software. The linked dissenting opinion is at least partly right; they're comparing apples to oranges.

Maybe they should try comparing open- and closed-source software that's actually trying to solve the same problem? That'd be a bit more valid of a comparison...

Even worse.

[khasim]

He's comparing "bugs" in a project such as Apache with "bugs" in the software controlling a jet engine on an airplane.

He refuses to accept that different projects have different requirements. When the project results in people dying if it fails, you spend a LOT more money and time finding all the "bugs".

When the worst that happens is that you don't see a web page, your money/time requirements are not so high.

Even so, from his finding, Open Source is, on average, better than the closed source projects (not counting the closed source projects that result in loss-of-life in the event of a failure).

He's an idiot for confusing the different requirements.

Re:Even worse.

[phantomfive]

Don't listen to the slashdot summary. It's terrible. The author is not against open source, he talks about the "brilliant open-source community."

What this guy is trying to say (besides 'buy my software') is that open source can do better (the title of his article is "...what open-source developers can learn....."). He wants people to use stricter development practices; things like automatic testing, nightly builds, etc.

Furthermore, he is probably right, automatically testing code ala j-unit or cpp-unit is a great idea when you are getting contributions from many different people. If that became common practice in the open-source world, the code quality would improve. He's not saying open-source is bad, he's saying it could get better.

This guy is not an idiot, you just didn't understand his point.

Lies, damned lies, and statistics

[Ibag]

If you look at the summary, you come to the conclusion that proprietary software is five times less buggy than open source. It is also unclear how software can have five times as many bugs but be of higher quality. However, if you read the article, you find:

In our research using automatic bug-hunting technology, no open-source project we analyzed had fewer software defects (per thousand lines of code) than the top-of-the-line closed-source application. That proprietary code, written for an aerospace company, is better than the best in open source--more than five times better, in fact. That company's software won't let you down when you're flying from New York to London.

If we ignore that the automatic bug finding algorithms might not be a good measure for anything, we have a few issues with the summary. The richest american is twice as rich as the richest Swiss man. Does it follow that Americans are on average twice as rich as Swiss people? No. In the same way, the statement does not imply that the average open source software has five times as many bugs as the average proprietary software does. The coding practices of mission critical apps like flight control systems are different from those of most of the industry, and it is almost wrong to lump them together with everything else.

The problem with statistics is not that they give an inaccurate picture, or even that selecting the right statistics can give a skewed picture, but that people who don't appreciate what statistics actually give use them to form opinions, make decisions, and summarize articles. Statistics don't lie, but the people who misreport them do, even if they don't realize it.