Bug Hunting Open-Source vs. Proprietary Software

PreacherTom writes "An analysis comparing the top 50 open-source software projects to proprietary software from over 100 different companies was conducted by Coverity, working in conjunction with the Department of Homeland Security and Stanford University. The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy. On the other hand, the open-source software was found to be of greater average overall quality. Not surprisingly, dissenting opinions already exist, claiming Coverity's scope was inappropriate to their conclusions."

Comments

So how did they test the proprietary software?

[pembo13]

I scanned through the article, it didn't seem to mention how they tested the top proprietary software. I can well understand that there are are a lot of bugs in open source code since it is written by humans. But human also right the proprietary code. How did they test it?

What's a bug?

[BadAnalogyGuy]

Knuth used to have this great offer where he'd send you a check for pi or e or something if you managed to find a bug in his code.

Well, what is a bug?

I doubt he'd send me a check if I told him that TeX doesn't have an easily accessible iconic user interface. No, his concept of a bug is a deviation from the specified functionality.

But what if that functionality is wrong or sucks?

Apple does really well at creating functionality that doesn't suck. They suffer from the same problems of deviations from the spec as much as anyone, but they manage to mold their spec around what users want. Microsoft, to some extent, does the same and they release products that conform to what users want (generally) because they change the spec as necessary when customers demand change.

If you are implementing towards a standard (like most OSS projects with any traction are wont to do), then you are necessarily restricted by what that spec says. If the spec says to do something inane, the standard-follower must implement it that way.

I don't really have a point here except to say that unless they say "this is what we mean by bug", there can be no way to really examine their results.

Re:How much is it really true?

[hkmwbz]

You are assuming that "a whole lot of people" actually check the code and submit patches for FOSS projects... My guess is that most testing, even of FOSS software, is done with the compiled program, not by reading the source code.

Number of Bugs vs Bug types

[Alien54]

The problem is that there are different types of Bugs. things like a typo in a help file, or American spelling vs British spelling, vs a bug were the app crashes the system when installed on a system with an early version of Quicktime are clasdsified differently.

The summary just says all bugs, which is not fair if the proprietary has 5 times the number of critical or super-critical bugs.

Re:Number of Bugs vs Bug types

[LetterRip]

Coverity scanner only checks for programming errors. Ie things that cause crashes, etc.

However as others have pointed out they are comparing mission critical software to non mission critical software. What should have been done (as has also been pointed out) is to cluster by usage case or software field. So databases to databases, browsers to browsers, generic office usage to generic office usage, etc.

LetterRip

Re:Number of Bugs vs Bug types

[marty-heyman]

I couldn't agree more. But it's really only interesting if they stop grandstanding and compare comparable products. In our case, Coverity shouldn't make any statements about Open versus Closed source unless they have some degree of comparable data for OpenLDAP versus Netscape/Red hat, Sun, IBM, CA, Novell, and Oracle (at a minimum) Directory Server products. Comparing the bug level in OpenLDAP to that of a Jet Engine control program is not only misleading (because they don't give you a measure of the cost per thousand lines of code to achieve the defect level) but irrelevant because people evaluating Directory Servers don't care what the defect level is in some other irrelevant discipline's closed soource code.

I find it unacceptable that they publish all this pretty, and generally positive, information about which projects in Open Source they scanned and then draw conclusions without telling the reader which non-Open Source Code they're comparing it to. One suspects, in the dark of night and in a paranoid frame of mind, that they're comparing to what Homeland Security COULD give them which is stuff nobody's much heard of and no Enterprise would think of deploying. Just s suspicion but one that's left on the table by the one-sided reporting.

Exactly what constitutes a software bug?

[dapsychous]

Somebody please explain to me exactly what kind of software bug can be found by automatic scanning that isn't found by standard debugging and compile-time checks. If a computer can ascertain exactly what the programmer intended to do, why do we need programmers?

The simple answer to this is that they can't. That's the point behind hiring human codeslingers to write applications. Considering that most software bugs are logic bugs (off by one, etc) that can't be directly seen in the code without actually, you know, RUNNING the program, I find it difficult to believe that AI has come to the point where it can guess the coder's intentions and infer the purpose of an application.

Additionally, where exactly did this proprietary code come from? I don't know which companies these programs were released by, but other than the SharedSource initiative which mostly caters to programming students (if they still even have this program), Microsoft would call their next o/s Windows Sh*tburger (I think I'll trademark that) before they would release their code, especially to a third party consulting firm.

I find this article highly suspect

Re:Exactly what constitutes a software bug?

[dgatwood]

Somebody please explain to me exactly what kind of software bug can be found by automatic scanning that isn't found by standard debugging and compile-time checks. If a computer can ascertain exactly what the programmer intended to do, why do we need programmers?

Security holes. Coverity specializes in programmatic detection of buffer overflows.

On a related note, as a programmer, I find open source software much more valuable than closed source because WHEN (not if) I find a critical bug, I can usually fix it myself instead of being dependent on someone else. I would GLADLY pay more for proprietary software to be able to get the source code along with it, even if I had to sign a nondisclosure agreement to get it. Nothing annoys me more than having to wait for somebody else to decide that my problem is important enough to fix. To -me-, it is the most important problem in the software.

Re:Exactly what constitutes a software bug?

[EvanED]

There's a difference though between "a code change can break the code, despite the surrounding code's quality" and "the surrounding code is very brittle; tread lightly." Stuff like "check preconditions of functions, especially if they are on a module boundary" are important and should be present. It's the idea of failing early. If I pass you bad data, and you fail in an assert, that's good; if I pass you bad data, and you corrupt your internal variants and continue running for a while, that's bad. Functions that can't trust their input and yet don't verify it are brittle.

Code like

int A[SIZE];
memset(&A, 0, SIZE);

is brittle. If the first line is changed to

int* A = malloc(SIZE * sizeof(*A));

the second line breaks. Better is to have

memset(A, 0, SIZE)

. In the first case it does the same thing, and in the second case doesn't break.

Why is this surprising?

[Chairboy]

Why does this surprise anyone? Propriety software traditionally undergoes a formalized, designed testing process. It's not perfect, but it's an ordered approach to boundary testing, design level implementation of quality, and more. Open source software must rely on after-the-fact testing in the form of "this broke when I tried to do this".

In the end, it comes down to black box vs. white box testing. Commercial software has a strong QA engineering component. Open Source software relies primarily on a black box testing approach.

Open source has MANY benefits and MANY advantages over commercial software. This just doesn't happen to be one of them, but unlike the commercial software, the bug fix cycle on open sourced stuff can be a LOT quicker, so it evens out in the end.

Not quite...

[Timothy Brownawell]

The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy. On the other hand, the open-source software was found to be of greater average overall quality.

No, *popular* open-source software is 5x as buggy as *safety-critical* closed software. The linked dissenting opinion is at least partly right; they're comparing apples to oranges.

Maybe they should try comparing open- and closed-source software that's actually trying to solve the same problem? That'd be a bit more valid of a comparison...

Re:Not quite...

[maxwell demon]

I guess all compilers will quickly stop with "syntax error", "parse error" or similar if you throw random input at them. It's highly unlikely that this way you'll trigger a bug in the compiler even if the compiler is very buggy.

Open or Closed ?

[quiberon2]

Open-source software is expensive if you want a commercial support contract (because you are asking a professional to spend a lot of time learning).

Closed-source software doesn't have the function that you want, and you cannot fix it to add the funcion that you want.

You pays your money and you takes your choice. You can always stick to pencil-and-paper, and not use this 'software' stuff at all, if you prefer.

I will bestow more wisdom upon this forum

[The_Abortionist]

It should come as no surprise that people here cannot accept the results of any study that shows concrete flaws in the open source software philosophy.

However, what this article says can only make sens.

Developping software is much more than coding. There is planning, designing, testing, etc involved. It's a whole organisation that requires a lot of organising. Open source people only put emphasise on the coding aspect, because that's what they care about most. Hence the result is often applications that look like crap, are difficult to use, are not documented and crash on startup 50% of the time.

In the end, the last bit of testing is always done by the user. I call it the test of time. This part of the software development cycle happens too early with open source software, almost all the time.

Of course, I am talking here of community developped software, not software where the company releases the source. Opening the source in itself is immaterial to the source.

Even worse.

[khasim]

He's comparing "bugs" in a project such as Apache with "bugs" in the software controlling a jet engine on an airplane.

He refuses to accept that different projects have different requirements. When the project results in people dying if it fails, you spend a LOT more money and time finding all the "bugs".

When the worst that happens is that you don't see a web page, your money/time requirements are not so high.

Even so, from his finding, Open Source is, on average, better than the closed source projects (not counting the closed source projects that result in loss-of-life in the event of a failure).

He's an idiot for confusing the different requirements.

Re:Even worse.

[phantomfive]

Don't listen to the slashdot summary. It's terrible. The author is not against open source, he talks about the "brilliant open-source community."

What this guy is trying to say (besides 'buy my software') is that open source can do better (the title of his article is "...what open-source developers can learn....."). He wants people to use stricter development practices; things like automatic testing, nightly builds, etc.

Furthermore, he is probably right, automatically testing code ala j-unit or cpp-unit is a great idea when you are getting contributions from many different people. If that became common practice in the open-source world, the code quality would improve. He's not saying open-source is bad, he's saying it could get better.

This guy is not an idiot, you just didn't understand his point.

Re:Even worse.

[phantomfive]

Yes there is some terrible proprietary code out there, but that was not my point. My point was that open-source code CAN improve with automated testing, and really there is no reason not to do it. That some proprietary code is uglier than open source is not important; what is important is that we can improve. Pointing fingers will accomplish nothing, we can do little to improve their code, but we can improve our own code and here is a way.

Re:Even worse.

[kz45]

"The many-eyeballs approach doesn't guarantee good code (alas!) but it does stop at least some of the worst practices."

This is not true.

The following are examples: Open Office, oscommerce, and mysql. All are fairly popular in the open source community and each one's sourcecode is a complete mess (oscommerce especially).

Open source can actually increase messy code, because not everyone follows the same practices. At least with proprietary software, it is usually written by the same X amount of people (X=number of people working on the project).

Re:Even worse.

[MikeFM]

I think the obvious point is that open source is a process which is evidently working since we have these independent third parties donating help to find and fix these bugs in the open source software. Yes, you may find bugs in the open source but then you are finding and fixing the bugs in the open source. It's a matter of time before the open source has fewer bugs.

Please find and report bugs whenever possible. Fix some bugs if you can. This is the process that does make open source better in the long run.

Nice way to generate publicity

[wannabgeek]

This is just smart marketing. Imagine they put up a survey that did not make any controversial claims (something like, open source and proprietary software are comparable), then would that generate as much heat? Now many people hear about the company because more people talk about this now than if the survey said something less controversial.

Now to compare every open source software application to aerospace software is really comparing apples to oranges. There is a big difference in the expected quality between an editor and an aerospace application. It's alright even if my editor crashes once in every 20 times I invoke it. Is that acceptable with an aeroplane?

I'm sure the folks at Coverity understand all this. But if they really speak what is right, they will not get all the eyeballs and publicity. In classic slashdot lingo:
1. Do something (anything) that involves open source and proprietary software
2. Make claims that sound outrageous / controversial
3. Profit! (with all the free publicity)

Lies, damned lies, and statistics

[Ibag]

If you look at the summary, you come to the conclusion that proprietary software is five times less buggy than open source. It is also unclear how software can have five times as many bugs but be of higher quality. However, if you read the article, you find:

In our research using automatic bug-hunting technology, no open-source project we analyzed had fewer software defects (per thousand lines of code) than the top-of-the-line closed-source application. That proprietary code, written for an aerospace company, is better than the best in open source--more than five times better, in fact. That company's software won't let you down when you're flying from New York to London.

If we ignore that the automatic bug finding algorithms might not be a good measure for anything, we have a few issues with the summary. The richest american is twice as rich as the richest Swiss man. Does it follow that Americans are on average twice as rich as Swiss people? No. In the same way, the statement does not imply that the average open source software has five times as many bugs as the average proprietary software does. The coding practices of mission critical apps like flight control systems are different from those of most of the industry, and it is almost wrong to lump them together with everything else.

The problem with statistics is not that they give an inaccurate picture, or even that selecting the right statistics can give a skewed picture, but that people who don't appreciate what statistics actually give use them to form opinions, make decisions, and summarize articles. Statistics don't lie, but the people who misreport them do, even if they don't realize it.

Re:So how did they test? -- badly

[fatphil]

The '11 of the top 15' is also grossly misleading. There were twice as many proprietory projects as OSS projects - you'd expect them to take 10 out of the top 15 slots. Deviation by 1 from that is lost in the noise.

I agree with your analysis - I've been on the fixing end of a lot of these kinds of reports, and have known that the flagged error can never occur, but the linting nazis insist that there must be zero warnings at any cost.

I thought Voyager was the ultimate in stable code, not the space shuttle?

FatPhil

Re:Smell Microsoft?

[canuck57]

I smell some Microsoft conspiracy behind this. :-P

Me too, as how can this statement even be verified:

The study found that no open source project had fewer software defects than proprietary code. In fact, the analysis demonstrated that proprietary code is, on average, more than five times less buggy.

This might give a hint on what is up:

Working with the Homeland Security Dept. and Stanford University, my firm, Coverity...

The US government politicians control government, including Homeland Security, this we know and we all know of Microsoft soft and hard contributions to politicians.

But the key is in what Coverity is trying to sell, a code analyzer.

I guess we cannot blame Micro$oft (directly) for this, but their results on IIS and Outlook were suspiciously not posted. Or perhaps they didn't even look at Micro$oft? In fact, we don't know what commercial source it was compared to do we?

Another flawed and cooked report for sure. Certainly the "dissenting opinions already exist" link in the original post has more merit.

Re:Smell Microsoft?

[Anonymous Coward]

> I guess we cannot blame Micro$oft (directly) for this
Or indirectly, or even mention them anywhere on the same page.

Just because something shows Open Source software in a bad light don't automatically assume that Microsoft is behind it.
I don't think Microsoft actually care that much about 99% of Open Source software out there. They probably even use a whole bunch in house.

There's a whole lot less conspiracies in the world than you think.

This is basically nuts

[vtcodger]

What they seem to have done is run a bunch of software through some sort of automatic bug checker that may or may not be a a pile of manure, identified the "best" product which chances to be what the military would call mission critical proprietary software. Then they proclaim that open source isn't as good (Duh) and doesn't meet their high standards.

What they have not done is compare comperable projects -- IE to Firefox, Open Office to MS Office, Windows to OS-X to Linux-KDE. There is, as far as I know, no Open Source software product that is really intended for mission critical applications -- I guess maybe SSH might qualify, but I don't see it in their list.

So, I think what we have here is a comparison of Apples to Turnips using a dubiously calibrated error-o-graph machine that uses an unknown technology to perform undefined tests on software.

Don't get me wrong. I sure as hell wouldn't run a nuclear power plant with Linux-X-Windows-whatever. Nor with Windows -- neither Windows 9 nor NT based Windows. They don't meet my admittedly subjective standards of quality either. But if we waited for near perfect software quality, we'd still be trying to get text mode right. Personally, I 'd vote for that because I think building major structures on weak foundations will likely lead to big trouble a decade or three out, but I think I'd lose that vote about 93 to 1 with maybe 6 abstensions.

Re:not to mention...

[linuxci]

I hate reports like this, there's so many reasons that bug counts don't prove anything. This all reminds me of the times MozillaQuest used to delight in posting Mozilla bug counts as a measure of quality (now MozillaQuest doesn't seem to mention Mozilla anymore, but a good parody of their Mozilla reporting is here).

Now these days you often get studies claiming that proprietary software is less buggy than free software, but it misses some very significant points, the ones we used to respond to MozillaQuest articles still apply very much to today:

• Free software projects very often have an open bug database so it's easy to see how many open bugs are in a project, most proprietary software doesn't have an open bug database so you have to trust the manufacturer and your own testing
• Not all bugs in open databases are really bugs. Some are requests for enhancement, some are duplicates and some are rants
• In some cases one persons bug may be another persons feature (e.g. if an application does something differently to the platform guidelines, some people may like this alternative behaviour, others will consider it a bug).
• The profit motive - companies have a lot to lose by letting people know about bugs, volunteer led projects tend to want people to know about bugs in the hope someone will help fix them (this is getting a bit blurred now that more and more organisations are making money off free software but the fact still is with proprietary software you can't fix the bugs so they gain nothing by telling you about them)

Sorry if this is redundant, I'm working on call at the moment and was halfway through typing this when I had some work to do!

Re:not to mention...

[belmolis]

Coverity's study is based on their analysis of the code itself, not on bug reports, so the considerations you mention are not relevant.

Coverity is involved, so ignore the study

[The Man]

The whole purpose of the study is to ingrain in the minds of readers the idea that Coverity makes software that can count the number of bugs in a piece of software, leading to the obvious conclusion that it can also identify them and therefore is an extremely valuable product for developers. Of course, this is not true. Coverity's products cannot tell you that your program includes an infinite loop (because it cannot solve the Halting Problem), they cannot tell you that your program will perform at a snail's pace (because the performance characteristics of a piece of software depend on the algorithms it uses, which cannot be reliably determined by examining code, as well as the performance characteristics of the machine on which they are run, which simply cannot be determined in any way by examining source), and they cannot tell you that your program is logically wrong (because they do not know what your program is supposed to do). These are, in the real world, the kinds of problems that occupy virtually all bug-fixing effort. Worse still, many of the problems that Coverity's products, like all other automated source checkers such as lint and gcc -Wextra, do report are in fact false alarms.

Coverity, of course, knows that reports like this will be written up in exactly the way this summary was, clearly associating their company with the idea of enumerating the bugs in a piece of source code. While not illegal, this type of marketing is of course deceptive; while published papers describe the type of defects (or non-defects) actually detected, the overwhelming volume of commentary will reflect the broader, and incorrect, view that Coverity == bug-finder. It would be just as meaningful (which is to say, not very) to publish the number of lint warnings or missed opportunities to qualify pointer arguments with the const keyword, and neither would require an expensive piece of overhyped software.

Just Say No to Coverity's marketing gimmicks.