450 Million Lines of Code Can't Be Wrong: How Open Source Stacks Up

An anonymous reader writes "A new report details the analysis of more than 450 million lines of software through the Coverity Scan service, which began as the largest public-private sector research project focused on open source software integrity, and was initiated between Coverity and the U.S. Department of Homeland Security in 2006. Code quality for open source software continues to mirror that of proprietary software — and both continue to surpass the industry standard for software quality. Defect density (defects per 1,000 lines of software code) is a commonly used measurement for software quality. The analysis found an average defect density of .69 for open source software projects, and an average defect density of .68 for proprietary code."

Comments

Correction

[Sla$hPot]

"450 Million Lines of Code Can't Be Wrong"
should have been
"450 Million Lines of Code Can't ALL Be Wrong"

Re:Correction

[hduff]

It mean over 300,000 lines of code are wrong, most of it in the app I keep trying to use.

65 million lines of HOST file can't be wrong

[Anonymous Coward]

Just ask apk!

it contradicts the definition

[nimbius]

the very definition of 'proprietary software' indicates you dont have access to the code to calculate defect density, and even if you did you cannot independently verify the code you have is production code. how did the researchers quantify it?

Re:it contradicts the definition

[GrugVoth]

We use coverity where I work on proprietary code and part of their service is to report, anonymously obviously, the defect count, type and lines of code etc back to coverity if you want to. Via this they can get an idea of the defects found using their tool over a very large code base.

Re:it contradicts the definition

[Chris Mattern]

We use coverity where I work on proprietary code and part of their service is to report, anonymously obviously, the defect count, type and lines of code etc back to coverity IF YOU WANT TO.

Am I detecting a selection bias here? Coverity can run their tests against all of open source. Coverity can run their tests only against that proprietary code that decides to use it and report the results--and it strikes me that only the better, and more open, proprietary shops would be doing this. Is Mircrosoft reporting their code? I doubt it. Is Oracle?

and all the children are above average

[Anonymous Coward]

"Code quality for open source software continues to mirror that of proprietary software — and both continue to surpass the industry standard for software quality."

What is this third kind of software that is neither open source nor proprietary which is bringing down the average industry standard for software quality? Because if there is only open source and proprietary then they can't both be better than average. Or perhaps the programmers are from Lake Wobegon?

Re:and all the children are above average

[ZorroXXX]

The selection of sample projects is biased. For proprietary software, the data is taken from projects that at least cares as much for code quality that they run some tools (e.g. at least Coverity) to analyse it. I would suspect that the industry standard is below that because there exists some companies that mostly only consider "get the product out the door". For open source the selection is probably also somewhat scewed, in that they have analysed relatively large, mature and highly successfull projects. I would assume those have higher quality than the average sourceforge/github project.

Re:and all the children are above average

[Gr33nJ3ll0]

This is a good point. To build on it, the results reported from the propertiary code has had coverity at least run against it, and usually the problems that it reports fixed. This does not appear to have been done in the case of the Open Source software, which was just scanned, but never given a chance to fix. In that circumstance I would have expected a much much higher result for the Open Source software, because Coverity often reports on very pedantic issues, which are often not important to overall software quality. Further these issues would not show up in anything other than Coverity, making the initial scan the first time these issues were brought to life.

Re:Defects fixed for proprietary may differ.

[Cenan]

Propietary defects are ones that may cause financial harm. FOSS defects are ones that cause annoyance.

I know that our code has more defects than we'd consider fixing purely because the CBA isn't there.

I'm guessing you mean defects in propietary software only gets fixed if they have an impact on the bottom line? Otherwise that whole reply makes no sense.

Anyways, that is not much different from the OSS model. Whoever cares about the sub-system that has a bug, fixes it, and if nobody cares (or has the skills to fix it) it can go ignored for years. The selector for OSS is different, but the end result is the same: nobody gives a fuck about the end user unless it directly affects their day/paycheck/e-peen.

What else is in the "industry"?

[ZahrGnosis]

and both [proprietary and open-source software] continue to surpass the industry standard for software quality

... What else is there? And why is this unknown third type of code dragging down the "industry"?

Re:Some things can't be measured objectively

[gorzek]

You are wrong, and here's why.

With no measurements at all, you cannot make informed judgments about the quality of your software. You can only guess. This means you would be unable to convince anyone (sane and intelligent) that your product has n bugs. "Because I say so" is not a metric.

With a poor measurement--such as one that ranks all defects equally--you have information, but now it's bad information. If you share the information but not the method(s) used to gather it, you can convince people you're right, because you have data about it. Never mind if you are stacking up Product A with 1 show-stopping bug against Product B with 50 cosmetic bugs or unhandled corner cases. By this bugcount-only metric, Product A looks better, and that's just stupid.

You need good measurements, and sometimes that includes measurements which cannot be quantitatively calculated without human intervention. A human programmer (or QA or other support person) who is familiar with a product will know just how severe a given bug is in terms of its impact. It is why, after all, bug tracking systems generally allow you to prioritize work by severity, fixing the worst bugs first.

Poor information is worse than no information because it can lead you to make the wrong decisions with confidence. With no information, at least you know you are shooting in the dark.