Slashdot Mirror

Those types of tools are generally free if you are developing using a qualifying open source license. Try http://scan.coverity.com./ Yes, I'm affiliated, but I don't get any compensation when people use the service.

For example, this: https://scan.coverity.com/proj...

They state 0.45 defects/kLOC. Of course, they won't "find" all defects... and there might be some false positives in there. But you get the ballpark.

Use your favourite search engine (hopefully not Google and its ilk).

Kids, these days. When I was young, I queried Altavista with telnet. Tsk, tsk.

We can require everyone to use formal methods, but don't expect any updates to OpenSSL/LibreSSL this decade.

It would cost $2.4 billion to reduce the bug density in the Linux kernel to 0.00045 or less and keep it there for a year.

Current status: https://scan.coverity.com/proj...

That's very nearly bug-free. It would actually be 100% bug-free in all components that don't require features that are inherently unreliable. The government could afford it, most corporations could not.

I would actually like that for Linux, have a huge program to perform a proper detailed clean-up of the entire kernel. No loss of functionality, just a loss of bugs. It's used in many important areas and no system can be more reliable than the OS it uses.

But you can't ask people to design KDE that way (although they could design it better), nor could you ask a commercial vendor like Oracle to get their database to that standard. Only a government has the money needed and even then only for a few projects.

When it comes to encryption, it's worse. We don't know what constitutes good, we only know some things that constitute bad. Same for authentication. Ergo, we can define minimum standards by defining what is bad, but we can't define anything better.

Open source doesn't help, since nobody does test driven development and almost nobody tests. Documentation is dreadful. Want to show otherwise? Sure, go ahead. Reply to this with a file in CPNTools format that shows the full state machine for the IPv6 stack. That should be easy, you have RFCs showing the datagrams and state changes.

Such a diagram can be drawn, but not by anyone here in any sane length of time. That's full-time work for a large team of high-end experts.

Coverity certainly does *not* work one compilation unit at a time. It builds up an entire callgraph and then does a depth-first post-order traversal. Static analyzers like Coverity get confused for one of two reasons. The first is that they are approximating an NP problem with a polynomial one. This means that sometimes a heuristic has to be used instead of an exact calculation. The other reason is because Coverity (and other good static analyzers) use an AST as part of their input and have their own compilers. If the compiler misunderstands a commonly used construct, you'll get a lot of very similar FPs. In terms of libc it has been run through tools like Coverity so many times that you're unlikely to find many real defects left unfixed. https://scan.coverity.com/proj...

That's what their competitors actually do. https://scan.coverity.com/

If you want something better, there's Coverity. Free if you qualify. If not, it's even more expensive than PVS-Studio, but does a heck of a lot better job.

FreeBSD has been analyzed by Coverity for years... did it not catch the problems that PVS-Studio found?

You seem to be somewhat confused here:

Coverty is static code analysis that find potential stability and security problems. https://scan.coverity.com/
Think "Unix Lint".

Many FOSS projects like the Linux kernel and LibreOffice uses Coverty. It isn't a magic wand, but it plain works nevertheless. No systemd contributor needs a licence in order to contribute to systemd, even if they get commit access. Coverty can trivially be replaced or complemented by other static code analysers, or even be omitted; it is something helpful, not something systemd depend upon.

The Jenkins builder provides: "Continuous Integration with automated test execution", something that really improves code stability since it ensures that eg. a patch isn't committed if it breaks the build.

Again, no contributor needs any kinds of licence for this, and again, this is a nice to have thing, not something systemd depends upon.

You seem to be arguing against code discipline and the use of automated tools, just because systemd have them (while SysVinit doesn't). That is just bizarre.

Anyway, systemd's developer community is extremely vibrant and growing all the time; there are hundreds of people who have contributed already with new ones coming every month. So there doesn't seem to be anything that scares contributors away, unlike, sorry for putting the boot in, the tiny and deteriorating non-systemd developer eco-system.

They were vulnerable because openssl took specific measures to counteract the defense mechanisms present in openbsd. See this writeup.

Even Coverity could not detect the problem.

Critical pieces *do* get checked. And have been checked. The Covarity Code Checker(tm) is a software application that tested large parts of code (funded by the United States Government Department of Homeland Security). The checked open source software and (because they are part of the US government) closed source software. Critical parts of Open Source (specifically the Linux kernel among other pieces of software). The results reveal that, yes, Open Source (at least the pieces tested) *really are* more secure and are created to higher standards than the commercial software also tested. The commercial software included operating systems from leading vendors (you can probably guess who they are). So its not just FOSS doing code reviews, the DHS is doing them too (as are others).

Sorry to break it to you, but Coverity's free-open source scanning was originally funded by the DHS. :-)

After the DHS grant expired in 2009, Coverity continued the service pro bono.

This new program seems like a step back, though. Now, if the DHS was instead investing in improving the open-source tools, it would make sense.

I trust Coverity's Scan program far more than I'll trust the organization that continues to promote security theater. DHS has no business in this area. This is typical over expansion of a bloated bureaucracy.

Coverity has a blog post describing the problem and why their static analysis methods currently can't detect it.

No dangerous construct warning would have caught heartbleed

Coverity has already come up with such a contruct warning. It is very hindsight oriented.

I'm not sure which part of your long post is the most optimistic. Belief in perfect test coverage? By definition bugs come from things you (and/or the fuzzing developers) didn't think to test. That's at least a noble goal. Can't say the same about deciding to build custom VPN software. You've got some hubris dude.

I believe that Coverity did not find the defect, since it was external information that was tainted.

Using an in-house version, they were able to modify their scan to detect the issue

OpenSSL is on the list of projects scanned by Coverity.

I wonder why exactly Coverity did not catch the heartbleed bug. Most likely, the scan wasn't set up to deal with OpenSSL's use of it's own internal heap management routines. That's something that I would have thought should be fixed right away.

Indeed, who would review other people's code for free or for fun?

Well, right offhand, Coverity will. They're not perfect, of course, but they're pretty good. Their system didn't flag Heartbleed, but Heartbleed showed them how they could add a new test that would and that has reportedly found other possible issues, which are being investigated and will either be fixed or found to be false positives and used to refine the new test. Either way, not a bad thing.

Indeed, who would review other people's code for free or for fun?

Well, right offhand, Coverity will. They're not perfect, of course, but they're pretty good. Their system didn't flag Heartbleed, but Heartbleed showed them how they could add a new test that would and that has reportedly found other possible issues, which are being investigated and will either be fixed or found to be false positives and used to refine the new test. Either way, not a bad thing.

Disclaimer, I work for Coverity. There's a write-up on why Coverity didn't find it out of the box here:

http://security.coverity.com/b...

Bitch about this instead. A fucking static checker found heartbleed.

No, it says, "Coverity did not find the heartbleed bug itself", which very clearly means that Coverity did not find Heartbleed. And Coverity themselves confim that Coverity does not detect the problem (though in response, they've added a new heuristic that does detect it, but no word on how the new heuristic affects the false positive rate).

Warning: pdf
http://wpcme.coverity.com/wp-content/uploads/2012-Coverity-Scan-Report.pdf
explains much if not all that you ask

For a good article and a fun read that goes into the background of Coverity and what it does, see
http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext
it's written by some of the developers and founders

If only there was a way to get more information, somehow.

Find and fix defects in your C/C++ or Java open source project for free.

A pity it's free. They must be implying that open source software isn't worth any money, right?

So, a private company has been helping 400 open source projects with code quality (usually considered important) for quite some time now using their tools which find many different code defects. It had been started with government money, but now they take it out of hide. And do you shed any light on it? Provide more information? No, you just make uninformed comments about things that have easy to find answers and whine. What a waste.

Some of the better-known projects scanned include Apache, Firefox, GIMP and a number of forms of Linux and BSD.

Open Source Is Better Than the Closed Stuff (Until You Hit 1 Million Lines)

A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

So, a private company has been helping 400 open source projects with code quality (usually considered important) for quite some time now using their tools which find many different code defects. It had been started with government money, but now they take it out of hide. And do you shed any light on it? Provide more information? No, you just make uninformed comments about things that have easy to find answers and whine. What a waste.

Some of the better-known projects scanned include Apache, Firefox, GIMP and a number of forms of Linux and BSD.

Open Source Is Better Than the Closed Stuff (Until You Hit 1 Million Lines)

A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

So, a private company has been helping 400 open source projects with code quality (usually considered important) for quite some time now using their tools which find many different code defects. It had been started with government money, but now they take it out of hide. And do you shed any light on it? Provide more information? No, you just make uninformed comments about things that have easy to find answers and whine. What a waste.

Some of the better-known projects scanned include Apache, Firefox, GIMP and a number of forms of Linux and BSD.

Open Source Is Better Than the Closed Stuff (Until You Hit 1 Million Lines)

A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

Here is the data sheet (.pdf) that should help you understand.

Here is some addition detail on the common problems (.pdf) it looks for.

Here is a background article: A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

Here is the data sheet (.pdf) that should help you understand.

Here is some addition detail on the common problems (.pdf) it looks for.

Here is a background article: A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World

Coverity sells software that does static analysis on source code and looks for patterns that suggest defects. E.G., a code sequence that allocates memory, followed later by something that de-allocates that memory, followed later by something that de-allocates the same memory again (a double-free).

The product is not open source software, but a number of open source software projects use it to scan their software to find defects: https://scan.coverity.com/ It's a win-win, in the sense that Coverity gets reports from real users using it on real code, as well as press for their product. The open source software projects get reports on potential defects before users have to suffer with them.

I see where you're going with this: high-quality software doesn't exist. Touché.

Oh, yeah - there really isn't much of a distinction of quality whether closed or open. There is a distinction of freedom.

Open source projects, Coverity says, tend to have .69 bugs per thousand lines of code, virtually the same as proprietary software, which tends to have .68 errors per thousand lines.

See the Coverity report.

Heh. Haha. BWAHAHAHAHAHAHAH!

Not really, most of each of thousands of projects have at most a few core developers and extraneous people who occasionally submit patches to fix specific itches. There is no "A team" scouring all open source for vulnerabilities from the simple fact such vulnerabilities most certainly do exist as innocent bugs and have not been reported by such teams.

To illustrate this point the linux kernel is developed by armies of smart people yet an automated tool found a laundry list of shit that has been around for years nobody noticed.

http://www.coverity.com/library/pdf/linux_report.pdf

First, from the very report that you linked to:

The results show that the number of defects detected by the Coverity analysis system has decreased from over 2000 to less than 1000 while, during the same period of time, the source code has quadrupled in size and the power of Coverity's detection capabilities has increased markedly. We conclude using this data that the Linux kernel is a robust, secure system that has matured significantly.

You want a real eye opener? Check out Coverity's current press release:

Code quality for open source software continues to mirror that of proprietary softwareâ"and both continue to surpass the accepted industry standard for good software quality. Defect density (defects per 1,000 lines of software code) is a commonly used measurement for software quality. Coverityâ(TM)s analysis found an average defect density of .69 for open source software projects that leverage the Coverity Scan service, and an average defect density of .68 for proprietary code developed by Coverity enterprise customers. Both have better quality as compared to the accepted industry standard defect density for good quality software of 1.0. This marks the second, consecutive year that both open source code and proprietary code scanned by Coverity have achieved defect density below 1.0.

(snip)

Linux remains a benchmark for quality. Since the original Coverity Scan report in 2008, scanned versions of Linux have consistently achieved a defect density of less than 1.0, and versions scanned in 2011 and 2012 demonstrated a defect density below .7. In 2011, Coverity scanned more than 6.8 million lines of Linux code and found a defect density of .62. In 2012, Coverity scanned more than 7.4 million lines of Linux code and found a defect density of .66. At the time of this report, Coverity scanned 7.6 million lines of code in Linux 3.8 and found a defect density of .59.

(snip)

While static analysis has long been cited for its potential to improve code quality, there have been two significant barriers to its adoption by development organizations: high false positive rates and a lack of actionable guidance to help developers easily fix defects. Coverity has eliminated both of these obstacles. The 2012 Scan Report demonstrated a false positive rate for Coverity static analysis of just 9.7 percent in open source projects. Additionally, the 2012 report noted more than 21,000 defects were fixed in open source codeâ"more than the combined total of defects fixed from 2008-2011.

The real conclusion that you should draw is twofold. First, if you're relying on software that isn't doing static code analysis, you're probably relying upon insecure code.

Second, Every. Single. App. Has. Bugs. The difference is that open source lets anyone do the analysis and fix the bugs. The same can't be said when of any closed source package.

So, which is safer? The OSS app where everything is publicly discussed and bug fixes generally get acted upon fast, or the closed source app where the vendor may be handing the known vulnerabilities off to the NSA or its equivalent in the country of your choice? I know which way I choose. :-)

But equally there are thousands of really talented programmers who examine the source code very thoroughly, many of whom contribute back.

Not really, most of each of thousands of projects have at most a few core developers and extraneous people who occasionally submit patches to fix specific itches. There is no "A team" scouring all open source for vulnerabilities from the simple fact such vulnerabilities most certainly do exist as innocent bugs and have not been reported by such teams.

To illustrate this point the linux kernel is developed by armies of smart people yet an automated tool found a laundry list of shit that has been around for years nobody noticed.

http://www.coverity.com/library/pdf/linux_report.pdf

If there were back doors then there is a high chance that they would have been detected.

There is no difference between a backdoor and a vulnerability. The logic that deliberate backdoors would be detectable in source code when we know from experience innocent bugs having the same effect as a backdoor have a proven track record of not being detectable is simply wishful thinking and wrong.

Plus anyone really paranoid about it CAN go and check the source code to make sure for themselves.

I suppose anyone can drain the earths oceans with an eye dropper as well.

What are they counting as a "defect"?

Their FAQ lists example, but ends with "and many more".

Which leads us to the question of who set the "industry standard" at 1.0, and what did THEY define "defect" to mean? If it is a standard there should be a standard list of defect types.

Considering that Microsoft is listed as a Coverity customer here (select the "Software and Internet" tab), it's actually quite possible that they do have access to Windows metrics. I know it's gonna seem impossible to believe, but Microsoft isn't still shipping Windows ME. Their software has dramatically improved in recent years. It's not perfect, but it's quite a bit more stable than it used to be.

And for all the people whining about selection bias, there isn't any more bias for FOSS than there is for Proprietary: FOSS projects have to register with coverity to be included in their scan service. Any project reporting to Coverity (FOSS or Proprietary) must value quality enough to USE Coverity and report metrics. Coverity is not just spidering SourceForge and going "LOL FOSS SUCKS."

Here's the list of open source projects that Coverity covers currently - looking through the list, there's quite a few heavy hitters in there (including the Linux kernel, which their report notes as a "benchmark" for FOSS quality), and not a lot of one-off SourceForge abominations.

Considering that Microsoft is listed as a Coverity customer here (select the "Software and Internet" tab), it's actually quite possible that they do have access to Windows metrics. I know it's gonna seem impossible to believe, but Microsoft isn't still shipping Windows ME. Their software has dramatically improved in recent years. It's not perfect, but it's quite a bit more stable than it used to be.

And for all the people whining about selection bias, there isn't any more bias for FOSS than there is for Proprietary: FOSS projects have to register with coverity to be included in their scan service. Any project reporting to Coverity (FOSS or Proprietary) must value quality enough to USE Coverity and report metrics. Coverity is not just spidering SourceForge and going "LOL FOSS SUCKS."

Here's the list of open source projects that Coverity covers currently - looking through the list, there's quite a few heavy hitters in there (including the Linux kernel, which their report notes as a "benchmark" for FOSS quality), and not a lot of one-off SourceForge abominations.

Apparently you suffer from ding-battery. There was a long series of tests done about 5 years ago sponsored by the United States Government and done by the University of California at Berkeley and Coverity. A summary of the results are here. The lie perpetuated by guessers and those suffering from ding-battery gets killed firmly. Open Source is *BETTER* than closed source proprietary software. It has, in general, about 10x fewer bugs, and the severity of those bugs are 10x less severe or critical than proprietary software. Nincompoops asserting otherwise, are talking through their hats. They can say "open source is bad", but "et suppositio nil ponit in esse" --sayin' it don't make it so--.

I've been pretty impressed by the training my company has been able to put together lately.

• Seth Hallem, founder and former CEO of Coverity came to teach us about their static analysis tool.
• Dan Saks came to teach us about embedded software best practices.
• Scott Meyers came to teach us about using the STL effectively.
• James Grenning came to teach us about test driven development.
• Michael Barr came to teach us about real time scheduling.

Most of these guys are well respected in their fields, and while not exactly famous, are names I had seen more than once in connection with those topics. All of them spent some time looking at our company's needs specifically before doing the training in order to customize it for us. Our company isn't small, but not huge either. We have around 1600 employees, a few hundred of which took the training. It has really helped us revitalize a lot of our old school techniques. If a company our size can put a line up of training like that together, it ought to be within reach of most mid-size organizations.

Citation needed.

Ok then, the first hit from "openbsd auditing" leads to an OpenBSD Security page which has a section claiming that OpenBSD has a continual audit process and that it is successful..

I'm not necessarily thinking the opposite, but is OpenBSD really that much audited? Are we talking about the kernel? The network stack? Or the encryption protocols?

As I understand it, OpenBSD refers to the whole release, everything they ship.

Now, I'm not sure if claims from the OpenBSD marketing department actually translates to a citation.. I am of the feeling that an "audit" would imply that there were specific procedures followed (searching for particular algorithms known to be problematic for instance) and specific records kept of the results but I have never been able to find public records of those.. Yes, there is the OpenBSD CVS repository but that includes things which are not part of any audit. I found a quote from Theo de Raadt "Most bugs in software are the same ten to fifteen mistakes made over and over" but I don't know what he thinks those mistakes actually might be.

So in conclusion I [as a NetBSD devloper] would be interested to see such records.. I know that many open source projects are subject to Coverity scans which are more public, though it seems that OpenBSD is not listed at this time..

http://scan.coverity.com/

I don't believe Google would have paid an outside firm. After all, people at Google view themselves as "the best of the best."
That doesn't mean they have all the skill sets. They don't have a static analysis tool.

I must be missing the link to the study results. Oh, won't be out until next year, to allow for patching.

So, maybe something, maybe nothing.

There are better release from Coverity's site, http://coverity.com/

I made my submission after first seeing a story in El Reg. While I saw it in several other places, I thought the Dark Reading story was a bit better in highlighting the findings than most. You're right, though. It's very light on the methodology. The press release on VeraCode's site has a bit more information:

...1,600 Internally Developed, Open Source, Outsourced, and Commercial applications analyzed when first submitted to Veracode...

...the first report of its kind to provide security intelligence derived from multiple testing methodologies (static, dynamic and manual) on the full spectrum of application types (components, shared libraries, web and non-web applications) and programming languages (including Java, C/C++ and .NET) from every part of the software supply chain on which organizations depend.

...analyzing billions of lines of code submitted to Veracode for independent verification of software security from more than 15 industries.

(emphasis added)

So. The sample consists of approximately 1,600 applications consisting of billions of lines of code from self selecting organizations; organizations who have an interest in writing the most secure code possible or they wouldn't be subjecting themselves to this process or paying for the service. And still, 60% of all these apps fail the first time through!

I've been following testing results for FOSS for years. I've waded through thesis papers, press releases, magazine articles, Coverity's Scan site, you name it and I've dug into it. Virtually everything else that I've come across covered just a single means; static or dynamic code analysis, pen testing, fuzz testing, bug report analysis, mathematical breakdowns that attempt to address the "why" FOSS works so well, etc. The press release defines a sample size that is at least within shouting distance as the largest two that I know of; a study commissioned by the European Commission to analyze the economic impact of FLOSS which briefly discusses bug fixing, Coverity, and Coverity again.

At most, they might have tackled two methodologies in a single article. This is the first such announcement that I've been able to find that covers multiple methodologies. IMNSHO, that's what makes this an important story. Slashvertisement or not.

(BTW, note that the original announcment was at RSA Conference 2010. I suppose that makes it a RSAvertisement first? Nahhh. Doesn't trip off the tongue. ;) )

Bugs are an error in the process, not the code. If you find a bug, you need to find the process error that allowed that bug to occur.

Agreed!

I read, with interest, the referenced article. I was expecting FUD - but I didn't find much, until I reached the Conclusion.

eg.

The many eyeballs argument is neat, tidy, compelling, and wrong.

The article starts with

Eric S. Raymond wrote , “Given enough eyeballs, all bugs are shallow.” He calls this Linus’ law.

and then attempts to refute. Fair enough. Except - the link leads to The Cathedral And The Bazaar - where I cannot find the quote... Hmmm

Now this might be relevant if the "many eyes" routine was the only form of audit used in GNU/Linux - but is not the only form of review/audit used. I'm sure other, more knowledgable posters will be able to provide more evidence than I could find in a quick search.

I call FUD

http://scan.coverity.com/faq.html#static

Some examples of the defects include:

        * leaked resources
        * references to pointers that could be NULL
        * references to pointers that are guaranteed to be NULL
        * use of uninitialized data
        * array overruns
        * unsafe use of signed values
        * use of resources that have been freed

It won't fix misplaced decimals (sorry Mr. Bolton), but it'll probably catch if you try sticking an item into $array[11] when you've defined @array[10].

(Yes, I know I used perl syntax which doesn't give a whit, but it's what I'm familiar with. It's the theory I'm trying to convey.)

From the press release:

Since 2006, more than 11,200 defects in open source programs have been eliminated as a result of using the Coverity Scan service.

While this is good for open source and demonstrates the value of static analysis, it is not surprising that if you fix the issues found, the number of issues remaining will go down.

According to the coverity report http://scan.coverity.com/rung2.html Perl does not have the lowest defect rate.

I think you should try Coverity Prevent. You will be amased how one pointer set to NULL only under certain set of circumstances and passed through several functions gets dereferenced - and Prevent will catch it. I have been in the industry for many years and had a clear idea what is possible what is not. Well, my ideas had to change after using Prevent. See http://www.coverity.com./ There are some other tools on the market, but their depth is not even close.

valgrind, BoundsChecker, and I believe the others mentioned, are all run-time error checkers. These require a test case that execises the bug. The static analysis tools the poster was asking about, like those from Coverity and Green Hills, don't need test cases. They work by analyzing the actual semantics of the source code. I've found bugs with tools like these in code that was hard enough to read that I had to write test cases to verify that the tool was right. And it was! The bug would have caused an array overflow write under the right conditions.

That was Coverity. We've found the results very useful for Parrot. You have to be smart about how you interpret the results and how you fix actual bugs, but their tools did reveal many dubious constructs and actual bugs.

It's hard to test whether you've broken a driver when you don't have the hardware to test with. Perhaps the future will be Qemu emulation of all the different hardware in your system : )

This is not to say that there need to be tests for things that can be caught at compile time or run time regardless of hardware but there is only so far you can take it.

It's not like the kernel doesn't have any testing done on it though. There's the Linux Test Project which seems to test new kernel's nightly. If you ever look in the kernel hacking menu of the kernel configuration you will see tests ranging from Ingo Molnar's lock dependency tester (which checks to see locks are taken in the right order at run time), memory poisoning, spurious IRQ at un/registration time, rcu torture testing, softlockup testing, stack overflow checking, marking parts of the kernel readonly, changing page attributes every 30 seconds... Couple that with people like Coverity reporting static analysis checks on the code. Tools like sparse have been developed to try and so some of the static checks on kernel developer machines while they are building the code.

But this is not enough. Bugs STILL get through and there are still no go areas of code. If you've got the skills to write tests for the Linux kernel PLEASE do! Even having more people testing and reporting issues with the latest releases of the kernel would also help. It's only going to get more buggy without help...

If you are involved in said projects, please contact coverity through the website and get involved. I don't see any reason why a project would not want to have this scan done.

Rung 0: http://scan.coverity.com/rung0.html

There are industry estimates that say average code in production contains 2 bugs per thousand lines of code. Some say that number is much higher. How many lines do you think are in Vista?

Yes, OSS has bugs. Everything from compilers to content management systems, surely. So do proprietary programs.

The more qualified eyes you get on a bug, the better chance you have of finding and fixing it. You can do that by having a big staff that pores over code again and again. You can do it by having lots of outside help, like in the case of popular OSS projects. One thing that helps is to have a fresh set of eyes look over something, which is much easier in OSS that in closed-source applications.

BusinessWeek had an article from a guy at Coverity back in 2006 about this. In that article, Ben Chelf said that 4 of the top 15 programs on the quality scale measured by defects per thousand lines of code were OSS. He said that on average, the major-project OSS software they tested was indeed higher quality software than average. He said, though, that the absolute highest quality code was the cream-of-the-crop proprietary, closed source code from places that make things like fly-by-wire systems. Well, yeah. I'd want my airliner's fly-by-wire system completely bug-free, too.

Commercial software tends to harbor anywhere from 1 to 7 bugs per 1000 lines of code according to the National Cybersecurity Partnership's Working Group on the Software Lifecycle. Voluntary testing by Coverity requested (and probably paid for) by MySQL AB revealed that project to have all of 97 flaws, one of which could be a serious security issue. All 97 were to be fixed for the next release.

A similar study (same link) found 985 bugs in over 5,700,000 lines in the Linux kernel, or fewer than one bug per 10,000 lines of code. TFA has data on a newer version of the kernel -- 0.127 bugs per TLOC.

In Apache, 22 bugs total, 0.14 per TLOC, and three fixed so far.

PostgreSQL had 0.041 per TLOC, and have so far fixed 53 of the 90 bugs.

The glibc team fixed 83 of 83 bugs found.

OpenVPN had found one security-related bug in over 69,000 lines of code. As of later yesterday, it's officially security bug free according to the same testing people.

The list of officially security-bug free software includes Amanda, NTP, OpenPAM, OpenVPN, Overdose, Perl, PHP, Postfix, Python, Samba, and TCL.

So with Linux (0.127), glibc (0.000), Apache (0.140), PostgresSQL (0.041), Perl (0.024), PHP (0.000), and Python (0.000) powering a web server (numbers according to Coverity), you have 0.0474 defects per thousand lines of code across the server. I'd say that's pretty good.