US DHS Testing FOSS Security

Stony Stevenson alerts us to a US Department of Homeland Security program in which subcontractors have been examining FOSS source code for security vulnerabilities. InformationWeek.com takes a glass-half-empty approach to reporting the story, saying that for FOSS code on average 1 line in 1000 contains a security bug. From the article: 'A total of 7,826 open source project defects have been fixed through the Homeland Security review, or one every two hours since it was launched in 2006 ...' ZDNet Australia prefers to emphasize those FOSS projects that fixed every reported bug, thus achieving a clean bill of health according to DHS. These include PHP, Perl, Python, Postfix, and Samba.

Comments

RTFA

[Pinckney]

The important point here is that proprietary software manufacturers aren't telling you how many security flaws they had. I bet it's more than 1 per 1000 lines, that is an incredibly excellent figure for the first time a scanner like coverity is run. Actually, the first line of the article reads "Open source code, much like its commercial counterpart, tends to contain one security exposure for every 1,000 lines of code, according to a program launched by the Department of Homeland Security to review and tighten up open source code's security."

Re:Looking good, too bad the press didn't understa

[grcumb]

The important point here is that proprietary software manufacturers aren't telling you how many security flaws they had.

Indeed. FTFA:

"Our commercial customers wouldn't like it too much if we aired the number of defects found in their code," said Maxwell, when asked about the results from scans on 400 product lines of the firm's private customers.

One can only speculate about the, er, source of their discomfort.... 8^)

I bet it's more than 1 per 1000 lines, that is an incredibly excellent figure for the first time a scanner like coverity is run.

1 per 1000 lines is even more impressive as an average across all 180 FOSS applications tested. Most impressive of all are the highlights:

• SAMBA: 236 defects in 450,000 lines of code. 228 already fixed.
• Linux Kernel: 0.127 security faults per thousand lines of code. The kernel scan covered 3,639,322 lines of code.
• Apache: 135,916 lines of code, which yielded a security defect rate of 0.14 bugs per thousand lines of code. Or 1.4 per 10,000 lines of code, if you prefer. 8^)
• PostgreSQL: 909,148 lines of code, with a 0.041 per 1000 defect rate.
• glibc: 83 bugs in 588,931 lines of code, all since fixed.

Even some of those with more bugs have at least responded well:

• KDE: 4,712,273 lines of code, fixed 1,554 defects, verified another 25 and has only 65 to go.
• GNOME: 430,809 lines of code, fixed 357 defects, verified 5 and has 214 to go.

And my favourite 'backslider' of all, OpenVPN, has yet to fix 100% of the bugs found during this exercise. Of course, that's only 1 bug in over 69,000 lines of code....

These results should be viewed as excellent, by and large. This doesn't mean all this software is bug-free, just that there aren't a lot of easily preventable bugs in the code base. Most encouraging, though, is how fast they got addressed and fixed by the healthier FOSS projects.

Pessimism in article

[filbranden]

Not only did the article say much like its commercial counterpart, but most of the numbers it shows are actually good for open source software.

For instance, most of the projects discussed had less than 1 bug for 1000 lines of code. For instance, the Linux kernel had .127 bugs per 1000 lines, and that on over 3 million lines of code.

Also, the article talks about key projects, such as the glibc (which is basically used by everything on a Linux system) that already fixed all the issues.

Even something huge and complex as Firefox has already fixed half of the issues, and is showing progress on the rest of them (by the fact that some were already verified).

Overall, I didn't get the half glass empty tone that the summary is implying. And what I found strange is that even the comments on the site itself, and many of them on /. itself, are also taking the pessimistic view.

I thought that this news are great for open source software. Shows that it has less security issues than average, that the issues are fixed quickly, and still that some programs are certified by a company for use in security related departments such as the DHS. What could be better than that?

Re:Wow... FOSS looks pretty pathetic

[mr_mischief]

There are industry estimates that say average code in production contains 2 bugs per thousand lines of code. Some say that number is much higher. How many lines do you think are in Vista?

Yes, OSS has bugs. Everything from compilers to content management systems, surely. So do proprietary programs.

The more qualified eyes you get on a bug, the better chance you have of finding and fixing it. You can do that by having a big staff that pores over code again and again. You can do it by having lots of outside help, like in the case of popular OSS projects. One thing that helps is to have a fresh set of eyes look over something, which is much easier in OSS that in closed-source applications.

BusinessWeek had an article from a guy at Coverity back in 2006 about this. In that article, Ben Chelf said that 4 of the top 15 programs on the quality scale measured by defects per thousand lines of code were OSS. He said that on average, the major-project OSS software they tested was indeed higher quality software than average. He said, though, that the absolute highest quality code was the cream-of-the-crop proprietary, closed source code from places that make things like fly-by-wire systems. Well, yeah. I'd want my airliner's fly-by-wire system completely bug-free, too.

Commercial software tends to harbor anywhere from 1 to 7 bugs per 1000 lines of code according to the National Cybersecurity Partnership's Working Group on the Software Lifecycle. Voluntary testing by Coverity requested (and probably paid for) by MySQL AB revealed that project to have all of 97 flaws, one of which could be a serious security issue. All 97 were to be fixed for the next release.

A similar study (same link) found 985 bugs in over 5,700,000 lines in the Linux kernel, or fewer than one bug per 10,000 lines of code. TFA has data on a newer version of the kernel -- 0.127 bugs per TLOC.

In Apache, 22 bugs total, 0.14 per TLOC, and three fixed so far.

PostgreSQL had 0.041 per TLOC, and have so far fixed 53 of the 90 bugs.

The glibc team fixed 83 of 83 bugs found.

OpenVPN had found one security-related bug in over 69,000 lines of code. As of later yesterday, it's officially security bug free according to the same testing people.

The list of officially security-bug free software includes Amanda, NTP, OpenPAM, OpenVPN, Overdose, Perl, PHP, Postfix, Python, Samba, and TCL.

So with Linux (0.127), glibc (0.000), Apache (0.140), PostgresSQL (0.041), Perl (0.024), PHP (0.000), and Python (0.000) powering a web server (numbers according to Coverity), you have 0.0474 defects per thousand lines of code across the server. I'd say that's pretty good.