Slashdot Mirror

While there isn't a lot of research on the topic, what there is seems to indicate that open source software and "commercial" (i.e. not open source) software have similar defect densities. Here's a paper on the topic:

http://www.reasoning.com/pdf/MySQL_White_Paper.pdf

I've worked on both "commercial" projects and opens source projects. My personal experience has been that willingness to fix bugs is much higher on the open source teams. Usually the authors actually use the product in their everyday life and bugs affect them personally. They are highly motivated to fix them. On "commercial" software projects, my experience is that the authors of the software rarely use it in their every day life. Selection of bugs to fix usually comes from a project manager.

Both open source and "commercial" projects usually have large backlogs of bugs that never get fixed. The difference is that open source projects usually fix bugs that directly affect the authors, while "commercial" projects fix bugs that directly affect customers who have bought enough units to gain the right to complain (i.e. thousands of copies). However, with an open source project, if the authors decide not to fix a bug you usually have a number of options. You can complain on the developer's mailing list and plead your case. If that doesn't work, you can fix the bug yourself, or hire someone else to do it. With "commercial" software once you file your bug you usually don't even know if they will decide to fix it. You usually don't even know if it was fixed in the next version without buying it and trying it for yourself. If the program manager decides not to fix your bug, you have no recourse at all unless you have already bought thousands of copies of the software and can threaten not to upgrade to the next version unless they fix your bug. Even then they might decide not to.

"Well that's the thing with volunteer made software isn't it?" is what you wrote. Yes. That's the thing with volunteer made software. You have direct access to the developers to report your bug. You get informed whether your bug when your bug is being looked at. You can make a case for having your bug fixed. And if that doesn't work, you can fix it yourself. What is there that needs fixing again?

I work on a C/C++ code base that is a lot bigger than 500k lines. I've worked with results produced by Klocwork and also with the output from Reasoning. Both of these services/packages will cost you money but both provide good insight into your code. The commercial packages generally produce more focused results with less false-positives, so while they cost you money up front, your developers will spend less time weeding out the noise.

If paying money out for a commercial package isn't your thing, don't overlook the old standby lint or splint, an updated successor.

Also well worth investigating to see how your code is actually running is Valgrind and it's associated tools. The Valgrind toolkit will give you a good idea where memory is being leaked, where variables and pointers are going off the rails. Valgrind hooks into a running program, so it's important to make sure that you test all the corners of the codebase if you go this route.

Cheers,
Toby Haynes

MySQL Defect Report: http://www.reasoning.com/pdf/MySQL_Defect_Report.p df
MySQL Metric Report: http://www.reasoning.com/pdf/MySQL_Metric_Report.p df
MySQL White Paper: http://www.reasoning.com/pdf/MySQL_White_Paper.pdf

MySQL Defect Report: http://www.reasoning.com/pdf/MySQL_Defect_Report.p df
MySQL Metric Report: http://www.reasoning.com/pdf/MySQL_Metric_Report.p df
MySQL White Paper: http://www.reasoning.com/pdf/MySQL_White_Paper.pdf

MySQL Defect Report: http://www.reasoning.com/pdf/MySQL_Defect_Report.p df
MySQL Metric Report: http://www.reasoning.com/pdf/MySQL_Metric_Report.p df
MySQL White Paper: http://www.reasoning.com/pdf/MySQL_White_Paper.pdf

Think about it--why does the Open Source model produce better code?

You haven't been keeping up on current events, have you? :-)

The first study you mention, and its conclusions, were obviously flawed in several ways, as the discussion here and elsewhere pointed out at the time. The new Reasoning study, currently under discussion elsewhere on Slashdot as well, finds rather differently, though apparently still suffers from several of the same flaws as the original, and thus has to be taken with the same pinch of salt.

I agree with you that communal development projects, such as much of the Open Source world, tend to attract a certain type of developer, whose motivation and skill level may be higher than average for a number of reasons. However, since many of these guys are also professional developers during the day, and still have a good attitude and skillset then, it doesn't follow that Open Source necessarily produces better code than an equivalent closed source offering.

Nice flowery speach. Unfortunately, correctness and validity outweigh passion in a lot of manager's (and customer's) minds.

DEFECT CLASS: Null Pointer Dereference DEFECT ID 1
LOCATION: httpd-2.1/modules/aaa/mod_auth_basic.c :291
DESCRIPTION The local pointer variable current_provider, declared on line 235, and assigned on line 257, may be NULL where it is dereferenced on line 291.
PRECONDITIONS The conditional expression (res) on line 253 evaluates to false AND
The conditional expression (!current_provider) on line 264 evaluates to true AND
The conditional expression (!provider || !provider->check_password) on line 268
evaluates to false AND
The conditional expression (auth_result != AUTH_USER_NOT_FOUND) on line
282 evaluates to false AND
The conditional expression (!conf->providers) on line 287 evaluates to false.

Metric Report

They make you fill out a form that asks for your email and then do an opt out checkbox at the bottom of the form (you have to check it to NOT get spam from them). The site's a bit slashdotted right now though.

Metric Report

They make you fill out a form that asks for your email and then do an opt out checkbox at the bottom of the form (you have to check it to NOT get spam from them). The site's a bit slashdotted right now though.

1. Apache 2.1 (dev) is a mere 76,208 LOC.
2. No memory leaks detected
3. 29 NULL pointer dereferences
4. 2 Uninitialized variables
5. No bounds errors, no bad deallocs
6. otherchild.c had a rate of 7 NULL pointer dereferences per 1000 KSLC
7. 
   
   One of the explanations (given by Reasoning) for a NULL pointer dereference is "can occur in low memory conditions," which I think means the original allocator did not check for malloc failure.
   
   So you can get a sense of what a defect looks like, here is #21. The orignal uses bold and fonts improve readability, but I don't know how to reproduce that in slashcode:
   DEFECT CLASS: Null Pointer Dereference
   
   DEFECT ID 21
   
   LOCATION: httpd-2.1/srclib/apr/misc/unix/otherchild.c : 137
   
   DESCRIPTION The local pointer variable cur, declared on line 126, and assigned on line 128, may
   be NULL where it is dereferenced on line 137.
   PRECONDITIONS The conditional expression (cur) on line 129 evaluates to false.

   CODE FRAGMENT
   124 APR_DECLARE(void) apr_proc_other_child_unregister(void *data)
   125 {
   126 apr_other_child_rec_t *cur;
   127
   128 cur = other_children;
   129 while (cur) {
   130 if (cur->data == data) {
   131 break;
   132 }
   133 cur = cur->next;
   134 }
   135
   136 /* segfault if this function called with invalid parm */
   137 apr_pool_cleanup_kill(cur->p, cur->data, other_child_cleanup);
   138 other_child_cleanup(data);
   139 }

The above link wants your email address. Bah.

The direct URLs for the reports are:
Defect Report
Metric Report

The above link wants your email address. Bah.

The direct URLs for the reports are:
Defect Report
Metric Report

Umm, Apache 2.1 hasn't been released yet. Current latest stable is 2.0.46.

I can only assume that they're looking through the current DEVELOPMENT codebase -- finding a higher ``defect density'' in such a development codebase compared with commercial offerings is not exactly unexpected.

They're also some automated code inspection product; the press release doesn't go into details as to the severity of the defects found or the testing methodology.

It'll be necessary to read through the full report before drawing any sound conclusions.

Here are the links to the Apache Open Source Inspection Report you requested:

Apache Defect Report: http://www.reasoning.com/pdf/Apache_Defect_Report. pdf
Apache Metric Report: http://www.reasoning.com/pdf/Apache_Metric_Report. pdf

Reasoning provides the world's leading automated software inspection service. We boost the productivity of development teams by finding software defects faster and at a far lower cost than traditional approaches. Please let me know if you would like additional information. Thank you again for contacting Reasoning!

Sincerely,
Reasoning

Here are the links to the Apache Open Source Inspection Report you requested:

Apache Defect Report: http://www.reasoning.com/pdf/Apache_Defect_Report. pdf
Apache Metric Report: http://www.reasoning.com/pdf/Apache_Metric_Report. pdf

Reasoning provides the world's leading automated software inspection service. We boost the productivity of development teams by finding software defects faster and at a far lower cost than traditional approaches. Please let me know if you would like additional information. Thank you again for contacting Reasoning!

Sincerely,
Reasoning

But Apache isn't in that boat - it's considered to be mature. Which version were they studying?

Don't laugh. They studied apache 2.1 .

IOW. an open source product tagged alpha is comparable to a typical closed source product.

27 fspec=malloc(strlen(dir) + 2 + 1);
28 strcpy(filespec, dir);
30 if ((ix = strlen (fspec) ....

First of all it looks like line 29 is disappeared, but maybe it was just whitespace. Looking at the code above show me that the strlen function is called with the pointer returned by the malloc in line 27. That's great, since strlen is looking for the "null termination byte" in the string it will return the position of whatever random zero byte will be next in the allocated memory because nobody was writing anything to the allocated space. I suspect, that line 28 should refer to "fspec" instead of "filespec" but since the program obviously compiled it can also be that filespec is a valid global identifier.

Anyway, this simple example from them shows, that their automatic tool doesn't find all bugs and so the numbers returned can be just a sort of wild guess. BTW: I would really like to know what their code inspection tool will report if they use it on their own code inspection software. :-)

The interesting thing is that ATT is a client of Reasoning.

Makes you wonder if they may have tested OSs like Plan9 or Inferno... They probably haven't tested the original UNIX code, though.

Well, if we take a quick look at reasoning.com we get..
Illuma identifies defects that cause application crashes and data corruption. Examples of the C/C++ error classes covered include:

• Memory leak: a reference to allocated memory is lost.
• NULL pointer dereference: a dereference of an expression that is a NULL pointer.
• Bad deallocation: a deallocation is inappropriate for type of data.
• Out of bounds array access: an expression accesses a value beyond the end of an array.
• Uninitialized variable: a variable is not initialized prior to use.

Looks like your standard memory checker. We also get:
Illuma is hardware-independent, does not require running code or test cases, and does not affect code size or execution speed.

It's a code analyzer that watches your variables. The question is, is code analyzing enough? I have a feeling that other methodology (ie. Cleanroom) may bring better results. The Meta-Level Compilation referenced by the parent post looks neat as well (though the two are radiacally different).

Does anyone else have an opinion about reducing number of bugs?

Given that on more than one occasion "independent institutions" which conducted similar studies (and concluded that closed source is superior) were revealed to have been sponsored by the other side, how do you convince other people of your neutrality? Since you are selling a service, not a product, I would guess that the confidence of your customers in your independence is pretty important from a business perspective. How do you win and keep that confidence? The article notes that you agree with ESR's pro open-source reasoning. Wouldn't the perception of your having a OSS bias be something you'd want to avoid?

The want you to request the actual paper by filling in a form. This is the URL they sent me http://www.reasoning.com/downloads/Open_Source_Whi te_Paper_v1.1.pdf.

*smile* I believe it. I used to work there and know the guy that wrote the original white paper. He can't name the other companies for the obvious reasons. But they do get LOTS of code from different companies. There haven't been any problems with code falling into the wrong hands. Trust issues are one of the important things for Reasoning to deal with. Reasoning has been offering code inspection services for the last 3 years and has built up quite a database of error rates across a wide variety of companies. They can't tell you specifics, but they say that your code has X errors/kloc and other people writing this kind of code have an error rate of Y.