Slashdot Mirror
Ask About Proprietary vs. Open Source Code Quality
Scott Trappe is CEO of Reasoning, a company that has gained a certain amount of noteriety (and a Slashdot mention) by running its Ilumna automated inspection service on several versions of TCP/IP -- and concluding that the Linux version has fewer bugs than most proprietary ones. Why is this? Let's ask Scott, and also ask him any other question you can think of about software quality and how to achieve it since, after all, that's his business. We'll send him 10 of the highest-moderated questions and post his answers when we get them back.
What kind of open source code do you prefer? GPL, BSD, or the "here, take it" license?
What is your opinion on this topic? In your experience, does having the source closed make it any harder to find bugs and security flaws?
reference to your research? Whose inclusion of your work most impressed? Most confused? Most disturbed? And was there anyone who referenced your work but totally misunderstood what you were saying/doing and their conclusions were way off?
Would a volunteer house builder make better houses? Would a volunteer Fireman put out fires better than a paid fireman? How about volunteer surgeons? I can see one certain contradiction - volunteer web editors choose mych dumber stories than their paid counter parts!
Very popular slashdot journal for adul
"Reasoning declined to disclose which operating systems it compared with Linux, but said two of the three general-purpose operating systems were versions of Unix"
So did you go cherry picking to find OS's that had more bugs than linux, or was it random or what?
Too often the Open vs Closed argument turns into linux vs windows, and then criteria is arbitrarily picked. Since the two OS's are designed largely for very different purposes, the comparison is by definition never fair, no matter who conducts it.
Saying that one product is better doesnt necessarily mean that the way it was created is inherently superior.
Implementing properly documented standards is something the OS community is great at, since they're all on the same page. Creating from scratch is different.
Hence, TCP/IP is rock solid in linux, yet development on the desktop crawls along in 100 different directions at once, gaining little ground.
I don't need no instructions to know how to rock!!!!
Some proprietary products like Microsoft Office partially maintain there dominance by not disclosing the details of the file format, or modifying standard formats to reduce compatability. Do you think that competetive free products would be more widely accepted if the file formats were open/standardized, or would the dominance and familiarity of the current packages would maintain their market control?
That what was all this school was for... to teach us how to solve our own problems. -- janeowit
What is the best piece of code you've seen? What about the worst? Did the best and/or worst come from open source code, or proprietary? I'm just curious.
Ah am not a crook! (\(-__-)/)
Because there are obvious differences in the cultural enviroment of developing open source versus proprietary software, what is you opinion what factors affect the quality of code that is produced from these two enviroments and how?
Memories become legend, Legend fades to myth, and even myth is forgotten by the time that age comes again.-Robert Jordan
OK "Your TCP stack is cleaner than theirs" but what metrics are being used? How do we know bugs in their testing software doesn't skew the numbers?
Trolling is a art,
Would it be fair to ask if you are an advocate of any particular type of software, or you merely promote use of the "right tool for the right job"?
Smokey, this is not 'Nam, this is bowling. There are rules.
Is a certain percentage of bugs that result from the interaction of two or more otherwise bug free components?
If open source has such a direct correlation to better quality, why do you feel companies are still keeping their source proprietary? Do you think that we should try and convince them to open source their code in every case, and if so, what do you think needs to happen before they can be convinced to change their minds?
Overrated Moderation: This posts sucks... because.
In my book, quality is broken down as:
50% Stability, efficiency
33% Form, structure
17% Ease of build
Stability and efficiency, of course, is the most important thing. Does the code work? How well does it cover all cases? Does it do it efficiently? Does it make 10 copies of a string just to return a substring?
Form and structure are important too. This is key for maintainability. Is the code broken down unto logical modules? Or is the entire 50000 line code base contained all in one monster if/else function? Does the code itself follow sensible, consistent conventions? Or did the developer purposely obfuscate it to prove how smart he is? Or did the developer hack the whole thing due to a failure to understand the actual problem to be solved? How well documented is the code?
Ease of build - how many #define's (or their analogues in other langs) do I need to get the thing to compile? Does it come with a makefile or build script? Do I need to install a 100MB SDK because the author decided to use 1 small function he could have written himself?
These are the factors by which I use to measure the quality of source.
Do you believe the 'evolutionary' pressures that led the Linux tcp/ip implementation being better are in action in other areas of opens source activity? I can see the tcp/ip implementation getting a lot of attention from coders as linux is primarily a server platform but are less obviously important areas performing similarly?
If so, which areas do you think are benefitting and which need more community action / peer pressure to excel?
Are there any areas you think this phenomenon will never apply? (eg areas in which proprietary code will always be better)
Sometimes I think Linux takes more that its fair share of the limelight. Generally when I see some aspect of Linux compared to other OSes, I'm interested in seeing how BSD fares as well. Not so much to decide which is better, but it's interesting to see how the two do against each other, given that they're both open source projects. It seems to me that they both have many different and similar goals, and take different approaches at doing things. I'd like to see how it all adds up.
So did you take a look at the BSD tcp/ip implementation, if so, how did it compare to the rest?
If you didn't, why not?
-kidlinux.
The parallelizability of bug-fixing is quite clearly very effective for high-visibility projects such as the linux kernel and apache. However, considering that most open-source projects have only between 1 and 5 developers, how popular do you think a project needs to be for it to significantly benefit from people looking at the source code?
It sounds like your company focuses on analyzing the code bugs, and not necessarily the perceived bugs. What are your opinions on this? I know that locating and eliminating the bugs *is* a critical part of software QA, but do you feel that bug-free ensures true quality? A bug-free Open Source project may still be too difficult to use or confusing for the non-technically inclined.
My beliefs do not require that you agree with them.
Ironically, the reason most companies will opt for closed source solutions is because they have large companies behind them: Microsoft, Sun, IBM. Although this gives the illusion of having someone to hold responsible, the EULA usually contains a clause relinquishing the vendor of any real responsibility or culpability. Whereas with open source software, you have no legal recourse if the latest release of sendmail or bind has an exploit, but rest assured that within 24 hours a fix will be released. Compare that with response times from commercial closed source vendors...
It is natural to expect the number of bugs to go down when more people look at the source. However the downside to being open source from the security viewpoint is that possibly makes it easier for the bad guys to find bugs. Have you measured the effect of this? Is it actually easier for crackers to find bugs when they have access to the code? If so, do you think the smaller frequency of bugs adequately compensates for their increased exploitability?
The tcp/ip findings were interesting, but were they really relevant. Let's say that one of the tcp/ip stacks that had more "bugs" was Solaris. Now I assume that the Solaris tcp/ip stack has not had significant changes in a very long while, also, that if these "bugs" actually negatively impacted the working of the code, that they would have repaired. So my question is, how does your application mark a bug? And from the original /. article, it would appear that you treated all "bugs" the same. Is a failure to check for a null pointer "a bug". Having a more explicit rundown of the type and scope of bugs found would be much more meaningful.
It says they found so many bugs per 1,000 lines, but did they submit the errors, or fixes, to CVS, or to the vendor?
Do you find that the quality of the programming depends upon the geographic location of the programmers? So, for instance, an open source program will be troubleshot and combed over by people from potentially a dozen different countries. Closed source software is checked by people where it is written. Since, as a general rule, education varies in quality and areas of emphasis around the world, does it help having people attacking a program from many different angles (i.e., open source, cheked world wide) rather than simply drawing from a set of people who may share many of the same abilities, backgrounds, etc.?
What do you mean by "defect rate"? Is it a measure of bugs your group found for the first time or were you looking at already discovered and documented bugs? In either case how do you ensure that you have enumerated all the defects in the code?
Why did you choose to look at the TCP/IP code over any other particular subsystem? Do you have plans to review any other portions of the code? For instance, I think it would be very interesting to see a similar comparison which examined the code for file systems or virtual memory. Have you reported the bugs you found back to the authors of the code?
I assume some of this information may be "company secrets" but I'm very interested in learning what metrics are used to determine which source code is "buggier" than others. Is this something as simple as running lint + "gcc -Wall -ansi -pedantic" then piping the output to "wc -l" ?
Are there checks for use of unsafe functions like gets and the str* family of functions in C? Are there more complex data flow analysis algorithms at play here like those in the used in Stanford's Meta-level compilation techniques?
Inquiring minds want to know. A pronouncement like OS foo is has more/less bugs than OS bar is meaningless without a definition of what having more/less bugs means.
Please compare and contrast the services your
company offers with those offered by Checker
or Smatch.
They seem pretty similar. In fact, do you
use Checker or Smatch internally? It would
seem logical.
- Dan Kegel (dank@kegel.com)
Given that on more than one occasion "independent institutions" which conducted similar studies (and concluded that closed source is superior) were revealed to have been sponsored by the other side, how do you convince other people of your neutrality? Since you are selling a service, not a product, I would guess that the confidence of your customers in your independence is pretty important from a business perspective. How do you win and keep that confidence? The article notes that you agree with ESR's pro open-source reasoning. Wouldn't the perception of your having a OSS bias be something you'd want to avoid?
Open source works for itself. I don't understand why people think it would rule the world. It's simple an alternative way to code and offer the work done to others.
I agree with : Still, I like the sound of Open Stalin. Can we kill people who decide to reinvent the wheel, to encourage OS developers to pull in the same direction? Stop the holy wars for once and for all..
People just wants to do what they like more.
Complaint mode on The only complain I'd like to make is that opensource developers don't report the development platform where they built their apps.
So often you download something - and the library depends are respected - but the app fails. Just writing: developed with Redhat 7.3, full install would suffice. Complaint mode off
How much of a role do you think peer review plays in software quality?
In proprietary source systems, there is generally formal peer review, as per CMMI. But I have seen this done rarely (almost exclusively for CMMI level 3+ projects). There seems to be a disincentive to do formal peer review. There seem to be various reasons for this, cost, workplace environment, and group dynamics. Which do you think are most significant?
Whereas in open source projects, there is not the formal peer review, but rather seems like a mass informal peer review. This seems to foster an enviroment of besting each other, trying to find the most and most obscure bugs.
What do you say?
SCO to Hell
Do programming methodologies actually increase code quality, in your opinion?
What I mean is this: over the years there have been numerous methodologies that to some extent all claim to make programmers write better code in less time. eXtreme Programming is a recent and - imho - fairly impressive example. All of them boil down to a slightly different approach to the task of programming.
So if you find fewer programming defects in the Linux IP stack, would you think that this indicates that there is something that works well about the way open-source programmers approach programming? Or could it be simply that people willing to donate their time to a project tend to be talented?
Is this really just a problem of the resources that can be brought to bear on producing the final product? Does the quality simply come from the shear number of people that plays on the law of averages/big numbers ? ie the open source got 10,000 hours of development time vs 2,000 hours in a closed source environment restricted by cost/budget/time etc. If the resources to producing the "product" were the same would the quality be any different ?
Something that interests me is the trend for code quality analysis tools to pick the Linux kernel or other well-known free program as an example. So researchers developing the tool get to boast 'we found 47 bugs in Linux', which is a statistic people can understand (even if it may not always be strictly true), while Linux benefits from some extra bug reports.
Strictly speaking, static analysis tools measure what is called kwalitee, a property which isn't the same as code quality but is usually closely correlated with it. In other words the tools do make mistakes, but most of the time they are on the right track.
It would also be possible to have a big online 'databank' of C source from many projects - the top thousand on Sourceforge plus the GNU programs, or something like that - and make this a standard 'corpus' for code analysis tools.
Hmm, I have to get a question out of this. Do you think that code analysis tools like Splint could improve free software quality further? What sort of infrastructure could be created for doing code kwalitee checks across a whole Linux or BSD distribution?
-- Ed Avis ed@membled.com
What errors are currently hard to detect automatically but which you would really like to be able to find?
What is the next category of errors that you're trying to detect with automatic code inspection?
To give you some ideas, what about:
When comparing implementations of TCP/IP, how did you define what constituted a bug? Was any attempt made to verify whether an actual problem could result? For example, one can read from unitialized memory w/o causing a problem as long as one doesn't then use that value later with the expectation that it is valid.
I have used several source code analysis tools, and they are very valuable for catching defects. However, they also tend to catch a lot of things that are fixes to the code that don't effect the output of the function (e.g. unreachable code, etc.).
Are you willing/able to post the actual details of the bugs found?
Does the choice of the implementation language affect the number and/or severity of bugs found? Obviously, the skill of the programmer will affect the quality of the code, but perhaps a study like this can lend credibility to the idea that the choice of a language can predict a certain level of quality in the code. ie, maybe it's easier to write bug free code in some languages. Any data that would suggest this?
What do you think about the new "test first" software development methodology? For those that haven't heard of it, it's a method wherein the test cases for a program are written, and no code is written that doesn't cause a failing test case to pass. All test cases are automated and run after every code change. Would you advocate this in an open-source project? This would mean every contributor would write test cases for each new feature, and add it to a project's common test case repository... What do you think?
What opinion you have about this?
There seems to be a lot of effort in automated testing which goes into trying to determine what the program is supposed to do. An automated tool can never find all of the bugs, since some of them will be that the program doesn't crash or anything, but fails to follow the specification, which isn't given to the automated tool.
How would you extend C/C++ to include information about the intended behavior of programs, so that programmers can tell the tool directly what is supposed to happen?
I would like to move my company towards creating s/w that will operate on Linux, both as a client and as embedded (target) s/w. Our clients are the large primes for militaries around the world.
Most primes and militaries are moving towards COTS products to reduce costs and improve reliability and support. If we were to port our product s/w to run on Linux, how on earth can we achieve similar value and benefits of COTS-like s/w, s/w like WinRiver's Tornado, that have great robustness, standard (purchasable) support, and carry the perception (remember: perception is reality here) of greater security?
For those of you who think support is not important, market data has shown that for larger organizations, the number one "care about" is support. And since Sept 11, security is moving to the top of the list of care abouts for the militaries and primes.
"Content's a bitch."
I wonder if there were marked similarities in the bugs in Proprietary code compared?
Were these similarities found in FOSS code that was looked at, or did the dendritic peer review process handle that to some degree?
Were bugs found in the proprietary code that were already (verifiably) marked as things to be fixed, and if so, what was the average lag time (Bug turn over)? Do these companies keep track of their bug turn over periods, and what is the empirical comparison with that of FOSS?
Was there pro-active debugging done in the FOSS code that were results of known bugs in the proprietary code base, and if so, were these bugs addressed in the proprietary code?
Was there a verifiable process for maintenance in the proprietary companies that had changed in the 3-6 months prior to the testing?
I think that will do for now. Plenty more where that came from. :)
Taran