Slashdot Mirror
Ask About Proprietary vs. Open Source Code Quality
Scott Trappe is CEO of Reasoning, a company that has gained a certain amount of noteriety (and a Slashdot mention) by running its Ilumna automated inspection service on several versions of TCP/IP -- and concluding that the Linux version has fewer bugs than most proprietary ones. Why is this? Let's ask Scott, and also ask him any other question you can think of about software quality and how to achieve it since, after all, that's his business. We'll send him 10 of the highest-moderated questions and post his answers when we get them back.
How can any conclusions about the relative virtues of two development methodologies with a universe in the millions of components be drawn from a single sample, and one as small and atypical as a TCP/IP stack?
I don't know what Scott's opinion is on this, but I know I've found the specifications to be the biggest point of failure. I can't tell you how many times I, or someone I know, have written the perfect program that nobody wants because it didn't follow what the customer actually wanted. --Jason
One of the bigger challenges facing open source projects as compared to their proprietary equivalents is how to manage confidentiality of test cases. With companies such as Red Hat and Ximian involved, it's certainly less of an issue for their core products and projects they over-see, but there will always be cases where there is friction when the best/only person who can fix a particular problem is on the outside, unable to work with the test cases in question.
What are your thoughts on this trade-off between test case management and confidentiality as it relates to proprietary v.s. open source code development?
--The more you know, the less you know.
Since you are a small team rather than the entire open source community, don't your own conclusions imply that you are likely to be detecting only a small fraction of the defects, invalidating the study?
Silly peon.
"open source" works because your owner needs something done and may realize that it makes more sense to spend labor on the problem rather than money. There also may be no compelling reason to maintain ownership over the results.
Software is a tool, not fools gold.
Software is valuable for what it can do for people who don't have any interest in selling it.
A Pirate and a Puritan look the same on a balance sheet.
I meant, as far as "enforcing" code quality was concerned.
Hence, TCP/IP is rock solid in linux, yet development on the desktop crawls along in 100 different directions at once, gaining little ground.
Actually, the Linux desktop has gained a lot of ground, as have the distro installers. If anything, public perception lags far behind the reality. That's not too surprising given that the OS community isn't pumping millions of dollars into marketing.
Where can I get the source code to these automated inspection tools?
That is 1.6 million IDLE connections vs. 6000+ REAL connections that are actual used to transfer data. If you read Terry Lambert's posts, you will see that he merely opens 1.6 million connections: his test program/tuning doesn't actually try to get real work done with these.
Since the kernel address space was probably on the verge of exhaustion in his test, I doubt you could have run a benchmark with it. If you could, would it outperform AIX or would it flunder???
Please don't be a "fan-boy": read and understand exactly what is being claimed. Having 1.6 millions idle connections is useless for a server.
Do you think part of the difference in resulting code quality is due to the developers' motivation for working on the project -- that perhaps closed-source programmers are more likely to be doing it just to earn a salary, while open-source programmers are more interested in the art of coding itself?
Compare it to any group of people working on something together for everyone to share - especially professionals doing something they like. For example, a group of carpenters building a summer house together because they love doing it and therefore do it pretty damn well and are intent on sharing it together. Except that with open source everybody gets their own copy of it instead of eg. everyone gets to spend X weeks of your vacation here. And in addition to them getting a copy many others do too - and they thus get their eternal admiration and gratitude :)
Karma. Moderation. Is my
It is possible to have a proprietary model and to have code reviews required (and documented) done by competent system architects and security experts. It is also possible for proprietary developers to do no reviews and to lack the skill and experience and coding standards and automation to produce reliable code.
It is possible to have an open source model and have the code reviewed by no one but the original coder. Or to have 15 reviewers of varying competence looking at ever line and debating it vigorously.
It is possible in the same OS to have source files or code fragments from various sources with various development and review methodologies. Some can be as extreme as using/requiring automated tools to find potential errors and requiring skilled reviewers. Some as lax as no review by anybody or anything.
Given this diversity, how can the terms open and proprietary be used to usefully describe software quality? Doesn't it depend not on the open/closed but on the amount of skill of the coder, automation of the review and experience of the reviewers. And isn't that independent of open/proprietary?
Do you study the makeup and practices of the development team as part of your analysis? Would you find it useful to know if a team favored one lifecycle methodology over another - and are there any correlations you have seen along these lines?
Lodragan Draoidh
The more you explain it, the more I don't understand it. - Mark Twain
Bug-free software is obviously an ideal goal, but it's not the only thing that measures code quality, in my mind.
Do you forsee any metrics in the (near) future to measure other aspects of code quality? Performance is obviously important, but what about things like code style, modularity, and 'cleanliness?'
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
The volunteer examples don't jibe because closed source is unmodifiable by the user. The anaolgies above are non-revisionary. Or difficult to revise at the very least.
And in any case, Apache has been more stable than IIS forever. And Linux has been more stable than Windows forever*.
*Not including the shiny GUI, which has only really gotten started in the last few years on Linux. But the improvement rate in Gnome and KDE is far better than Windows', in any case.
This is because, although being open makes it possible to involve many more people, it is not necessarily true that many people will look at your code. Coding is not an easy task, it takes time. In general, many open source projects are maintained by few people, which is actually worse than the commercial applications, since these commercial companies can hire the top people in their area, and they can hire as much as possible if that's needed to compete with any other product, including open source applications. So being "open" does not translate into anything. It is the number of people, their quality, their time to dedicate for the project, not the license of the product.
I am partially open source advocate, and I really appreciate people working for open source. But there are big problems associated with it, and I think instead of trying to cheat people to use open source, we need to focus on the problems of the open source itself. Otherwise it will be a hobby for people, geeks, but nothing more.
In short, can you explain your logic behind this conclusion, because it just seems to me either you or Slashdot is making it up.
Think about that.
If Stack A is 3 times as large (bloated code) but has only 2 times the bugs as stack B, then stack A (worse in all respects) gets a better grade!!!
You can halve your defect count by doubling the number of lines of code in your module. What a rip! How could so many people read and write about this and not see the problem.
I18N == Intergalacticization
thats a bit short sighted, and its not so obvious.
One the one hand we have open source which is subject to large amount of peer review.
On the other hand we have closed source where no way near as many people can check the code, end users can't help much in finding them either.
A persistent h4x0r maybe helped a little by the O/S, but security through obscurity has been proven to fail, time and again