Slashdot Mirror
Open Code Has Fewer Bugs
ganns.com writes "Reasoning, which sells automated software inspection services, scrutinized part of the code of Linux and five other operating systems, comparing the number and rate of programming defects. Specifically, Reasoning examined the TCP/IP stack and found fewer errors in Linux. 'The open-source implementation of TCP/IP in the Linux kernel clearly exhibits a higher code quality than commercial implementations in general-purpose operating systems,' the company said in a report released last week. Reasoning also compared the code with that used in two special-purpose networking products and found it superior to one of them."
Really? But I thought most commercial OSes derived their TCP/IP stacks from BSD code in the first place. And since BSD is open-source, shouldn't these commercial OSes show roughly the same level of quality then? Or are they arguing that the Linux TCP/IP stack is superior to the BSD one?
Cheers,
Ian
Over time open source has the *possibility* of having fewer bugs. There are plenty of older open source projects that have TONS of bugs, moreso than equally old MS suites.
"I have an odd craving to whisper about those few frightful hours in that ill-rumored and evilly shadowed seaport of dea
I find the fact that they did not say what OSes they compared to be very... suspect. What about Mac OS X, FreeBSD, and other open source OSes that have Open Source TCP/IP implementations in their kernels? Since they did not say what OSes are being used...
"Reasoning declined to disclose which operating systems it compared with Linux, but said two of the three general-purpose operating systems were versions of Unix."
How lame. For all we know, they could have tested the Amiga OS, Mac OS 9, Windows 3.1, A/UX, and NeXTStep! Other than this, the article is pretty vague and does not seem to give me much meat on the subject, nor a link to the study (you have to go through some forms and give up personal info to get it at www.reasoning.com).
Conglom-O: We Own You (TM).
Most cryptographic algorithims do not gain acceptance without being open to peer review to spot flaws and potential weaknesses...
So why should any of this article be a suprise or even particulary note worthy?
--- Users are like bacteria -> Each one causing a thousand tiny crises until the host finally gives up and dies.
Much more interesting would be a code comparison between open source samba and the micro$oft CIFS code..
If they found 0.1 errors per 1000 lines of code, did they approach Linus and Co. to point them out? Has Reasoning submitted any kernel patches to address the errors they say they found?
Qu'on me donne six lignes écrites de la main du plus honnête homme, j'y trouverai de quoi le faire pendre.
This is not the first such study, there was a paper published in the early nineties which tested various standard unix command-line tools from a variety of vendors. They subjected the tools to horrendous stress and abuse, and found (to their suprise) that the GNU tools were the most reliable, with approximately a 1% failure rate in their bank of tests. The second best was HP, with about 8% failure rate, and everyone else was between 12-20%.
I don't have a link, but the paper was pretty widely publicised at the time, and should be fairly easy to track down. It was the first major study to really show an emperical link between openness and reliability, but it was far from the last. This latest one is merely one more in a long list.
in other news....duh
Yup. Note that this doesn't tell you anything about the overall quality of the software, though. So much open source software tends to be writter by students with little experience ("this was my first large project") and it shows. Just because other people find and fix the bugs doesn't change this.
Basically, the business logic goes something like this:
We can build your application in one of two ways.
- $5000 for proprietary products (app servers, IDEs, etc.), and $5000 for our time and effort (total = $10000), or...
- $1000 for proprietary products (the rest are all open source), and $7000 for our time and effort (total = $8000)
Needless to say, this goes over well for the client ($8000 expense is better than $10000 expense), and also for us ($7000 revenue is better than $5000 revenue ).Obviously, I'm just picking numbers at random, but I think you get my point.
Not every client is eager to jump on open source tools, but more and more they're finding that it's a really good idea. Especially when a major consulting company with an excellent reputation (ie. us) comes along and tells them that this is a good idea. People tend to listen to us, because we tend (historically speaking) to be right a lot of the time.
PHBs might tend to be stuck in the mindset that "if it's free, it must suck, if it's expensive, it must be worth it". But when they pay a high-priced consultant to come in and give them advice, and that consultant says "you know, you can buy IBM's WebSphere Portal Server for $140,000 per CPU, or you can use the open source Jetspeed, which is practically the same thing, in fact, WebSphere Portal is basically just Jetspeed repackaged with some extra tools that you probably don't even need," even PHBs can understand that kind of logic.
"You cannot simultaneously prevent and prepare for war." -- Albert Einstein
Now if they could please point out the filenames and line numbers in question, perhaps we could eliminate the bugs altogether...
This is still an argument for the open source method, but I think that the code quality should be attributed to a different source. Perhaps it is not about an inherently good or inherently bad method. What if age is the key factor?
The Linux networking code has been in for a long time. Not in it's present form, obviously, but each change builds on the last; as it must in open source - it would be foolish to start afresh when you have something that works. So a cylcle develops and at each stage the code gets better. Compare this with proprietary; can they look at a competitors code? No. They must start afresh and so their code is effectively younger.
Further, if we measure software age not in units of time but in units of updates, open source has the advantage that there are many updates, there is always someone new to look at the code. No company can compete with the sheer quantity of viewings and therefore updates that occur in open source developments.
Carpe Daemon
Hmm..
Say you have a load of CS students, and some of them code OS programs for a hobby. Now intuitively the ones who code for a hobby are more likely to be better coders than the average CS student.
When the hobbiest-student-coder has to do his 6 month computing project (I assume that most uni's have similiar projects - or at least some coding projects) then you have to produce plans and documentation etc for it. They are also far more likely to make their said software open source.
When in the work place, then you have all the coders together producing code - resulting in a lower average quality of code.
I'm obviously making some assumptions here, but you get my point..
Btw, as for kernel, it's not quite so clear cut. I do not dispute that there isn't enough documentation, but sometimes people take up issues in the wrong places.
For one example, when AA wrote his memory manager, and non-coders complained and booed because it wasn't documentated. One big reason for this was that the algorithm used was a standard well documentated algorithm, and anyone that understood the algorithm well, would be able to easily understand the code.
Anyway, I have a quantum computing lecture to attend.. bye!
... so with the luxury of time, it _should_ be less buggy.
I don't think releasing the source is necessarily a good thing for a commericial app. How would you control updates and mods? Where would the configuration control come from? I just had my first encounter with CVS at Sourceforge. _NOT_ straightforward. I don't think you could scale that up to a million purhcasers.
"Consensus" in science is _always_ a political construct.
How lame. For all we know, they could have tested the Amiga OS, Mac OS 9, Windows 3.1, A/UX, and NeXTStep! Other than this, the article is pretty vague and does not seem to give me much meat on the subject, nor a link to the study (you have to go through some forms and give up personal info to get it at www.reasoning.com).
Reasoning is not a research entity, they are commercial code analysis contractor. They make their money by having large companies pay large dollars to run lint against their code and perform some additional analysis.
The reason they don't reveal to what they compared Linux is that they are under heavy NDAs from their customers. They couldn't very well say that "Linux stack has fewer bugs than Solaris'" and expect Sun to keep paying them. (I don't know that Sun is a customer, it's a hypothetical.)
The likelihood is that stacks used in comparison are current versions from large corporations and not obsolete code. Large corporations are the only entities with both the means and motivation to hire Reasoning and that's how Reasoning gets the source in the first place.
Finally, Reasoning doesn't have any interest in free software or Linux so this isn't blind advocacy. It's likely a straight comparison of numbers running their standard suite against Linux and comparing to their records of current customers.
but these professional programmers/coder might also be the ones programming the more buggy commerical closed sourced programs.
Also, why is it that they won't name the commercial software they scanned on their home page? Why is it that I have to provide contact information to view their report? Since everyone here is so critical of BS moves MS makes, why are they not asking the same questions of this for-profit entity?
Well, judging by your reference to MCSEs, I'm forced to assume that you are assuming that my reference to open source products necessarily equates to choosing Linux over Windows. Which it does not.
Regardless, this "vendor lock-in" is really not an issue. Basically, because we are not the creators of the open source software in question, we actually have little advantage over our clients in terms of knowledge and resources for support. We have to pour through the same newsgroups that their own IT departments would have to pour through in order to diagnose a problem. So there's really little advantage for them to insist on continually hiring us to support the system, when all we would do is precisely the same thing their own IT people would do. Granted, we wouldn't recommend a specific open source solution if we didn't have some experience with it, but over time their own IT staff will acquire that experience as well.
On the other hand, if we were to sell them a proprietary solution, we have the benefit of partnerships and certifications which we can use to "lock them in", or at least give them the illusion of being "locked in".
To put this in perspective, let's look at a real example. We do a lot of J2EE development. We could sell a client a complete proprietary IBM package, including WebSphere for the application server and WSAD for the IDE. This means they will primarily rely on IBM for the bulk of their support, or else turn to us, as we have lots of WebSphere certified people (myself included). Or, we can sell them an open source solution that includes JBoss for the application server and Eclipse for the IDE. Eclipse is open source, but it's primarily backed by IBM, so they would still have IBM available for support, as well as us, as well as the Internet community (it's all too easy to assume that "open source" equals "some virgin hacker in a basement", but that's not always the case). JBoss comes with plenty of readily available support -- lots of books on the subject, newsgroups, etc.
As far as application servers go, JBoss is no more complicated than WebSphere (WebSphere requires a certain amount of "command-line configuration" and "regular updates"). Eclipse and WSAD are actually pretty much the same tool (WSAD is built out of Eclipse). I don't see how using tools such as these locks our customers into relying on us to support them.
Which is not to say that "locking them in" is a bad thing, from a business perspective. I just don't think it's an accurate assessment in this case.
Your response makes me sad. How are we to get PHBs past the perception of open source as sloppy unsupported crap slapped together by idiots in basements, if we can't even get geeks past this perception. Yes, some of it is. The same is true of some of the crappy closed source software that is for sale these days. We don't recommend crappy unsupported software to our clients, whether it's open source or proprietary.
"You cannot simultaneously prevent and prepare for war." -- Albert Einstein
People coding something because they want to (and because they need it for something for themself) leads to better code. I know when I do something for myself, I don't half-ass it.
Coding for the end result = quality
Coding for a living = paycheck
Any questions?
In SOVIET RUSSIA... erm...NSA AMERICA, the Internet logs onto YOU!
Aye. It could be that the TCP/IP stack that the article mentioned has "flourished" (become better software) because the people who develop it are VERY MUCH using it.
Linux geeks grok TCP/IP networking, and Linux users DEPEND on TCP/IP (not 'it would be nice to have web access and surf porn while I type this memo') for practically all of its market share. Like gcc, TCP/IP is part of the Linux deal.
It would be biased to regard this as conclusive evidence of the superiority of open-source unless other, less sexy areas of Linux development are compared to their commercial counterparts in the same way.
As evidence that certain commercial companies have not put priority on the TCP/IP stack of their OS, this could very well be good evidence.
But this doesn't necessarily mean the commercial companies are inferior; they may very well be right in having different priorities.
For example, for a Windows user it's more important that the Media Player works perfectly than having an efficient TCP/IP stack. Even on the server side it's not a big issue on their market. It's under so many layers of software, appearances and priorities that their clients would never notice if they made it better anyway.
Freedom is the freedom to say 2+2=4, everything else follows...
If a company could package O/S software with nice manuals and guaranteed 'support' then it'd gain much more acceptance, however, I suppose it would stop being 'free' software then.
This post reminded me of a question I was pondering last week. What makes one software program better than another software program? Is there a way to quantitatively measure how good a piece of software is? Would we measure the "goodness" of software by, the number of bugs it has (or rather the lack there of), the number of lines it took to write it, how long it took to develop, the type of license (open vs proprietary), the efficiency (how long to takes to run), the language it was written in, ease of use, etc, etc, etc? My guess is we would have to come up with a convoluted mathematical formula to measure the goodness of a program. Anyone care to take a stab at it?
The thing is that open source products aren't necessarily made by a company whose primary purpose is selling software. Alot of open source is worked on by people who make companies work. Eg, Company X makes widgets, they need a widget inventory control. Company Y makes car parts, and have written an inventory control program. They release this as open source, Company X uses it, and there internal guys find & fix a bunch of bugs. Because it is open source both companies are gaining the added benefit. I think you will find that for most open source projects (especially those that are not high profile) this is how they are being financed. Remember 95% of code written these days is for internal systems that are not released onto the market.
Go out and get sailing!