Open Code Has Fewer Bugs

ganns.com writes "Reasoning, which sells automated software inspection services, scrutinized part of the code of Linux and five other operating systems, comparing the number and rate of programming defects. Specifically, Reasoning examined the TCP/IP stack and found fewer errors in Linux. 'The open-source implementation of TCP/IP in the Linux kernel clearly exhibits a higher code quality than commercial implementations in general-purpose operating systems,' the company said in a report released last week. Reasoning also compared the code with that used in two special-purpose networking products and found it superior to one of them."

Comments

Maybe, maybe not

[Raul654]

Companies such as Oracle and Microsoft typically sell binaries incomprehensible to humans rather than the comparatively understandable source code.

After seeing this, I think that statement is being a bit generous

Statistics

[Caractacus Potts]

How about using a larger sample of code before making such bold statements. It's probably true that the code has fewer bugs, but when you abuse statistics it just makes things look dishonest.

Re:Statistics

[Xtifr]

This is not the first such study, there was a paper published in the early nineties which tested various standard unix command-line tools from a variety of vendors. They subjected the tools to horrendous stress and abuse, and found (to their suprise) that the GNU tools were the most reliable, with approximately a 1% failure rate in their bank of tests. The second best was HP, with about 8% failure rate, and everyone else was between 12-20%.

I don't have a link, but the paper was pretty widely publicised at the time, and should be fairly easy to track down. It was the first major study to really show an emperical link between openness and reliability, but it was far from the last. This latest one is merely one more in a long list.

Re:Statistics

[dglo]

there was a paper published in the early nineties which tested various standard unix command-line tools from a variety of vendors. They subjected the tools to horrendous stress and abuse, and found (to their suprise) that the GNU tools were the most reliable, with approximately a 1% failure rate in their bank of tests. The second best was HP, with about 8% failure rate, and everyone else was between 12-20%.

I'm guessing you're probably referring to Bart Miller's Fuzz Testing software. They did a survey in 1990 and a followup in 1995. They've even got the software available if you want to do the 2003 version!

Title of post misleading

[jfrumkin]

Over time, successfull open source projects which address a particular issue will most likely have fewer bugs; just being open source doesn't mean fewer bugs (or better software). It just means that it has a better chance, if it survives, of being better software.

But aren't TCP/IP stacks mostly BSD?

[mccalli]

'The open-source implementation of TCP/IP in the Linux kernel clearly exhibits a higher code quality than commercial implementations in general-purpose operating systems,'

Really? But I thought most commercial OSes derived their TCP/IP stacks from BSD code in the first place. And since BSD is open-source, shouldn't these commercial OSes show roughly the same level of quality then? Or are they arguing that the Linux TCP/IP stack is superior to the BSD one?

Cheers,
Ian

Re:But aren't TCP/IP stacks mostly BSD?

[gurps_npc]

While the commercial OSes derive from BSD code, it is not the same thing. Related to that, there are three sources of bugs that Closed OS's will have but Open OS's will not. 1) Errors in the derivation of the BSD code - that is they generally have to make minor changes in the BSD code to get it to work with their product. 2) Bugs in the Non-BSD code that is wrapped around the BSD code. 3) Errors found in the BSD code after the Closed code was written. Usually the closed Os will NOT upgrade the BSD code for a bug found in it because either 1) they are lazy, 2) they are ignorant of the bug, or 3) doing so would require a re-write of the Non-BSD code.

Bah.

[KefkaFloyd]

I find the fact that they did not say what OSes they compared to be very... suspect. What about Mac OS X, FreeBSD, and other open source OSes that have Open Source TCP/IP implementations in their kernels? Since they did not say what OSes are being used...

"Reasoning declined to disclose which operating systems it compared with Linux, but said two of the three general-purpose operating systems were versions of Unix."

How lame. For all we know, they could have tested the Amiga OS, Mac OS 9, Windows 3.1, A/UX, and NeXTStep! Other than this, the article is pretty vague and does not seem to give me much meat on the subject, nor a link to the study (you have to go through some forms and give up personal info to get it at www.reasoning.com).

No Suprise There

[Greyfox]

The attitude I've seen in the corporate world is that open source products are made by amateurs and is therefore in some way not blessed by the magical corporate coding fairy which makes all the shit churned out by corporate code shops stink less. This attitude is arrogant and does not take into account the simple fact that all those people who got into programming just for the money tend not to work on open source products. When you've got code that is both written and reviewed by legions of people who love to code and who find good computer programs to be beautiful, you're going to get better code.

Re:No Suprise There

[mrpuffypants]

I'm encountering this in my new job that I recently took. I walked into a company that was using an antiquated MS Exchange system for most of their communication, old networking hardware(which is another issue entirely), and software packages that hadn't been updated in about 5 years because the company that originally wrote them has gone under in recent years (.bomb)

After looking at everything I suggested a lot of open-source alternatives to all the current software. The prices to buy it all was zilch, and upgrading all the hardware can be done in-house, without the help of "contractors" that charge out the ass just to support their own software. The system would work great, a lot better than the currently antiquated crap we are using.

After presenting my ideas to management they shot it down totally. They, with their mind for the bottom line, couldn't understand how people would release software totally for free. They kept asking me when they would pull the bait and switch on us. It's two whole different schools of thought, and the only way that I can implement it now is to do it slowly behind their backs until they don't even know what hit them when they don't have to reboot the server daily anymore =]

Claim is too general

[Peter_Pork]

Open Code Has Fewer Bugs

The study looked at a single part of an operating systems (TCP/IP stack) and then the posting made a very general claim about open source software. This is cheap engineering (a.k.a. bad science). Period. You need a much larger sample to make such a claim. A single data point is meaningless. In fact, I believe that code bugs are much more a function of programmer performance and code complexity than open vs. close source development model. Opening the code may have a positive impact, but it is not the major factor to consider. The last thing Open Source needs is this kind of marketing strategies...

Stating the obvious

[seanadams.com]

'The open-source implementation of TCP/IP in the Linux kernel clearly exhibits a higher code quality than commercial implementations in general-purpose operating systems,

Well of course it does! The Linux and BSD IP stacks are benchmarks. This is where practically all protocol research happens - how would anyone be able to verify your results otherwise? Furthermore, only the free stacks are useful for compatibility testing because they are so configurable.

So obviously it stands to reason that this code is much more complete and bug-free than any commercial implementation. THOUSANDS of people are studying every single line of this code on an ongoing basis.

I've worked on a number of commercial IP stacks - some from scratch, and some based on Linux. Any IP stack written from scratch is understandably simpler, but it's not that hard to implement the essential RFC requirements (i.e. the "MUST"s) and make it stable. Now, making it FAST and making it use all of the bleeding-edge TCP stuff... that's another story. Only Linux/BSD are there (and of course any other OSes which use their stacks).

Space shuttle code is closed

[MondoMor]

The code for the shuttle's GPCs is closed, and it's regarded by many as probably the most bug-free code around with any degree of complexity. It's been upgraded several times since the '70s, and rarely have errors been found.

It probably had one of the longest development times for its size, too. Which helps a lot.

Quality has nothing to do with whether code is open or closed source. It's got everything to do with the environment in which it was written. Code written under extreme management pressure from a profit-hungry megacorp is just as bad as code written by an ignorant or uneducated dork in his basement.

Re:Fewer bugs than what?

[LizardKing]

Microsoft pinched their TCP/IP stack from *BSD

Not exactly true. I can't find the link off hand, but I read an explanation of the background to this myth quite recently. If you Google around you should be able to find it.

Back when MicroSoft were keen to add TCP/IP support to Windows, they contracted another firm to to do the work. That firm took the BSD licensed stack (from 4.3BSD as I recall), and did tyhe necessary porting work. This they then delivered to MS, meeting the original deadline. Since then, NT has gained a new TCP/IP stack written from scratch by MS engineers.

As a result, the TCP/IP stack currently used in Windows owes little or nothing to the BSD implementation.

Chris

This only makes sense.

[SatanicPuppy]

The code I write for myself is the cleanest stuff in the universe. I get freaky about extra lines or lines that look "ugly" or inelegant.

Now when I'm at work I toss out functional, ugly code. Doesn't work quite as well, but 90% of the users will never know that. I'll write catch statements for the most obvious errors, but I don't sit and brood about what some hypothetical idiot might want to do with the code. If there are enough people who hit an error there, I patch it, and move on with my life.

By and large, high production commercial code is sloppy. There isn't any profit to be made in making it pretty or elegant, and we all know how (for a random example) MICROSOFT feels about profit.

Open source is just the opposite; if you're not making any money on it, you're doing it for your own personal satisfaction, and I think most people find it more satisfying to have clean baddass code, rather than sloppy junk code. Heh. Especially when your NAME is on it, and the SOURCE is available.

Just my .024 euros.

Yes, but the code has diverged.

[Jimithing DMB]

Actually, you've inadvertantly stumbled upon an excellent point.

No code is perfect to begin with. The BSD stack is still improved from time to time. The BSD stack that companies folded into their code years ago has since had some major changes and the companies haven't bothered to take many of those changes into account.

Had they been required by license (GPL) to keep the code open, then it could be fixed by other people. Instead, the implementation has languished. This in fact is one of Stallman's great resons for keeping all code free.

However, the reality of it is that our current environment still favors closed source software. With any luck, people will slowly start to wake up and realize that source code needs to be open for all software projects. Think about it. If it was normal to receive source with binaries, nobody would really think twice about it. It's only seen as a bad thing because it's not what Microsoft does. But the reality is that Microsoft has a business model that works well for them, a giant monopoly. The reason their competitors fall on their asses is because they are trying to play as if they were MS, which they are not. It's not impossible to compete with Microsoft, it's just impossible to compete head-on.

Re:Yes, but the code has diverged.

[Eccles]

However, the reality of it is that our current environment still favors closed source software.

I'd say it's not environment, it's economics. Apache has flourished because the people who develop it are also people who use it. But what percent of graphic designers are really using the Gimp vs. Photoshop? Maybe Photoshop has more bugs, but it has more usable features (performance also?), and that's what its users want. Unless you can come up with a scheme to fund development of open source in the same way that software purchases fund closed source, closed source is going to be the only way to develop software where the users generally aren't also the developers.

I develop commercial closed-source software. I'd absolutely love it if some sugar daddy came up to me and said, keep doing what you're doing and I'll keep paying you what you're getting paid, except we're making the code open source. But it isn't going to happen.

Re:Yes, but the code has diverged.

[Bodrius]

Aye. It could be that the TCP/IP stack that the article mentioned has "flourished" (become better software) because the people who develop it are VERY MUCH using it.

Linux geeks grok TCP/IP networking, and Linux users DEPEND on TCP/IP (not 'it would be nice to have web access and surf porn while I type this memo') for practically all of its market share. Like gcc, TCP/IP is part of the Linux deal.

It would be biased to regard this as conclusive evidence of the superiority of open-source unless other, less sexy areas of Linux development are compared to their commercial counterparts in the same way.

As evidence that certain commercial companies have not put priority on the TCP/IP stack of their OS, this could very well be good evidence.

But this doesn't necessarily mean the commercial companies are inferior; they may very well be right in having different priorities.

For example, for a Windows user it's more important that the Media Player works perfectly than having an efficient TCP/IP stack. Even on the server side it's not a big issue on their market. It's under so many layers of software, appearances and priorities that their clients would never notice if they made it better anyway.

Which is the cause and which is the effect?

[Skapare]

What I am wondering is which is the cause and which is the effect:

Microsoft source code is defective because it is closed.

Microsoft source code is closed because it is defective.

Is age the key factor?

[realnowhereman]

This is still an argument for the open source method, but I think that the code quality should be attributed to a different source. Perhaps it is not about an inherently good or inherently bad method. What if age is the key factor?

The Linux networking code has been in for a long time. Not in it's present form, obviously, but each change builds on the last; as it must in open source - it would be foolish to start afresh when you have something that works. So a cylcle develops and at each stage the code gets better. Compare this with proprietary; can they look at a competitors code? No. They must start afresh and so their code is effectively younger.

Further, if we measure software age not in units of time but in units of updates, open source has the advantage that there are many updates, there is always someone new to look at the code. No company can compete with the sheer quantity of viewings and therefore updates that occur in open source developments.

Or...

[intermodal]

People coding something because they want to (and because they need it for something for themself) leads to better code. I know when I do something for myself, I don't half-ass it.

Coding for the end result = quality

Coding for a living = paycheck

Any questions?