Linux Has Fewer Bugs Than Rivals

sushant_bhatia_progr writes "Wired has an article stating that according to a four-year analysis of the 5.7 million lines of Linux source code conducted by five Stanford University computer science researchers, the Linux kernel programming code is better and more secure than the programming code of most proprietary software. The report, set to be released on Tuesday, states that the 2.6 Linux production kernel, shipped with software from Red Hat, Novell and other major Linux software vendors, contains 985 bugs in 5.7 million lines of code, well below the industry average for commercial enterprise software. Windows XP, by comparison, contains about 40 million lines of code, with new bugs found on a frequent basis. Commercial software typically has 20 to 30 bugs for every 1,000 lines of code, according to Carnegie Mellon University's CyLab Sustainable Computing Consortium. This would be equivalent to 114,000 to 171,000 bugs in 5.7 million lines of code."

Comments

Mistake

[StevenHenderson]

Windows XP, by comparison, contains about 40 million lines of code

I think they mean "40 million lines of bugs" :)

Re:Mistake

[chrish]

Somehow I doubt that the XP kernel is 40 million lines of code. I know it's not good "news" but they should really compare apples to apples in their study.

This just in! "Hello world" has 0 bugs per three lines of code! Most stable and secure software ever devised!

Re:Mistake

[jacksonj04]

I disagree - some bits of Windows actually work as intended without glaring problems. In fact, the vast majority of Windows (especially in 2000 onwards) does what it's supposed to.

Deleting half the DLLs in sys32 then trying to run applications does not constutute a bug, especially when Windows shouts at you that you really don't want to be deleting them. If the user being able to cause problems was a bug (some would say it is), then Linux is more buggy than anything else. Windows has the decency to complain if you're deleting anything essential, Linux at best goes "Y/N", and even that can be overridden with a switch.

Lots of bugs maybe, but you can't say the entire codebase is badly written.

Re:Mistake

[Allen Zadr]

I actually agree here. I'm a big Linux proponent, but the whole text seems fishy to me. And if they are certain there are exactly that number of bugs in the code, then they probably have been addressed. Yet, there are probably more bugs yet to be found.

Then, when speaking of XP, they don't quantify the bugs, but merely say, "more are being found daily". Great... a pear.

Re:Mistake

[Kjella]

This just in! "Hello world" has 0 bugs per three lines of code! Most stable and secure software ever devised!

Actually, hello world has the highest ratio of bugs/program complexity I've seen. Depends on who is doing the implementation, I guess.

Kjella

Re:Mistake

[phats garage]

Don't forget, when sizing up XP's kernel you have to add in IE and media player 10.

Re:Mistake

[FrYGuY101]

Okay, why?

Those are integrated into the Operating System. They are not part of the kernel.

Re:Mistake

[fitten]

If that's the only qualification, every device driver ever written has to be included in the line and bug count. Also, pretty much any program that does anything interesting has to call into kernel space to have something done for it, so it's kind of running in kernel space too, just not all the time. Well... that pretty much pulls in every application ever written. I think that research group has a bit more work to do. If 5.7 million lines of code took them 5 years, they should only have about another 1000 years to go before they can publish :)

Re:Mistake

[blowdart]

And of course to do a proper comparision they would need to actually have access to the XP source code to analyse in the same way they analysed the linux source. Then, if they're doing a like for like compare you can't, for example, analyse IE if Firefox wasn't in the linux analysis. If X wasn't analysed, well there goes the Windows GUI.

Hardly a fair comparison (just like certain TCO studies) but that's not going to stop Linux vendors using it and slipping down to MS marketing levels.

However it's not clear if the comment

Windows XP, by comparison, contains about 40 million lines of code, with new bugs found on a frequent basis.

comes from Wired, or from the Stanford report.

Re:Mistake

[dbacher]

X.Org, Mesa and your choice of KDE/GNome and the tools necessary to have the same base application set as Windows has have over 5m lines of code, so it's safe to say Apache 2, Samba, etc. weren't included in this count, either, nor the GNU C compiler suite.

There's no central Linux repository for reporting bugs in ancillary packages, while at Microsoft's site, all reported bugs go into a single database that can be queried. Each individual Linux distro and each individual Linux package maintains its own bug lists, which would have to be some how amalgamated.

In order to do the stated comparison, you would need to state what distribution you were using, you would have to state which patches you were using, and you would have to document where the information on bug counts came from for both products.

On top of all this, bugs per 1000 lines of code, while an industry metric, isn't a valid measurement of code quality, either, by the way.

One good code metric is the number of control points per function. The more control points per function, the more likely that you have a bug. It's always possible to take a complicated function and break it up into smaller pieces, and with C/C++ (but not .NET or Java), it's possible to do so with no performance impact at all in all ISO compilers.

Another good metric is the distribution of severity of the reported issues. Draw a bell curve of severity of issues, compare percentage to percentage. You're not concerned with 1000 bugs vs. 100 bugs, you're concerned about how many of those bugs compromise your system or cause a crash.

Another good metric is delta bugs over time. There will always be bugs, everyone knows that there will be bugs. The question is the bug count going up or down.

Another good metric is delta time between open and close. Again, there will always be bugs, but how fast they are getting closed is a measure of whether a product is good or bad.

Another good metric is distribution of the number of days that bugs have been open and reported (by severity).

The reason for this is that the number of lines to implement a given function using a given algorithm varies between programmers and programming styles. Some will use two lines or statements to do what can be done in one, to make code easier to read. The line number is something that a developer can manipulate easily.

However, these other measurements are tied directly into the quality of the code. Nobody cares how big the Linux kernel is compared to the Windows kernel, or vice versa. What they care about is how well the Linux kernel works compared to the Windows kernel.

Any count related to bugs, also, needs to take into account the fact that on Windows, you have billions of users any of whom could find and report a bug. On Linux, bugs are more likely to go undiscovered for a longer period of time, simply because there aren't as many people trying to hit them.

The Windows and Linux kernel tend to be very similar in bug counts. The kernels of both OSes tend not to have bugs, because kernels tend to have simple code that's hard to mess up in any meaningful way.

It's only when you start including all these ancillary subsystems, device drivers, etc. that you start to see significant percentages of the bug.

And it is exceptionally hard to get an accurate count of those on Linux. On vMac, I documented approximately one bug in ten that I fixed, and I think that would be typical of other open source projects (although I can't swear to that).

I think that on these other, valid measurements that aren't dependent on lines of code, that if you could collect the statistics on a package per package basis, and compare them, that Open Source would still come out ahead of Windows.

It's just a matter of using a meaningless metric on widely divergent code bases to prove any point is irresponsible, and by reporting it, the media is doing a disservice to the Open Source community by perpetuating the image of the community as a bunch of college students who don't understand real development practices.

Re:Mistake

[Thundersnatch]

But you're wrong, they don't. IE and Media player run in user space in Windows. As do the vast majority of "buggy" programs in Windows.

It would be far more fair to compare XP with the Linux kernel + X + Gnome/KDE + browser + email app + xine + a bunch of other equivalent user-space utilities. But that would be a lot of work, and deciding where to draw the lines would be difficult. The two OS are architected very differently... shouldn't the horrendously buggy third-party video drivers that ship with most linux distros be included, since so many third-party drivers are included with Windows?

I'm not saying Linux would lose such a "fair" comparison, it would probably still be found less buggy than Windows. My point is this referenced article is stupid, useless propaganda generated by someone with an axe to grind.

Re:Mistake

[MyLongNickName]

Excessive information hiding exacerbates any bugs in inherent in the system attempting to do the hiding.

Excessive verbage inclusion exacerbates recipient misunderstanding of the intended message.

Re:Mistake

[Dread_ed]

Dammit dbacher, stop making sense and being all rational and stuff!

We're trying to bash the dogshit out of MS products here and you are messing it up!

Go to your cubicle!

Congratulations...

[kjones692]

...but while they were going through all those 5.7 million lines of code, would it really have killed them to debug them while they were at it??

Re:Congratulations...

[MadKeithV]

From TFA

Seth Hallem, CEO of Coverity, a provider of source-code analysis, noted that the majority of the bugs documented in the study have already been fixed by members of the open-source development community.

Re:Congratulations...

[lphuberdeau]

I doubt they looked up all the code. They probably only made statistics to compare the amount of bugs based on what has been reported and archives.

As a side note, at 20 bugs per 1000 lines, the 40 millions lines of Windows would contain 800000 bugs. I'm not a M$ fan, but this sounds a little excessive.

Re:Congratulations...

[chialea]

I'm not a programmer-type either, but I'm familiar with some of this research. There are a few techniques that I've seen, but don't consider this to be complete, my research is in crypto, not code.

1. code patterns -- if you see something that looks like a pattern, it is probably a bug... "if(x = 0)", for example. of course, you have to check that it actually IS a bug, but you can catch certain common things that way.

2. type safety -- tools can go through your code (either statically or while it's running) and look for type violations. for example, you might write an int to an unsigned int, or mix up pointers and ints, which could be bad. you can catch a stunning number of bugs this way.

3. pointer analysis -- another annoying bug can be in aliasing, where you have multiple pointers that may or may not be pointing at the same memory. are you really /trying/ to overwrite that data structure pointed to by another pointer? In general, this sort of analysis is hard (undecidable, off of the top of my head), but feasible in limited cases.

I'm not sure what sorts of current tools are released by these researchers, but this is a very basic overview of the techniques I've heard about people using recently. (Repeat disclaimer: I'm a theorist.)

Lea

Re:Congratulations...

[LiquidCoooled]

so, if they're so good at identifying them, is it much different to fix them?


Yes, its VERY easy for somebody to notice a bug or unexpected effect.
it may also be very easy to suggest a possible fix for that bug.
However, in a large system with lots of interconnected parts, you cannot just impliment the fix without being certain that you haven't screwed up other parts.

Take for example the slashdot rendering bug in firefox. Plenty of people have reproduced the bug, plenty of people have suggested work arounds and fixes, however none has been stable enough to go into the proper branch (may have changed since I last looked, 1.0 is still problematic).

Now, another source of bugs and errors is compiler updates. Problems with buffer overflows and other "standardised" bugs can often be removed by changing the compiler switches or obtaining an update to the compiler, this is in effect how MS managed to remove a great many problems with SP2.
The inverse is also true, code which worked as expected on one version of the compiler may begin to fail during recompilation many years down the line leading to new unnoticed bugs.

Re:Congratulations...

[Khazunga]

I've not RTFA, so this may not be true for this particular case. However, the bug figure is usually derived statistically. You assume bug finding follows an inverse exponential (picture a capacitor charging function). After some time, from the found bug data, you can deduct the function limit when time is infinite. That limit is the total bug count.

Re:Congratulations...

[Octagon Most]

"I doubt they looked up all the code. They probably only made statistics to compare the amount of bugs based on what has been reported and archives."

Exactly. The article is nearly useless. According to the CNet article covering the same issue:
"Proprietary software, in general, has 1 to 7 flaws per thousand lines of code, according to an April report from the National Cybersecurity Partnership's Working Group on the Software Lifecycle, which cited an analysis of development methods by the Software Engineering Institute at Carnegie Mellon University."

The Wired article says,
"... 20 to 30 bugs for every 1,000 lines of code, according to Carnegie Mellon University's CyLab Sustainable Computing Consortium."

Hmmm, both cite Carnegie Mellon University as the source. So which is it, one to seven, or 20 to 30? That's a big difference. It's either 5,700 to 40,000 flaws or 114,000 to 171,000.

The bottom line is that the Linux source code can be viewed and has 985 visible bugs of various identifiable types. The Windows source code cannot be viewed and may have anywhere between 5,700 and 171,000 flaws based on some questionable extrapolation using two widely divergent methods.

Re:Congratulations...

[chgros]

A very common bug in the kernel:

p = malloc();
if(!p)return;
q = malloc();
if(!q)return; <= Memory leak if this happens

How can one be sure

[UltimaGuy]

How can one be sure about closed source kernels like Windows XP. Even though I agree that it has to contain more bugs, without the actual source how can any one make any judgement in this matter?

Re:How can one be sure

[Deviate_X]

Actually "Windows XP" isn't a Kernel. The kernel of Windows XP is called the actually called the "NT Executive" - which is composed of the Hal (Hardware abstractiomn..), Microkernel and kernel services ( device drivers,.. ).

Windows XP Architecture

Re:How can one be sure

[Burb]

Also, take a close look at the statement Windows XP, by comparison, contains about 40 million lines of code, with new bugs found on a frequent basis

Well, the number of lines of code in the code base isn't strictly relevant, but comparing an unquantified rate of bug discovery against a quantified absolute number of bugs doesn't sound like sound practice to me.

It also makes no statement about the relative severity of the bugs - 100 trivial bugs with simple workarounds would certainly be worth fixing, but how can you compare them against, say, one bug that trashes your principal hard disk?

OK that's my opinion, now to R the FA.

Re:How can one be sure

[maxwell demon]

The kernel of Windows XP [...] which is composed of the Hal [...]

This may explain the Windows crashes. "Sorry Dave, I cannot let you do that."

Conflict of interest...

[BJZQ8]

The problem is that there is very often little vested interest in fixing bugs in closed software...if it can be covered up, then so be it. In open software, there's always a reason, even if it is just to keep people from pointing at your code and laughing.

Re:Conflict of interest...

[akadruid]

If it can be covered up then it's not a very serious bug. Why spend money fixing bugs that aren't a big deal?

See Also: Diebold

Re:Conflict of interest...

[BJZQ8]

Perhaps it's not a serious bug at first examination...but someday an exploit is discovered, and it becomes a VERY serious issue. With open software, there is no burying. If it's there, it at least CAN be found, even if it isn't immediately. Closed software depends on the goodwill of the software manufacturer, where there often is none.

Re:Conflict of interest...

[Malc]

"Bug" is a very generic term. Some scaremongers like to use it in the worse sense of the word, but it could just be something simple like setting a screen area to the wrong colour or having a spelling mistake in a piece of text displayed to the user. A bug could just be the use of an inefficient algorithm leading to poor performance and making the user stare unnecessarily at a busy cursor.

Why do people always look for the worst in things? Yes, you have to have a certain level of paranoia when coding as it is hard to see some of the ramifications, but I think this goes beyond that. In this case it's a sensationalist story...

What about the ones they missed?

[Ironsides]

Not to be a downer but, how do we know they didn't miss anything? 5.7 million lines is a lot of code to go through and analyze. I'm also curious where they came up with the 20-30 bugs per thousand lines of code that proprietary software suffers from since they can't see the code.

Of course, we must remember, "It's not a bug, it's a feature!"

Re:What about the ones they missed?

[AccUser]

Not to be a downer but, how do we know they didn't miss anything?

Like the man said...

There are bugs we know we know.
There are bugs we know we don't know.
There are bugs we don't know we know.
There are bugs we don't know we don't know.

Apple != Orange

[kin_korn_karn]

Talk about misleading stats...

The Windows XP code base includes all of the extraneous crap that gets bundled with and on top of the kernel.

The "Linux" code base just includes the kernel.

Re:Apple != Orange

[abb3w]

The Windows XP code base includes all of the extraneous crap that gets bundled with and on top of the kernel.

This is what you get for integrating your web browser into your operating system. Legality aside, there was a low cunning to that business move when M$ did it. Now, however, that decision is coming back to bite them on the tender bits: the browser is part of the OS, ergo bugs in the browser count as bugs in the OS.

Re:Apple != Orange

[hackstraw]

The Windows XP code base includes all of the extraneous crap that gets bundled with and on top of the kernel.

The "Linux" code base just includes the kernel.

MS gets dinged all the time (at least by us UNIX guys) for what crud they put into the kernelspace, and much of that "extraneous crap" has caused a number of security and stability issues over the years.

Windows and Linux are fundamentally different in design and purpose. And I do believe that most of the "extraneous crap" code should be included in the kernel lines of code count if it runs in kernel space.

Re:Apple != Orange

[falsified]

Forgive me, O Moderators, for being blunt:

Do you use only the Linux kernel? No. You run Slackware, or Fedora, or SuSE (my personal choice), or Gentoo, or something. A fresh copy of the latest version of a distribution is what should have been analyzed. All of the other stuff that's piled onto the kernel is what you use, such as the GUI, time-killing games, and all the other shit that a standard XP install disc contains. You don't click on line 11359 of the Linux kernel to open up Konqueror. You click on an icon, which starts a process that you use. The stuff piled onto the kernel is far more buggy than the kernel would be. I'd bet that a distribution of Linux is still far less buggy than Windows is, but it's certainly not 985 vs. 800,000. For example, when I minimize my windows on XP, there isn't hard drive activity for the next five minutes, as can happen with SuSE with not very much open. I'm sure something's misconfigured, but if a distribution, in 2004, is that poorly configured by default, then that's a bug.

Re:Apple != Orange

[MojoRilla]

You are suggesting that they should have sacrifice usability for a meaningless statistic. What a typical dumb PHB statement.

Exactly how does making the browser part of the operating system make Windows more usable?

Because they did this, I now can't remove IE if I want to. I have trouble with outlook if I set mozilla as my default browser. I can't upgrade IE without patching the OS (which means other programs that use IE components might break). There is no way for me to downgrade to earlier browser versions. There is no way for me to have multiple versions of IE installed, so it makes it hard to test. Because IE is in the operating system, Microsoft has delayed releasing new features until the next version of the OS is shipped.

I can't think of a single thing that integrating the browser into the OS buys you, except some code reusability. From my engineering perspective, this wasn't done for any legitimate engineering reason, but because of legal and business reasons.

Re:Apple != Orange

[truthsearch]

Start up time and rendering speeds are fast for IE.

And both can be achieved without OS integration. Rendering for any 3rd party app can be direct to the video driver if the OS allows it. That's not integration.

It's already been proven that startup time for all Office apps is from hidden API calls near the start the executable code. They load the visual interface before the application's actually ready for use. That plus pre-loading of DLLs gives fast startup. Office isn't considered part of the OS, yet IE is. Therefore fast startup times have nothing to do with integration.

Try supporting 5 different applications vs. one. Over the phone. With a user with no training or previous knowledge.

I have. An entire office of old-fashioned accountants who prefer ledgers and pencils. How is blending 2 apps tightly together better than having 2 separate apps? If there's a problem with Firefox I can tell someone to not launch it. If there's a problem with IE parts of it are in memory whether I choose them to be or not. If there was less integration in Windows then it could be trimmed down to a minimal size for each user. Instead everything including the kitchen sink must continually be supported. You're only increasing your headache by using Windows and its tight integration.

I'm questioning the statistic mentioned is valid or not. Can this number even be trusted?

Not as purely fact. Yet someone who reads the study may determine that it's better to have the code open to all who can fix bugs instead of one select group. Or it may give insight to management that security can be better achieved when they can have their own people analyze the code. When read properly I don't see how anything but good can come from a study such as this.

Not completely scientific

[The-Bus]

First off, what does this statement mean?

"[Linux has] 985 bugs in 5.7 million lines of code, well below the industry average for commercial enterprise software. Windows XP, by comparison, contains about 40 million lines of code, with new bugs found on a frequent basis."

So Linux has 985 bugs. Windows has bugs that appear frequently. Ok that doesn't really tell me anything. I tried to dig a bit deeper and came up with: "Coverity has not analysed the source code to Microsoft Windows because the company does not have access to the source code, Hallem said. Apple Computer's Mac OS X has a great deal of proprietary programming, but the core of the operating system is based on BSD, an open-source operating system similar to Linux."

So everything is based on estimates. Now, you know and I know that the Linux kernel has less bugs... but this is a tentative (at best, shoddy at worst) way of presenting that idea.

Linux Kernel vs Windows XP

[vasqzr]

What about if you throw in KDE or GNOME, Mozilla, etc, everything that you'd have to add to really equal the features of Windows XP....

Re:Linux Kernel vs Windows XP

[Realistic_Dragon]

Crashing KDE won't kill your system. System services will keep on rolling just fine.

Crashing the Windows shell will nuke the whole box, web servers, ftp servers, application servers and all.

Obviously the distinction for a desktop user is minor, since your desktop is gone and your work with it, but if you are running servers the separation is VERY important. A KDE crash (unlike a hard lock from windows shell bringing down the system) doesnt lead to:

Service unavailability for customers.
Possible disk corruption (disk writes not completed).
Potentially having to rebuild a volume from raid or journal.
Loss of state based data (eg from a web app).

And so on. Add to this that users for whom stability is critical won't even be running a desktop on a Linux system and you can see why this is a real world metric for users who aren't relying on all of the extra crap that they cannot disable in the NT kernel.

Re:Linux Kernel vs Windows XP

[beuges]

the windows shell is explorer.exe, microsoft's equivalent to kde. if explorer.exe crashes, web servers, ftp servers, database servers, and anything else running on the box, even user applications that are not tied in to the shell, will continue to run unhindered. so, to respond to your first statement, crashing explorer.exe wont kill your system (do it yourself on windows xp, go to task manager and kill the explorer process). system services will keep on rolling just fine. contrary to /. belief, the windows shell is not "tied in" to the windows kernel any more than kde is tied into bzImage.

Re:Linux Kernel vs Windows XP

[Al Dimond]

Well, right. The architectures are different so they crash and burn in hilariously different ways.

On a Windows machine I used to use at work if I printed to a printer that was down from MS-Word the computer would completely lose it to the point where explorer.exe had to be killed (actually it would pop up the alert saying that explorer.exe is not responding, would I like to end task, so I clicked yes). I've actually never seen a Windows computer that ran smoothly after killing explorer.exe, every time I've done it I've had to do a restart to make it work well (that's probably because there was something else going wrong too). At least you can still save documents, etc., after killing explorer.

On this here Linux box, if I ssh to University Solaris machines and use -X instead of -Y and start a remote X app, my system goes to a grinding halt. To get out of that I have to switch to the console (which takes about 10 seconds) and kill ssh, because X is completely unresponsive (keystrokes in the console are slow as well, but it's at least usable). Although once I kill ssh it runs perfectly well, and I remember to use -Y next time. I've never had my Linux box crash to the point where it had to be restarted, but I don't run KDE or Gnome, so if those can really take a computer down I wouldn't know.

Re:sigh...

[ceeam]

Note how the (/.) article does NOT state the number of bugs in WindowsXP code. It just states the number of lines in XP code (supposedly, courtesy Microsoft Corp.) and some _industry_average_ bugs per line numbers. I would call that "propaganda" if I weren't on their side ;)

20-30 bugs per 1000 lines???

[phasm42]

Commercial software typically has 20 to 30 bugs for every 1,000 lines of code, according to Carnegie Mellon University's CyLab Sustainable Computing Consortium.

I'm gonna call bullshit on this figure. This sounds like a number someone pulled out of their ass. A rate of 20-30 bugs per 1000 lines would render most programs unusable.

Re:20-30 bugs per 1000 lines???

[phasm42]

If you're doing this 20-30 times per 1000 lines, it's bound to show up pretty quick. And that's an average -- figuring in highs and lows, maybe it'll hit 100 bugs per 1000 lines every now and then, 1 line out of every 10 is defective. Ridiculous. I understand what you're saying about state C when I made my comment, and I still believe this figure is BS.

Re:20-30 bugs per 1000 lines???

[Bachus9000]

A rate of 20-30 bugs per 1000 lines would render most programs unusable.

Sounds like Windows to me! :-)

It's a joke, laugh. :)

Re:20-30 bugs per 1000 lines???

[Rob Riggs]

I still believe this figure is BS

It's not.

There are bugs that affects applications to a level the user notices, and there are bugs that affect the maintainability of the code, the reusability of the code, and the ease of use of the code. Most bugs are hidden from the user's view.

Does a stack overflow in a piece of code that only occurs when a craftily created exploit is executed constitute a bug? Yes it does -- even when no exploit exists. Under normal operations it does not affect the use of the system -- at least not until an exploit is developed and widely used. The fact that Windows, especially the older versions of the OS, were vulnerable to so many simple exploits illustrates the bugginess of the code. And most of the bugs are not even in close enough to the surface to be so readily exploited.

Does an end user see API bugs? Does anyone outside of Microsoft experience architectural flaws in the Windows OS? No, but that does not mean they do not exist.

What about code that misbehaves when presented with certain data. That may not be a problem for the original application that the code was written for, but any time that code gets reused, the programmer must know about these "hidden features" (read: bugs). Again, the user never sees this sort of bug.

The number they came up with wasn't pulled out their ass. See this article on measuring bugs for a more detailed discussion on the topic.

When reading it, bear in mind that most commercial software is produced for in-house use, receives very little QC, and frequently does not even compile cleanly. It's usuall just "good enough" to get the job done.

Re:20-30 bugs per 1000 lines???

[Iphtashu Fitz]

I'm gonna call bullshit on this figure.

Keep in mind that you need to know the definition of a bug. It's not necessarially what you think it might be, but what the researchers defined. By their definition a condition that could never occur could be considered to be a bug. For example:

int foo ()
{
if (0)
return;

do_something();

return (0);
}

This overly-simple example could be considered to be a bug. If the condition is ever true the function will return an undefined value, but the condition will never be true so you couldn't possibly return an undefined value. It's not at all uncommon to find code with similar logic scattered throughout - improperly defined loops, conditionals, etc. could result in theoretical bugs that no path of execution can actually get to.

Then there are the kinds of bugs that only occur in extremly specific situations. About 13 years ago I had to track down a bug that caused a report package to crash. It took me a while to figure it out but eventually I did. The program would crash only on specific days. It'd only crash on Wednesdays. It'd only crash on certian Wednesdays - Wednesdays in September. Even more specifically, usually only the 3rd or 4th Wednesday in September.

The bug was that whoever wrote the code that printed a header on the reports was extremely anal about memory usage. He calculated exactly how many characters it would take for a buffer to hold the full date. The problem was he miscalculated by 1 character. With "Wednesday" being the longest day spelled out and "September" being the longest month, a 2 digit date (eg. Wednesday September 23) meant that the full date string would overflow the buffer by 1 character. This kind of bug wouldn't show up very often - only a few times a year - but it was a pretty nasty one when it did.

Re:20-30 bugs per 1000 lines???

[jeremyp]

Bzzzt wrong

jeremyp@dhcp-2-1-56:jeremyp$ cat foo.c
#include <stdio.h>

int main ()
{
if (0)
return ;
printf ("Hello World\n") ;
}
jeremyp@dhcp-2-1-56:jeremyp$ gcc foo.c -o foo
jeremyp@dhcp-2-1-56:jeremyp$ ./foo
Hello World
jeremyp@dhcp-2-1-56:jeremyp$ gcc --version
gcc (GCC) 3.3 20030304 (Apple Computer, Inc. build 1666)
Copyright (C) 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Re:20-30 bugs per 1000 lines???

[julesh]

Sounds like Windows to me! :-)

It's a joke, laugh. :)

It's a sure sign that the moderators are getting stupid when you have to point out the jokes to them. ;-) <--- NOTE SMILEY. :) :)

Re:20-30 bugs per 1000 lines???

[Macphisto]

jeremyp@dhcp-2-1-56:jeremyp$

Dude, letting dhcpcd set your hostname is extremely lame.

Four Years?

[!the!bad!fish!]

according to a four-year analysis of ...
... the 2.6 Linux production kernel

The 2.6 kernel isn't even a year old yet. How'd they do a 4 year analysis?

No I didn't RTFA.

Statistics

[amigoro]

• Typical Commercial Software: 20 to 30 bugs for every 1,000 lines of code (Kloc)
• Linux kernel : 0.17 bugs per Kloc
• Windows XP: 40-50 bugs per Kloc Source

Moderate this comment
Negative: Offtopic Flamebait Troll Redundant

Positive:Insightful Interesting Informative Funny

Interesting, but the comparisons are flawed..

[pcardno]

Sounds like it was a pretty dull thing to do, but reasonably interesting results. I would question though that the "bugs" they found would seem to be pure programming bugs, since they just analysed the source code. The majority of bugs found in systems are usually found by actually using the software and often come about as a result of either unexpected circumstances, unexpected input or compatability issues. Merely reporting the straight programming errors really isn't the same thing.

Also "Windows XP, by comparison, contains about 40 million lines of code, with new bugs found on a frequent basis" isn't exactly very scientific either. How frequent? How severe? XP has been released for roughly 3 years. According to the poster, it's roughly 8 times the size of the code these guys analysed, in which they found 985 errors. So to be at the same level, that would allow for around 7880 bugs, or about 8-10 bugs being found per day since its release. Is that the frequency that's implied here?

It sounds like a good bit of initial research, but probably only just to Bachelors degree level. They need to apply this research correctly in fair comparisons to other operating systems before the results they came up with are meaningful.

Re:Kernel is not the problem

[jacksonj04]

Amen to that. I have never been a big Apple fan, but one thing I will say in their favour is that it just works.

Even on Windows machines, Apple software just works. iTunes shares music across the network with a single checkbox and everything else just works. I plug my iPod in and it just synchronises, and comes up with a playlist based on what I listen to and what I like.

Doing something similar with a combination of vendors? Not a chance. Doing something similar on Linux based systems? Possible certainly, but I don't want to have to write it.

Linux Kernel is solid. Sadly, once you put useful applications on it (like the ones that make WXP 40 million lines long) it will fall apart.

That's not just funny - it's TRUE!

[Anonymous Coward]

Why do you think code inspections work so well?

Yes, the extra eyes looking at your code helps. But so does the fact that you know that there will be extra eyes looking at your code.

So you do a better job.

And the correllary is therefore that one of the big reasons some programmers don't like code inspections is that they don't want others to see their code.

Gee, if you worked for me and didn't want others to see your code I'd wonder why.

Re:More /. FUD

[phasm42]

Mod parent up. Although I may believe Linux has fewer bugs, they don't really present any facts to support this other than some vague statement that Windows has "new bugs found on a frequent basis", and some blanket statement with a figure they pulled out of their ass about commercial software, "20 to 30 bugs for every 1,000 lines of code. The reporter for that article did a crappy job in his research. I'll bet that if CyLab is as good an institution as it's made out to be, the people doing the research are probably pissed about how full of it this Wired article is. The reporter probably twisted a couple vague statements he got from someone at CyLab.

The real question...

[fizban]

Did they file bug reports on those 985 items?

Mistaking symptoms for bugs

[MoebiusStreet]

In proprietary closed code, it's impossible for an outside party to determine the number of bugs.

The best you can hope to do is to count the number of *symptoms*, but that's not the same thing as bugs. It's possible, even likely, that a single bug in the code manifests itself as 100 apparently different external symptoms.

So this comparison is apples to oranges and completely meaningless.

So in all this code auditing...

[eno2001]

...have they found any SCO bugs? ;P I kid, I kid! Because I love!

new vulnerabilities + exploits

[NoSuchGuy]

Here are some new vulnerabilities:

Linux kernel IGMP vulnerabilities von Paul Starzetz

Linux kernel scm_send local DoS von Paul Starzetz

Update

[Tom]

Update: 18 hours after posting the study, the Linux kernel team had eliminated all the bugs documented in the study, forcing the researchers to correct the bug count down to 0 per 10,000 lines.

Where do they get these numbers???

[gUmbi]

Commercial software typically has 20 to 30 bugs for every 1,000 lines of code, according to Carnegie Mellon University's CyLab Sustainable Computing Consortium

I'm going to call bullshit on this. Maybe this number is based on some rule that every variable must be asserted, everything exception checked, etc. (even if these conditions rarely or never happen).

If they're counting bugs like I count bugs - i.e., in a normal operating environment the software loses data, produces incorrect results or limits operability then there is no way that a commericially viable product can have this number of bugs.

Re:Where do they get these numbers???

[p3d0]

I think they must be counting compile-time errors.

smells like BS

[OldAndSlow]

This post pisses me off:

How do you analyze 5.7 million lines of code except to run it through a static analyzer? Static analyzers can't detect most errors, which tend to be data dependent. So a company selling an uberlint donates their tool, 5 academics run it and write a paper. *Blech*

Commercial software typically has 20 to 30 bugs for every 1,000 lines of code Bull. Software with 20-30 errors per KSLOC doesn't work. (I know, I just spent a year trying to save a project that had 10 errors per KSLOC. Even at that defect density (found and fixed) it was undeliverable. Notice that the link in TFA doesn't take you to anything that supports this assertion, just to the organization's home page. I could't find anything to support these numbers.

Weasely non-comparisons: 2.6 Linux production kernel, ... contains 985 bugs in 5.7 million lines of code as opposed to Windows XP, by comparison, contains about 40 million lines of code, with new bugs found on a frequent basis.What, bug in Linux are found on an infrequent basis?

This is all hot air.

OK

[hackstraw]

Linux software ... contains 985 bugs in 5.7 million lines of code

I hope they submitted a patch :)

Re:My thought, too

[chialea]

(Err... just a note. The reason he's had trouble getting people to use them is not code-related, it's more ego-related. This, for me, really reinforces the moral -- be nice to people who patch your code as part of their research.)

Lea

20 to 30 bugs for every 1,000

[dnhughes]

I'd like to know what consistutes a bug.

Bug This!

[mikers]

Karma to burn, karma to burn...

The report, set to be released on Tuesday, states that the 2.6 Linux production kernel, shipped with software from Red Hat, Novell and other major Linux software vendors, contains 985 bugs in 5.7 million lines of code, well below the industry average for commercial enterprise software.

Commercial software (at this point in time) has its priority on releasing new versions often. Because each release is a salable item. Linux on the other hand gets forked or changes version whenever "Linus feels its ready". BIG DIFFERENCE. Here's why.

Commercial software decide how much value is on each bug, if the bugs are cheap (not show stoppers), but minor things they can't forsee as causing them to lose money... They will ship it. Acceptable known bugs. Project management decision.

Open source has time on their hands. They can look over the code carefully, waste time on bugs that commercial outfits wouldn't even bother... But the problem (like with software project management) is that you can't tell which bugs will be the nasties when you choose to ignore them. Less bugs == more secure software, less nasties.

If commerical software decided to play the careful release, minimize bug game... They would make less money initially, but in the end it would work out. Microsoft and ilk can certainly compete with linux, but they made a choice long ago not to. They made a choice to RELEASE FAST and MAKE MONEY FAST! (hey, that sounds like spam).

m

Re:They counted bugs per lines of code

[KarmaMB84]

They could compare it with the Xbox version of Win2k :)

Re:Now tell us what the bugs are

[1u3hr]

RTFA: they were working on the code from 2000; took them four years to analyse it. So the code they were looking at has been patched in current versions: "Seth Hallem, CEO of Coverity, a provider of source-code analysis, noted that the majority of the bugs documented in the study have already been fixed by members of the open-source development community."

Flawed comparison

[willgott]

The article claims that "...the Linux kernel programming code is better and more secure than the programming code of most proprietary software..."

Of course it is.

Most proprietary software are user-level apps where a bug here or a bug there isn't critical. The economic loss that can be atributed to a bug in MS Word's bullet-point-algorithm isn't as great as a datacenter going down due a Oops in the linux-box running the SQL-database. To put it simply: there are different requirements for different tasks. A comparison between the quality of WinXP's GUI and GNOME for example would have been much more interesting.

I don't understand how this story could make slashdot's frontpage.

Re:Flawed comparison

[Ignignot]

With apologies to Douglas Adams:

You don't understand how the story could possibly make the frontpage. To those that know how slashdot really works, it was inevitable that once this story was written, it would reach the front page.

oh man I messed that up bad.

Breaking News!!!

[Class Act Dynamo]

This just in: Microsoft has purchased Stanford University. These rogue researchers have been fired and put in jail on nebulous theft charges. Stanford University announces that Windows is better than Linux according to its research.

IEFBR14

[iBod]

When I walked with dinosaurs (ok, IBM mainframes) as a sysprog, there was a utility program named IEFBR14.

The purpose of IEFBR14 was to do exactly nothing, and pass a zero return code to the caller after doing the 'nothing' (branching on the return address in register 14 - thus BR 14).

This was actually more useful than it sounds and was used frequently in MVS JCL (Job Control Language) to make JCL do its thing without having to run a real program in a JCL 'step'.

Thing is, this program that had to do precisely nothing, had no less than 3 patches issued from IBM. Mostly to do with not clearing R15 (the return code register) correctly.

Go figure!

Re:IEFBR14

[iBod]

Yes, as you say, It allowed the JCL to work its dubious magic with file allocations etc.

Did you ever figure out condition code interpretation in JCL? It always seemed backwards to me COND(0,4) LE and all that???

Re:20-30 bugs for each 1000 lines???

[Derleth]

void main...

I found a bug in your code already.

Re:partisan hackery

[pjrc]

Vaporware marketing strikes again.

Believe it or not, we're supposed to give Microsoft some credit for their recent good efforts. We're supposed to accept that someday in the future, they will ship secure, high quality software.

That may yet happen, and indeed XP SP2 was a good step, but a soild Microsoft system is still vapor.

Faced with superior products already existing on the market, Microsoft has consistently promised and promised and promised vapor... in the hopes of persuading everyone to keep on using their shoddy wares in hope that they'll someday, at an unknown time in the future, provide improvements. After all, they're "serious" and making a good effort.

We're also supposed to believe that Microsoft's process is somehow superious, despite flawed results, because of a 3 year roadmap. It's all part of the vaporware marketing tactic... don't buy something better NOW, instead wait for us. Never mind their poor track record of slipping their own dates and dropping those ambitious planned features.

Re:Now tell us what the bugs are

[mahdi13]

How could they analyse the 2.6 kernel 4 years ago when 2.6 is less then a year old?
ReadingTFA: "The Linux source-code analysis project started in 2000"...

This is an ongoing study, to find the bugs checkout Bugzilla.kernel.org it's not like they hide them or anything

Not a true comparison

[OhHellWithIt]

Where did they get the Windows source code to examine? If they didn't get the code, then how do they know they found all the bugs in Windows?

There's also the question of defining a bug. Sometimes, one person's feature is another person's bug. Take allowing scripts in email, like Outlook Express used to do. (May still do, for all I know.)

Classic case for the do-nothing argument

[ClosedSource]

For most proprietary software, the source is not available. So when contemplating this study the authors should have realized that it couldn't possibly be valid and decided to go do something else. The fact that they went ahead anyway, is a strong indication of their lack of dedication to scientific principles.

Re:Kernel is not the problem

[tclark]

When I upgrade a kernel I don't want to recompile my Nvidia driver. It should just _work_.

You might want to talk to Nvidia about that. They are able to produce a driver that does this, but they choose not to.

Plus

[Sycraft-fu]

If someone can confidently say Linux has 985 bugs, and ONLY 985 bugs, and point them all out, then one would think in short order Linux would have 0 bugs, since they'd all get fixed.

The whole thing kind of strikes me as people talking out of their ass.

Another obvious biased stance

[Sycraft-fu]

Is comparing the Linux kernel to the Windows OS as a whole. The Linux kernel by itself is essentially worthless as an OS. You at the very least have to add drivers, a shell, and some basic programs and services. In reality to have a workable desktop OS you need X, a good WM, and a ton of programs on top of that.

Now I can see maybe comparing the kernel mode code. Windows throws a lot of stuff in the kernel, much of which UNIX types disagree with. Basically the whole graphics subsystem, including rendering engine, is all kernle mode. Microsoft decided that the speed increase was worth it and it didn't matter since if it goes down, the system halts anyhow since Windows doesn't drop to shell. UNIX is different, if X crashes, big deal, you now have a console and you restart X.

However even with kmode comparisions it gets hairy. I mean what is really necessiary and proper there? Talk to a hard core microkernel fan of soemthing like QNX and they'll say NOTHING excpet the kernel, not even drivers. Linux put a LOT in kernel space as compared to some OSes.

Is anyone else bothered by this?

[rd_syringe]

Is anyone else bothered by the idea that an article entitled "Linux Has Fewer Bugs Than Rivals" is posted that compares 5 million lines of code that make up just a kernel to 40 million lines of code that make up a kernel, an operating system, a desktop, its integrated components, and its included applications? 40 million lines has more bugs than 5 million? Who'd have thunk it?

Why does "Linux is just a kernel" suddenly not apply here? Is it because we're bashing Microsoft? This is kind of lame and makes the community look silly and vitriolic.

Re:Is anyone else bothered by this?

[ColMustard]

This guy has a valid point, and to mod him down as if this argument isn't valid is just silly. Mod points shouldn't be used to kill people who you happen to disagree with.

Re:Is anyone else bothered by this?

[Anonymous Coward]

Read the article again.

According to the author: Linux has fewer lines of code AND fewer bugs per thousand lines of code when compared to typical commercial software.

Currently linux has 985 known bugs in 5.7 million lines of code. If linux is to compete with commercial code it will need about 114,000 to 171,000 bugs in 5.7 million lines of code.

Re:Is anyone else bothered by this?

[JudicatorX]

Why does "Linux is just a kernel" suddenly not apply here?

Linux is a kernel. The problem is that this is comparing the linux kernel to all of windows XP

And yet

[Anonymous Coward]

And yet, Linux has more security advisories than Windows each month.

I'd like to see a serious discussion of the methods and funding sources in this study. You know, the same kind of discussions that appear whenever a Microsoft study is posted. Mysteriously, that critical thinking is absent in articles like this.

40 million lines of code != 5 million lines of code. If Microsoft attempted such a skewed study, I can already picture the kinds of posts people would be writing, asking about methods, funding, and more. Those posts aren't appearing in this article. Why? Well, that's because this isn't about the quest for finding out which operating system actually has less bugs (sorry, I forgot, "Linux is just a kernel!"...funny how that oft-uttered phrase is missing in the article discussion), this is nothing more than a Microsoft-bash article intended to generate Microsoft-bashing replies.

OSTG, the corporation that owns Slashdot, makes money off of OSS products. It's rather convenient that they run a "tech news" site that just so happens to post news stories that are all critical of their competitors, isn't it? Imagine if Microsoft owned a tech news site that posted nothing but anti-Linux articles? You guys would be all over it, but your biases against Microsoft make you have a double-standard. So you ignore that Slashdot is a corporate-owned website with a major bias and an editorial staff that can barely read and write correctly, much less bother to read their own website.

Linux Has Fewer Bugs Than *One* Rival

[MavEtJu]

That's only compared to WindowsXP.

How about comparing it with MacOS/X, FreeBSD and others?

I said -pedantic gives warnings.

[Inoshiro]

From the manpage:
-pedantic
Issue all the warnings demanded by strict ISO C and ISO C++; reject all programs that use forbidden extensions, and some other programs that do not follow ISO C and ISO C++. For ISO C, follows the version of the ISO C standard specified by any -std option used.

-- That seems to be exactly what I described it as doing.

Now, you choosing to make standards un-conformance into a warning rather than an error is something else. (Old) C defines the default return type to be int, but C++ has it as undefined. Forcing it to a warning is not the way to solve that problem; you should've fixed the code, which is why g++ defaults to making the type checking as errors, instead of warnings.