Lack of Testing Threatening the Stability of Linux

sebFlyte writes "Andrew Morton, a Linux kernel maintainer, has said that he thinks that the lack 'credit or money or anything' given to those people who put in long hours testing Linux releases is going to cause serious problems further down the line. In his speech at Linux.Conf.Au he also waded into the ongoing BitKeeper debate, saying 'If you pick a good technology and the developers are insane, it's all going to come to tears.'"

Comments

Vacation for Linus...?

[tquinlan]

...does it seem like Linus might need a vacation?

TFA states that he's starting to take as much pride in rejecting patches as he does accepting them, and with this whole BitKeeper thing, it seems to me like he might need a small break.

Of course, I'm not one to really talk, as I don't do nearly as much as he does with Linux...

Also, with regards to testing, those of us who use it daily are testing all the time. I know it's not structured QA, but still, it's a lot of testing.

Also, maybe slowing down the kernel releases a bit might help. I know that I do an emerge world on my Gentoo boxes about once a week, and it seems like there's a new kernel release every week. If there's a need for more testing, perhaps a little less time releasing and more time testing is in order.

Re:Vacation for Linus...?

[SecurityGuy]

Also, with regards to testing, those of us who use it daily are testing all the time. I know it's not structured QA, but still, it's a lot of testing.

Isn't that the essence of Microsoft's QA?

Should we be doing what we rightly criticise them for?

Re:Vacation for Linus...?

[ShieldW0lf]

Also, with regards to testing, those of us who use it daily are testing all the time. I know it's not structured QA, but still, it's a lot of testing.

Isn't that the essence of Microsoft's QA? Should we be doing what we rightly criticise them for?

Say what you want about Microsoft and the stability/reliability/security of their software, but they have many full time (and paid) people devoted exclusively to testing and trying to break their software so that it can be fixed.

He's saying that they DON'T test their product enough, and that they DO ship it broken and have the community test it, and that the open source projects that are so critical of MS for doing this should aspire to something better.

You totally missed the point.

Re:Vacation for Linus...?

[ArbitraryConstant]

"Also, with regards to testing, those of us who use it daily are testing all the time. I know it's not structured QA, but still, it's a lot of testing."

When having users stumble into bugs is your primary method of finding them, your QA has already failed.

Because they do active development on the 2.6 branch, new bugs are introduced all the time. Even if they're only there for one version, there's always more bugs in the next version, which is a big disincentive for upgrades. And not minor stuff, big things like the ability to burn CDs.

Without proper regression testing stuff like that will continue to haunt users. The assumption is that distros will do it, but the simple fact is that they aren't. The kernel developers must take responsibility for it.

Re:Vacation for Linus...?

[ajs]

The point is that they don't come "remotely close", and there's the fact that these things do not scale linearly. In order to support 12 platforms as shipped, you have to do far more than 2x the work of supporting 6 platforms. Why? Because those first 6 are the ones that are most alike, and used by the largest intersection of developers. The other six are used by niche develpers and tend to be less like the first six.

This, of course, ignores the work that goes into special subsystems for popular platforms, special hardware, obsolete hardware, new protocols and standards, etc.

But again, this does not mean that FreeBSD is bad or poorly organized or useless. It's a fine OS that I recommend to people all the time. It's just that there's a different audience.

Re:Vacation for Linus...?

[bit trollent]

He's saying that they DON'T test their product enough, and that they DO ship it broken and have the community test it,

Yes, but he is full of shit. Lots of people are missing the point around here today.

Re:Vacation for Linus...?

[ajs]

It's different in the Linux world, for the reasons that you pointed out before.

The kernel team tests and releases in one pass, which is roughly akin to unit testing in a large project that has several sub-projects.

Then distributions pick up the changes (it's really not that clean a separation, but let's say it is for sake of simplicity), and incorporate it into their OSes. Each distribution has its own unique QA/release process, so let's look at Red Hat as an example. They take some internal things, some external things, the "official kernel" and start testing it with their system. They make some changes, give some feedback/fix bugs/etc. and eventually they come up with a collection of patches that they feel brings the kernel to the point they want it (they repeat this for hundreds of packages, some larger, most smaller than the kernel).

The original source+patches is packaged in two forms: an SRPM, which contains all of the discreate pieces and an RPM which contains the result of unpacking the SRPM, applying the patches inside it to the base code inside it, building and installing along with any pre- or post-install steps that are required.

That's all shunted into Red Hat's final release process which I know almost nothing about, but I presume it involves test farms which they use to stress the new OS in various ways. This might, of course, result in bugs discovered, and further iterations of the process.

Contrapositive

[fistfullast33l]

"If you pick a good technology and your developers are insane, then it's going to come to tears."

Contrapositive is:

"If you don't pick a good technology or your developers are not insane, then it won't come to tears?"

Just some food for thought.

Re:Contrapositive

[Anonymous Coward]

failed logic, huh? no, the contrapositive is: if you don't want it to come to tears, once you've picked a good technology make sure the developers aren't insane.

Open Source Means More Eyeballs?

[xxxJonBoyxxx]

I thought the point of Open Source was to allow more people to read through the code. You mean thousands of people aren't really doing that for fun? I'm shocked.

More seriously... I think many of the people who DO eyeball the code are looking for security problems these days (where you do get recognition, etc.). For the record, I know I won't get any HR props for putting OS bugs that I've uncovered on my resume, but the security bugs I've found are always good conversation pieces.

lack of testing?

[Cat_Byte]

I thought the whole Fedora project WAS mass testing of "cutting edge technology for Linux". Have I been wasting my time submitting bugs? Most have been fixed that I submitted so far.

It's also frustrating to test a moving target.

[Richard Steiner]

Linux is constantrly improving, but that means it is also constantly changing, and that makes it a constantly moving target.

That applies to most distros as well as the kernel itself.

It's hard to put a lot of effort into testing something when it's possible those tests will be invalidated a few months down the road...

"They get no thanks or credit or money... or anyth

[Senor_Programmer]

ing," he said.

Wait a minute here...

I thought the whole scheme was structured thusly...

I crank up the latest greatest kernel. I find a bug. I report it. My bug gets fixed. THAT's MY REWARD! The friggin bug gets squashed. What more could one ask for, with a clear conscience and a straight face.

As for those guys who fix the stuff. Well sanity is a relative term as we should all realize in light of the Japanese influence and emergence of cargo cults in WW-2 Niu Guinea. AFAIK, most Linux users view the kernel developers as some mysterious force from which benefit is derived through clever creation of effigy's.

That's odd

[radiophonic]

I was under the impression that by using Linux, I was, in a sense, testing Linux.

Re:Bugzilla

[Mad Merlin]

If the average Linux user were educated on how to recognize a bug, and file a meaningful bug report it would mean a lot to developers, and likely speed up development and stability. ...and scare away 99.9% of potential new users.

Re:Hmm..

[kfg]

I have this tendency to respond to serious posts with a joke, and jokes with a serious post. I tend to come at problems from angles of perception that other people do not see.

This is what the best developers do, otherwise they would simply come up with the same mediocre to bad solutions that everyone else does, no?

They do, however, have this really annoying tendency to see everything from the same "Man from Mars" perspective, not restricting themselves to viewing only code differently than most do. This can make them appear "insane" to the general populace.

In the land of the blind the one eyed man is a paranoid schizophrenic.

Insanity is percieving things not as they really are. If the majority percieve things not as they really are the man who does so will give the perception of being insane when he acts upon his perceptions, those acts being unintelligable to the majority.

And thus is born the image of the "quirky" genius. All will hail his new invention, but titter quietly about how he wears his socks, never for one minute stopping to take the obvious point of view that there just might be something of genius in the way he wears his socks, because he wears his socks differently than the majority do.

And being the same is sanity, right?

Nevermind that we innately wipe out genius in one swell foop with that attitude. It enforces a regression to the median, if it weren't for the fact that half the populace would have to progress to the median somehow, which, trust me, they just ain't gonna do. So instead of a regression to the median we get a regression the "really dumb."

Take the current fad for "Playskool" interfaces. . . .please.

Of course, some "geniuses" really are just insane and "luck into" some discovery through their insane perception of things.

So how do you tell the difference? Well, takes one to know one I'm afraid. It would be nice if it didn't seem as if the people who end up in charge of "mental health" weren't all, themselves dimwitted morons at best, and completely, utterly crackers at worst.

They're coming to take me away, HO,HO! HEE, HEE! HA, HA!

I think it's something about the way I wear my socks.

KFG

It's Free Software!

[Ulrich Hobelmann]

What are they expecting? It's based on voluntary work.

If anybody needs some guaranteed service, or commercial-grade testing, maybe they should hire some programmers to do it?

Re:*sigh*

[dsci]

If linux had the same user base as MS there would be 100,000 times as many patches, security holes etc.

You don't know this. How can you know this. It may well be true, but right now, it is a gross extrapolation with no data to support it.

And it's based on a very iffy premise: bugs = user base.

Re:In contrast to the MS method...

[Lumpy]

I'm really surprised no company really has used this as a business model.

you don't work with Vertical market software do you.

all of our critical sales apps are considered alpha testing. by the time an app becomes stable and useable they retire it and sell us their next abomination that does not work right, has 1/3rd the features the sales guy sold the CTO on and has stability problems that make any Admin cry (imagine a printer driver on your W2K box causing data corruption in an app... WTF is THAT!

look at applications used by small segments of industry, there is wher eyou will find the untested and crappy code sold for $2000-$6000 per seat.

Re:*sigh*

[Anonymous Coward]

Don't let the infantile MS bashers here bother you, the people that imply that microsoft has no testing are the same idiots who "hear" that MS products are insecure because NBC nightly news said so. If linux had the same user base as MS there would be 100,000 times as many patches, security holes etc. We may not all agree with MS's business practices, but its evident that they have a very serious testing regime, especially since products often get delayed for years because of minor things.

I thought the OPs article was very interesting. Comparing their testing process with their results is enlightening. I found his post really interesting.

You, on the other hand, are an idiot.

To the best of my knowledge the NBC Nightly News or any other mainstream press outlet have done nothing to help users understand the insecurity in MS products. If anything, their pet analysts have greatly downplayed any weaknesses in MSFTs software or business model leaving millions of uninformed users and investors with their dicks in the wind.

If Linux had the same user base as MS, there would exactly the same number of patches, security holes etc. More developers would slightly increase the number of patches and more users would increase the number of bug reports. The number of security holes is independant of both.

Microsoft does have a very serious testing regime: they need to to stay in business. They have never let fixing bugs interfere with their approach to competition, however. There is ample evidence that MS will ship a critically broken application or even introduce svere bugs rather than allow a competitor to gain market share. There is also abundant evidence that they will not bother to fix a product,no matter how broken, if there is no competitive advantage.

The fact that you still think that most of these "delays" aren't planned from day one suggests that you are watching to much TV and not reading enough (real) IT sites. When MS announced the original ship date for Longhorn, every reputable trade site said "MS claims that they are going to ship X, Y and Z in timeframe T. This is not possible." Do you think that these trade sites (having since been proved right) know that much more than the strategists at MS or do you think that there is another reason that MS might announce a ridiculously optimistic roadmap? How do you think the market would have reacted if, in 2003, MS had said "Longhorn will actually just be SP3 for XP and will ship sometime in 2006."

Re:Hmm..

[Cthefuture]

I have this tendency to respond to serious posts with a joke, and jokes with a serious post. I tend to come at problems from angles of perception that other people do not see.

Heh, I do the same thing. I often consider myself the ultimate Devil's Advocate.

When others are stressed, I'm calm. When others are calm, I'm stressed. Bizarre, but that's the way I work. Unfortunately this causes problems for me even in geek circles because they don't see what I see and often I can't explain why I know the things I do.

As for the socks thing, I think it just depends on what you think is important enough to put energy into caring about.