Time for a Linux Bug-Fixing Cycle

AlanS2002 writes "As reported here on Slashdot last week, there are some people who are concerned that the Linux Kernel is slowly getting buggier with the new development cycle. Now, according to Linux.com (Also owned by VA) Linus Torvalds has thrown his two cents in, saying that while there are some concerns, it is not as bad as some might have thought from the various reporting. However he says that the 2.6 Kernel could probably do with a breather to get people to calm down a bit."

Comments

I preferred the old odd/even split

[WillerZ]

As a user, I preferred the old odd/even unstable/stable code split; I'd run .even at work and .odd at home.

I suppose if you buy your linux off the shelf you can complain to your vendor, but for home users looking to do some DIY kernel building the new way is a bit worse. However, I suspect we're a dying breed...

Re:I preferred the old odd/even split

[gowen]

I was wondering could someone explain to my why this (IMHO) good development model was abandoned in favour of continuous feature-adding in the 2.6 kernel?

It was very, very slow, (ironically, even Andrew Morton complained about this). This meant that desirable new features would be backported to the stable branch anyway, either in mainstream or vendor kernels (with all new bugs), which kind of defeated the object.

So it increased the workload, didn't seem to offer massive stability benefits (although, maybe it did, in retrospect), it reduced the amount of testing the new features got, and limited the workloads on which they were tested.

Personally, I find the present -stable branch of non-bleeding edge kernels to be as solid as 2.4 and 2.2 ever were. I do think we've a tendency to look back at that dev-cycle with rose-tinted glasses. It's not as if 2.4 or 2.2 were reasonably bug-free until the twentieth cycle or so.

Re:I preferred the old odd/even split

[s31523]

I wouldn't so much say we're a dying breed... Rather, I would say that the numbers of people that do their own Kernel building is growing, but the number of people that just buy a distro and install and "hope everything just works" is growing much much faster, which can be viewed as a good thing, since the more people that use Linux will cause commercial Vendors to take note and support Linux more readily. Although, I will miss being that nerdy guy who doesn't run Windows...

Re:I preferred the old odd/even split

[Mr Z]

No, the 2.6.x.y are patch releases of 2.6.x. The development releases are 2.6.x-preY. The release candidates are 2.6.x-rcY.

Makes sense to me at least.

--Joe

Re:I preferred the old odd/even split

[diegocgteleline.es]

The "stable/unstable" development model does not work so well with huge projects like the linux kernel is.

With the old model, the linux kernel would start a unstable release and people would start adding stuff which not the care you'd put into merging something in a stable tree, is not tested a lot, etc...

Now keep this for one, two years. When you decide to release the unstable tree as the next stable version you realize that your unstable tree is full of crap, and you need to waste months or years (Vista) trying to stabilize it. Even when you release the .0 version it's still unstable, so people has to wait even more months to start using it.

The "new" development fixed that. In the current linux development model people is allowed to put new features in the kernel even if they're invasive. But programmers are not allowed to put crap in the kernel, they need to be VERY WELL tested (in the -mm tree) and reviewed, show numbers that back your words if neccesary, document things, etc. Of course no code is free of bugs, so the released version will not be 100% stable as current 2.4 is, but it's QUITE stable.

Because the features are merged progressively, it's MUCH easier to find and fix bugs. Even if there're new features in every release, there're not a LOT of new features - it's much easier to find out what feature broke something between two releases. Compare it with a stable/unstable development model: People keeps adding things for years, when the user switches from 2.4.x to 2.6.0 his kernel doesn't boot. How do you find out who broke that with so many changes?

IMO, from a Q/A POV, the new development model has more sense than a pure stable/unstable development model. It's about "progressive" vs "disruptive", and for projects with several millions of lines and so many contributors it may have sense. Of course, because new things got added there're always some bugs, which is what people is bitching about today. Maybe this could be fixed by leaving the current tree as "stable" and start a new tree - but instead of a "unstable" 2.7 tree, a 2.8 "stable" tree. A pure unstable release doesn't works that well with huge projects like the linux kernel. Remember the hell that FreeBSD 5.x was and how much has affected to the FreeBSD project, remember windows Vista. Maybe it works for some people, but I don't thing it's the best development model for such projects. Solaris is also using this model to some extent - they release things into opensolaris, but what you see in opensolaris is not the "official stable release", it only becomes "stable" after a while.

Re:I preferred the old odd/even split

[Rattencremesuppe]

2.6.16 fixed a critical vulernability over 2.6.15. It also breaks several network drivers.

Stable driver APIs anyone?

Oh wait ... stable driver APIs promote binary drivers ... EVIL EVIL EVIL

Standardize the Kernel API!!

[mlwmohawk]

I have been using Linux since the early 1990's and I've been a software developer for almost 30 years. The one ting that concerns me, and I think this recent indictment is just a symptom of a larger problem.

The problem is that the drivers have to remain in constant flux because the kernel API is always changing. Now, when there are a limited number of drivers, this means that you can move quickly on the kernel. As you add more and more drivers, you add more and more work to keep the drivers updated. Eventually, there is more work needed to update the drivers than modify the kernel, and the drivers become your sticking point.

This is where I believe Linux is stuck. Linus and the kernel team has to look at the various kernel APIs and standardize them with the next release.

Sorry guys, time to grow up. Linux *is* mainstream!

Re:Standardize the Kernel API!!

[cortana]

http://lxr.linux.no/source/Documentation/stable_ap i_nonsense.txt

Re:Standardize the Kernel API!!

[John_Sauter]

As a software developer whose experience goes back more than 40 years, to the Stanford Time-Sharing System on the DEC PDP-1, I can assure you that the only way to keep the kernel API from changing is to kill the project. Just as you wouldn't expect a driver written for Microsoft's MS-DOS to be effective on a modern NUMA machine, you shouldn't expect any driver interface standardized today to be effective 10 or 20 years from now. An attempt to freeze the driver API would hamstring the kernel developers, making the kernel less interesting to work on. Somebody would fork it, to lift the compatibility restriction, and the new kernel would work much better with modern computers, causing everyone to migrate to it.

The only way to keep Linux relavent it to let it evolve. Yes, that creates a burden on driver writers. Linux has a partial solution: keep your drivers in the kernel source tree, and test each kernel to be sure your driver still works. When it breaks the cause should be obvious, and easily fixed. If you are lucky, the person who changed the API will also update your driver, but you can't count on that, which is why you must test.

Re:Standardize the Kernel API!!

[MROD]

I disagree.. mostly.

There needs to be a stable API for drivers PER MAJOR RELEASE so that the driver maintainers can keep stable, well tested and debugged drivers.

The API should be allowed to change with every major kernel revision but any change should be made with a great deal of thought and, unless it's very difficult to do, the old API should be supported for backward compatability.

Not only this, but I would argue that it would be good hygene to separate the core kernel from the drivers. Doing this would make developers think hard about the bounderies between the two and not have one polluting the other. It would also make the developers think long and hard about whether changing the API for something is such a good idea just because it would be useful for the "ACME USB SLi Graphics card programming port widget" interface.

The the kernel is the kernel, the drivers are merely plug-ins to virtualise the hardware, the two should be as separate and distinct as they are logically.

Re:Standardize the Kernel API!!

[just_another_sean]

The operating system MUST provide a standarized [sic] API.

People who code free software MUST not do anything unless they feel like it. Sure some of them might get paid by Company X to develop Driver Y or Application Z but they do so on the shoulders of what's already been put in place by free software developers.

If Linus and the rest of the kernel developers decide at some point to provide an ABI that proprietary companies can use to build their drivers, all the while clinging to their dated business methodologies and obsession with "IP", then great, that's their choice. It might take a Herculean effort to get all those copyright holders to agree and do it but if they can then that's up to them.

Conversely, if they choose not to, they under no obligation to provide anything. Nobody on the kernel team, IMHO, ever got together and said "we need to start coding and provide some free software so companies with no interest in participating in the process can take our free software and make some money selling hardware". They do it for themselves, their friends and family, their community. Whether or not ATI and NVIDIA want to be a member of that community entitles them to exactly nothing.

Re:question

[zootm]

As the previous article pointed out, there's no lack of developers, just a lack of developer interest in fixing the bugs. Many of the larger contributors are paid by companies to ensure that specific features are put into (or at least developed for) the kernel. And let's face it; bug-fixing is not fun. Regardless of how hard-working the people are on average, bugfixing is generally the sort of thing that people shy away from unless the bugs directly affect them, especially when working voluntarily.

All large systems have a danger of bugs creeping in over time, and it can be easy to let their numbers get out of control as time goes on. The fact that the people in charge are point it out now is basically an example of good management — attempting to address a concern before it becomes more serious.

Re:question

[Vo0k]

yep, the previous poster is right.

I thought I know C until I tried to fix a bug in the kernel.

It was a simple syntax bug. Somebody put xxx[...]->yyy instead of xxx->yyy[...] in one line, and the compiler was protesting about type mismatch. One single line. But it took me 4 hours or so and I figured out only what the correct syntax for that piece of code would be, by analysing types of the variables used. I have no idea if the fix really corrected the problem, it just made the line lexically correct and let the compiler go on. In the meantime I had to crawl about 4 levels of header files for each of the variables/records used in the line to reach primitive types of given variables and macros, from which the structures, pointers etc were derived, and generally was totally dizzy. And I was doing it the code monkey style, I didn't really understand the workings of the kernel, what was the line I edited meant. I was purely checking that a pointer to float isn't directly assigned a value of float, just pointer to it etc.

Kernel is too difficult for us average coders. Only the elite can fix these bugs for us.

Re:Typical monolithic kernel problem

[tadmas]

Any kernel with upwards of 2.5 million lines of code is going to be incredibly buggy, perhaps it's time to rethink and go back to the microkernel

Splitting any software into external pieces is exactly the same as splitting the software into internal pieces. Microkernel is not the answer -- encapsulation is the answer.

Besides, converting the kernel will not get rid of the bugs; it will just make different ones. 2.5 million lines is a lot to rewrite, and any rewrite will lose all the bugfixes already in place.

Re:question

[bhima]

This is nearly the same as my own experience... which makes me enjoy using, in my case, OpenBSD. I use C professionally but it's an order of magnitude (or two) less complex than the Linux kernel. It's just amazing to me it all comes together despite how many people are working on it.

Back to the point what can spending some time and having a bug fixing cycle hurt? I don't see a downside...

Dave Jones take on the story

[greppling]

can be found in a post in his live journal. He reports that with every new kernel release, the number of kernel related bug reports in the Fedora bugzilla goes up substantially.

(Davej is a long time kernel hacker and currently the Fedora kernel maintainer.)