Time for a Linux Bug-Fixing Cycle

AlanS2002 writes "As reported here on Slashdot last week, there are some people who are concerned that the Linux Kernel is slowly getting buggier with the new development cycle. Now, according to Linux.com (Also owned by VA) Linus Torvalds has thrown his two cents in, saying that while there are some concerns, it is not as bad as some might have thought from the various reporting. However he says that the 2.6 Kernel could probably do with a breather to get people to calm down a bit."

Comments

I preferred the old odd/even split

[WillerZ]

As a user, I preferred the old odd/even unstable/stable code split; I'd run .even at work and .odd at home.

I suppose if you buy your linux off the shelf you can complain to your vendor, but for home users looking to do some DIY kernel building the new way is a bit worse. However, I suspect we're a dying breed...

Re:I preferred the old odd/even split

[gowen]

I was wondering could someone explain to my why this (IMHO) good development model was abandoned in favour of continuous feature-adding in the 2.6 kernel?

It was very, very slow, (ironically, even Andrew Morton complained about this). This meant that desirable new features would be backported to the stable branch anyway, either in mainstream or vendor kernels (with all new bugs), which kind of defeated the object.

So it increased the workload, didn't seem to offer massive stability benefits (although, maybe it did, in retrospect), it reduced the amount of testing the new features got, and limited the workloads on which they were tested.

Personally, I find the present -stable branch of non-bleeding edge kernels to be as solid as 2.4 and 2.2 ever were. I do think we've a tendency to look back at that dev-cycle with rose-tinted glasses. It's not as if 2.4 or 2.2 were reasonably bug-free until the twentieth cycle or so.

Re:I preferred the old odd/even split

[Skuto]

2.6.16 fixed a critical vulernability over 2.6.15. It also breaks several network drivers.

There was a time when you could grab the next stable kernel, for example when there was an exploit and you really had to, and you'd know you'd only get *more* stability. Now it's exactly the opposite. If you have to upgrade, you're just screwed.

This started around the time they added reiserfs in the stable series although it was far from stable yet. It's not new in the 2.6 series, really. It's a wrong philosophy.

Compare this to FreeBSD release engineering with RELENG, STABLE and CURRENT. FreeLinux anyone? :-P

Re:I preferred the old odd/even split

[s31523]

I wouldn't so much say we're a dying breed... Rather, I would say that the numbers of people that do their own Kernel building is growing, but the number of people that just buy a distro and install and "hope everything just works" is growing much much faster, which can be viewed as a good thing, since the more people that use Linux will cause commercial Vendors to take note and support Linux more readily. Although, I will miss being that nerdy guy who doesn't run Windows...

Re:I preferred the old odd/even split

[Mr Z]

No, the 2.6.x.y are patch releases of 2.6.x. The development releases are 2.6.x-preY. The release candidates are 2.6.x-rcY.

Makes sense to me at least.

--Joe

Re:I preferred the old odd/even split

[diegocgteleline.es]

The "stable/unstable" development model does not work so well with huge projects like the linux kernel is.

With the old model, the linux kernel would start a unstable release and people would start adding stuff which not the care you'd put into merging something in a stable tree, is not tested a lot, etc...

Now keep this for one, two years. When you decide to release the unstable tree as the next stable version you realize that your unstable tree is full of crap, and you need to waste months or years (Vista) trying to stabilize it. Even when you release the .0 version it's still unstable, so people has to wait even more months to start using it.

The "new" development fixed that. In the current linux development model people is allowed to put new features in the kernel even if they're invasive. But programmers are not allowed to put crap in the kernel, they need to be VERY WELL tested (in the -mm tree) and reviewed, show numbers that back your words if neccesary, document things, etc. Of course no code is free of bugs, so the released version will not be 100% stable as current 2.4 is, but it's QUITE stable.

Because the features are merged progressively, it's MUCH easier to find and fix bugs. Even if there're new features in every release, there're not a LOT of new features - it's much easier to find out what feature broke something between two releases. Compare it with a stable/unstable development model: People keeps adding things for years, when the user switches from 2.4.x to 2.6.0 his kernel doesn't boot. How do you find out who broke that with so many changes?

IMO, from a Q/A POV, the new development model has more sense than a pure stable/unstable development model. It's about "progressive" vs "disruptive", and for projects with several millions of lines and so many contributors it may have sense. Of course, because new things got added there're always some bugs, which is what people is bitching about today. Maybe this could be fixed by leaving the current tree as "stable" and start a new tree - but instead of a "unstable" 2.7 tree, a 2.8 "stable" tree. A pure unstable release doesn't works that well with huge projects like the linux kernel. Remember the hell that FreeBSD 5.x was and how much has affected to the FreeBSD project, remember windows Vista. Maybe it works for some people, but I don't thing it's the best development model for such projects. Solaris is also using this model to some extent - they release things into opensolaris, but what you see in opensolaris is not the "official stable release", it only becomes "stable" after a while.

Re:I preferred the old odd/even split

[Rattencremesuppe]

2.6.16 fixed a critical vulernability over 2.6.15. It also breaks several network drivers.

Stable driver APIs anyone?

Oh wait ... stable driver APIs promote binary drivers ... EVIL EVIL EVIL

Re:I preferred the old odd/even split

[diegocgteleline.es]

With FreeBSD 5.x, if you had a working system

I'm not saying you couldn't choose a stable FreeBSD version - you can run a 2.4 kernel if you don't like 2.6, aswell.

I was talking about development models. 5.X was a disaster, and this is something that even the core FreeBSD developers have accepted (they have changed a bit their development model to avoid the 5.x disaster again, you know): Too many time, too unstable, too many time to stabilize. 6.1 (which was released today, BTW) is great, sure. That doesn't means the development model is the best

Re:I preferred the old odd/even split

[Leadmagnet]

This is why I run WinXP-SP2 and I NEVER have any viruses, crashes, or spyware. Rock solid and recognizes all my hardware, and all software runs flawlessly

Standardize the Kernel API!!

[mlwmohawk]

I have been using Linux since the early 1990's and I've been a software developer for almost 30 years. The one ting that concerns me, and I think this recent indictment is just a symptom of a larger problem.

The problem is that the drivers have to remain in constant flux because the kernel API is always changing. Now, when there are a limited number of drivers, this means that you can move quickly on the kernel. As you add more and more drivers, you add more and more work to keep the drivers updated. Eventually, there is more work needed to update the drivers than modify the kernel, and the drivers become your sticking point.

This is where I believe Linux is stuck. Linus and the kernel team has to look at the various kernel APIs and standardize them with the next release.

Sorry guys, time to grow up. Linux *is* mainstream!

Re:Standardize the Kernel API!!

[cortana]

http://lxr.linux.no/source/Documentation/stable_ap i_nonsense.txt

Re:Standardize the Kernel API!!

[John_Sauter]

As a software developer whose experience goes back more than 40 years, to the Stanford Time-Sharing System on the DEC PDP-1, I can assure you that the only way to keep the kernel API from changing is to kill the project. Just as you wouldn't expect a driver written for Microsoft's MS-DOS to be effective on a modern NUMA machine, you shouldn't expect any driver interface standardized today to be effective 10 or 20 years from now. An attempt to freeze the driver API would hamstring the kernel developers, making the kernel less interesting to work on. Somebody would fork it, to lift the compatibility restriction, and the new kernel would work much better with modern computers, causing everyone to migrate to it.

The only way to keep Linux relavent it to let it evolve. Yes, that creates a burden on driver writers. Linux has a partial solution: keep your drivers in the kernel source tree, and test each kernel to be sure your driver still works. When it breaks the cause should be obvious, and easily fixed. If you are lucky, the person who changed the API will also update your driver, but you can't count on that, which is why you must test.

Re:Standardize the Kernel API!!

[MROD]

I disagree.. mostly.

There needs to be a stable API for drivers PER MAJOR RELEASE so that the driver maintainers can keep stable, well tested and debugged drivers.

The API should be allowed to change with every major kernel revision but any change should be made with a great deal of thought and, unless it's very difficult to do, the old API should be supported for backward compatability.

Not only this, but I would argue that it would be good hygene to separate the core kernel from the drivers. Doing this would make developers think hard about the bounderies between the two and not have one polluting the other. It would also make the developers think long and hard about whether changing the API for something is such a good idea just because it would be useful for the "ACME USB SLi Graphics card programming port widget" interface.

The the kernel is the kernel, the drivers are merely plug-ins to virtualise the hardware, the two should be as separate and distinct as they are logically.

Re:Standardize the Kernel API!!

[just_another_sean]

The operating system MUST provide a standarized [sic] API.

People who code free software MUST not do anything unless they feel like it. Sure some of them might get paid by Company X to develop Driver Y or Application Z but they do so on the shoulders of what's already been put in place by free software developers.

If Linus and the rest of the kernel developers decide at some point to provide an ABI that proprietary companies can use to build their drivers, all the while clinging to their dated business methodologies and obsession with "IP", then great, that's their choice. It might take a Herculean effort to get all those copyright holders to agree and do it but if they can then that's up to them.

Conversely, if they choose not to, they under no obligation to provide anything. Nobody on the kernel team, IMHO, ever got together and said "we need to start coding and provide some free software so companies with no interest in participating in the process can take our free software and make some money selling hardware". They do it for themselves, their friends and family, their community. Whether or not ATI and NVIDIA want to be a member of that community entitles them to exactly nothing.

Re:Standardize the Kernel API!!

[EvilGrin666]

What the kernel really lacks is a good standard for coding practices, like say adding comments and indenting at least somewhat sensibly [yeah I know for some of you "elites" you can take reading a complete lack of consistent indentation but for the rest of us ...]

The kernel includes a document detailing the coding style to use. It lives in Documentation/CodingStyle.txt You can read the current version from Linus' Git tree here. If you spot anything in the kernel that doesn't follow CodeingStyle.txt you should submit a patch to the kernel janitors to fix it up.

Re:Standardize the Kernel API!!

[LordOfTheNoobs]

Point 1 - Your post contradicts its own supposed respect of the GPL.
Point 2 - Linux is FSF free, share and share alike by license. BSD is not. You can't generalize them together on this issue. If you don't get the difference, you don't know what the hell your talking about.
Point 3 - The operating system doesn't _have_ to do shit. If the companies want their shit to run in Linux, the should submit GPL'd drivers or suffer their rightful hell for being miserly with their code in a project based on sharing. To hell with them.
Point 4 - There is a fairly standard API. And when they change it they fix the GPL drivers. There is not an ABI, `application binary interface' since you obviously don't know, which is not require or desired as Linux runs on many different types of hardware. Should we instead suffer to create an ABI for each hardware platform that each driver must uphold? There is more than x86 out there. Hell, even in x86, should we make all drivers have to suffer to a 16 bit driver interface, or create different ABIs for 32 and 64 and the future 256 and 1024 bit systems?
Point 5 - Cry more noob.
Point 6 - If a hardware manufacturer wants to sell their hardware to us, they will either suffer intolerably or they will give in and release some GPL'd code. If coders want it bad enough, someone will reverse engineer it and create free code on their own. It's not like we're going to start using our DVD-Rs to burn off graphics cards. Well, not until my chinese GFX-RW comes in anyway.

Re:question

[zootm]

As the previous article pointed out, there's no lack of developers, just a lack of developer interest in fixing the bugs. Many of the larger contributors are paid by companies to ensure that specific features are put into (or at least developed for) the kernel. And let's face it; bug-fixing is not fun. Regardless of how hard-working the people are on average, bugfixing is generally the sort of thing that people shy away from unless the bugs directly affect them, especially when working voluntarily.

All large systems have a danger of bugs creeping in over time, and it can be easy to let their numbers get out of control as time goes on. The fact that the people in charge are point it out now is basically an example of good management — attempting to address a concern before it becomes more serious.

Re:question

[Vo0k]

yep, the previous poster is right.

I thought I know C until I tried to fix a bug in the kernel.

It was a simple syntax bug. Somebody put xxx[...]->yyy instead of xxx->yyy[...] in one line, and the compiler was protesting about type mismatch. One single line. But it took me 4 hours or so and I figured out only what the correct syntax for that piece of code would be, by analysing types of the variables used. I have no idea if the fix really corrected the problem, it just made the line lexically correct and let the compiler go on. In the meantime I had to crawl about 4 levels of header files for each of the variables/records used in the line to reach primitive types of given variables and macros, from which the structures, pointers etc were derived, and generally was totally dizzy. And I was doing it the code monkey style, I didn't really understand the workings of the kernel, what was the line I edited meant. I was purely checking that a pointer to float isn't directly assigned a value of float, just pointer to it etc.

Kernel is too difficult for us average coders. Only the elite can fix these bugs for us.

Re:Linux is BUGGY so it IS about TIME !

[TerminaMorte]

While I'm not sure if he meant it this way, it sounds to me that he's saying that it's not considered terribly odd for Windows to crash; not that Windows constantly crashes.

If a desktop user sees a blue screen of death (device driver, bad hardware, what have you) it's nothing incredibly shocking; we've grown used to it over the years.
 
Linux has certainly crashed on me (mostly when trying out drivers that arn't exactly stable), and when it happens it is a much rarer (and stranger ;)) occurance.

Certainly you agree that Windows (he didn't specify XP/2003, remember, just Windows in general) is known for problems like that more than Linux is?

Re:Typical monolithic kernel problem

[tadmas]

Any kernel with upwards of 2.5 million lines of code is going to be incredibly buggy, perhaps it's time to rethink and go back to the microkernel

Splitting any software into external pieces is exactly the same as splitting the software into internal pieces. Microkernel is not the answer -- encapsulation is the answer.

Besides, converting the kernel will not get rid of the bugs; it will just make different ones. 2.5 million lines is a lot to rewrite, and any rewrite will lose all the bugfixes already in place.

Re:question

[bhima]

This is nearly the same as my own experience... which makes me enjoy using, in my case, OpenBSD. I use C professionally but it's an order of magnitude (or two) less complex than the Linux kernel. It's just amazing to me it all comes together despite how many people are working on it.

Back to the point what can spending some time and having a bug fixing cycle hurt? I don't see a downside...

Dave Jones take on the story

[greppling]

can be found in a post in his live journal. He reports that with every new kernel release, the number of kernel related bug reports in the Fedora bugzilla goes up substantially.

(Davej is a long time kernel hacker and currently the Fedora kernel maintainer.)

Re:Typical monolithic kernel problem

[diegocgteleline.es]

Any kernel with upwards of 2.5 million lines of code is going to be incredibly buggy

You mean that a microkernel is magically going to implement the same funcionality than linux, with all the thousand of driver, with its support for docens of hardware platforms, in less of 2.5 millions of lines of code?

Sure, a "microkernel" itself doesn't takes a lot of code. But BECAUSE it's a microkernel, drivers, filesystems, networks tacks etc. need to be implemented as servers. Implementing servers that implement the same funcionality than linux has today would take more of 2.5 milliones of lines, for sure. And those servers can have bugs, you know. And hardware bugs exist - it's completely possible (too easy, in fact) to hang your machine by touching the wrong registers no matter if you're using a microkernel or not.

Also, I don't understand why a microkernel would be magically more maintainable than a monolithic kernel. As far as I know, software design is something that doesn't depends in whether you pass messages or not. Sure, a server running in userspace can't take the system down. But that's completely unrelated to modularity and mainteinability. Microkernels were in fact invented because people though that hardware complexity wouldn't allow to continue running monolithic kernels, ignoring the fact that it's perfectly possible to write a mainteinable monolithic kernel with modular design - which is how Linux, Solaris internals etc. are today - just like it's completely possible to write a unmainteinable, non-modular microkernel. It all boils down to software design. And guess what: Current general-purpose monolithic kernels (linux, *BSD, Solaris, NT, Mac OS X - no, a operative system that implement drivers, filesystems and network stacks in kernel space it's not a microkernel) have had a lot of time and resources ($$$) to become mainteinable and modular, extensible, etc.

It's fun how when a monolithic kernel has a bug it means microkernels are better, like a microkernel model magically makes coders bug-free, or like it's not possible to write a microkernel server with a bad API that forces all driver developers to patch their drivers to fix a security bug. I'd love to hear what development model would use the Hurd/QNX/whatever guys to maintain six millions of lines of code, be it driver for a monolithic kernel or drivers implemented as microkernel servers.