Slashdot Mirror
2.6 Linux Kernel in Need of an Overhaul?
toadlife writes "ZDNet UK reports that Andrew Morton, the head maintainer of the Linux production kernel, is concerned about the amount of bugs in the 2.6 kernel. He is considering the possibility of dedicating an entire release cycle to fixing long standing bugs." From the article: "One problem is that few developers are motivated to work on bugs, according to Morton. This is particularly a problem for bugs that affect old computers or peripherals, as kernel developers working for corporations don't tend to care about out-of-date hardware, he said. Nowadays, many kernel developers are employed by IT companies, such as hardware manufacturers, which can cause problems as they can mainly be motivated by self-interest."
A lot of times, the old debate of Windows Vs Linux covers how often the OS fails miserably. Yes, we all know the famous "blue screen of death" and I think that that single concept connected with Windows makes it unappealing. I believe that Linux has the ability to handle internal errors more elegantly but that's only because I've only seen it fail from hardware errors. Granted, I don't know enough about the inner workings of Windows or Linux but let's face it, Win95 & Win98 first editions would crash if you looked at them wrong.
Here's a possible horror story:
While the debate rages on, Linux gets more complex. Linux gains more bugs. Linux begins to aim for more end-user features. Developers get sick of maintaining other developers code and focus on making new features (asked for or un-asked for) because it gives them pride to make something new. The Linux kernel hits the same pitfalls as the Windows kernel.
If it takes an entire developement cycle to simply improve the current version's bugs, I'd gladly accept and encourage that.
My work here is dung.
One problem is that few developers are motivated to work on bugs
Yeah, this is one for the "no shit sherlock" column. What did you expect to happen when you eliminated the stable/unstable cycle? At a minimum the individual parts of the kernel would achieve stability at different times so that the kernel as a whole was never stable.
This frustrates me immensely at work. I hung on to 2.4 as long as I could. Hardware compatibility pushed me to 2.6 and it just isn't as reliable.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
I think at some point you need to draw the line regarding support for older hardware and peripherals. I mean, excessive backwards compatability has retarded advancement of the industry IMHO.
End of Line.
If a free software developer doesn't want to do something, like fix old bugs, they won't. If this is made to be the only way they can contribute, they probably won't contribute. It's better to get some shiny, unasked for features than nothing at all in my opinion, even if it is not as good as added stability.
When Argumentum ad Hominem falls short, try Argumentum ad Matrem
The number of bugs. "Bug" is a countable noun, like "chair", not an uncountable noun, like "salt".
# cat
Damn, my RAM is full of llamas.
So, there are two relevant aspects to it. Probably more.
The 2.6 Kernel has been plagued by bad bugs. On the other hand, one way or another you need it for a multimedia-enabled desktop on more modern hardware (compared to 2.4). From that point of view, the proposal is fantastic. Otherwise we see the quality of the kernel of our beloved OS going down.
2.6 has never seen a phase of consolidation, really. Therefore, the proposal is almost overdue.
It would be badly short-sighted to think of quick ROI (as the IT companies usually aspire), since the troubles only multiply with further advances.
Yes, please, Andrew, get stability back into 2.6 - Though I have no single word of say in this, I thrust up both hands in favour !
Maybe there are some thumb-screws needed for the contributors: As long as the bug level stands above a certain threshold, no enhancements will be accepted.
There is also a political aspect to it: we have always argued about re-use of legacy hardware. This becomes even more important with Vista on the horizon. The kernel must not lose the 'caring' attitude. It must be trustworthy and trusted by the general public to care for more than greedy hardware manufacturers and their sick quest to replace functional hardware with most recent hardware.
Man, it's crazy but we have this thing where I work. Uh, what do you call those things again?
... I think they're called 'leaders.'
They are very good at convincing people to do things regardless of what they get out of it
If Andrew Morton doesn't have leadership skills, I suggest he step down and let another manager step up.
If I were in his position, I'd get everyone who's even mildly important in a room (or, failing that, an e-mail) and:
"Guys, remember back to the reason you first joined in the contribution to develop a free operating system. Now, think of all the hard work you've put into it and other people have put into it. Now, that's all in jeopardy and here's why..."
Spend some time reasoning with them and pointing out the bugs that are really really hurting the kernel. In the end, wrap up with:
"Look, I know this sucks and you're going to have to tangle with a lot of bugs that aren't even your own. But what have got if we haven't got a stable operating system? We've got another Windows, that's what. You just don't have to pay for our piece of malware. Just see this one development cycle through, I promise we'll make it as quick and painless as possible and after all is said and done, we'll have another meeting like this were anyone can suggest any crazy-ass feature they want to add. Once we pick out what we want, we'll spend the next development cycle letting our imaginations run wild. We'll make a kernel so unstable that the user'll have to re-flash their BIOS when it crashes! Then maybe we'll work on solidifying that. Right now, we just owe it to ourselves and our fans to give them something that's 100% stable and reliable."
If you can't reason with them like that, maybe you just have to accept they can't be persuaded and let them do what they want but prune their work if it detracts from your goal end system.
My work here is dung.
Nice house. Did you build it yourself?
In a way Linux as a whole (the kernel) is now suffering from the same problems as Debian stable once was, at least from my perspective. Do you guys remember the previous Debian stable? It remained stable for such a long time that eventually you simply needed websites like Backports to be able and run some current software since everything included with Debian was way ancient. Naturally you could run Unstable but it wasn't exactly the best approach for servers. I eventually ended up running Testing and keeping a close eye open for bug reports, exploits, etc. while not updating the box every time something new came out.
/usr/X11R6, so why couldn't they add /usr/X11R7 ?
And that is what I see happening here as well. The last really stable kernel is IMO 2.4.32. There are no new features being added, only bugs being fixed. Which is IMO exactly what is needed for serious usage since every beginner programmer knows that when you add new features to your software you will also increase the risk of more bugs popping up. These could be bugs resulting in the addition of new code and the way it cooperates with the existing code, or simply bugs which only manifistate themselves in the new routines. Unfortunatly the kernel developers don't see this or they don't care resulting in a rather stable kernel 2.4.32 which unfortunatly lacks some hardware support and certain features when compared with the rather unstable 2.6.x kernel branch.
Personally I'm worried about the future. When looking at the 2.6.x kernel I don't like what I see. When looking at the current Debian Sid and the rather rough way they implemented the new X environment I also can't help wonder if there aren't more bugs than "usual" popping up.
Anyway, this is all moot now since I have lost fait in Linux all together when it comes to server usage for quite some time. I think that Linux is suffering from its own success and it may well proof fatal in the end, although I really hope it doesn't. I still enjoy running Linux on my workstation and I'm not planning to stop. But when it comes to serious work, like my server, my trust is now put into Sun Solaris 10. While Solaris is also moving into the Open Source environment Sun still uses their common sense and as such split their software into 3 parts: Open Source, Unstable and Stable.
Of course it's more rewarding to create a new feature. First of all, no coder enjoys working on foreign code. It just doesn't "look right", doesn't "feel right", simply because everyone has his own style.
... zip.
And don't forget bragging rights. Hey, I invented some feature. Sure, some guy debugged it, but I get to slap the label to it. I might even name it after me (Hello Mr. Reiser, if you should read this...). The guy who debugs it gets
This has to change first if we want people to put in time to hack through other people's code. Appreciate the work done to get it fixed. After all, appreciation, bragging rights and "making a name" is everything you get from writing free software.
Few people do it out of generosity or because it "feels good". They want to be known. Linus might not have gotten much out of writing that Kernel, but he sure as hell has a killer paying job now. I doubt the people who wrote the original implementation of iptables/ipchains are worse off. But the debuggers? Lot of work, no name.
Pull the debuggers in front of the curtain, and you'll see people debug. If we only appreciate the people who wrote a feature in the first place, even if that feature doesn't work 100%, we won't see people debug.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
The painful truth is that very few developers, in open source or otherwise, like fixing old code or old bugs. This is very true if the bug fix isn't going to be noticed by a great number of people. Face it, most of us like to write new code or improve on something that isn't working the way we want it even if it is working right.
This is what separates professional developers from the rest. We work on it regardless of how much it benefits us. We might gripe a bit but in the end we do what is asked. Sure that backend has flaws and is going to be replaced down the road but it does not excuse us from making it work now.
When you go look at some of the bugs listed in even current applications you start to see the age some have accrued. Some are rightly passed over as 1 in a million occurences but too many are skipped because it just doesn't have any allure. Note, I am not singling out people who work on Open Source, I am pointing out that the article fails to touch an area that exist but most don't want to acknowledge.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
Yes, it's a good idea.
But don't waste time on bugs that only affect legacy hardware.
It would also be a good idea for some effort to be spent on consolidating, corrrecting, and updating the various lists of "Hardware supported by Linux". There are lots of such lists on the web, for example:
- not to mention the distro-specific compatible hardware lists maintained for Redhat, Mandriva,Suse, and others.
We need one correct, maintained list, not dozens of nearly-correct, usually out-of-date lists. And it seems to me that the list should depend only on the kernel version, not on the distro.
My experience is that stability is dropping, even on modern hardware. You can no longer take the latest '2.6' stable kernel and expect it to keep your server running stably.
Now, you can take a Redhat or SuSE packaged kernel and find those are pretty stable.
But there is a problem; if you report a bug in a Redhat/SuSE kernel on the lk.ml you get a
'that's Redhat/SuSE problem - speak to them'.
As the 2.6.x stable tree becomes less stable, less people use them on production servers and instead
use packaged kernels. As less people use them, they get tested less - and less bugs are actually reported for them.
It is also not just a case of old hardware; in the last few kernels I've had leaks that make
a simple firewall die repeatedly after a few months, I've got a machine with a slow RAM kernel leak
that makes a simple DHCP server fall over every few months, and I've had a 2.6.1x kernel that couldn't
run an NFS server for 24 hours without falling over.
It ain't nice - but these are my experiences.
Dave
Here Here! Seti at home had a gazillon(tm) people contributing cycles to the effort (many times in teams) to see who could place highest on the list of contributors.
How about a BFoD - Best Fix Of the Day? Each day, post the name of the submitter and some details about the item debugged and fixed:
This could be further improved by posting a Bug Of the Day (BoD) where there is a daily bug that is to be fixed. The first fixer gets recognized as well as anyone who provides an especially elegant solution. Award bonus points for fixing related bugs in the area so as to promote more complete fixing in that area.
Post these prominently for all to see and I'd be willing to bet that there would be a groundswell of support.
This is just off the top of my head - please post any suggestions for enhancements or (gasp) any problems you see in it!
Mr Moreton is a very, very wise and informed gentleman, and "leadership" skills aren't the only useful ones -- in fact, they can easily become crippling handicaps as every rational response is knifed in favour of a justifiable "leadership response", effective or not.
If you see a need for a leadership character, please engage them in addition rather than in place, else Linux will overall lose even if a relative genius is so employed. There is much comment in this post WRT Linux vs Microsoft development models; if y'all want to be as vulnerable and inconsistent as Microsoft products are well-reknowned to be, then you will need to make many of the same mistakes are they are making, and the-leader-wins-all is their primary fubar.
Got time? Spend some of it coding or testing
Ahhh, but a good developer doesn't use increased CPU speed as an excuse to write slow code. Ideally; the faster the CPU, the faster the code runs. NOT: The faster the CPU, the "less slow" the code runs. In terms of CPU-hungry code, whatever "CPU-hungry" is defined as, depends on the task at hand.
how is babby formed?
As an application developer, it really irks me that I have to release software that I *know* has bugs, choosing instead to complete whatever features were supposed to be in the release. As a consumer of applications, sometimes I wish that instead of adding all the new wizbang stuff, someone would devote an entire release to fixing *all* known bugs and improving performance. Maybe this will finally happen w/ the kernel.
I agree that you cannot support old hardware indefinately. At some point you've got to say that this new version of the kernal just isn't going to run on a 486 or a pentium 1 computer.
BUT
That should be a decision that is made not just the result of accumulated bugs. That way when the decision is made code specific to the old hardware can be scrapped. Also people will know that kernal fu.bar.fu must have this minimum hardware to run, instead of learning by installing it and running into endless instabilities.
-- QED
given that some microkernels are only about 3500-6000 lines of code (as opposed to Linux's million or so)
Oh yes, but the microkernel doesn't implements almost any user-visible functionality - TCP/IP stack, VFS, filesystems, USB, random devices....
You know, the linux core kernel is also quite stable. They're the drivers who hit more bugs. A microkernel itself can be perfect, but the userspace daemons implementing funcionality will also have bugs, and those daemons will take more or less the million of lines that linux takes. IOW: microkernels doesn't fix magically bugs.
Or you could use nooks. Nooks will protect the OS from driver crashes and restart failed drivers transparently.
The big overhaul is a good idea, Andrew. Dotting the i's and crossing the t's should be an obligatory part after architectural changes, like the kernel has seen recently. It's often forgotten, just like management forgets to do it after reorganisations.
:-( ), and encounter few problems. Most of those concern reverse-engineered drivers like that puta broadcom bcm43xx crap :-(
Anyway, users have some resposibility, too. If they use old, scarce hardware and encounter bugs, at the very least they should
submit bugs somewhere. Better yet, fix it, too, and submit a (proto-)patch. Hey, I am as lazy as the next guy, and my code doesn't aim for correctness, just fixing problems, so I don't submit it. But at least I help myself, and people nearby.
People, for the kernel dev's this is a lose-lose situation. If they don't fix new hardware, they get complaints. Lots of complaints. If they don't fix the old stuff, they get lots of complaints, too. Strangely, even kernel dev's are only human, and have only 24h/day. They need us, to spot the bugs, and to fix them.
Andrew, for moral support, I maintain 70+ boxes, from early 2.4 to the latest and greastest, on lots of different generations of x86 hardware (PII to latest opteron; since OSX no more PPC
may the source be with you all
I've ran the Linux kernel since the 2.0 series. In 2.0-2.4, I'd run into one minor bug (a corner-case in which something in /proc was misreported). In 2.6, I run into bugs regularly. If you configure up a kernel, most of the time it doesn't even compile. Calls for 2.7 have been rejected, so what I'd suggest is the following:
We tag 2.6.16 as "stabilizing." We keep developing 2.6, but backport bug fixes to 2.6.16.x. Once 2.6.16 is stable, we mark (at that point) 2.6.45 as "stabilizing," and continue to maintain 2.6.16.x. One 2.6.45.x stablizies, we mark that as stable, discontinue support for 2.6.16.x, and make 2.6.98 as stabilizing. That way, at any point in time, there's at least one stable kernel.
C doesn't offer enough abstraction to deal with new levels of complexity. It is now far from the best language available for systems programming: bitc and prescheme are especially worth looking at if you haven't heard of them.
If you put resources into making the newest kernel compatible with old peripherals that resource could not be used for bugfixes and new features.
The new kernel probably will not bring anything new to the old hardware, either. So why don't just use the stable 2.4 kernel with security patches?
I see a lot of hand waving about how buggy 2.6 is but I do not see any references to bug databases or particular reproduceable bugs. How about some data?
So far 2.6 has been just as solid for me as previous kernel versions but I try really hard to avoid using bizarro hardware and drivers that probably do not get much testing, and rightly so.
I think we need to distinguish between bugs in the core kernel (code that everyone runs) and bugs in drivers. The vast majority of the Linux kernel code is drivers.
I think the fundamental difference between windows and linux, is the community. Linux is subject to this kind of scrutiny where I don't think window's gets nearly as much. So you will hear the occasional outcry about there being too many holes or this gaping security flaw or this critical error in linux kernels and distrobutions but it's something the community will take care of and fix themselves where we don't have to wait on Apple or Microsoft or IBM or anyone to come up with a fix. Another thing that gives linux the advantage in this kind of debate is the degree to which it can be customized for the user. The user does not have to install each and every single component where in windows it can be fairly difficult although not impossible to leave things out.
There will always be people that care about open source around to take care of their favorite distrobution and if you think your favorite distrobution needs a little bit of a change and you can't write the needed code theres always a place to suggest it and then discuss why or why not its a good/bad idea. Personally I try to donate whatever I can to my distro of choice because I think its that good and it fits. In my attempt to quit smoking I've started donating what I would have spent on patches or smokes to slackware. If you like your distrobution donate and I doubt it will become an issue.
If this kind of thing is a concern of yours there is always something you can do to pitch in even if a simple hello world! is beyond your ability.
I've said this, here and elsewhere, over and over and over. Quality is something that has to be in software FROM THE START. It's not something you can retrofit.
As soon as the kernel dev team decided that Linus' kernel didn't need to be stable anymore, as soon as they started waving their hands in the air and expecting 'the distros' to magically fix their problems, OF COURSE quality took a dive. One of the kernel devs said that it was okay for only one out of three 'stable' kernels to actually be stable! Stability takes a long time... they now refuse to support a given kernel for more than a couple of months. The 2.4 kernel still has a few problems, and it's been around for, what, six years now? Supporting a given kernel release for only a couple of months is impossibly stupid from a stability perspective.
They're doing it this way because they're tired of doing the painful, annoying, tedious task of making sure the kernel always works. And the 2.6 kernel has, as a result, been a steaming pile of crap. Features don't matter if the fucking kernel doesn't stay up. No kernel since about 2.6.8 has worked in APIC mode on my ASUS KT333 board. 2.6.15 crashes my Intel 865 chipset servers randomly; they rarely stay up more than an hour or so. 2.6.14 broke traceroute. And with the constant stream of patches to their security fuckups, my system uptimes rarely exceed two weeks. Remember being proud of your kernel uptimes?
The social contract with Linux for many years was essentially: "The official kernel tree is as stable as we know how to make it. You can trust this code." And that is what got Linux as far as it has gotten... the fact that you could TRUST IT. It NEVER fell over. The 2.2 kernel was one of the finest pieces of software I've ever run. 2.4 took a huge dive in terms of stability, and was a total mess until Linus branched off to 2.5 and let the poor harried 2.4 maintainer, Marcelo Tosatti, take it over. He finally whipped it into shape. He has done an outstanding job.
What Linus et al need to do is GO PLAY IN THEIR SANDBOX IN 2.7. Let 2.6 fucking stabilize. They're shoving new features down our throats so fast that it's a part-time job just keeping up with the new stuff... and obviously NOBODY understands the security implications of moving this fast, or we wouldn't have so many goddamn security patches. We're gonna be having those security patches for YEARS because of this bullshit. The number of possible interactions in a system goes up exponentially with the number of features... so adding features should slow down over time, not speed up.
Go BACK TO THE OLD SYSTEM. People crying about 'too slow release schedules' is a HELL of a lot better than people crying about Linux being unstable. Linux *owned* the word stability for many years, and it's in very real danger of losing it, right at its height of popularity. The old system worked. It got Linux where it is today.
A simple 'bugfix release' won't do shit... it's the process that's broken. It'll fix some of today's bugs, but what about next week?
If I have old hardware that doesn't run 2.6, I can and do drop back to an older kernel. Hell, 2.0.40 came out in 2004. And note the size! That kernel boots as fast on my 133MHz machine as 2.6 does on my 1GHz frankenstein. New features on a new kernel mean nothing on hardware that can't use it. If you want to keep running a new kernel on old hardware, obviously you're going to suffer plenty of bloat, as evidenced in the Windows world. And speaking of that, if MS had kept their old version on the market. They could have slimmed down the new versions considerably. Of course, most of us know that older versions are MS biggest competition, so that's why the lockdown, all made possible by our gracious IP overlords. So be it. I don't need them anymore. Even Apple put up their old old versions for free. But it doesn't run on new hardware. And their new software doesn't run on old hardware. And furthermore, wouldn't it be easier to troubleshoot and fix bugs in the older, smaller kernels? My general rule is to use a kernel that is approximately 6 months to a year newer than the hardware it's running on. We shouldn't try to make a single kernel to run on all hardware. We have lots of them, one for each specific time period. This also applies to the distros. The older ones are still available for your old hardware.
FTA: Nowadays, many kernel developers are employed by IT companies, such as hardware manufacturers, which can cause problems as they can mainly be motivated by self-interest.
Am I supposed to be surprised by this? Even the most altruistic of us are generally motivated by self interest. We all want some kind of return for our efforts....even if it's a simple "Thank you".
What?
This might hurt my karma, but here goes:
Yes, imagine. But imagine seriously. Would it really be such a bad thing? It would be different, for better and for worse. Not just for worse.
Face it, there have been lots of kernel problems surfacing lately, and not only bugs, but design features. There's more and more kernel options that are incompatible with each other, and harder and harder to make the correct choices for a given system. A bad side effect of the feeping creaturitis that's been allowed to take place after the odd/even cycles were abandoned, and people were allowed to add things almost unhindered. Perhaps someone in the lead who isn't afraid to kill off someone's babies and say "heck, no" wouldn't be such a bad thing.
Regards,
--
*Art
>A computer that is firewalled doesn't have to worry so much about remote exploits and worms.
Famous last words...
Do us all a favor and go get the patches...
A machine that is powered off, disconnected from the network, and buried at the bottom of a disused diamond mine does not need to worry so much about worms and remote exploits.
The rest are potential zombies.
I can explain the whole deal if you want it spelled out (defense in depth), but it should be enough that MS knows their OS better then either of us knows it, and you should believe them when they say something is critical. (They also have a reason to downplay risks, so you should REALLY believe them since you know it has been bounced around Redmond for some time before it leaves the building as a critical vulnerability.)
Failing to apply the issued critical security patches on purpose (without a damn good reason) is professional legligence plain and simple.