Slashdot Mirror
2.4, The Kernel of Pain
Joshua Drake has written an article for LinuxWorld.com called
The Kernel of Pain.
He seems to think 2.4 is fine for desktop systems but is only now, after a year of release, approaching stability for high-end use. Slashdot has had its own issues with 2.4, so I know where he's coming from. What have your experiences been? Is it still too soon for 2.4?
2.4.x has been OK, albeit not totally stable. I've got 2.4.17 running and I like it quite a bit. As for me, it has probably been benifical since it got me reading a bit of the LKML, and learning more of how to do stuff with my kernel.
--Josh
There are exactly 42,935,718 letter sized sheets in a square mile.
I'll tell ya, I tried the preemptive patches, and all the -ac stuff naturally, and well, the desktop just isn't snappy ... I mean, Windows (follow me here) just feels better. I don't need a force feedback mouse or anything, it just doesn't not show me that it is rendering a window... and that's something that Gnome was doing even on 450 mhz machine.
Also, even with the preemptive patches, I could hold down a key in, say, star office or abi word, and it would stutter! Hold down the arrow key, and it stutters.
These are basic inferace issues that could use some due attention before Linux is ever ready for the desktop.
We've got a 2.4.7 kernel with RTAI real-time extentions for a house automation system running for several months now without ANY problem. Besides the house automation stuff, this box also acts as a mail,web,ftp,file,whatever server. 2.4 unstanble? I don't think so!
As a sysadmin, I have to state that the 2.4 kernels have ruined whatever reputation that existed before about the 2.2n series kernels being stable. Atleast in the 2.0 and the 2.2 series, you had islands of stability where really careful distributions could pick a kernel version as their default kernel. One of the main problems with Debian not finalizing a 2.4 kernel has been due to the fact that there hasnt been an island of stability so far in the 2.4 series.
And I've been waiting a long time now. The early 2.4 series didnt really work out on my SMP servers. The 2.4.6 onwards kernels broke Tulip support for me. Then came the VM switch. Then just when I decide, ok, 2.4.16 seems stable enough, we have the OOM problem. And I also keep hearing statements being made about the new VM being more friendly to desktop systems than servers.....
Now if only 2.2 offered iptables.....
There is no such thing as luck. Luck is nothing but an absence of bad luck.
We're running the Red Hat 2.4.9-13 kernel on several SMP database servers and they have been perfect (not rebooted since 'rpm -U' of the new kernel) for several weeks. Before that, we were running 2.4.7-something from Red Hat and they were the same -- ran straight from the day we installed the kernel to the day we updated without needing to be restarted.
On my desktop machine, I've taken more risks (installed pretty much every official 2.4.x-linus release as they have come out) and some have been good, while others have been total dogs.
I'm running 2.4.17 right now. It seems okay; I've only had a freeze-up once over the last couple of weeks, though it was a total hard freeze (i.e. no ping, no magic SysRq, no nothing), which I haven't had in Linux for several years.
The obvious issue is VM; if you keep lots of memory (768M, or preferably 1.0G+) in your system, things to much more smoothly, though MP3 playback still skips a little.
Right now, I'd prefer some work on the RAID and IDE performance issues. One or two of the 2.4 series have had disk performance 100%+ better than the current 2.4 kernels. Why? I'd like to get the disk I/O back to reasonable levels.
STOP . AMERICA . NOW
There was a bad period where the Soundblaster Live driver (particularly mixer settings) was broken. That lasted through at least three kernel releases. There was a worse period where the VM had fits, and where performance degraded way too rapidly if the system had to swap. That lasted at least six kernel releases. There were at least one or two releases where I discovered that Alan Cox's (usually more bleeding edge) tree was being better behaved.
Of course, whenever I'm playing around with this stuff I don't delete my "last known good" kernel, so if after a couple hours or a couple days I noticed a problem, I just booted back to what worked. The default (albeit heavily patched) Red Hat kernels were good, so "last known good" always existed for me.
To summarize: this hasn't been a source of inconvenience for me, but it has been one of vicarious embarrassment. I've only been using Linux since 2.0.somehighnumber, but this is the worst mess I've seen the "stable" kernel tree go through in that time. Don't get me wrong, I've experienced system-crashing bugs (a tulip driver that freaked at some tulip chipset clones, some really bad OOM behavior a couple years ago) before, and pragmatically I guess that's worse... but those problems were always fixed fast enough that the patches predated my bug reports. Watching even the top kernel developers seem to flounder for months over bugs in a core part of the OS like the virtual memory system just sucked.
I have a kernel 2.4.16 + preempt patch.
It is the most stable config I ever had using this kernel generation.
I explain :
Before, with kernel 2.2.1x I only had "some" preformance issues (mostly disk access related) and what I thought were apm problems (this is a laptop).
Since I have been using kernel 2.4 I happened to have good times but mostly bad surprises.
pcmcia (I use the pcmcia-cs package) is not quite plug'n play (system even hanged once) but symptoms vary from version to version.
So, the big PROS is that, yes, I boot a much quicker way.
The CONS is that since the 2.4.6/7, I bitterly regret upgrading this kernel since the functionality I gained was compensated by the new bugs.
Note that I don't mention the APM because besides the Windowmaker apm applet, I don't even imagine using the suspend/resume on this laptop.
BTW, when I see the difference with and without the preempt kernel, I wonder why this is not implemented in the official tree (radio button : "server or desktop" ?).
Trolling using another account since 2005.
Having said that, there are some serious issues with 2.4 on some 8-way 8GB machines that I manage. They have been running 2.4.13-ac7 since November, because that is the last kernel that is usable for me (-ac11 would probably be ok). Newer kernels have terrible behavior under the intense IO load these machines go through. They get 14-30 days of uptime, and then hang or get resource starved or something and have to be rebooted.
I think part of the issue is that there simply aren't that many people running 8-way boxes, so bugs aren't found as easy, this is of course on top of having 8-way SMP being much more complex than a defacto single user, single processor desktop machine. To make it even worse, the machines are pushed hard. They move around GBs of data every day, and often will run for extended periods with loads over 25.
Of course, it is still mostly ok. While the machines are working they mostly work fine. Of course 20 days of uptime is totally unacceptable. I have an alpha running Tru64 pushing 300 days of uptime, and the last time it was down was due to a drive failure, not an OS problem.
My only remaining issue with Linux on "small" machines is an oscillation problem in IO. Data will fill up all available memory before being written to disk, and then everything from memory will be written out, and then memory fills up again before anything new is written to disk. This is a bit inefficient, and the machine's responsiveness at the memory-full part of the cycle is poor.
What are my options though? I guess I could try FreeBSD, but a bit of lurking on their lists and forums reveals plenty of problems there, too. Do I switch and hope things get better, or wait out 2.4 and hope it comes around soon? Aside from a few nasty bugs in some releases, pretty much each successive 2.4 kernel has been better than the previous one, at least on small systems.
Several years ago I was having a hard lockup problem with Tru64 (Digital Unix, at the time) and that was very scary. It took time to get the problem escalated to the OS engineers, instead of just sending an e-mail to lkm. Even then I could only hope that the issue was being addressed, but I had no way to know if anybody was doing anything about it or not. (Turned out to be an bug in the NFS server that would cause the machine to lockup when serving to AIX.) For all of its problems though, it is extremely reassuring for me to be able to monitor the development process of Linux through the linux-kernel-mailing list, and other specialized lists. If I feel that people aren't aware of some problem I am experiencing, I can raise the issue. I am not in the dark about what is happening, and what fixes are being made. I know what changes have gone into each kernel update, so I know if there is a chance of it fixing my problems.
jakob@unthought ~> uptime
9:21am up 181 days, 13:25, 3 users, load average: 3.57, 3.33, 2.79
jakob@unthought ~> uname -a
Linux unthought.net 2.4.0-test4 #1 SMP Fri Jul 14 01:56:30 CEST 2000 i686 unknown
I suppose that ain't too bad. Other than that, with real 2.4 kernels, on UP and SMP systems, I've been fairly satisfied.
There was a RAID bug (RAID-1) in 2.4.9 or there about, which they forgot in the article. I think, except for the fs/raid corruption problems (which are horrible when they happen), that the 2.4 kernel has been a nice experience.
Think back for a moment: How would you like *not* to have iptables, reiser, proper software RAID, etc. etc. etc.
I think I would miss 2.4 if I went back, although the fs/raid corruption bugs made me "almost" do that.
The problem is the "release" level kernels usually aren't really ready for release. Most hard-core linux people tend to know this, but those that are coming in from elsewhere expect that a "release" product is, well, ready for release... maybe with some hesitation on a .1, but by .2 or .3 the thing should be good.
Maybe holding on to "beta" status for a little longer, or having a "unstable", "testing" and "stable" like debian. So that when someone wants the latest stable kernel, they don't end up with something the kernel guys think is stable... till they release the next "stable" version a day later...
Some people have already mentioned this in passing, but I think its an important point to make.
I've found that the kernel is pretty stable for me. I use my system mostly for code development and as a server for files and web pages.
I find that the kernel itself is pretty stable, although as the article says, it does seem less stable that 2.2 series did. But even so it's not bad for the use I've made of it.
The old style applications are also very good. The command line tools, and the development tools (gcc etc) are all totally solid and are why linux gained it's early reputation for reliability.
*BUT* I find that much _new_ software. Both gnome, kde and others GUI software to be terribly unreliable. Say what you like about microsoft outlook but it rarely just crashes. On the other hand, every "modern" mail program I've used on linux tends to end with a crash eventually. And it's not just mail programs. I find that many of the programs I would use tend to crash quite a lot. Not all the time, but just once is too much.
It's rather sad in my opinion that such a solid base of reliable code is being let down by the stability of some of the more modern software. Frankly it doesn't matter how stable the kernel is if the programs that run on it crash.
This isn't indended to be a complaint, and I realise that before applications can be considered reliable the kernel needs to be, but it does concern me that the overall reliability of linux systems does seem to be going downwards.
Sig is taking a break!
I agree with you re: a small, stable kernel with memory and processor management, but when you think about it, USB hardware is very common. I bought my box 3 years ago and it has USB (even though I have never used it).
This is my opinion:
optional hardware, devices, peripherals -> modules
hardware found on most x86 machines -> built in
FreeBSD has a rock-solid VM system. A GPL'd fork of this code can be created as long as we acknowledge the original authors. Why don't we do that instead of constantly trying to reinvent the wheel?
Yeah, you nailed it down.
Now remember that the whole point of microkernels is to enable flexibility -- not only for development's sake but also to be able to adapt to different loads and usage characteristics.
So the HURD seems to be the answer. That or Linus (or someone else, or a group of kernel hackers) over a reasonable amount of time manages to get better at (1) understanding modifications to Linux and their consequences and (2) based on this understanding only release "stable" kernels once they're done.
Given the complexity of the task, I doubt it's doable. The free flavours of BSD never scaled as far as GNU/Linux; the proprietay Unices have basically choosen to scale up and up, forgetting the small systems situations and accepting bloatedness in order to cater for stability, resilience and other high-iron stuff. Even single-server microkernels like Windows NT and Apple Darwin haven't much to offer, being bloated, slow and not flexible at all.
We still have a free software and open systems advantage, because POSIX OSs can cover the whole gamut of computing systems simply by having different kernels with the same APIs; standard bodies and GNU libc are the real heros here, not Linux or BSD. Proprietary software usually will be even more fragmented, with all those slightly incompatible, underperforming, unstable versions of Microsoft Windows, Mac OS Classic and so on. But still it would be nice to have a free, copylefted common kernel. Unless the Linux situation improves dramatically soon, the only answer in the horizon is the Hurd -- and it still needs to be finished.
Leandro Guimarães Faria Corcete DUTRA
DA, DBA, SysAdmin, Data Modeller
GNU Project, Debian GNU/Lin
Is it not significant the new experimental kernel series 2.5 is being launched just as the `stable' 2.4 series finally becomes stable?
:)
Maybe even number kernels should only be declared stable when they have an experimental branch
Saying that, I've not had many problems with 2.4, bar terribly IDE performance on my hpt370 controller.
Before 2.4.10, I had serveral issues with my own little server box. It would always randomly crash. The error always told me about 'swapper'. Needless to say I had known about the rough edges in the kernel's VM, but maybe with my hardware and software , I was an extreme case. Since Linus incoporated Andrea's VM patches in 2.4.10 and higher, everything has been smooth.
The problem I had with 2.4 was just that, it crashed for me. I managed roughly 6 linux servers for an ISP all running 2.2. Would I even consider sticking 2.4 on those after what I witnessed with my personal machine? No. But now things are much different now. My box is flawless, and I've since upgraded those boxes that had 2.2 to 2.4.
-reid
My hardware is: ASUS P3B - PIII-550 (slot 1), 2 x 256mb Hitachi RAM, Nvidia TNT2, Maxtor HDs.
almost from the start (2.4.3 if I remember correctly). We patched XFS into the kernels and some other stuff we needed and the transition to 2.4 was relatively painless using a fresh install (for XFS).
We ran into some trouble with a number of Athlon systems but that was due to the 'Athlon bug' and was soon fixed. More worrisome was the performance of pre-2.4.9 kernels on the desktop: sometimes they slowed down to a crawl (and i'm talking about lightly loaded ~750MHz machines here).
We got over that with the -ac kernels however, and it's been a breeze ever since. We currently use 2.4.14 with XFS patched in (although we're ditching it in favor of ext3 now that it's been integrated and the RH installer supports it) and we're looking at 2.4.17 now.
Why use 2.4 on servers (as some have asked)? Well, iptables is a good reason, for one. Other security-related things count heavily too. And XFS seemed a good reason to do it at the time too. It can deliver very good performance.
Some stats:
zuse [1] > uname -a
Linux zuse 2.4.14-xfs_MI10 #1 Tue Nov 6 17:34:04 MET 2001 i686 unknown
zuse [2] > uptime
2:25pm up 61 days, 21:21, 1 user, load average: 1.07, 1.02, 0.93
--
2.4.5 has fried my box twice. It corrupted the partition, comes up with a freakout message and such. Ran fsck, but it never was quite right afterwards, so I re-installed both times. Never seemed to happen with later versions (9 through 16) Then again, I probably should be blaming the VIA chipset ;0
Don't call my crazy, that's what they called me back in the home!
Mandrake puts a lot of patches in the kernel too. I think both Mandrake and Redhat tended to base their kernels off of the -ac series rather than the vanilla Linus kernel.
However, you are absolutely right that Mandrake focuses on ease of use and new desktop-like features while Red Hat focuses on stability at the expense of "coolness."
Both can be frustrating if they're used in the "wrong" places.
I know BSD can run many Linux binaries, but what about kernel modules?
At least for nVidia, it is being worked on: FreeBSD NVIDIA Driver Initiative
I'm beginning to feel like a broken record, and maybe Linus should just change the terminology so that people stop making the same assumption over and over, but people: wake up and smell the bogomips!
"Stable", in the context of a kernel release refers to the interfaces. When Linus releases 2.<leven>.0, he is saying that this kernel is one that has reached some arbitrary plateu of development stability, and it's now ready for others to begin actuall release engineering on.
You have to understand that the Linux Kernel is released by Linus in a state that is very reasonable for a development team, but that will never be "production quality". Debian puts a lot of realease engineering work into a Kernel. As does Red Hat. As does SuSe, etc, etc.
If you just grab 2.4.x and install it, you're acting as Linux Q/A, and I applaud your effort, but when it breaks in your environment, you should not be stunned.
Once again, production release != stable release. A stable release is just one the developers are happy with (and I've yet to see a 2.4 kernel that I can say developers should not be happy with).
So, maybe next time, 2.6.0 should be called the "post-development" release so that people don't go off half-cocked installing it on production systems.
If the code is unreadable for most advanced users (people with enough C background to follow execution flow in readable code), then you really can't get the benefit of having a large community debugging the software.
What you have here is a LARGE body of (l)users, and a small cadre of kernel hackers who are separated and out of communication. The three times I've found a problem in FreeBSD-STABLE (to be honest, I think one was the 2.2.6, and two were 3.x branches), I sent my bug report in with instructions on how to repeat the problem, and a patch for the ugly hack I made to make the problem go away. I never beat the committers to the bug. Now, before I do all the work, I watch the WebCVS for commits for a couple of days and email the committer who touched the effective files last with a quick question "hey, I see this problem.. have you?"
I tried to read some Linux (the kernel) code a long while ago. There were some funny comments, error messages, and variable identifiers, but otherwise it gave me a headache. I just felt being a Linux participant was beyond my tolerance. I browsed the FreeBSD source code though, and even in the midst of reorganization, both organising schemes were apparent, well documented, and there is a clean style to all FreeBSD code I've seen that makes for (relatively) easy reading.
I guess that the difference is culture, but in this case it seems to be a serious problem. I have to wonder why the Linux kernel people haven't broken the Linux VM code out for modularity and borrowed the FreeBSD VM as an option? I mean: FreeBSD is free-as-in-beer and also free-as-in-software. I'm not volunteering, but after all these years wouldn't it make sense to hack out some Linux wrappers for the FreeBSD VM system? I think I remember reading a Matthew Dillon interview where he talks about all the good stuff he's done to FreeBSD VM recently...
--- Nothing clever here: move along now...
If my memory serves me correctly, one reason to release 2.4 a year ago is to get more people to use the damn thing.
There is a relatively small number of people that use the odd-numbered, experimental kernels. At the end of 2000, it was becoming clear that having the same people running the development kernels on their hardware wasn't fixing many more bugs - I remember a post from Linus to LKML to that effect.
2.4 was stable enough for mass consumption, and so it was released. However, it is important to remember that this is free software, and frequent incremental updates are the rule. Free software can't work if it is not constantly evaluated by users, bugs reported and fixed, and new versions shipped out.
Software is an evolutionary process; it is important to remember that free software (especially the Linux kernel) fully embraces this notion.
I almost took this guy seriously until this part:
The kernel seemed to show more stability. Then we hit kernel 2.4.15.
Linux version 2.4.15 contained a bug that was arguably worse than the VM bug. Essentially, if you unmounted a file system via reboot -- or any another common method -- you would get filesystem corruption. A fix, called kernel 2.4.16, was released 24 hours later.
Look, anybody who is deploying a kernel on the day it is released on a production server deserves what they get. One day turnaround on a bug fix is phenomenal. Even if these are marked as "stable" kernels, trying to track the new versions in real time is a dumb thing to do.
This guy has written a moan and groan article based on a small set of bugs, some of which he only could have experienced if he is experimenting on his production system. He obviously requires extreme stability and says he needs this over the new 2.4 features (SMP, 2G memory, 2G files), which makes me ask: why was he putting new kernels on his production system before emperical evidence was there about high stability?
Open source will fix bugs faster the proprietary. It doesn't change reality to make bugs impossible. This is true even in "stable" releases, especially if you are talking about highly stressful production environments.
Excellent point about the difference in maintaining stable vs. development series.
While watching the 2.4 drama unfold, I've been wondering if it wouldn't be a better idea to switch kernel development to a Debian-like model -- a simultaneous Stable, Testing and Unstable (experimental) series. It may just be too much to ask bleeding-edge hackers to wait *years* before trying out their latest ideas. At the same time, those who excel at tweaking, optimizing and clarifying shouldn't have the ground shifting under them all the time.
It would probably take some more resources, but it seems that there is a surplus of good minds drawn to kernel development anyway.
Kill, Tux, kill!
So true. I often have to reboot my home machine, because X crashes and locks up the console. The kernel itself is ok, I just can't type anything. On the occasions when I have my laptop available, I can just telnet in and reboot the system cleanly. Unfortunately, most of the time I have to hard-reset it and experience the joy of fsck.
While the facts in this guy's article are correct, he is way overstating stability problems. My company, a high-traffic dot-com survivor, has been running on 2.4 since the pre-releases, and we have never had a single incident on any of our servers.
Although known problems with 2.4 might account for his troubles, it's equally likely he has been having hardware problems. These "bugs" may very well be a straw man, drawing attention away from faulty hardware, which is usually the problem when a machine just suddenly locks up.
I, for one, welcome our new Antichrist overlord.
An article by Michael J. DeMaria over at networkcomputing.com.