Operational Testing of Linux Kernel 2.5.x

G3ckoG33k writes "The Open Source Development's Lab has begun operational testing of the 2.5.x Kernel: "The staff at OSDL has been involved with development and testing of 2.5 since the beginning and we've noticed that it seems to be very stable for a development tree. So good, in fact, that we think it is ready to be tested in a production environment. We have planned and begun execution of a project to test the 2.5 kernel in our data center using our production environment. The project includes lots of testing and lots of escape hatches so we don't run recklessly off the edge. We began with some of the simpler, less critical servers and, as we build confidence, are moving to the more complex servers. Today we have several servers running 2.5 and within a month we'll have most of the data center migrated to 2.5." Can anyone say Dare Devils?"

Comments

2.5 impressions

[yokem_55]

I've been running the 2.5 kernel on my laptop for a couple of weeks now to get the new cpufreq support. It seems to work really pretty well. Getting pcmcia-cs to build took some work, but I finally got it up and running and the performance of this new kernel is really nice, especially for the desktop.

Re:2.5 impressions

[SuperBanana]

I've been running the 2.5 kernel on my laptop for a couple of weeks now to get the new cpufreq support.

This reminds me- one problem I've always had is that new stuff that gets thrown into the kernel isn't clearly explained- in the most basic ways. Ie, what the heck is it? I remember lots of versions of 2.4 had features and options with no help to explain what they did. Google searches don't always turn up anything handy- often they turn up lots of hits on patches or posts talking about the feature, but not describing what it actually is.

Anyway, For those wondering what the heck cpufreq is...From a kerneltrap interview:

JA: You also mentioned working on the x86 side of Russell King's cpufreq code. We spoke with Russell King in an earlier interview, but we didn't talk about cpufreq. What is it?

Dave Jones: Quite a few CPUs these days allow changing of the voltage/multiplier/bus speed through software. Russell and Erik Mouw did a bunch of work on the ARM CPUs that support this feature, and started writing a generic framework for this type of technology so that he wouldn't have to duplicate code that for eg, recalculates loops_per_sec in every speed scaling.

etc.

Good for them (and Us)

[Anonymous Coward]

I've been trying out 2.5 for quite a while now with varying degrees of sucess.
It would be great to hear from more people like OSDL that it's working well.
Unfortunately, unless RH9 comes with module-init-tools, it will still be a pain to try out the 2.5 kernel.

This is great.. but don't forget:

[mindstrm]

The reason it's not for production use isn't because it is necessarily crash prone... it's because it can break drastically between minor versions as features are added/changed.

Re:This is great.. but don't forget:

[ahkbarr]

Not really. I thought 2.5 had frozen, right?

I like 2.5... I just wish I could clear up the FB

[miketang16]

I've tried compiling several different 2.5 versions, and yes they're very stable, when run in normal VGA mode... but I prefer the higher res FBdev mode. Unfortunately whenever I boot into a higher res, my screen either scrambles, or totally blacks out...

I'm using 2.5.66 right now...

[Dr.Dubious DDQ]

I've been playing with the 2.5 series off and on mainly for the USB Storage support (devices that don't seem to work in 2.4 seem to work fine in 2.5 - at least the two or three that I've tried.) For the longest time, there was always ONE of the features that I really wanted that wouldn't compile or work, either the USB, or Video 4 Linux, or something else...

I came back and tried it again at 2.5.63. That was the first version what compiled and ran everything I used perfectly. .64 and .65 seem to have had a timing glitch that messed up my scheduled recordings (by mencoder via V4L), but that seems to be fixed again in 2.5.66, which has been working beautifully for me so far.

I honestly expect to see "2.6.0preXX" versions start appearing in the relatively near future...

Daredevils? How about idiots.

[Anonymous Coward]

There's a reason people don't use 2.5. It's the DEVELOPMENT kernel. You SHOULD NOT be using it for production use. Often things will break. Sometimes it will cause hard disk corruption. It wouldn't be the first time.

Please, fellow slashdotters, don't be tempted to use 2.5 for your important systems. It's good that it's tested more, but if you do use it, please don't bitch and whine about how it destroyed all your data.

2.5

[Spy Hunter]

I have also been using 2.5 on my desktop. I got it at first to test out the supposed desktop performance improvements, but I haven't really noticed any improvement. What I have noticed is the increase in quality of the sound drivers. The new drivers for my card can suddenly mix 2 channels together in hardware, allowing me to run XMMS or mplayer and still hear my Gaim sounds in the background or visit a Flash site, without running a retarded sound server, or having programs choke and die because they can't open /dev/dsp. If only ALSA would implement a kernel-mode audio mixer so everyone could have as many channels mixed together as they wanted. We could get rid of this rediculous proliferation of bloated, incompatible "media servers" that use complicated IPC schemes to achieve basically the same result less efficiently. Here's hoping.

Re:2.5

[glenebob]

Yeah! Let's put the X server in kernel too while we're at it. NOT!

It seems to me that media services should either become embedded in the X protocol, or standardized as a sister protocol. The problem lies in the failure to properly standardize the protocols. The actual mixing can still be done in hardware when possible, but the applications should have a very standard way of writing to the sound stream and that it shouldn't be done in kernel. I'd rather write to $DISPLAY:$SOUND_PORT than to /dev/dsp or whatever. Try going the /dev/dsp route while using a remote X server (which is how I always access my Linux box). It might sound a little... distant.

Re:2.5

[Spy Hunter]

No, the driver for my card is just getting that feature. In fact, the Windows driver doesn't even have it (but Windows has a kernel mode mixer so programs can still play sounds at the same time, the CPU does the mixing). Other Linux drivers have had it for some time, I understand. And I'm talking about 2 PCM streams here, as in two programs playing sounds at the same time. It has always been able to mix the microphone, line in, MIDI, CD, and other channels of course.

Re:2.5

[sydb]

I don't understand. I run 2.4.20, standard Debian package. I am listening to the Ozric Tentacles with XMMS as I write. As I test I installed mpg321 and played an Eddie Izzard track from the command line at the same time. No problem., mixed seamlessly.

Now, let's try more channels...

Now, I've mixed Wagner, Ride of the Valkyries into that too. I'm kind of dizzy, but it all works.

Maybe this is a feature of the EMU10k1 driver, or something, but it just works for me.

Woooah my head is spinning! Stop!

Re:2.5

[PhoenixK7]

Most desktop apps at least support going through arts, esd or some other software mixer so while its kindof a crappy solution its not that much of an issue.

I'm not sure about most of the cards available these days, but I do know that at least the SBLive (and the linux drivers, both alsa and the old oss ones) allows for hardware mixing of multiple channels (not sure how many but its way more than just 2).

The ALSA drivers ARE of a much higher quality though. Has anyone else noticed that if you put the PCM volume all the way up on pretty much any sound card with the old OSS drivers you start getting nasty distortion (well not really nasty, but for anyone thats picky about audio its pretty nasty). Main volume is fine all the way up, and its definately not a speaker issue. Going on about the same volume level, PCM down Main all the way up vs PCM all the way up and Main down there is most certainly a difference in the quality of sound.

I've seen this with SBPCI 128, SBLive, and on my 800MHz iBook as well, so its not even limited to one platform.

Re:2.5

[Spy Hunter]

An audio mixer is hardly rocket science. It's actually a very simple thing. It just demands very low latency and very high efficiency, which the kernel can easily provide, while user-space servers have a little more trouble. There is a humongous difference between putting the X server in the kernel and putting an audio mixer in the kernel.

Putting media services in the X protocol would probably be a good idea, but it is orthogonal to a kernel mode audio mixer. Some programs will always want to access the hardware more directly, and the server itself still needs to access the hardware. The kernel API will still be used, it needs to work well.

Re:2.5

[Spy Hunter]

Apparently you've discovered that the EMU10k1 driver from the 2.4 series also has this ability. Some cards have the ability to mix up to 8 streams, I've heard. If you keep opening up more mpg123s eventually you will hit a wall. The mixing was new to me since my sound card driver (ESS something-or-other) didn't have this capability in 2.4. The new ALSA driver for my card in 2.5 does. Also it doesn't have the annoying bug where every time XMMS switched songs, it swapped the left and right channels :-) The old ESS driver really wasn't very good.

Re:2.5

[Doug Neal]

The emu10k1 chip that the SB Live is built around can mix 32 channels together like that in hardware. the Linux driver provides support for that ability by letting you open /dev/dsp for output up to 32 times simeltaneously.

Simple :)

2.5 is pretty good but...

[Fnkmaster]

Yeah, it seems to run very well, the preemption patch is fabulous, new scheduler and whatever other magic made it into 2.5. Subjectively, X feels more responsive on my older hardware (okay, it's a PIII 650 but still a generation behind the times), which is running Mandrake 9.1 with KDE 3.1 now, on the 2.5.66 kernel.

This is however still a DEVELOPMENT kernel. I put that in big letters because it's very, very true. Lots of kernel modules won't compile still. Documentation for what has changed is somewhat spotty, and it took me some time to get everything working decently. And getting a system that can boot into 2.4 or 2.5 seems quite difficult with the new modutils package (or at least I haven't gotten it working yet - have to reinstall modutils RPM if I want to boot into 2.4).

Also there's a major bug with ext3 right now in 2.5.66 - if your computer doesn't shut down cleanly, the journal recovery in 2.5 seems completely broken - I have to reboot into 2.4, let the 2.4 kernel do the journal recover, do a clean shutdown, and THEN boot back into 2.5. Pain in the ass, especially since I've had two hard crashes since I upgraded to 2.5. Also 2.5.66 doesn't compile out of the box with default config. Had to patch one file with a patch from LKML.

So in short, 2.5 may be more stable than usual devel branches, but don't delude yourself about what you are getting into. If you want the latest and greatest in performance for your desktop machine, give it a try. But I wouldn't run even a low uptime-requirement server with it yet.

Re:I like 2.5... I just wish I could clear up the

[DaveAtFraud]

Try pulling down the XFree86 source and recompiling it against the 2.5 kernel headers. Seems to work fine on my dev box at home (Tyan Tiger MPX mobo, 2 x 1900+ Athlons)

[dave@bend ~]# X -version

XFree86 Version 4.3.0
Release Date: 27 February 2003
X Protocol Version 11, Revision 0, Release 6.6
Build Operating System: Linux 2.5.61 i686 [ELF]
Build Date: 04 March 2003

At some point I'll rebuild X aginst the current (2.5.66 for the moment) dev kernel. Its hard enough keeping up with the various kernels so I only recompile X when I have to.

Not really that daring

[ahkbarr]

These are folks who don't include every driver and feature available. They probably won't be running preempt, which has been at times problematic. You can get a very stable 2.5 series kernel by being prudent.

All in all, my experience at running 2.5 has been positive, and my only problems have really been with features not likely to be used by folks running special purpose servers.

2.5.x

[Dante]

I've been running 2.5.x on both my station at work and at home. For the most part it's been pretty stable.

I've run 1.3, 2.1, 2.3 and now 2.5 kernels as they came out and 2.5.5x and on have been a pleasure. I had a 2.1.x kernel eat my file system, I've had nothing like that so far.

Now the caveat: don't run a 2.5.x kernel unless your willing to lose everything, backup regularly! and most important because I don't think anything bad will happen, be prepared to write bug reports correctly! READ THIS AFTER DOWNLOAD! linux-2.5.x/Documentation/BUG-HUNTING

Not for long...

[Anonymous Coward]

"we've noticed that it seems to be very stable for a development tree."

And, in other news, Kernel developers worldwide learned that the development tree was too stable and announced sweeping changes to the VM, IDE, and Scheduler modules.

Said one developer, "it's not bleeding edge unless someone is bleeding. It pains me to think that we've actually got this thing stabilized with an odd-number dev version. We normally don't go for that until we go to the even-number release versions, usually at a x.y.5 or x.y.6 release."

Tread carefully on any kernel release...

[Ho Kooshy Fly]

Having gone through high cpu/disk load crashes over multiple kernels, I would suggest a good test plan before embarking on any new kernel.

Our most recent experience with 'stable' kernels (specifically drivers in our case) was the default kernel in RH 8. It had some very subtle issues with Intel's GigPHY/MAC chipset that caused crashes only under specific high load every three to four days. Crashes were not repeatable in specific time frames but would eventually happen. I suggest finding a characteristic set of applications/loading of disk/mem/CPU applications and then test out your favorite kernel under all those circumstances. Many programs that run huge FFTs or other number crunching applications are many times too specific to cause failures. We in this example used a program to calculate huge FFTs while doing looping network file transfers to test without issues... nothing beats the real thing!

Also don't think that even 2.4.x series kernels are above this... as I stated earlier even a heavily patched 2.4.18 kernel could be your downfall... so maybe a 2.5.x kernel is okay but beat the crap out of it before putting both feet in.

-Ho

Question

[RedBear]

There are those of us who like to mess with our Linux systems but aren't exactly experts and probably never will be. Some of us would really like to dabble a bit with the new 2.5 kernel on our personal systems, but we'd rather not hose our system in the process. Is there anyplace out there where someone periodically puts together a "semi-stable" version of the development kernel, that us dabblers can download and be reasonably sure that it will be free of such things as major filesystem bugs?

Everyone says, don't run the development kernel if you don't know what you're doing, and of course any particular 2.5 kernel grabbed off of kernel.org can be majorly broken, right? So it would be really cool if one of the real kernel developers could put together something inbetween the 2.4 "stable" kernel and the 2.5 "careful!" kernel. There are just so many cool new features in 2.5, like that huge improvement in interactivity that could really make the desktop more usable, but those of us who aren't experts are really leery to just grab the source and start compiling, because who knows what might be broken in any particular development sub-version.

Does anyone make a habit of doing this "semi-stable" thing with the development kernels? Failing that, are cool things like that interactivity improvement being backported to the 2.4 kernel already?

Re:Question

[oxfletch]

I actually *do* try to do that in the -mjb tree already ... I collect bug fixes, performance improvements, and diagnostics tools (so if it does break, you can find what went wrong).

Staying one kernel release back will help you too ... ie we're on 2.5.66 now, so run 2.5.65-mjb2 (the latest 65 version). And don't turn on preempt (it's broken in my tree by something I did).

The interactivity tweaks are against the O(1) scheduler, so won't do you much good in 2.4 .. unless you run 2.4-aa or something.

If you're going to upgrade from 2.4 to 2.5, make sure you compile import support in (not off, or as a module), and turn VT console support on explicitly. Those are the usual tarpits for new 2.5 people.

Re:NVidia Drivers

[pokryfka]

one can compile and load nvidia binary module, though one needs to apply some patches

i am using 2.5.66/2.4.20 alternatively while using nvidia binaries

2.5 works prety well though small changes fool some user space programs

for example since the name extension of modules are different i had to hack my debian debconf script to work properly
also gkrellm memory monitor doesn't work (free works though)

Answer

[BlowChunx]

vmware.

If it hoses your virtual machine, you are out nothing. If you aren't up for the kernel screwing up your*real* machine and having to reinstall everything, leave it alone.

2.4.xx is perfectly fine. You really aren't missing anything. You'll get it soon enough, without the pain. Besides, anticipation makes you appreciate it more.

Re:Answer

[Anonymous Coward]

at > US$299.00 vmware is not for everyone...

Re:NVidia Drivers

[jamesh]

I'm running the latest NVidia drivers with 2.5.65 and they work fine (after patching of course). The patches can be found on the internet if you look around.

Re:Daredevils? How about idiots.

[iabervon]

You should only use a development kernel in a production environment if you've already tested it extensively and found it to have no problems with your particular load on your particular hardware with the options you're using. Of course, if you're OSDL, you can actually do this sort of testing, but practically everyone else doesn't have the spare hardware and test suites necessary.

Re:Daredevils? How about idiots.

[Anonymous Coward]

Don't sweat it. Many ppl here use MS windows and are plenty use to lots of downtime, crashes, and loss of data.

2.5 more reliable than 2.4 for me!

[jamesh]

I've been running 2.5.xx on my home server since xx was >30ish. not really by choice either. 2.4.xx (for all xx) is grossly unstable for my combination of hardware. I got ide irq timeouts which brings the machine crashing down often before it had finished booting. It would run in uniprocessor mode but what's the point of that!!!

I have one problem with hostap (wireless access point drivers) and my sound card sharing an interrupt which causes a crash occasionally, but if i don't load the sounds drivers it never crashes.

My hardware is:
ABIT BP6 mb using onboard ata66 ide
2 x Celeron 400 (SMP kernel)
TV card
sound card
NVidia gfx card
wireless card
network card

Anticipatory Scheduler

[Pegasus]

I'm running 2.5.65-mm4 on my home box because i wanted to find out whats all the excitement and nice numbers about the new scheduler. After i got all the modules right, i did some tests ... and was a bit dissapointed. You see, it's not all that faster ... it just feels different. Yes, programs do load somewhat faster, but at the same time doing a ls -l in my home dir was kinda slower that with excellent WOLK patchset for 2.4.18. On the other side, i was completely able to browse my large inbox (~20k mails in maildir) while checking md5 of the latest knoppix iso on the same disk.

I have a lot of expectations of the Alan/Andre team with their ide work ... i just can't wait to test the 'fixed up' promise driver and ide tcq code! Right now ide tcq on promise is somewhat borken. If ide tcq shows some numbers, that would be the last argument down for scsi vs. ide in our servers...

Re:Anticipatory Scheduler

[Nicopa]

Your expectations were wrong. The new Linux won't make things faster, it will just (as you put it) make things feel different for interactive processes. The idea is to be able to work without one second pauses when a background process is performing some heavy task.

Re:Bugs?

[sloanster]

Nah - the whole purpose of ntfs on linux is to share data with windows, especially on a dual boot setup - no production linux server is going to be using ntfs for anything serious.

Not really daring

[ChaoticCoyote]

I've been running the 2.5 kernels since about 2.5.30 -- on my primary workstation, no less. In other words, my livelihood is depending on a development kernel.

It works. For me, I've had almost no trouble, save for some difficulties with the radeonfb driver not liking my DFP when it's attached to the DVI. Overall, though, performance is excellent -- though I do keep studious backups in case soemthing "goes wrong."

2.5 is really a solid pice of work. Yes, it had bugs; follow the kernel mailing list, watch what people say, read the patch lists, and skip releases that seem a bit flakey.

What I did

[DaveAtFraud]

Regardless of whether you use lilo or grub, you can have the option of booting multiple kernels as long as you have room for them in /boot. When you install a new, development kernel, edit the appropriate boot loader configuration file to make sure you can still boot to a stable kernel (e.g., 2.4.X). I have only had a couple of instances where a new development kernel either wouldn't boot or was unstable once it did. I documented the bug, in a couple instances helped test the patch and could always drop back to my stable kernel while things didn't work. Also, once you get a development kernel that seems stable with your rig, that joins the stable production kernel in your boot configuration. If nothing else, putting "milage" on even a less than current development kernel helps since there *could be* a lurking time dependent error (haven't hit one but could happen).

So you end up with usually three and sometimes a few more kernels to choose from when you boot:

1) stable production (2.4.X)
2) seems to be stable development
3) current development

When a "current development" kernel seems to be stable, it becomes your new "seems to be stable" dev kernel and you can drop the old "seems to be current" version. Just be sure to weed out old kernels from /boot so you don't fill the partition.

Unless your rig is a completely stock retail box, chances are your specific combination of peripherals and software are unique. So there is no guarantee that your specific configuration will be stable with a development kernel. The beauty of it is, that's a question only you can answer.

Re:Modules/drivers

[Zippy the Pinhead]

The kernel modules sytem was revamped, and is much nicer. But loading modules is now so easy to do, it would've been more hassle to emulate the old interface than to write new utilities.

The new module-init-tools are under:
ftp://ftp.kernel.org/pub/linux/kernel/peop le/rusty /modules/

If you're running an RPM-based distro, you can get the latest modutils.src.rpm from there and use rpm --rebuild to recompile it. Bonus: it still has the old modutils, so you can dual-boot.

If you're running Debian Sid, you can apt-get install module-init-tools.

Mixed results

[be-fan]

Interactive performance - Pretty sharp. I/O background load really doesn't put much of a burden on foreground stuff, but then, 2.4 + preempt patches didn't either. Resizing is weird. Resize slowly, and the effect is like kernel 2.4 (canvas lags behind window frame). Resize fast, and the effect is like OS X, the window frame lags while the canvas catches up. Both kinda suck. CPU background load (MP3 compression) causes the machine to feel like an XP machine -- big 10-15 pauses.
CD drivers - They suck. Certain CDs (Evanescence's Fallen) will cause the CD drive to go into spasms. This doesn't happen under 2.4.
I/O scheduler - Gimpy. Under heavy CPU load (the aformentioned MP3 compression) starting an app that isn't in cache will take tens of seconds.
Compile performance - awesome. I use Gentoo, and I've noticed big improvements.
Power management - Mediocre. APM is alright. ACPI sucks. Causes weird beeping noises when I try to load the "processor" module. It's probably a fault of my Inspiron 8200's fsck'ed DSDT, so I won't bitch, but WinXP has no problem with it.
Stability - Surprisingly good, for development code. A far cry from 2.4, crashes maybe once a week, but much better than the 2.5.20-something releases, which once hosed my entire partition when I burned a bad CD...

Re:I like 2.5... I just wish I could clear up the

[cymen]

I used to keep the console simple too except I've always had problems with this Dell Inspiron 4000 and display corruption when switching back and forth from console to X Windows. Recently I installed Slackware 9.0 and it uses the vesa fb by default (even for X!). I've been using it for a while and no display corruption at all. It works wonderfully and as a side benefit I get a nice big 1024x768 console fulling my LCD instead of some little odd looking 640x480 centered portion.

I'm a convert to the fb world. Of course I had to move X off of that vesa fb to the ati 128 driver but it was a pretty neat installation setup.

Don't Use Odd Numbers

[abcxyz]

You've got to stay away from odd numbers, remember all the problems with everybody's 3.0 4.0 7.0 8.0 releases?

Oh no, those were the even numbers.......