Ensuring That 2.6 Will Perform Better Than 2.4

Jeremy Andrews writes "Con Kolivas, a practicing doctor in Australia, has written a benchmarking tool called ConTest which has proven to be tremendously useful to kernel developers, having been designed to compare the performance of different versions of the Linux kernel. In this interview on KernelTrap he explains the project, noting that "a good 2.5 kernel (and that's not all of them) feels faster than 2.4 in most ways and this bodes well for 2.6." Also discussed is his high performance -ck patchsets, adding performance to the 2.4 stable kernel with the O(1) scheduler, kernel preemption, low latency, compressed caching and more..."

Comments

Beta testing 2.5!

[Blkdeath]

I certainly hope 2.6 out-performs 2.4, I mean, who'd want to take a step backwards, right? But we have a problem - the kernel team are implementing lots of top-notch functionality, but don't have enough people testing it. There are still compilation failures, ACPI buglets, and small quirks with several other sub-systems (speaking from my own experience here). Linus pleaded to the LKML not so long ago that he hopes lots more people will start plugging away so they can find and fix all the kinks before we go to 2.6 (or 3.0, like, whatever. :) ).

So download away and start testing!

Re:Beta testing 2.5!

[dpilot]

It hasn't been that long that the IDE subsystem has been sufficiently stable for wider testing on commonplace hardware.

But according to the list, IDE is better now, and the 2.5 kernel is ready for wider testing.

Which brings up the key question... What are the other system requirements for running the 2.5 kernel? Back in the days of transition out of 2.2, there was a fairly decent list: this compiler, that modtools, etc. I know the new native Posix threads will require glibc 2.3, but is that merged, and what other requirements are there?

A brief kernel 2.5 HowTo by someone in the know would be welcome, about now. It would help others of us begin testing.

Re:Beta testing 2.5!

[Blkdeath]

Which brings up the key question... What are the other system requirements for running the 2.5 kernel?

Present system requirememnts are still listed in Documentation/Changes (referenced in the README file, which of course everyone reads before they attempt to compile a kernel, right?) and go as follows;

o Gnu C 2.95.3
o Gnu make 3.78
o binutils 2.9.5.0.25
o util-linux 2.10o
o modutils 2.4.2
o e2fsprogs 1.29
o jfsutils 1.0.14
o reiserfsprogs 3.6.3
o xfsprogs 2.1.0
o pcmcia-cs 3.1.21
o PPP 2.4.0
o isdn4k-utils 3.1pre1
o procps 2.0.9

I couldn't post a direct copy/paste from the file, because apparently it contained too many "junk" characters and wouldn't pass the lameness filter. But if you download 2.5.43, you could just open up the file yourself. :)

I wait anxiously to test the new(er) 2.5 kernel; 2.5.39 compiled and runs for me, but I get a mini-oops on startup (haven't had time to report it yet. :/ ). 2.5.40 wouldn't compile, I missed .41, .42 had a flurry of "WAIT! IT'S STILL BROKEN!" messages on LKML, and I have yet to grab .43 at home (I've got it at work, which does me no good at home, see. :) ).

As soon as I get a chance, I intend to start hammering at 2.5.43 and file a bug report for any anomale I see.

Re:Beta testing 2.5!

[leviramsey]

I'm planning to start testing 2.5 once it's in feature freeze.

Re:Beta testing 2.5!

[Blkdeath]

I'm planning to start testing 2.5 once it's in feature freeze.

Since it would be infinitely more helpful to test features as they're added (ie; "Whoops! That broke it!") - why do you plan to wait? The 'stable' kernels are the ones in feature freeze, and 2.2 and 2.4 are being tested more than amply already. Why not throw caution to the wind and give 'er a go?

Re:Beta testing 2.5!

[leviramsey]

Because I'm only able to install it on a production machine, I can't totally throw caution to the wind.

Re:Beta testing 2.5!

[Blkdeath]

Because I'm only able to install it on a production machine, I can't totally throw caution to the wind.

You don't have a test machine? Like, not even one? How do you test software updates before putting them live?

Re:Beta testing 2.5!

[leviramsey]

Being a poor college student, my production machine isn't that mission critical (it's just a workstation), but I lack a testbed. Hopefully, on obtaining sufficient dough to upgrade my creaky Duron 650, I'll have a decent testing machine.

Re:Beta testing 2.5!

[Anonymous Coward]

Yeah right, Linus calls for testing. Within a few days I downloaded linux-2.5.40 and began running with it on our embedded system. I found 3 show stoppers. I emailed the kernel mailing list. I offered to help if someone wanted me to try stuff. I included enough detail then was ready to answer questions if someone asked for something I had left out.

The result?

No response whatsoever. I guess they don't want to hear that there are actually bugs in their shiny new kernel.

Aside from that after tweaking I got it to work, and it did outperform 2.4.18 in everything we cared about. Specifically the IDE system, it appeared much more efficient at scattered disk reads by multiple threads.

Re:Beta testing 2.5!

[gmhowell]

The question is, why bother? I have yet to see a compelling need to switch to 2.5/6. Hell, without improved USB and journalling filesystems, what was the point in 2.4? I read the kt digest, but I'm really not seeing any new wonderful things. Perhaps there are others in a similar situation?

Re:Beta testing 2.5!

[Blkdeath]

No response whatsoever. I guess they don't want to hear that there are actually bugs in their shiny new kernel.

Perhaps you just gave them enough information that they didn't need anything further from you, and instead went hunting for the cause of your problems. Speaking as a former programmer myself, often times I wouldn't consult directly with the user(s) who submitted bug reports, but would instead fix the problems and ship out a new, more stable version.

Aside from that after tweaking I got it to work, and it did outperform 2.4.18 in everything we cared about.

Excellent! I hope you've posted a follow-up to your initial message to LKML so people will know the solution as well as the problem.

Cooperation for a better OS, etc.. ;)

Re:Beta testing 2.5!

[Blkdeath]

Hell, without improved USB and journalling filesystems, what was the point in 2.4?

The new VM was one of the big kickers (then they replaced it mid-stream with a better one, which was risky at best, but I digress). I've found that on all workstations and servers on which I've installed the 2.4 kernels performance is through the roof. It responds faster under all load conditions than any 2.2 kernel ever could.

I read the kt digest, but I'm really not seeing any new wonderful things. Perhaps there are others in a similar situation?

Pre-emption, new I/O scheduler, new paging, all sorts of wonderful stuff that makes the kernel more suitable for heavier loads. Most everyone who's tested it thus far sees it performing far better than 2.4 so far, and it's got a whole slew of improvements yet to come.

There's also talk of replacing LVM1 with LVM2, and supporting various other logical disk managers which will make it more desireable on the enterprise front.

Re:Beta testing 2.5!

[Anonymous Coward]

The truth comes out. You don't have a "production" machine, son. You just want to sound cool.

Re:Beta testing 2.5!

[thegoldenear]

"creaky Duron 650"!!??

for most of us, surely, the "creaky" Duron is the high-end workstation. and there are infinite numbers of Pentium133 age machines to be found in skips in cities to use for testing and learning. hell, in Britain, a Pentium-II 266 with 3GB HDD, 128MB SDRAM, AGP Matrox Millenium video, 3C920 netcard, sound card, CD-ROM, USB, keyboard and mouse is only 60 UKPounds+VAT+15delivery; throw a monitor out of a skip onto that or pay 35UKPounds for a 17" one and yr sorted

Re:Beta testing 2.5!

[leviramsey]

If I could mod you, Insightful, my dear AC, I most definitely would.

Re:Beta testing 2.5!

[melee]

Except if you have a flaky VIA/Athlon combo. IT doesn't like anything much more advanced than 2.4.6.

Re:Beta testing 2.5!

[badhack]

Have you taken a look at Linux Kernel 2.5 Status page?
Listed there are the major changes since the 2.4 fork.

badhack

Re:Beta testing 2.5!

[gmhowell]

I've looked at it, but nothing is really jumping out at me. Due to maturity of 2.4 and business reluctance to upgrade, I think that adoption of 2.6 may be the slowest yet.

Re:Beta testing 2.5!

[morgajel]

The parent post made me think of something.
I'm a geek type with a little(very little) experience programming, and I'm competent with linux... what/how exactly could I help beta test a kernel? what does it entail? I'm guessing it's more than just compiling it on my system and going "ok, it worked!" Perhaps someone can give me some pointers so "people like me" can help too.

Re:Beta testing 2.5!

[Blkdeath]

I'm a geek type with a little(very little) experience programming, and I'm competent with linux... what/how exactly could I help beta test a kernel? what does it entail? I'm guessing it's more than just compiling it on my system and going "ok, it worked!" Perhaps someone can give me some pointers so "people like me" can help too.

Compile the kernel, boot the kernel, report what breaks. If something either breaks or significantly slows down (performance loss), report it with great details to the appropriate places.

See the file REPORTING-BUGS in the Linux kernel source directory for more information.

Re:Beta testing 2.5!

[badhack]

In theory most of the list adds up to a more robust kernel. I can't say anything for speed, I haven't actually tried the kernel. But what were you really looking for? Flashing lights?

2.4 IS mature in the regard that it provides all the functions that a kernel should. The only thing 2.6 can do is reimplement what exists in a better way, or provide new tech:

• IDE Cleanup
• Preemptive Kernel
• Opteron Support
• ACPI
• TCQ/Serial ATA
• Software Suspend
• XFS

badhack

Re:Beta testing 2.5!

[BlackHawk-666]

Is this performance noticeable by boring old workstation users (like me) using it as a desktop, or will I only notice if it's a server? Just interested, since I don't have any compelling reason to switch from a kernel I know is working for me, and working well.

Re:Beta testing 2.5!

[MikeBabcock]

Because you're helping the developers. You want to help the people who gave you your free OS kernel, right? Well, they would really appreciate it if you would download the latest 2.5, compile it, boot it, and send them bug reports. They may ask you a bunch of questions or give you some patches to try. You may have time to do those things, and help out the rest of the community in the meantime.

If you end up staying on 2.4.x for most of your work (as you should) for the forseable future, that's great, but at least you'll have helped the effort.

Re:Beta testing 2.5!

[sirsnork]

Hey, you found my old machine... I want it back :-)

compressed caching......

[oliverthered]

"compressed caching offers nothing to machines with heaps of ram",
If memory transfer and access speed is causing a bottleneck then well designed compressed caching can give a good performance increase by decompressing into the cpu cache.

Streems with small blocks would probably give the best performance increases.

Re:compressed caching......

[Theom]

Also not all machines have heaps of ram.

Re:compressed caching......

[Anonymous Coward]

Also affected,
SMP, less page faults, synchronisation requires less synchronisation etc....

GPU's and peripheral devices, the obvious bus speed and latency, as well as performance gains that GPU's can get using RLE.

Networking, never ever design a protocol that sends large amounts of uncompressed data.

Types of RLE can also increase performance in binary tree searches, e.g. in database indices, or filing systems.

Do they include use of compression in design pattern and refactoring books?

Re:compressed caching......

[Hast]

Regarding SMP the interview state that the compressed caching isn't compatible with SMP. (At least not now. No details are given.)

And regarding network protocols: It depends on which level you are building. On the application level a simple and slightly verbose protocol is nice from a debugging and future safe perspective. (Think HTTP.) When designing protocols for the link level it's not as much use though. (Most likely you'll just simulate that in any case. "Hot debugging" isn't all that much fun there.)

Good stuff

[truth_revealed]

I'm very grateful that this guy took the time out of his busy schedule to quantify what every Linux user has suspected for a long time - CPU performance degradation during heavy IO. I have always felt that Linus put too much emphasis on pure CPU-bound tasks and that's why he resisted raising HZ above the ridicously low value of 100 for so long - to the detriment of desktop applications. Hopefully this a start of a trend to create a more universal general purpose kernel for interactive desktops, web servers and number crunchers.

a good 2.5 kernel?

[SoftwareTechie]

As a Linux newbie I don't really understand what that comment means.

Is it that some versions of the Kernel are better or worse than others (e.g 2.5.2 is better than 2.5.3)?

Or does it mean that some distros of Linux (even with the same kernel version) are better or worse than others depending on how the kernel was patched, built and configured by the supplier?

Re:a good 2.5 kernel?

[Charlton Heston]

The answer is yes, some versions of the kernel are better than others. But these differences are sometime subtle. As a newbie, your biggest concerns are probably does it run my programs, and does it stay running without crashing. Beyond that you can satisfy your curiosity to get the details. Read the kernel development mailing list archives. Read the kernel source. There's about a zillion different patches that people have written that aren't in the kernel. You can find them and try them out on your own kernel. Take notes, and maybe build a webpage with what you learn.

Re:a good 2.5 kernel?

[cartographer]

The 2.5.x set of kernels is the development branch (the key being the odd minor number). It's where major feature changes are taking place, and new ideas are tried out. As you'd expect, things don't always work at the first (or fifth) go, so not every 2.5.x kernel is going to be 'good'.

'Good' probably depends a whole lot on how well the bits you need for your exact machine configuration are currently working, so your good kernel may be someone elses nightmare.

The lesson to all this is 'don't use a development kernel if your not ready for breakage'. That means back up all your file systems and don't even think of putting one on a production machine. But if you have some new widget that isn't supported in the stable kernel series (2.4.x currently), you might want to see how the development series is shaping up and even offer some feedback so that the next stable series really is.

Re:a good 2.5 kernel?

[Anonymous Coward]

Linux Kernels are like Star Trek Movies. The even numbered ones are always the best (2.0.x, 2.2.x, 2.4.x). The odd numbered ones are a bit unstable and are really made for development purposes. However if you don't mind testing and reporting bugs, feel free the try the odd-numbered sub-versions.

general purpose kernel

[oliverthered]

Why have a GP kernel, Linux comes in source code form, you should be able to have a taylored kernel give the best performance for you uses.

You could even stat the kernel and do a rebuild with what the stats say is best for you usage pattern.

Re:general purpose kernel

[Anonymous Coward]

The average Windows user (or even Linux user) does not know how to do the things you mentioned.
Ideally the kernel would be self-tuning based on usage patterns rather than statically built for one specific pattern.

Re:general purpose kernel

[oliverthered]

Well the argument I was replying to is that there should be one kernel for servers and desktops.

My preferance would be to ship a does everything desktop tuned kernel, if you want to run a server then build a kernel to meet you server requirements (if you can't do that then you shouldn't be running a server)

Self-tuning is the best way, but generating the stats will slow things down a bit, at least on an x86 arch.

Re:general purpose kernel

[BurritoWarrior]

I use Linux. I can recompile my kernel and take out most of the things I don't need (I leave it in or build it as a module if I am not sure). But I do not know what it means to stat my kernel nor how to do it.

Can anyone provide a reference, as this seems like it could be quite useful.

Multitalented

[abdulla]

"Con Kolivas, a practicing doctor in Australia, has written a benchmarking tool called ConTest which has proven to be tremendously useful to kernel developers..."

Let me guess, he cooks too. Ladies this is your man.

Re:Multitalented

[janda]

What's this "ladies" crap?

Re:Multitalented

[NoMercy]

Con Kolivas: I'm 32 years old, live and grew up in Melbourne Australia, am very happily married and have a 9 month old son. I'm a little embarrassed people get me confused for a kernel hacker, as my real profession is very remote from IT. I'm a doctor; a specialist in anaesthesia.

Eg, he's already taken, sorry ladies :)

C(++) is great

[stevenp]

> After applying each patch I had to sort out the problems with each merge and found that looking at
> the code it made a lot of sense to me and I could sort out the problems - mind you I can't program in C at all.
> Look at the code for long enough and you start understanding what it is doing.

C is great! Even doctors can read it and fix the bugs!