Byte Benchmarks Various Linux Trees

urbanjunkie writes: "Moshe Bar has an interesting article, essentially benchmarking the standard kernel (with aa VM) against the -ac kernel (with Rik's VM)." He also raises some very interesting points about how patches (and entire development trees) interact.

Comments

Interesting conclusion...

[NOT-2-QUICK]

The author came to the quite informed conclusion of:

I prefer the 2.4.17 or 2.4.18pre2aa for my heavy-duty servers. The reverse-mapping patch by Rik, however, has great promise once it has stabilized. Finally, the Red Hat 2.4.9 is a very good kernel, fast and reliable.

While I personally may not have agreed with this synopsis prior to reading the article (and am still not completely sold...), there are certainly some interesting facts and figures to ponder the next time you reload your system or update your kernel...

Re:Interesting conclusion...

[peripatetic_bum]

What a nice article and it seems to come at a time just then everyone is talking about "Fork this and Fork that" that in fact this is exactly what is needed in this healthy debate.

I think perhaps that we should start having "kernel" races if you will, and we could have various categories (ie, I/O races, Stress test races) in which the various trees would compete and by-and-by the best kernel trees would become known.

Please, I would like to hear your comments!

Re:Interesting conclusion...

[Rik van Riel]

What a nice article and it seems to come at a time just then everyone is talking about "Fork this and Fork that" that in fact this is exactly what is needed in this healthy debate.

Indeed, forks are (IMHO) the best way of doing development. Doing your development in the main kernel will just lead to contradictory code being integrated and the code never working quite right because it's missing fixes (guess why RH's 2.4.9 runs faster ... it does have the fixes).

One minor nitpick though ... I never released an -rmap VM against 2.4.18-pre3, the latest is still against 2.4.17. I suspect that the crashes Moshe saw are due to some change in 2.4.18-pre3 conflicting with the -rmap VM patch, especially since rmap-11c has survived the kernel torture lab at RH. ;)

[ot] So what's the best kernel to get right now?

[wrinkledshirt]

Sorry for the dumb, offtopic questions.

I'm sitting at home with my fresh install of RH 7.1 and I'm wondering what kernel to upgrade it to. Any suggestions? Is there a stable one in there somewhere that I should go with? Should I stick with the default kernel that's on their now?

If I'm regularly compiling new programs using gcc or g++, is it safe to go from one tree to another, as long as they're all 2.4.x, or what? Do I need to recompile with a new kernel? Or is that a red herring?

Red Hat 2.4.9 is a very good kernel, fast...WHY?

[srw]

Can anyone explain this? Was the stock 2.4.9 faster and more reliable than our current stable kernel? If there are stability and speed patches in the RH kernel, why haven't they been adopted in the standard kernel? How close is RHs 2.4.9 to Alan Cox's kernel? I'm assuming he has a strong influence on RHs kernel.

Re:Red Hat 2.4.9 is a very good kernel, fast...WHY

[Rik van Riel]

Thanks to Alan Cox, Red Hat (and most Linux distributions) do have the patches for my VM that Linus didn't have the time for.

Stable kernel?

[sitturat]

It is hard to believe that all this is going on with what is meant to be a stable kernel version, ie 2.4.x

So far the VM has been replaced twice, and now the rmap patch is apparently going to be added despite the fact that "something is seriously messed up in the reverse-map implementation".

Have they saved any experimental code for the 2.5.x kernels, or will that now be stable?

Re:The folly of Open Source

[Hagmonk]

Most of the 'forks' seem to actually be marshalling grounds for maintainers to prove the reliability of their work to the kernel community in general. Once things gain some faith in the community and in Linus' head, they get incorporated in the main tree. The process needs work, but it has gotten the community this far ...

Can you provide some high profile forking examples that have occured in the recent history of Open Source, out of interest? Or evidence of similar forks in Linux. Not, to coin a phrase(?), "soft forks" as I have mentioned here, but "hard forks". Fundemental changes in ideology or personality clashes that have seriously caused a split in the community?

Number of crashes in 100 days?

[alexhmit01]

It's good to see that Slashdot users still don't use computers for anything... I'm not looking for a system with better uptime than Win95, and that seems to be all you guys want.

I can't have multiple crashes in 100 days. If you are doing real load, spend real money, get real systems.

Don't build machines with your screw driver, get QA'd servers. Don't roll your own kernel, let Redhat test it.

These types of tests are useful as commentary and recommendations for what people should do in the development process.

Hopefully Someone Has an Answer...

[jschmerge]

Since I've seen posts from at least one kernel developer in response to the attached story, I figured that this might be a good place to ask the following question:

A little while ago, I wrote an application that uses an incredible amount of memory... A very space inefficient implementation of Eratosthenes' Sieve. In essence, what the algorithm does is cycle through the entire contents of memory sequentially many, many times (not a completely correct description). What I found with the following three kernel versions:

• 2.4.4
• 2.4.8
• 2.4.17

is that any time the program's footprint exceeded the physical ammount of available memory, performance degraded exponentially. I found this to be very suprising, considering that I was only exceeding the physical 1024 Megs of memory by less than 10 Megs. About the only difference between the three kernels I listed above is that the 2.4.17 kernel would kill of memory intensive processes a lot quicker than the other 2 versions.

My question for the Kernel gods out there is as follows: are there any stable 2.4.x kernel releases out there that would handle this type of stress without the performance degradation that I've experienced with these kernels?

Don't understand Moshe's conclusion

[DaveWood]

Under the section "Allocation and Swapping Results," I assume larger numbers are higher times and therefore worse. By the numbers, 2.4.18pre2aa (the Arcangeli kernel) seems to be the fastest overall, due to the 5th run (I would consider it the common case) results. Yet Moshe says:

"From the above figures it seems that the old van Riel VM is somewhat faster (considerably faster in the case of 2.4.9) than the new Arcangeli VM..."

Is my math wrong? The RVR VM in 2.4.9 is ever so slightly faster on the 2nd run and slower on the 5th, and the slowest of all is the newer one in 2.4.18pre3rmap. What's my mistake?

Moshe's politely indicating that van Riel was an ass when asked for comments; we can conclude either that Moshe didn't have a proper recent RMAP kernel to test with (as a result), or that the recent RMAP kernels are hit and miss.

From looking at van Riel's comments here, he vehemently believes his kernel is perfect and Moshe just got it wrong... The problem is that lots of people seem to "get it wrong" with that VM, including Linus... Overall in Rik I get the sense of an aggressive person who may have trouble admitting mistakes or accepting failure; not good traits in a programmer, since it's humility and communication skills which can often be the critical factors in a team programming effort... and lack of them can cause exactly the kinds of problems we've observed.

Re:Why is this funny?

[DrSkwid]

debian has it

freebsd has it

in fact both of those also upgrade third party software that's not part of the OS

windows update will never upgrade mozilla for you or KDE whereas apt-update & cvsup-portsupgrade upgrade EVERYTHING you've installed (if installed via apt-get or /usr/ports)

[my understanding of the debian process is through the grapevine so if I'm off the mark, be cool; not a fool]

This is nearly impossible to measure

[roystgnr]

If you see a Linux kernel crash, it is either hardware failure or a kernel failure. Since hardware failures (especially RAM and power supply) can be imposssible to repeat, the only way to prove that it is a kernel failure is to find the kernel bug. If you find the kernel bug, then the bug gets fixed, and suddenly your crash data is for an obsolete kernel.

You could try to take a statistical run at it, of course, but I suspect the number of machines required to give meaningful results would be on the order of Google's farm.

I thought about it some more...

[nusuth]

You should do something like starting with a large J (K*J comparable to but not exceeding total memory size) and read K*J pages without any preaccessing. E.g read values at offsets N,N+K,N+2K,...,N+KJ, then continue normal processing until you reach N+KJ. Repeat procedure with new N=N+KJ. This way you hint the VM that these pages are active, and should not be swapped out. K should be selected as large as possible, not exceeding a page boundary. Too large K will make fail to mark some pages as active, too small K will consume unnecessary processing time while hinting.

If everything goes according well, your kernel should swap out and swap in exactly M amount of memory for each preaccesing pass, where M is the amount of data that does not fit in the main memory. So if you have total memory T, total data T+M and K*J size prefetch, total swapped data per whole process would be M*(T+M)/(K*J).

This is all well and good

[The Bungi]

The raw performance of the VM is certainly important and all, but what I'd like to see are some *application* benchmarks among the various kernel trees. Star Office, the window managers, KDE, GNOME, etc. Graphics. Storage. Networking. Unles we're talking heavy metal servers running the usual suspect daemons the average user doesn't really give a hoot if the VM is well-designed or not - only if The Gimp runs quickly enough.

Comparision of evolution..

[kesuki]

In the context of evolution Torvalds represents natural selection, and kernel developers changing the code represent mutation, albeit a poor representation because the developers have some sense and purpose in what they do, while mutation "in nature" has absolutely none.

I have to disagree with the notion that this isn't Exactly how evolution occurs. Evolution isn't Random, Virus are the vector for almost all radical natural genetic manipulation. The purpouse of a virus is to use the resources of a host organism to replicate it's DNA sequences because Virus are too small to replicate themselves. In doing so It injects it's DNA inside a host cell and basically exploits the RNA strands inside to replicate it's own DNA. While doing this The virus can in fanct find new DNA within the host and borrow it for it's own protection. In many cases this is to make it resistant to antibodies produced by the body (HIV.) Now, not all Virus destroy the host cell especially when antibodies destroy it before it can complete it's task. In some instances the Virus may act as a Vecor Borrowing the DNA from one species, and inserting that code in another. Many virus can cross infect species. For example humans and pigs can catch influenza from each other. Geese and pigs can also catch influenza from each other, while humans and geese cannot infect each other with influenza.
Virus are acting for thier own self promotion and preservation. When a DNA stand from one species makes that species less able to destroy them they would try to splice that DNA into as many species as they can. Comparing that to kernel developers
is pretty straight forward. They try to 'infect' the kernel tree with the 'code' they've produced for any number of reasons. Being known for coding on linux, to get promotions at a linux friendly workplace, for the challenge and fun of contributing to the linux kernel, or just to fix something so that they can do something with linux that they were trying to do but couldn't.
This introduces variation along the same analog as virus changed DNA. As for 'uncorrected errors' in the DNA strand the only thing we can prove comes from that is cancer. Thus that type of 'mutation' is analog to 'bugs' in the code of the linux kernel. Unfortunately humans aren't anywhere near as good as the roughly 99.7% error correction rate of replicating double helix DNA strands, so code tends to get a lot more malignant tumors (root exploits) to cut out.

server this, server that

[paulbd]

it seems that the vast majority of commentary here all assumes that a linux machine is run as a server, or at best, some kind of generic desktop machine. while linux may be very good at running servers, and may be capable of acting as a good generic desktop machine, some of us are interested in it for much more specific tasks such as realtime audio processing, editing, synthesis and recording. we care about those extra 2 seconds spent in the VM code during a benchmark. we care about all the extra paging that goes on with some designs. we care about the internal operation of the buffer cache and how it affects attainable peak i/o performance because we stream, and when i say stream, i'm not talking about measly HTTP numbers, i'm talking about 64-128 streams of 24 or 32 bit 96khZ audio data. stop assuming that a top-end linux box is a server, please.

Re:Why is this funny?

[Pastis]

Let's compare Windows update with another typical Linux update.

1. Windows 2000
2. 
   I installed windows 2000 on my new Inspiron 8100 yesterday. And I used the so cool 'Windows update' function.
   Note that it requires to use IE (No IE, no windows update? At least the help doesn't describe any command line method.)
   
   • First I installed the OS (one reboot - pretty normal)
   • Then I installed Service Pack 2 (another one)
   • Then a group of security fixes (reboot)
   • Then another (missing ???) security fix (reboot)
   • Then IE 6 (reboot)
   • Then a new security fix, probably for IE6 (reboot)
   • Then DirectX 8.1 (reboot)
   • Then new drivers (reboot)
   • I also modified the fonts to large fonts: reboot.
   
   Until I had the drivers installed, everything was done with IE 5.0, in 640x480 mode, and (I think) 8 bits; so ugly!
   And if you do that from home on a modem, you have to restart your connection at every reboot. That's optimum.
   And when this is finished, you can now start installing non-windows applications!
   
   I forgot to mention that the first time I installed windows 2000 on a logical partition, is wasn't able to boot it after I installed Linux. Impossible to repair as my disk was not detected correctly.
   [Perhaps because w2k doesn't support DMA 100.
   When I face this kind of problems I am pretty happy to be able to recompile a new kernel.]
   
   I completely reinstalled it on a new Primary partition and, after I installed the drivers, I had a crash (blue screen) during reboot. [Note I was updating the win modem driver while using it to download the driver from the web]. Anyway, it was impossible to restart (crashing even in FailSafe mode or with 'last known good configuration'), and impossible to repair even with an up-to-date ERD. Had to restart the install from scratch for the third time! Damn!
   
   
3. Debian 3.0
4. 
   
   On the other hand, I have installed Debian 3.0 (aka testing) from floppies on the same machine. I rebooted once during install, and then another once again after I installed and tweaked the new kernel.
   Of course, you are not obliged to compile the kernel. I bet that if you use Mandrake or Red Hat, you will have a pretty good hardware detection process. I just like Debian and like to tweak the kernel.
   I did everything on the command line. Didn't have to have a full graphical installation (I don't like too much laptop built-in mouses).
   

To summarize:
Windows: requires graphical environment and a mouse to install correctly plus requires having at least 9 non-necessary reboots (with modem disconnections /reconnections)to have an up-to-date system. Don't mention the un-repairable crash and the non-recognized partitions...

Debian GNU/Linux: two reboots (one optional). All in command line, with a 15 seconds boot. No problem. There are perhaps GUI front-end to apt, but I do not need them. Can something be simpler than 'apt-get upgrade'?

I let you decide who is the easier and who is my winner.

So to come back to the subject of the thread: download kernel and patches.

• Note that windows update only updates your Windows stuff, none of your other applications. (and without applications and OS is nothing, right?)
• Note that possible dependencies problem may arise after you upgrade non-Windows applications.
• Note that you can automate the install of the updates on Linux (cron the apt-get command). That's even better than on windows, where you can only be warned that you need to update your system.
• 
  
  • 
    
    I could go on...