Slashdot Mirror
Kernel Benchmarks
kitplane01 writes: "Three students and a professor from Northern Michigan University spent the semester benchmarking a bunch of Linux kernels, from 2.0.1 to 2.4.0. Many functions improved in speed, but some did not. Total lines of code have tripled, and are on an exponential growth curve. Read their results here."
Thus "0" indicates byte alignment, "1" word (16 bit) alignment, "2" doubleword (32 bit), "3" quadword (64 bit), and "4" paragraph (128 bit). The other optimization of interest is the "-O" setting. Here arguments can take the value of 0, 1, 2, or higher. Personally, I found that -O2 was not necessarily the best setting, although it seems very common to find it set to that in Makefiles. I found using -O1 and tuning the alignment optimizations by hand provided better results.
My findings by benchmarking all the combinations of settings were that for a Cyrix 5x86, optimal alignment values were lower numerically lower than might be expected. For example, close to optimal settings as I recall were:
It wouldn't be a bad starting point for any Intel processor. On modern processors, it is more important to achieve high cache hits, which is thwarted by certain wrong optimizations such as aggressive loop unrolling and excessive alignment. One particular setting to avoid is -m486. It should be avoided for most processors other than a 486, because the 486 alignment requirements are less than optimal (i.e. tends to over-align) for both its predecessors and descendents. And if you don't need a debugging version of your code -fomit-frame-pointer is usually always useful as it frees up an extra general purpose register.That's a lot of work just to print out a negative number on your screen...
If tits were wings it'd be flying around.
Silly graphs is a pet peeve of mine. I hate it when my students give me graphs like these. Needless gridlines, unlabeled legends, connected dots, and poor statistical analysis.
I also find it ironic that they used MS Excel (which they don't say they did, but it sure looks like it)...
First, the university benchmarking team simply ran lmbench (a free, popular, old kernel benchmarking utility) on a variety of kernels. Claiming that:
...somewhat exaggerates this accomplishment
Three students and a professor from Northern Michigan University spent the semester benchmarking a bunch of Linux kernels
Second, no data were presented on the main areas of the kernel that were improved. How is SMP performance in kernel space? Did the finer grained locks help? How is the performance from the threaded IP stack? Does it prevent IO blocking?
THAT kind of information would have been interesting. They tested only things that the kernel has done forever.
Every evening I run a disk/memory intensive program that does a 3 year analysis of the US stock market. When moving from 2.2.x to 2.4.x I obtained a run time decrease from 270 to 190 seconds. This to me was a VERY impressive upgrade. The same code running on Win2000 takes 1300 seconds to run.
It's the same code running on the same box - a dual P2 400 with 0.5 GB of RAM. No ifdefs. Programs are invoked from the command line. Relatively small results datasets are saved to files. Because of the size of the input dataset, and the crappy indexes the main performance determinant is the efficiency of disk i/o and buffering thereof.
For this application the 2.4 kernel kicks butt up and down the street all day. YMMV.
Depending on point of view, that has already happened long ago...
To make the comparison meaningfull, you have to get systems of somewhat equal capacity. The linux kernel by itself is in no way comparable to Windows 2000.
In addition we need various fileutilities, an accelerated X11-server (with Mesa/OpenGL, the video-extension, and antialiasing), one of Gnome/KDE (filemanager, basic desktop utilities, a simple texteditor, something akin to COM (which would be Bonobo or Kparts)), a working web-browser (Mozilla or Konqueror), some userfriendly utilities to replace the control-panel, a user-friendly email-client and newsreader, a simple webserver, basic networking utilities (Samba with a user-friendly network neighborhood browser, telnet, ftp, ping, ...), a good media-player (capable of playing at least wav, mp3, CD's, mpeg, avi, mov and preferably asf and wmf), minicom, a ppp-dialer, and probably quite a few other goodies I've forgotten to mention.
If we put all this into a linux-distribution, I doubt we would do much better than W2k. But to make things even worse, that wouldn't make much of a linux-system. Most linux-users wouldn't be too happy without emacs, gcc with friends, perl, python, tcl/tk, and most of the common command-line utilities (sed, awk, find, etc...) (, and probably also apache, MySQL or PostgreSQL, gimp, etc...).
Line-count? Well, guess what... Linux has become bloatware... Even more than what's produced in Redmond!
Where are the results for networking?
I definitely noticed a jump in performance between 2.2.16 and 2.4.0 so they must be missing something here.
They note the large increase in hardware support, but don't seem to realise that this new support and improved support has given Linux much more performance than their benchmarks might show.
Maybe the improvements in X etc have helped but no real performance difference between 2.1.38 and 2.4.0? Put any such machines through real world work and you'll soon spot the difference...
"Don't get mad, get a monkey!"