Slashdot Mirror
Kernel Benchmarks
kitplane01 writes: "Three students and a professor from Northern Michigan University spent the semester benchmarking a bunch of Linux kernels, from 2.0.1 to 2.4.0. Many functions improved in speed, but some did not. Total lines of code have tripled, and are on an exponential growth curve. Read their results here."
http://euclid.nmu.edu/~benchmark/null_call.gif
This shows why computer guys are not scientists. My first year phys chem prof would tear his own arm off and beat you to death with it if you gave him a graph that looked that ugly.
The Excel defaults may be ugly, but you can change them.
Thus "0" indicates byte alignment, "1" word (16 bit) alignment, "2" doubleword (32 bit), "3" quadword (64 bit), and "4" paragraph (128 bit). The other optimization of interest is the "-O" setting. Here arguments can take the value of 0, 1, 2, or higher. Personally, I found that -O2 was not necessarily the best setting, although it seems very common to find it set to that in Makefiles. I found using -O1 and tuning the alignment optimizations by hand provided better results.
My findings by benchmarking all the combinations of settings were that for a Cyrix 5x86, optimal alignment values were lower numerically lower than might be expected. For example, close to optimal settings as I recall were:
It wouldn't be a bad starting point for any Intel processor. On modern processors, it is more important to achieve high cache hits, which is thwarted by certain wrong optimizations such as aggressive loop unrolling and excessive alignment. One particular setting to avoid is -m486. It should be avoided for most processors other than a 486, because the 486 alignment requirements are less than optimal (i.e. tends to over-align) for both its predecessors and descendents. And if you don't need a debugging version of your code -fomit-frame-pointer is usually always useful as it frees up an extra general purpose register.Except for the lines of code graph, I don't see how they justify fitting exponential curves to any of the other graphs. Since the resulting "exponential" curves that were fit are nearly straight lines there's really no basis for doing anything other than a linear fit.
They note that this was all run on the same hardware, but all that means is that the results are valid *for* that hardware. Some of the drastic changes in some areas might be due to, for example, the replacement of a generic driver with a specific driver optimized for one of the pieces of hardware they used. Obviously this change wouldn't carry over to all other systems.
All in all not bad though. It would've been nice to see some more rigorous data analysis though (the data analysis expected in a typical college freshman chemlab class is more extensive than this).
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
What about a web server using signal based on IO and a single process model handling quite a few connections? That can easily have thousands of signals per second.
--
Mike Mangino
Sr. Software Engineer, SubmitOrder.com
Mike Mangino
mmangino@acm.org
I knew it was boring way up in Northern Michigan, but until now, I never imagined just how boring it actually is. I guess in Manitoba they must be benchmarking DOS calls in various MS operating systems. I guess it beats watching caribu mate.
If tits were wings it'd be flying around.
The performance that I care about is "do it work???" and the NE2000 cards give me no trouble at all. 3c509 cards are also sweet and trouble free.
If tits were wings it'd be flying around.
That's a lot of work just to print out a negative number on your screen...
If tits were wings it'd be flying around.
Rule #4:
One can have a graph of any shape that he wants by carefully choosing the axis'.
I quote" Hardware compatibility is a large part of the growth."
I don't want a lot, I just want it all!
Flame away, I have a hose!
Only 'flamers' flame!
Silly graphs is a pet peeve of mine. I hate it when my students give me graphs like these. Needless gridlines, unlabeled legends, connected dots, and poor statistical analysis.
I also find it ironic that they used MS Excel (which they don't say they did, but it sure looks like it)...
First, the university benchmarking team simply ran lmbench (a free, popular, old kernel benchmarking utility) on a variety of kernels. Claiming that:
...somewhat exaggerates this accomplishment
Three students and a professor from Northern Michigan University spent the semester benchmarking a bunch of Linux kernels
Second, no data were presented on the main areas of the kernel that were improved. How is SMP performance in kernel space? Did the finer grained locks help? How is the performance from the threaded IP stack? Does it prevent IO blocking?
THAT kind of information would have been interesting. They tested only things that the kernel has done forever.
I use a "real world" benchmark (which of course might be completely irrelevant to you, however relevant it happens to be to me).
Here are some recent observations regarding this specific benchmark, ranked in order of effect:
- Changing BIOS memory setting from CAS 2 to CAS 3 : 3.7% speedup.
- Changing to a different brand motherboard, and matching the original's BIOS settings as well as possible : 2.1% speedup.
- Upgrading 2.4.3 to 2.4.4 : 1.1% speedup.
- Running under kernel compiled as "Athlon" rather than "i686" : no substantial difference.
Moreover, although I have not had time to test it, a well-informed friend tells me that using certain recent versions of gcc rather than certain older ones can give a whopping 30% slowdown, even using the same flags for compilation. (N.B. - He did not say "gcc is getting worse with time". He merely remarked re two specific versions, whose numbers escape me at the moment.)If performance tuning is your forte, then clearly you've got your work cut out for you.
--
Sheesh, evil *and* a jerk. -- Jade
Over three years it's still positive.
Every evening I run a disk/memory intensive program that does a 3 year analysis of the US stock market. When moving from 2.2.x to 2.4.x I obtained a run time decrease from 270 to 190 seconds. This to me was a VERY impressive upgrade. The same code running on Win2000 takes 1300 seconds to run.
It's the same code running on the same box - a dual P2 400 with 0.5 GB of RAM. No ifdefs. Programs are invoked from the command line. Relatively small results datasets are saved to files. Because of the size of the input dataset, and the crappy indexes the main performance determinant is the efficiency of disk i/o and buffering thereof.
For this application the 2.4 kernel kicks butt up and down the street all day. YMMV.
Annother more extensive linux evolution study is at:f
http://plg.uwaterloo.ca/~migod/papers/icsm00.pd
Well, along those lines of what I want on a system, for Windows, throw in VC++ (which i'm sure is huge), perl, python, VB (instead of tcl/tk, shell scrtipting, fortran, and all the other misc language support many of us have), MSSQL, and Photoshop or Paint Shop Pro.
(what I'd REALLY want on a Windows system is an X server and Cygwin, but for the sake of arguement, I'll leave that stuff out)
I'm guessing we'd be approaching some huge numbers on both sides, and all I can really speculate is that I think Windows would have more overlapping functionality in its apps, but I can't say as for lines of code.
Anyway, lines of code is not directly a measure of bloat. In my mind, bloat is lines of code divided by (functionality times stability times performance), but I realize that not everyone shares my view on that.
Yeah.
-ben.c
Win2K may be 30 million lines of code but the Win2K *kernel* is tiny compared to that amount. The 30 million lines includes everything from the kernel, logging, user management, dialup tools, solitaire to the file manager. Don't compare apples to oranges.
The other side of the story is on their site.
How we know is more important than what we know.
Yeah, right. The problem with this approach is that it leads to unnecessarily narrow definitions of functionality, and can prevent hardware manufacturers from doing things cheaper. Not only that, but the examples you chose are kind of screwy. "Current modems" without a qualifier implies the N+1 varieties of WinModems out there, which all do things differently. Many old sound cards did things their own way and had a small DOS TSR that provided SB compatability in software. The floppy, IDE, and ATAPI command sets, as well as the RS232 serial-port standards, are published and standardized, but these are properly communications protocols between devices, not the devices themselves. The PCI and ISA busses are, again, more like protocols to allow devices to communicate rather than devices themselves. I don't see too many non-PCI, non-ISA devices that plug into the insides of an x86.
Non-x86 hardware platforms have it easier; one vendor like Apple/Sun/IBM says, "This is the list of hardware that works on our platform," and you use it. The multitude of hardware vendors for x86 boards and devices has led to a large amount of conflicting standards and weird, proprietary hardware. (If a vendor can save $0.10 per unit on a device by leaving out hardware functions which can be replicated by a kludged binary driver, they will. Think WinModems.) This approach has also made x86 hardware cheaper than the alternatives.
Simply put, things will change and change quickly in hardware. Standards are a good idea, but they quickly become lowest-common-denominator, think "VGA".
Give a monkey a brain and he'll swear he's the center of the universe.
I hope you aren't using 2.4.2. It was buggy and crashed a lot on my system (reiserfs may have been the problem).
------
Guns don't kill people. Bulletskill people.
------
Obvious correction? Which correction is that? (What is CAS, anyway?)
------
2.4.0 has a dramatically improved mm system, most of the benefits of which don't show up on these tests, yet make a world of difference in real life.
Why the h*** does they list both RAM and a combination of base mem & extended mem on their 'resources' page. I would have mattered if they had tested MS-DOS 6.2, but not Linux!!
--larsw
I've always wondered why people say that. I can make several valid comparasions between apples and oranges:
- Oranges have a thicker skin than apples
- Apples grow better in northern regions than oranges
- Apples make a better pie than oranges
- Orange juice is thicker than apple juice
- Oranges have larger seeds than apples
I could continue on like this for some time and I don't think that I would ever get around to mentioning either Linux or Win2k whilst comparing apples and oranges (Though, I might get around to mentioning OSX and British cell phone users if I were to keep at it long enough)________________________
I don't want free as in beer. I just want free beer.
I'm not impressed.
[
I rented that video last week. Very racy.
:wq
Interesting... I think I do!
But there are still factors to consider. I think we at least need to multiply by the spaghetti ratio, but other factors, such as usefulness index, design cleanliness coeffecient and ugly hack quotient needs to be taken into account. :-)
Oh well.
Depending on point of view, that has already happened long ago...
To make the comparison meaningfull, you have to get systems of somewhat equal capacity. The linux kernel by itself is in no way comparable to Windows 2000.
In addition we need various fileutilities, an accelerated X11-server (with Mesa/OpenGL, the video-extension, and antialiasing), one of Gnome/KDE (filemanager, basic desktop utilities, a simple texteditor, something akin to COM (which would be Bonobo or Kparts)), a working web-browser (Mozilla or Konqueror), some userfriendly utilities to replace the control-panel, a user-friendly email-client and newsreader, a simple webserver, basic networking utilities (Samba with a user-friendly network neighborhood browser, telnet, ftp, ping, ...), a good media-player (capable of playing at least wav, mp3, CD's, mpeg, avi, mov and preferably asf and wmf), minicom, a ppp-dialer, and probably quite a few other goodies I've forgotten to mention.
If we put all this into a linux-distribution, I doubt we would do much better than W2k. But to make things even worse, that wouldn't make much of a linux-system. Most linux-users wouldn't be too happy without emacs, gcc with friends, perl, python, tcl/tk, and most of the common command-line utilities (sed, awk, find, etc...) (, and probably also apache, MySQL or PostgreSQL, gimp, etc...).
Line-count? Well, guess what... Linux has become bloatware... Even more than what's produced in Redmond!
You've just hit on the killer problem there. OS developers just take it for granted that they have to write drivers for every device out there. What I wish is that hardware manufacturers would just use one standard interface, then only one driver for each device would be necessary. Impossible you say? Look at current modems, old sound cards (all sound blaster compatible), NE2000 network cards (I won't buy any other kinds) ATAPI CD-Roms (all recent ones are) Floppy drives, and many more devices. If people would put their foot down an say 'I want compatibility' then driver problems under any device would be a distant memory, OSes would be far smaller, hardware would be truely interchangeable, and Windows wouldn't be the only option for those with exotic hardware.
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
The most important benchmark they showed was their charts--ugly products of Microsoft Excel. Even though a lot has changed in those 4.5 years, its still easier to make your charts in windows.
According to this graph page fault latencies suck in kernel 2.2. Is this true? I think I'm running a 2.2.17 AC kernel though and if I'm just doing development and not causing swapping then it doesn't matter though right?
So when will line count surpass Windows 2000?
----
There's no point in being grown up if you can't be childish sometimes. -- Dr. Who
If its the same code then it has nothing to do with his develpoment skills. Most calculations of that nature are done using programs that read input from a text file perform the calculation and dump it to the screen or another file. That should be completly portable with no #ifdef __POSIX. Now what could be to blame is the libraries that are being linked against.
--- Justin Dearing http://www.justaprogrammer.net/ We're just programmers.
Uh, maybe it's just me, but does anyone else think it's funny they used MS Graph (and presumably Excel) to draw the result graphs? You'd think they use StarOffice.
- I don't care if they globalize against free speech. All my best free thoughts are done in my head.
Yesterday I modded some of these Michael related posts WAY down. Why?
1. Because they are often insulting, and I don't like to read lame insults on my slashdot. :-)
If you make an offtopic comment about a delicate subject, it really doesn't help if you start insulting.
Just state your opinion calmly and have respect for other people. If you'd post like that I would mod it up. (But sadly i wasted all my points modding you down yesterday
2. You also always post so mysteriously. Why? I still don't really understand what all the fuss is about. And that's also really irritating. So would you please explain thoroughly what the problem is. Only if we all know what the problem is can we solve it.
So please post something abjective and insightful about this, so we can discuss and solve the whole thing. If you keep posting like this you will only get modded down > get frustrated > post more insults > ...
I've read that the Kernel Team has recomended use of egcs 1.1.2 as an alternative to gcc 2.95.2 for compiling the 2.4.0 kernel. How much affect does that have on the performance of an OS?
Is it worth the trouble?
Then they are saying that it will take twice as long for Linux to tell my apps that I have ordered them killed.... (-1) so maybe that extra 1.5 microseconds might prevent a -9 switch.
LedgerSMB: Open source Accounting/ERP
One thing that I wonder about: that huge performance hit on the page fault latency shown in 2.2.6. Is it still there as of 2.2.19? Did the fix make its way back into the 2.2 series, or is it only fixed as of the later 2.3's and the 2.4 series? 2.2.6 is the only 2.2 in their study, so the study doesn't answer the question.
-Rob
ie. There is not much fat to trim left...
Therefore the next dramatic improvements if they are to come will not be from tweaking this part or that part of the kernel, but rather from implementing entirely new classes of functionality.
ie. Linux has arrived. It's settled down, time for it to start exploring as yet unimagined new things to do instead of new ways to do old things.
The future will be, umm, fun.
This post is not designed or intended for use in on-line control of aircraft, air traffic, aircraft navigation or aircraft communications; or in the design, construction, operation or maintenance of any nuclear facility.
It would be nice to see updates to the data here as new versions of the kernel are released. For example, some users are not particularly concerned with newer versions of the kernel unless there are significant improvements. Consider this example: you're concerned mostly with performance aspects of the kernel. A new version is released that shows no improvement (or a decrease) in performance. No sense in upgrading immediately (of course, you may be one of those people who actually looks for and reports bugs) and you can wait until you see a downward trend in the graph before taking your time. There are other potential uses for "live" data such as this. I think it'd be nice if these guys would keep maintaining it. :)
Why bother.
Where are the results for networking?
I definitely noticed a jump in performance between 2.2.16 and 2.4.0 so they must be missing something here.
They note the large increase in hardware support, but don't seem to realise that this new support and improved support has given Linux much more performance than their benchmarks might show.
Maybe the improvements in X etc have helped but no real performance difference between 2.1.38 and 2.4.0? Put any such machines through real world work and you'll soon spot the difference...
"Don't get mad, get a monkey!"
I am a senior at nmu. Maybe you should try to communicate with the professors and other students. I have had no problems with it. In fact, I am currently starting a research project with Dr. Appleton this summer pertaining to linux file systems. I say, if you don't like it here, leave.... now.
For those of you who were interested in the "exponential growth" issue, I did a much more detailed study on the growth of the Linux kernel that was published in the 2000 Intl Conference on Software Maintenance. I think it's very readable by non-academics. Comments welcome. -- MWG http://plg.uwaterloo.ca/~migod/papers/icsm00.pdf