Slashdot Mirror

← Back to Stories (view on slashdot.org)

Linux Kernel Benchmarks, 2.6.24-2.6.29

Posted by timothy on from the impressive-span dept.

Ashmash writes "Phoronix has posted benchmarks of the Linux kernel from versions 2.6.24 to 2.6.29. They ran a number of desktop benchmarks from the Phoronix Test Suite on each of the six kernels on an Ubuntu host with an Intel Core 2 processor. The points they make with the new Linux 2.6.29 kernel are 1. there's a regression with 7-Zip compression 2. OpenSSL has improved significantly 3. a regression drastically impacting the SQLite performance has been fixed 4. the OpenMP GraphicsMagick performance is phenomenally better with this new kernel. In all of their other tests, the kernel performance was the roughly the same."

38 comments

  1. The OpenSSL benchmark by Anonymous Coward · · Score: 2, Interesting

    I find it difficult to believe that the 2x gain in OpenSSL performance for 4K RSA private key operations is solely due to this new kernel. Such operations are, at the core, just CPU-intensive modular exponentiations. Unless the kernel has become significantly better at making use of several cores (or processors) to parallelize such operations, I don't see how that can happen.

    1. Re:The OpenSSL benchmark by TheRaven64 · · Score: 3, Interesting

      There are lots of variables that may affect this. The new kernel may be preempting the CPU-intensive process less frequently, reducing TLB and cache churn, and context switching overhead. I don't think RSA should be using the FPU, but if it is (even for one operation in the 10ms quantum) then switching to lazy FPU context switching would give a performance increase (or, to put it another way, not doing lazy FPU context switching - either by accident or design - would give a performance penalty).

      I didn't read TFA (obviously), but on an SMP system, particularly an AMD machine, there are a few other issues that may arise. Without good processor affinity, the OpenSSL process may be being swapped between two cores, increasing cache misses (which can easily slow down a process a lot). The memory allocation routines may have been allocating memory from the memory controller attached to one processor while running the code on the other, increasing memory latency (and, therefore, cache miss cost).

      In short, there's nothing a kernel can do to speed up CPU-bound operations, but there is a lot it can do to slow them down, and stopping doing things that slow down operations looks a lot like doing things that speed them up.

      --
      I am TheRaven on Soylent News
  2. Re:Why no comparison to OS X? by wombat21 · · Score: 5, Insightful

    Comparison to OS X purely at the kernel level ? I'd prefer to see real-world benchmarks (a contentious area in itself..) across a range of operating systems, using identical hardware, but it would only spark endless debate that the methodology favoured one OS over another. Personally, I use all 3 and each has its pros and cons : such a benchmark would have to make a pretty compelling case for me to abandon any of my currently installed operating systems.

  3. well, we knew it all along -- almost no difference by Anonymous Coward · · Score: 0

    wake me up when you have an article that has some useful information that can drive my imagination or

    i can't get no ...
    no no no

  4. Is Phoronix even a credible source? by Anonymous Coward · · Score: 0

    I'm sure there are performance gains in the new kernal, but is Phoronix even considered a credible source for benchmarks?

    I remember one where they benchmarked the new opteron to compare operating systems but used different compiler versions.

    Looked at a Java benchmark recently and they were using different JDK versions on different OS's.

    I really am not sure if they know what they are doing. Can't you wait till anandtech, tom's hardware or some better site posts results?

    1. Re:Is Phoronix even a credible source? by macshit · · Score: 1

      Yeah, as you point out, some benchmarks on phoronix have been pretty sloppy.

      I think it's a great site for some purposes (where else can you find headlines on X development?), but I doubt many take their benchmarking seriously.

      While bad benchmarking is hardly rare, it's kind of disappointing in the case of phoronix, which is obviously aiming at a more technically savvy audience than most sites are, and often does benchmarks (such as this one) that are much more interesting to the FOSS crowd than the typical mouth-breather "win7 vs. xp" stuff you see elsewhere.

      --
      We live, as we dream -- alone....
    2. Re:Is Phoronix even a credible source? by Anonymous Coward · · Score: 0

      the benchmarks on tom's hardware will give you an overall result like this: "Linux kernel is faster because it's running on an Intel. Oh! ALL Hail our silicon overlord!"

      I still don't get why the benchmarks from TH are so valuable since they are obviously funded by MS and Intel advertising.

  5. Cure for my insomnia by fat_mike · · Score: 0, Redundant

    1. So I should just stick with gzip or bzip2?

    2. It went from 31.47 "signs per second" to 62.77. So the older kernel take one second less than the new one? News!

    3. Sorry, this is the part where I started nodding off.

    4. zzzzzzzz

    Wait, huh! Sorry, other than those four points every graph was within like .00001 signs or seconds or iterations of each other.

    Surely it is the year of the linux desktop!

    I'd like to see these results compared to a Windows machine.

  6. benchmark snd-hda-intel sound recording too! by multi+io · · Score: 2, Interesting

    Sure, it may have gone from working perfectly in 2.6.21 to not producing a beep in 2.6.28, but look how fast it has become! Priorities! :-P

  7. Why dang it? by MBCook · · Score: 4, Interesting

    Neat. They benchmarked a bunch of stuff and some real changes obviously took place. I can't help but be comforted by their conclusion (paraphrased): "Stuff changed."

    How about telling me why they changed.

    • Why did 2.6.29 double it's speed doing SSL signings?
    • Why did all the graphics tests speed up some?

    • Why did SQLite performance bomb for 3 releases?

    • What was the deal with 7-zip performance changing so much? What is it stressing that other tests aren't that cause it to vary?

    There are reasons for these things. You could test and find them out. You could read the mailing lists and see if someone else posted explanations (others must have noticed the SQLite thing).

    Heck, look at this list of new features and make guesses. I'd prefer "the newly added HyperScheduler v3.732 is probably the source of this" than the article's "things got faster, neat."

    That's why I love LWN and the kernel page so much. They post why things changed, or at least reasonable theories.

    --
    Comment forecast: Bits of genius surrounded by a sea of mediocrity.
    1. Re:Why dang it? by Sancho · · Score: 4, Insightful

      I'd like to know why, too. Drastic changes in performance may mean that faster ways to do a thing were discovered. It may also mean that codepaths are being skipped that are essential to things functioning correctly. Remember the Debian OpenSSL bug?

      That's why I'd like to know why SSL signing is so much faster under the new kernel. Seeing a 2x improvement makes me wonder if something's been screwed up that could compromise my certs.

    2. Re:Why dang it? by Anonymous Coward · · Score: 4, Insightful

      Hey guys. Michael (Larabel, whom owns and runs Phoronix) isn't omnipotent and he does a lot to keep his site running, OS's and tasks benchmarked, and news up to date on top of extra projects like the Phoronix Benchmark Suite of which he has brought togethor almost exclusively by himself. Do yourselves a favour and do some research, and maybe even post it to Phoronix for Michael to update. Assist him and, subsequently, the rest of us to reduce the number of questions.

    3. Re:Why dang it? by Anonymous Coward · · Score: 0

      So now you have to be a Linux kernel hacker to run a technology site?

      I love LWN and I'd love to know the answers to those questions, but nonetheless the "article" is interesting on its own, and without it raising awareness on these issues they'd probably remain unknown to Linux hackers (and LWN/Corbert).

  8. imagemagic libraries?? by gandhi_2 · · Score: 2, Interesting

    in almost all benchmarks, 2.6.29 did the same or a tiny bit worse than the others. Then in imagemagic operations, sometimes 2x faster? what mem / operation combination caused this?

    --
    THL phish sticks
    1. Re:imagemagic libraries?? by blitzkrieg3 · · Score: 2, Interesting

      If you're really so curious you can oprofile and find out yourself.

      Note: I'm not defending the Phoronix guys. As a previous poster pointed out, they are inherently bad at explaining the why things are slower and sometimes they are flat out wrong

    2. Re:imagemagic libraries?? by RAMMS+EIN · · Score: 2, Insightful

      ``Note: I'm not defending the Phoronix guys. As a previous poster pointed out, they are inherently bad at explaining the why things are slower and sometimes they are flat out wrong''

      In that case, the best they can do is to stop talking about it and just stick to what they know. Knowing just what is faster and what is slower is useful by itself. It can be used as a starting point for investigating what exactly caused the speed-ups and slowdowns. If the Phoronix folks can't or don't want to do this investigation themselves, it's perfectly fine to leave it up to other people.

      --
      Please correct me if I got my facts wrong.
  9. Re:Why no comparison to OS X? by blitzkrieg3 · · Score: 2, Informative

    I think you're missing an article.

    Besides, how do you expect a different binary to be fair? At least in this test the only variable was the kernel, whereas on OSX you have the kernel, application specific code, and the compiler to consider. They can't exactly copy the ELF binaries and libraries straight over from Linux.

  10. That is the question... by mtippett · · Score: 2, Informative

    The Phoronix benchmarking is intended to provide you the answers as to why. It is to highlight the stuff that has happened.

    If performance management is going on within the kernel community, then this shouldn't come up as a shock. The whole purpose of independent testing is that you see something that looks out of place, investigate and resolve. A perfect example is http://www.phoronix.com/scan.php?page=article&item=intel_atom_four&num=1 phoronix article, that showed that SuSE was trailing. This causes this http://lizards.opensuse.org/2008/12/16/comments-on-phoronix-benchmarking-opensuse-111/ discussion.

    The question and answer don't need to be provided by the same voice. It is when you have someone questioning, and then someone answering, then you have a discussion, then finally you have progress.

    To make it worse, there is virtually no reason that any number of the organizations supporting the leading developers can't invest a small amount of infrastructure and run the tests themselves. Phoronix Test Suite is absolutely trivial to use. The amount of "software development in autopilot" is frightening, this applies equally to Open Source as it does to Proprietary.

    1. Re:That is the question... by rackserverdeals · · Score: 1

      What's so special about that example you gave?

      openSuSE's disk I/O was slower because they enabled an option that the other didn't. Not enabling that option "runs the risk of severe filesystem corruption during a crash". Looks like they changed it to be like the other distros so they wouldn't look so bad during the benchmark.

      That's nice. Compromise stability for performance. This is the type of stupid crap that makes people wonder... Gee, why is such and such so much faster?

      The other issue was already reported as a bug on X11.

      So the benchmarks didn't expose anything that wasn't already known.

      Maybe it's a good benchmark suite, I haven't really looked at it, but maybe they should find other people to run it based on some of the comments here.

      --
      Dual Opteron < $600
    2. Re:That is the question... by Anonymous Coward · · Score: 0

      The point of the example was that SuSE had not conciously made the trade off. They didn't realize that they has made the safe option had that level of performance delta.

      Unfortunately, there are too few sites or groups doing benchmarking and so we are left with decisions made with qualitative vs quantative measurements.

    3. Re:That is the question... by oasisbob · · Score: 1

      openSuSE's disk I/O was slower because they enabled an option that the other didn't. Not enabling that option "runs the risk of severe filesystem corruption during a crash". Looks like they changed it to be like the other distros so they wouldn't look so bad during the benchmark.

      That's nice. Compromise stability for performance. This is the type of stupid crap that makes people wonder... Gee, why is such and such so much faster?

      It's not quite that simple.

      See that openSUSE bug:

      Since I wrote this, I came across comments by Andrew Morton, and a thread discussing barriers and why he refused a patch to make them a default. The sequential layout of journal, in general means the disk does the writes in right order, except when wrapping round (relatively rare); meaning in general running without barriers is a problem only for the unfortunate.

      If I found myself making a different decision than Andrew Morton, I might revisit that choice as well.

    4. Re:That is the question... by rackserverdeals · · Score: 1

      What makes you think SuSE unconsciously made the trade off? The default setting is to have barriers disabled. They would have had to override the default behavior.

      --
      Dual Opteron < $600
    5. Re:That is the question... by rackserverdeals · · Score: 1

      If you bought a new oven and it had an option to reduce gas consumption but in "relatively rare" situations it could explode while you were cooking, what setting would you leave it on?

      I've seen other things like this over the years, and that's why I am now deploying applications on Solaris. Like all software it's not completely bug free, but I haven't had any problems, and their philosophy seems to be to not intentionally make decisions that could blow people up.

      --
      Dual Opteron < $600
    6. Re:That is the question... by mtippett · · Score: 1

      Judging by your posts and your handle, you work in or around servers - a lot.

      You would probably be aware that security, stability, and all such things are a set of tradeoffs of risks and benefits/costs.

      You can make a system 100% secure, but it may not be useable. You can make a system five 9's stable, but you have to pay for it. You make the assessment of the risk (in this case data corruption), against the benefit/costs (double the speed in some cases).

      SuSE seemed to have made the assessment of risk without understanding the cost. They enabled barriers by default to take the high moral ground, but then didn't understand the cost of doing so.

      Your analogy about buying a new gas oven is interesting. You look at the manuals and there are *many* ways that you can blow up your oven. It is just that the risk (of someone naively or accidentally blowing themselves up) has been balanced against the benefit of lower consumption of energy. There are many ways of managing risks - redundancy, accepting the risk, etc.

      My prime point was that the benchmarking which yeilded questions - without the answers given - are extremely valuable. They allow the upstream people developing systems to understand that they need to consider the bigger picture and apply a risk/cost/benefit judgement and not close of all risks. I would expect that in later versions of SuSE they have turned off barriers now that the risk has been sufficiently understood and the costs determined as being commercially relevant.

      Or using your analogy. The tests that the oven may blow up but save 50% on the energy bill has been shown that the net benefit is on the side of the oven that may potentially blow up!

    7. Re:That is the question... by rackserverdeals · · Score: 1

      Or using your analogy. The tests that the oven may blow up but save 50% on the energy bill has been shown that the net benefit is on the side of the oven that may potentially blow up!

      How so? The average yearly cost of an oven is $42. You're telling me that you'd accept the risk of injury, possibly death to save $21 a year?

      From my experience, most people don't have a cage, or even a rack full of servers that can sustain the loss of a single server going down.

      Most don't even run at more than 40% utilization and the performance is not that important.

      What is important is that their system is reliable and that they don't have to waste time rebuilding it.

      The server and the operating system should just work. I'm more an application developer than anything else. That's were most of my time is spent. What I know about servers and operating systems I learned so that I can put my applications on something I don't have to worry about. A server being down means loss of revenue.

      In most cases, it's cheaper to buy a faster cpu, faster disks, use raid 10 than it is to suffer any significant downtime.

      --
      Dual Opteron < $600
  11. Boycott retards! by Anonymous Coward · · Score: 0

    Stop replying to stories where TFA is a click through.
    Boycott web designer retardation!

  12. Boot time is better. by Blice · · Score: 5, Interesting

    A lot of new code (and old code reformed) was added to try and speed up the boot process, I know that for sure. I saw some of the work Arjan did in the bootfast tree-

    fastboot: Asynchronous function calls to speed up kernel boot
    fastboot: make scsi probes asynchronous
    fastboot: make the libata port scan asynchronous
    fastboot: make ACPI bus drivers probe asynchronous
    fastboot: Make libata initialization even more async

    I don't know for sure that all of this made it upstream for this release but I know some of it did. I think you have to pass the "fastboot" kernel line for it, however. So check your kernel configs and update your grubs!

    Or LILOs, if you're weird...


    Oh one more thing.. I think the introduction of the asynchronous probing and various other things are going to start a whole new wave of bootfast tricks. For example, before it tries mounting the root file system and continuing on, it waits for device probing to finish. A comment above that code states "Waiting for device probing to finish... This is known to cause long delays in boots, for example this can take 5 seconds for a laptop's touchpad to initialize". The comment was written by Arjan, who obviously has intention to speed things up. So I think what might happen is instead of waiting for EVERYTHING to finish probing (Even if it is async), it'll just wait for the filesystem to become available (Perhaps try after IDE probes, then try after SCSI probes, then after USB, and so on.)

    I also remember there was a patch that didn't go upstream (I don't think so anyways) that added a function to be able to initialize things later on (After the boot was done). You changed the initialize() or whatever the function name was to initialize_later(), and then after you're done booting, whenever you want, you do a command and it then initializes anything you did the initialize_later() to. So you would be able to load up the webcam initialization or whatever else you know you don't use right when you boot.

    Well, where I'm going with this is that I would like to see them incorporate more of that stuff into the kernel. More boot hacks, more power saving, more efficiency. These things are only going to improve.

    1. Re:Boot time is better. by MichaelSmith · · Score: 2, Insightful

      Oh goody. A million bizarre race conditions.

      --
      http://michaelsmith.id.au
    2. Re:Boot time is better. by Blice · · Score: 2, Insightful

      They did it pretty good actually. They have a function that waits for certain things to sync up before continuing at places.

      And they *do* test the shit out of kernels before releasing, you know..

  13. Re:Why no comparison to OS X? by TheRaven64 · · Score: 1

    They can't exactly copy the ELF binaries and libraries straight over from Linux

    They can to *BSD or Solaris though, and that would make for an interesting comparison. Last time I saw anyone do this was a few years ago, and FreeBSD was around 5-10% faster running Linux binaries than Linux was, although I wouldn't be surprised if this has changed now.

    --
    I am TheRaven on Soylent News
  14. No, it's just Moronix by Anonymous Coward · · Score: 0

    Where they compare 5-year-old 32-bit Solaris x86 GCC to new 64-bit Linux GCC, then crow about how fast Linux is when compared to Solaris:

    Our build of Ubuntu 9.04 was depending upon the Linux 2.6.28 kernel, GNOME 2.25.5, X Server 1.6.0 RC 1, GCC 4.3.3, and we opted to use the EXT4 file-system. OpenSolaris 2008.11 is based upon Solaris Nevada Build 101b, uses the 5.11 kernel with X Server 1.3, GCC 3.4.3, and the ZFS file-system.

    Heaven forbid that Phoronix would benchmark an up-to-date version of Sun Studio compiler on Solaris. No, they use an old version of GCC.

    As I said, Moronix.

  15. Re:Why no comparison to OS X? by blitzkrieg3 · · Score: 1

    FreeBSD was around 5-10% faster running Linux binaries than Linux was, although I wouldn't be surprised if this has changed now.

    I would, actually. If history is any indication, the operating system gets slower as time goes on. Note that I actually have oprofiled some of these areas, and it's frequently due to things like processor ACPI enablement (processor sleeping) and security improvements like stack randomization.

  16. Benchmarks used x86-64 by chris-chittleborough · · Score: 1

    It was only on after reading the comments at Phoronix that I noticed the benchmarks used 64-bit (x86-64) kernels, not x86 as I had initially assumed. Maybe the use of kernel and compiler code that gets less testing than x86 is related to the odd performance quirks Phoronix found?