Phoronix Releases Linux Benchmarking Platform

KernelPie writes "The Linux hardware site Phoronix.com has announced the release of Phoronix Test Suite 1.0, a Linux-based testing platform designed for benchmarking software and hardware. This suite ships with 57 tests and 23 test suites, which contain everything from open-source games to file encryption to encoding software. In addition, they have a global database where users can submit benchmark results and more — with over 1,000 submissions already. This testing software is licensed under the GPLv3 and is available for download."

Comments

FINALLY

[PunkOfLinux]

A proper benchmark program for linux. Trying it now :D

Re:FINALLY

[Anonymous Coward]

I know this is a lame ass question and I'm prepared to take my lumps but can someone tell me how to get this started? So far I've downloaded it, installed it and did the $ phoronix-test-suite install-all to install the tests. I haven't gotten much past that. What next?

Re:FINALLY

[commanderfoxtrot]

It's taken a looong time to get here!

Many distributions already have some form of automatic test suites to check for broken/non-broken packages. Now we can run performance tests automatically as well.

This will be fantastic for seeing performance regressions in the code, maybe for every check-in.

Every Ubuntu etc developer should have a VMWare guest running this continuously...

Re:FINALLY

[ChuckDriver]

For my money hdparm -Tt is the best benchmarking tool.

Hooked on Phoronix

[GigaHurtsMyRobot]

Worked for me!

The future holds e-penii and flamewars.

[pwnies]

If this testing suite becomes decently popular, I predict a future full of flamewars about which distros are faster.

Re:The future holds e-penii and flamewars.

[powerlord]

If this testing suite becomes decently popular, I predict a future full of flamewars about which distros are faster.

Considering how many Linux Users have built their own machines, I expect a Flamewar over parts AND OSes.

"My Frankenstein 9000 can outperform your measly Borg 100!"

Re:The future holds e-penii and flamewars.

[QUILz]

Oh don't worry, I'm sure we all know Linux from Scratch is the fastest.

Re:The future holds e-penii and flamewars.

[Anonymous Coward]

> I'm sure we all know Linux from Scratch is the fastest.

CFLAGS="-O0 -mcpu=386"

Re:The future holds e-penii and flamewars.

[Anonymous Coward]

bu-dum-boom-chi!

Re:The future holds e-penii and flamewars.

[RiotingPacifist]

Its also the slowest tho

Re:The future holds e-penii and flamewars.

[krischik]

This is indeed a sad truth. I noticed a huge performance increase when I moved to x86_64 - an I think most of it is due to CFLAGS="-O0 -mcpu=586" for the 32 bit SuSE.

But it does not stop there - recompile parts of Cygwin or MinGW with "-O3 -mcpu=pentium4" and you will notice an significant as well.

And - letting the Apple fan boy out - I believe that part of Apples impressive performance is due to the fact that Apple knows which Hardware there system is going to run on and that they can tweak -mcpu= accordingly.

Of course this could be duplicated with Linux as well. The future will be a return to vertical vendors (like Apple) which produce hardware, operating and base software and can optimize how those three parts interact. And the OS will most likely be and optimised Linux.

Martin

Re:The future holds e-penii and flamewars.

[Bert64]

It makes sense, having read some of the hackintosh forums, it seems the core of OSX uses SSE3 instructions because no intel based macs use a processor older than that.

By contrast, binary based linux distros typically set a minimum hardware platform a lot further back, 386 or 486 perhaps... Consequently, they can't take advantage of features present in modern CPUs, which could explain the recent ubuntu vs xp benchmarks where media processing (something SSE features in modern cpus are designed for) was faster on xp...

Windows sits somewhere between the 2 extremes, it has multiple codepaths for certain performance critical areas (eg video routines/drivers), where it uses the sse version if it can... This is fairly common among closed source programs, but very rare in the open source world because you can configure the cpu at compile time.

The problem is a lot less with 64bit, because the oldest possible x86_64 system is an athlon64, which has far more in common with a more modern cpu than a 386 does... Tho this will change in time.

The same issue hit apple too, they used to have multiple codepaths for G3/G4 (altivec mostly), and there is talk of the same thing happening with 64bit x86_64 too.

I'm not sure what the answer is... It would take up a lot of space and effort to have different binaries for different processors.. It's possible to do it with some critical pieces of code (kernel, glibc, x11, media/crypto programs, games) but the relatively minor benefits from recompiling everything do add up, despite what anti-gentoo people say.

People might try stupid things with gentoo, like -O9 -funsafe_untested_optimizations etc, but just setting -O2 or -Os and the CPU type helps a lot, -Os can actually be faster if you're short of memory or have a small cpu cache.

Source distributions solve the problems, but cause some of their own too... What about a hybrid system, that recompiles itself at low priority in the background without any user intervention.

Re:The future holds e-penii and flamewars.

[kitgerrits]

This is what made Mandrake such a popular RedHat clone back in the day.
Same software, only compiled for pentium.

Statistics

[Meneth]

Damn lies aside, their website needs graphs, averages and stuff. It's not very useful when you have to browse through hundreds of individual benchmarks.

Re:Statistics

[Anonymous Coward]

A graph? A GRAPH?! That sounds like a graphical representation to me. This is for Linux you noob, everyone knows text is the only way to interface with anything.

Re:Statistics

[martinw89]

Yeah, regarding the "damn lies", looking through the information it looks like it wouldn't be too difficult to submit a fake result. Does anyone have more information about validation? I couldn't find anything saying it validates results.

Re:Statistics

[setagllib]

Which is pretty reasonable because there's no way to validate a benchmark submission like that. Professional benchmarkers actually go on-site to your server and certify their work in person. That's why nobody bothers with professional benchmarkers.

Re:Statistics

[Anonymous Coward]

While I admit to having not looked in great detail at their website I did see a number of results that simply say how long a test took to run. When I'm doing any form of benchmarking I want to know what is happening during the testing and that's why I always run collectl at the same time as any tests I may be running since it tells a much more complete story of what your system is really doing. It can interactively show performance data on most system components, log the data to a file for later review and even generate output in a form suitable for passing directly to gnuplot. This also allows one to save the results for later comparison with other tests.

Virtualiztion

[debatem1]

Virtualization should heat up on this news. I'm off to do a virtualbox-vmware head-to-head

Re:Virtualiztion

[4pins]

Please let us know how that turns out.

Re:Virtualiztion

[sirmonkey]

+1, actully heck this just awoke a sleeping linux geek..... sorry race car, its hot out and there is something new for the pc(s) :-D

PHP?

[Anonymous Coward]

It's written in PHP. I don't know anything about PHP other than it's blamed for something like a third of all webserver vulnerabilities. Is there a security risk in installing these dependencies?

Re:PHP?

[armanox]

PHP is just a scripting language, like PERL. The security issues come in with using it to handle web pages. (Although, I guess ASP.NET, PERL, C++, Java, Python, etc; all have that problem too).

Re:PHP?

[Bert64]

PHP is just a language, it's just as possible to write a vulnerable web application in C, ASP.NET, Java or Perl etc... PHP just happens to be very popular among the people writing webapps, because it's free, easy to learn, widely used and widely available from cheap webhosting companies.

PHP on the other hand has various features designed to mitigate web vulnerabilities, such as magic quotes, that some other languages lack. Tho these features can (and often are) turned off, while newbie coders often rely on them working.

So part of it's problem is that it's easy to pick up and learn, so a lot of people can create apps and often release them to the public, but you don't really want to be running apps written by an inexperienced coder...

Of more use than benchmarks is...

[AmiMoJo]

Of more use than benchmarks is the fact that there is finally a good Live CD for stressing your system. Very useful when building new PCs or overclocking.

Yahoo

[Anonymous Coward]

I was surprised today when browsing through Yahoo News! and Topix that I stumbled across this PR from phoronix. First time I witnessed a opensource project get a news announcement across the wire like that. Effective marketing strategy or just a piggy bank?

Comparisons

[Bert64]

The site doesn't seem to offer a good way to compare systems side by side...
You also need a lot of information to accurately compare linux benchmarks, not only the hardware configuration but also the compiler version and flags used to compile the kernel, test programs and dependant libraries.

For that reason, i'd like to be able to compare...
The same distro/programs on different hardware
Different distros / compilation options on the same hardware

I'm also curious about the speed and size of code output by various versions of GCC, from my limited testing 3.4.6 was much faster than 4.0 and 4.1, faster in *some* areas than 4.2 but gcc 4.3 was generally faster overall.

Re:Comparisons

[PunkOfLinux]

For this test, I believe that all tests ARE run under the same conditions (same flags) on every machine.

Re:Comparisons

[Bert64]

Same flags maybe, but not the same compiler version, nor the same flags used when compiling the kernel or the libs the benchmark programs link against..

Vertical Vendors

[krischik]

For Apple I suggest you read those two wikipedia entries:

http://en.wikipedia.org/wiki/Universal_binary
http://en.wikipedia.org/wiki/Bundle_(NEXTSTEP)

Apart from that a vertical vendor which compiles a custom made Linux optimized for his hardware won't have all those problems. If well done such a system would outperform most of what you can buy today.

Note that Linux is already quite successfully in this area. Think of set-top phones, mobile phones etc. pp.

Martin

Re:Vertical Vendors

[Bert64]

Yeah, apple do it but only for some performance critical programs... I believe Leopard was compiled for G4 and above only tho, while the x86 version is compiled for SSE3+...

A custom linux optimized for the hardware does make sense, but how long will that vendor want to keep maintaining it? A piece of hardware sold today will be obsolete in a year, and warrant it's own optimized distribution. If people want to keep using the hardware, do they run sub optimal binaries on it (which will make the already seemingly slow due to age hardware seem even worse) or stick with old software?