Slashdot Mirror
Free/Open Cross-Platform SPEC Substitutes?
ErnstKompressor asks: "In light of the endless debate regarding top-dog performance between all flavors of OS/Hardware, the failings inherent in the various methods of testing such diverse systems, and the cost of existing software solutions, I was wondering what the Slashdot community recommends in the way of free, open-source, cross-platform, performance evaluation software. Additionally, how should one go about testing systems to obtain the fairest results? What compilers are evenly matched amongst different hardware? What balance should one strive for regarding optimizations and platform specific enhancements versus results that represent real-world performance? Finally, should such tests take into account the sub-systems available, such as 3D performance and the various Quake-FPS metrics?"
It's a Sourceforge project and slated to be GPL.
The goal is to produce a valid, relevant, cross-platform benchmark that doesn't cost thousands of dollars.
Just compile the Linux kernel.
Go here to create your own Slashdot dis
LMbench is great for testing the subsystems of different UNIX systems. It is probably one of the most useful benchmarks other than whatever applications you will actually be running.
"For the Snark was a Boojum, you see." -From the Hunting of the Snark: An Agony in Eight Fits, by Lewis Carroll
The biggest problem with most benchmarks is that they try to reduce system performance to a single number. A good benchmark would produce a set of numbers, one for each subsystem such as integer performance, floating-point performance, logic performance, 2D graphics performance, and 3D graphics performance.
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
The HINT benchmark developed at Iowa State University was a fantastic multi-dimensional benchmark.
I think the code has been taken down from the web site and it's not maintained anymore but if you want the code. I can get it for you. It's GPLed so I can redistribute it. Unfortunately I can't login here, but my slashdot ID is "dlakelan" so use the email for that ID to contact me.
All benchmarks are fair unless you deliberately picked the criteria to favor one candidate over the other, or overgeneralize the collected results beyond reason. The bigger question is whether they are representative or real performance in real application.
What compilers are evenly matched amongst different hardware?
Compilers are not evenly matched among different source code, nevermind different hardware. I don't see how you can possibly achieve this, especially with a free compiler where developer interest and expertise (which is mostly beyond your control) direct how well it will perform on each platform.
Finally, should such tests take into account the sub-systems available, such as 3D performance and the various Quake-FPS metrics?
Depends on what you're testing. If it's a CPU versus CPU comparison, perhaps not. If it's a system versus system comparison, definitely.
I would suggest taking code from various free projects. For example:
- use Gecko or KHTML to render some known complicated web pages
- use POV-Ray to render some complex scenes
- use GIMP graphics filters to modify big images
You can also start with a CS data structures book, and implement various basic and well-known algorithms and apply them to big data sets.What is it that you want to be able to say when you're done testing? And who do you want to say it to? By defining some of this at the start the pieces will more readily fall into place for you. There are a lot of tests suites out there now already and you should be able to find them once you know what you want to measure and what you want to report. This might be a good place for you to start: http://opensourcetesting.org/performance.php
http://tinyurl.com/3t236
Bogomips?
http://www.tldp.org/HOWTO/BogoMips.html
I live in Soviet Canuckistan you insensitive clod!
That is kinda what I was aiming at...Something like SPEC2000, but free/open source(a good cross-platform capable suite) or like XBench for OS X(already free, but not cross platfrom). Both types of testing progs seem to give well-rounded complex test results.
I guess the meat of the question regards the dispute over using different compilers in relation to comparative testing...With the recent G5 debate focusing on the fairly noted use of GCC on both platforms instead of an intel-specific compiler, leading to lower SPEC scores for the Dell systems tested, I wonder how such discrepancies could be avoided in the pursuit of fairer/leveled testing scenarios...
We apologise for the fault in this post. Those responsible have been sacked. -- Signed RICHARD M. NIXON
It is hard to find a set of applications capable of testing every aspect of computer performance. Starting from scratch it demands a lot of time (and money) to make the applications portable enough to run on different architectures and operating environments (including compilers, filesystems, etc.) Then a database containing the results must be mantained for future reference. Only in these conditions the benchmark programs can be considered standard, it is perfectly justified to sell it for money. In conclusion, benchmarking is a task for especialists, it should be performed by computer and software makers using a (rapidly evolving) set of standard applications.