Slashdot Mirror
Open Source OS Benchmarking Competition
BenchmarkingFreak writes "OSnews is running a story about a new benchmarking competition: OSU Open Source Lab wanted to take the concept of benchmarking a little bit further with the Beaver Challenge 2004. In this competition they will be allowing a community of experts in each OS to tweak their configurations to ensure maximum performance. And they are running it all on wicked machines, just imagine... well you know."
Will they be benchmarking database performance, GCC compiling speed, I took at look at the methodology page and it wasn't particularly specific.
Whoever puts their apps in the latter sectors of disk and uses the first sectors for writing will win by a margin of 2:1, provided no one else does the same. On the other hand, if one group makes use of a nice mfs and no one else does then that group might win by a much larger margin. Depending on the test, selective use of processors or placing swap in just the right space may make a big difference as well. Maybe it'll depend on which team gets a Hacker to, in 3 days, recode specific routines in assembly just for that processor setup.
In short, welcome to the real world of benchmarking: whichever team figures out how to bend the rules just right will win.
This kind of benchmark rule bending happens quite often when the big players get rfp's for large orders from bigger players, and ppl make lots of money figuring out how to bend the benchmark rules, even when those rules cover well over a hundred pages of specifications.
Why not include Windows and perhaps others? I guess they wouldn't include non-open-source ones because it's a site about open source but I'd love to see the comparison. Have any other sites done that?
Comprison of various Linux distros (and of the 2.4 versus 2.6 kernel) is interesting. However, what is really lacking is an objective comparison of MS Windows Server 2003 versus Linux. I know Microsoft tries to prevent such benchmarking, but can they really enforce such a ban? It ought to be possible to find a team of Windows experts to tune Windows so the comparison is fair. Why not?
What's with the x86- and Linux-centric approach? Do we really need to see how 6 different distributions can be tweaked to behave like one another on the same $4300 piece of hardware? I'd be extremely interested to see a G5 Xserve entered into that mix, although you'd clearly have to add some unnecessary doo-dads to the Mac to bring the price over $4000 (even with hardware RAID and the inability to drop below an 80GB HD to the 18GB like the Dell has, I could only bring a single processor Xserve up to $3500). Include a PPC Linux or two while you're at it. As it stands, the results will probably be at least a 6-way yawn-fest.
Will they let you tweak Hyper Threading?
It'll be interesting to see how many people turn Hyper Threading OFF when doing some tests. I found that my database was 212% FASTER for read operations after I turned Hyper Threading off on the 2650.
I guess your feeling kind of angry over gentoo's general acceptance among some geek crowds. I assume you have your reasons.
But I truelly believe that gentoo is what slackware once was, i.e. a distro that was forgiving of modifications, and it gives a pretty interface for doing so.
Its tedious to set up, but once initial install is done, its almost painless for the life of the machine. To me, the theoretical (or proven) performance advantages are almost secondary, in most applications, to portages forgiving nature.. Its just so easy to administer.
You would argue against the 'Gentoo-zealots' having no discernable advantage performance wise... And then suggest that portage is the only advantage Gentoo(zeolots) have (infered as I read it) over slackware.
A fact to consider: Optimized binaries generally run faster than unopimized ( an unqualified 3% - 15%, got the charts to prove it).
By the fact that portage is being ported to slackware, I assume your chosen distro, and by the fact that you mention it here, means that portage is important in your eyes. I have long held that the defining factor of any distro is its chosen package managment system (excusing directory layouts).
So in a way, isnt slackware becoming more like gentoo in effect? I mean after all, we are all dealing with the same fucking code with some minor tweaks and major package maintainance differences.
Oh, by the way. Gentoo is faster. Its going to kick Slackewares ass.
Noted Debian and 'apt-get' fan
I think you underestimate just how much I just dont care.