Slashdot Mirror
Open Source OS Benchmarking Competition
BenchmarkingFreak writes "OSnews is running a story about a new benchmarking competition: OSU Open Source Lab wanted to take the concept of benchmarking a little bit further with the Beaver Challenge 2004. In this competition they will be allowing a community of experts in each OS to tweak their configurations to ensure maximum performance. And they are running it all on wicked machines, just imagine... well you know."
Will they be benchmarking database performance, GCC compiling speed, I took at look at the methodology page and it wasn't particularly specific.
Whoever puts their apps in the latter sectors of disk and uses the first sectors for writing will win by a margin of 2:1, provided no one else does the same. On the other hand, if one group makes use of a nice mfs and no one else does then that group might win by a much larger margin. Depending on the test, selective use of processors or placing swap in just the right space may make a big difference as well. Maybe it'll depend on which team gets a Hacker to, in 3 days, recode specific routines in assembly just for that processor setup.
In short, welcome to the real world of benchmarking: whichever team figures out how to bend the rules just right will win.
This kind of benchmark rule bending happens quite often when the big players get rfp's for large orders from bigger players, and ppl make lots of money figuring out how to bend the benchmark rules, even when those rules cover well over a hundred pages of specifications.
Why not include Windows and perhaps others? I guess they wouldn't include non-open-source ones because it's a site about open source but I'd love to see the comparison. Have any other sites done that?
Comprison of various Linux distros (and of the 2.4 versus 2.6 kernel) is interesting. However, what is really lacking is an objective comparison of MS Windows Server 2003 versus Linux. I know Microsoft tries to prevent such benchmarking, but can they really enforce such a ban? It ought to be possible to find a team of Windows experts to tune Windows so the comparison is fair. Why not?
What's with the x86- and Linux-centric approach? Do we really need to see how 6 different distributions can be tweaked to behave like one another on the same $4300 piece of hardware? I'd be extremely interested to see a G5 Xserve entered into that mix, although you'd clearly have to add some unnecessary doo-dads to the Mac to bring the price over $4000 (even with hardware RAID and the inability to drop below an 80GB HD to the 18GB like the Dell has, I could only bring a single processor Xserve up to $3500). Include a PPC Linux or two while you're at it. As it stands, the results will probably be at least a 6-way yawn-fest.
Will they let you tweak Hyper Threading?
It'll be interesting to see how many people turn Hyper Threading OFF when doing some tests. I found that my database was 212% FASTER for read operations after I turned Hyper Threading off on the 2650.
I know benchmarks are cool and all, but I'd much rather see which distro performed well on older machines, like Pentiums and Pentium IIs. I have so many of these to deal with, and most everything sucks. Funny, cuz I remember running Solaris x86 and OS/2 and thinking they were excellent (sort of). Now, everything seems bloated. Of course, I'm expecting too much...
I can pretty much tell you who's going to be the fastest and slowest without a doubt. Gentoo fastest OpenBSD slowest (I don't even use Gentoo btw). All those are for the most part general purpose distros meant to run on a lot of hardware. They are not meant to run as fast as possible. They basically are compiled -O -march=i386. Gentoo being the exception since it's a source compiled distro and are going to tweak the hell out of the cc flags. I know this is going to make a lot of people with distro partialness mad, but general purpose distributions were not meant for speed. Gentoo wasn't designed with ease in mind, it was designed for speed.
As for OpenBSD, it's focus is on security. From personal experience, it's noticibly slower than any *nix I've ever used. Not to say it's bad, it's just not built for speed.
I think that list needs to be reworked considerbly. Maybe put them into categories such as source compiled distros, general purpose distros, security focused distros. Because comparing a source distro to a general purpose one is no contest. Not to say speed is everything, but it's really apples an oranges. Comparing one source distro to many general purpose ones is like racing a porshe and a honda. The porshe is going to win, but that's not what Hondas are built for.
I guess your feeling kind of angry over gentoo's general acceptance among some geek crowds. I assume you have your reasons.
But I truelly believe that gentoo is what slackware once was, i.e. a distro that was forgiving of modifications, and it gives a pretty interface for doing so.
Its tedious to set up, but once initial install is done, its almost painless for the life of the machine. To me, the theoretical (or proven) performance advantages are almost secondary, in most applications, to portages forgiving nature.. Its just so easy to administer.
You would argue against the 'Gentoo-zealots' having no discernable advantage performance wise... And then suggest that portage is the only advantage Gentoo(zeolots) have (infered as I read it) over slackware.
A fact to consider: Optimized binaries generally run faster than unopimized ( an unqualified 3% - 15%, got the charts to prove it).
By the fact that portage is being ported to slackware, I assume your chosen distro, and by the fact that you mention it here, means that portage is important in your eyes. I have long held that the defining factor of any distro is its chosen package managment system (excusing directory layouts).
So in a way, isnt slackware becoming more like gentoo in effect? I mean after all, we are all dealing with the same fucking code with some minor tweaks and major package maintainance differences.
Oh, by the way. Gentoo is faster. Its going to kick Slackewares ass.
Noted Debian and 'apt-get' fan
I think you underestimate just how much I just dont care.
I've never used Mandrake but didn't it start out as a version of RedHat compiled for the Pentium instead of the 486?
First it was based off debian. The real motivation was KDE. Of course the first Mandrake I used was based off Redhat and I used Gnome. It was compiled for i586, but more importantly for me as a developer, it had more up to date packages. This meant I only had to upgrade the components I was working on and not every library it depended on. I still use Mandrake at work, but at home where I do more tweaking I use Gentoo, which is too bleading edge for day-to-day, but great for developing for newish hardware. I tried Gentoo on my laptop, but while it was faster, it was too much work to keep up to date, so I went back to Mandrake. My main box at home just runs a script every night to keep up to date, it wouldn't be a big deal if it were rooted so I sleep well at night.
If one of the reasons to run the benchmark is to show people how great F/OSS is, "round 2" seems like a natural. After all, don't we all know some people who have a couple of heavy paperweights sitting around? Wouldn't more people be willing to try something other than Windows if they saw that their old crappy box ran just as fast as their new shiny WinXP box?
okay if you say so:
/etc) in that one location. or branched off of it.
1) Python should not be used for a package management tool. It has to much process time involved in processing dependancies. (as an addendum python itself is DAMN slow when compared to a compiled language. google for it, the I/O handling on python is sub-par when compared to C/C++)
2) no useful installer.
3) to many damn scripts. send a coder to do the job you get one modular peice of code and a few libraries. send a scripter and you get hundreds of little scripts scattered from here to hell, that no other distro uses. Unix failed to penetrate the desktop market because of this vary thing. X users do A+B+C, y users do C+G+Y. Its moronic.
4) portage design. Why the hell is some of the stuff here and some of the stuff there ? why not make a central location put everything ('cept configs which belong in
5) the dependancy checking is half assed, I try to install db, db requires java, but i already have java installed (via portage) but it doesnt like HOW portage installed java, so the build dies complaning about needing some java item. (pick something, i have seen this with many different things: java include files, jar program, jvm etc...) this happens with many different things.
6) binary packages. The feature is there, yet they dont host any binary packages on there severs.
7) comments from ebuilds shouldnt be spewed out during an emerge, let alone an emerge world. they should be passed to a child proccess that retains the info and spits it out when the parent exits. or something along those lines.
Advice: realize that portage isnt fast, armak aint your daddy (neither is daniel) and the distro went to shit somewhere between 1.0 and 1.4.
And I wasnt trolling, Trolling is like this:
Gentoo blows small monkey's named fred, and its userbase rivals windows user base in common savvy.
"Two things are infinite: the universe and human stupidity; and I'm not sure about the the universe." --Albert Einstein
In short, welcome to the real world of benchmarking: whichever team figures out how to bend the rules just right will win.
Very interesting. It would be cool to have a benchmark with five times exactly the same distro, except the competing teams don't know this.
It would be nice to see if the differences achieved were comparable to what the differences between distros will be in this test.
I believe posters are recognized by their sig. So I made one.