Open Source OS Benchmarking Competition

BenchmarkingFreak writes "OSnews is running a story about a new benchmarking competition: OSU Open Source Lab wanted to take the concept of benchmarking a little bit further with the Beaver Challenge 2004. In this competition they will be allowing a community of experts in each OS to tweak their configurations to ensure maximum performance. And they are running it all on wicked machines, just imagine... well you know."

Comments

Missing One?

[elid]

We have selected the following distributions. This list is not final and if people want to ante in to try this with their favorite distro, let us know at bc2004 at osuosl dot org or in #beaverchallenge on the Freenode.net IRC network.

* Debian GNU/Linux
* Fedora Linux
* FreeBSD
* Gentoo Linux
* NetBSD
* OpenBSD
* Red Hat Linux
* Slackware Linux
* SuSE GNU/Linux

Where's Mandrake?

Re:Missing One?

[Anonymous Coward]

What I want to know is where is Emacs :-)

Sex on the brain

[Jason Earl]

First a story about screws, now a story about beavers. Apparently the /. crew had a slow weekend.

What wil they be benchmarking?

[MysteriousMystery]

Will they be benchmarking database performance, GCC compiling speed, I took at look at the methodology page and it wasn't particularly specific.

A Cool Idea, But...

[Hornsby]

This is a really neat idea, and it's a long time coming; however, I wouldn't expect overly divergent results among Linux distributions. Afterall, they're all going to use the latest 2.4 and 2.6 kernel and comparable glibc versions(with maybe the exception of Debian), so the only speed difference should be in the compiler flags used to build the packages. I'm not trying to negate the coolness of this competition because it should give a good measure of performance between the BSD distros VS Linux distros, but don't be surprised when the Linux distros all show comparable results. As a footnote, I do expect Gentoo to come in the lead of the Linux distros having tried them all and found it the fastest in empircal testing...

Re:A Cool Idea, But...

[Anonymous Coward]

However, I wouldn't expect overly divergent results among Linux distributions.

Whoever puts their apps in the latter sectors of disk and uses the first sectors for writing will win by a margin of 2:1, provided no one else does the same. On the other hand, if one group makes use of a nice mfs and no one else does then that group might win by a much larger margin. Depending on the test, selective use of processors or placing swap in just the right space may make a big difference as well. Maybe it'll depend on which team gets a Hacker to, in 3 days, recode specific routines in assembly just for that processor setup.

In short, welcome to the real world of benchmarking: whichever team figures out how to bend the rules just right will win.
This kind of benchmark rule bending happens quite often when the big players get rfp's for large orders from bigger players, and ppl make lots of money figuring out how to bend the benchmark rules, even when those rules cover well over a hundred pages of specifications.

Re:A Cool Idea, But...

[Aardpig]

It wouldnt take much to offer "up an running in 10 minutes" iso's to people with a default set of apps, portage & tree etc

What, you mean like the 2-disk Live CD option offered by, erm, what's its name, Gentoo Linux? Sure, there's a bit of tweaking to do, but almost everything is precompiled. Too much for you? Then perhaps Gentoo's not for you.

Re:A Cool Idea, But...

[scotch]

Nice Troll. You had me going until this bit:

I do expect Gentoo to come in the lead of the Linux distros having tried them all and found it the fastest in empircal testing...

Beautiful.

Back a while, when gentoo was still had the smell of pop novelty, you would hear all this great stuff about how gentoo distros were the fastest, something about being able to specify --funroll-all-the-bad-loopies and --enable-r0xor-opts and --omit-random-instructions to the compiler. Of course, all these claims of gentoo's speed have never been backed up. On the contrary, the only results we've seen published tend to indicate that you average gentoo distro is composed of slower-than-average or average applications.

These days, we hear the new mantras of the gentoo-fanboys: it's not the speed (good thing!) that they use gentoo for, but instead the ease of use or robust package management or configuration flexibility. That's great and all, but it's all a bunch of green-is-my-favorite-color kind of advocacy: opinionated, unsupported, and unconvincing. People who've gone through the long laborious pain of installing gentoo (reminiscent of slackware 3.0 and libc upgrades, what year is it again?), and then having wasted the effort on a system that will probably spend more cycles compiling itself than serving the users, they justify the waste with a belief that their system is better managed or more finely tuned or whatever. Emphasis on whatever.

Of course, none of the supposed benefits of gentoo are backed with anything approaching rigorous analysis. Instead we get vague anecdotes and slashdot fanboyism. When we inevitably learn that the gentoo portage system is riddled with problems, conflicting package maintenance mechanisms and policy, broken and overtweaked package scripts, and that the whole thing needs a certain amount of voodoo to work, the gentoo boys will probably come up with some other reason why it is the one distro to rule them all.

The rest of will just wait for the results of your empirical studies with smiles on our faces!

Re:A Cool Idea, But...

[1lus10n]

"When we inevitably learn that the gentoo portage system is riddled with problems, conflicting package maintenance mechanisms and policy, broken and overtweaked package scripts, and that the whole thing needs a certain amount of voodoo to work"

You know this is one of the better descriptions of portage/Gentoo I have heard. If I had the time/resources I would re-write portage using a bette langauge and more sane feature set. Portage was a good idea, and is a HORRIBLE implementation. however it still beats RPM.

PS somebody mod the parent up, I would have modded you up, but I already posted to this topic.

Re:A Cool Idea, But...

[acidtripp101]

Advice: Read the HUNDREDS of posts on the gentoo forums about this.

The fact of the matter is that portage is plenty fast. Any speed boosts given to the actual emerge program set would be negligable because of the sheer amount of time dedicated to compiling.

More Advice: Stop trolling.

When you say that "Portage was a good idea, and is a HORRIBLE implementation" you really need to enumerate WHY it's a horrible implementation.
Take this common troll as an example.
Example 1: Windows is a HORRIBLE OS.
Example 2: Windows is a HORRIBLE OS because being locked into the choices Microsoft made in my "interests" are usually counter-productive.

See the difference? Example 1, while in many people's oppinion is valid, leaves people wondering why Windows is horrible. Example 2 gives anybody reading specific evidence and also allows anybody that wants to defend the point areas to do so.

I love the way that gentoo handles packages. I, admitingly, have a BSD bias, but it still allows my system to be what I want.
The feature set is anything but 'insane,' but once again, I have no idea why you think so, so I can't exactly defend that against any reasons you have.

Reply to this post and we might actually have some decent points to give to the gentoo team to make inprovements.

Re:A Cool Idea, But...

[Anonymous Coward]

1. Unless you're using a computer more than 10 years old, you're not going to be waiting for Portage to calculate dependancies for any significant amount of time. I wouldn't mind Gentoo developers optimizing Portage for speed, but only trolls or the most anal-retentive of users will be put off by it's "inefficiency".

2. RTFM. Gentoo docs are very easy to follow, and it only needs to be done once. I'm sure that typing in a command or two is not going to kill you.

3. Emerge, as far as the end user is concerned, is the Gentoo package manager. The scripts that power Portage are completely behind the scenes. I've been using Gentoo since it was an early beta, and I've never had to go in and modify a Portage script in order to get it to work.

4. /usr/portage; /var/tmp/portage

Yep. Really disorganized.

5. Several options:
a.) tell portage to ignore the dependancy and install anyway
b.) remove java functionality from db
c.) install db manually and inject the package into portage

6. Gentoo is a source-based distro. It doesn't have very many binary packages, nor does it claim to. If binaries are so important to you, Gentoo offers a physical product that you can buy with all major packages precompiled for all major CPU architectures.

7. I'll give you this one.

It's not performance optimized.

[MysteriousMystery]

Mandrake is more of a "user friendly" distro then a performance optimized distribution. Someone might add Mandrake to the list, but it's not as tightly configured as say Gentoo and I can't imagine what purpose it would be to add it to the competition except for just representation.

Re:It's not performance optimized.

[zenyu]

I've never used Mandrake but didn't it start out as a version of RedHat compiled for the Pentium instead of the 486?

First it was based off debian. The real motivation was KDE. Of course the first Mandrake I used was based off Redhat and I used Gnome. It was compiled for i586, but more importantly for me as a developer, it had more up to date packages. This meant I only had to upgrade the components I was working on and not every library it depended on. I still use Mandrake at work, but at home where I do more tweaking I use Gentoo, which is too bleading edge for day-to-day, but great for developing for newish hardware. I tried Gentoo on my laptop, but while it was faster, it was too much work to keep up to date, so I went back to Mandrake. My main box at home just runs a script every night to keep up to date, it wouldn't be a big deal if it were rooted so I sleep well at night.

Beaver?

[Omni Magnus]

If they really wanted to make it a good competition, they would award beaver to the winner. That would get them fired up.

Better than what OSNews has been doing

[El Volio]

Given OSNews' recent penchant for poorly-done benchmarks (e.g. 1, 2), I'm glad to see them run an article about someone else's (hopefully well-done) testing. By having expert teams who know what they're doing tweak the configurations, this should be a much more representative result. Hopefully OSNews will learn some methodology from these guys...

Why not include Windows and perhaps others?

[enosys]

Why not include Windows and perhaps others? I guess they wouldn't include non-open-source ones because it's a site about open source but I'd love to see the comparison. Have any other sites done that?

Pity Windows is not included

[Mostly a lurker]

Comprison of various Linux distros (and of the 2.4 versus 2.6 kernel) is interesting. However, what is really lacking is an objective comparison of MS Windows Server 2003 versus Linux. I know Microsoft tries to prevent such benchmarking, but can they really enforce such a ban? It ought to be possible to find a team of Windows experts to tune Windows so the comparison is fair. Why not?

Linux Junkie:

[iammaxus]

This is clearly a plot by Microsoft and SCO to destroy OSS! They couldn't beat any one of us, but if they get us to fight each other...

Dude, where's my brain?

[SuperBanana]

Where's Mandrake?

Dude, I know you're not supposed to read the story, but at least read the text you quoted.

This list is not final and if people want to ante in to try this with their favorite distro, let us know at bc2004 at osuosl dot org or in #beaverchallenge on the Freenode.net IRC network.

Note what's implied...

• The list is not final
• Someone needs to carry the banner for each distro(ie, someone who knows the distro needs to set it up properly etc).
• You can volunteer your favorite distro AND volunteer to be the guy carrying the banner.
• They give you both an email addr and an IRC channel to discuss all of this

Beaver Challenge 2004

[tfoss]

I'm just sure there's a joke here somewhere.

-Ted

I don't see Darwin

[droleary]

What's with the x86- and Linux-centric approach? Do we really need to see how 6 different distributions can be tweaked to behave like one another on the same $4300 piece of hardware? I'd be extremely interested to see a G5 Xserve entered into that mix, although you'd clearly have to add some unnecessary doo-dads to the Mac to bring the price over $4000 (even with hardware RAID and the inability to drop below an 80GB HD to the 18GB like the Dell has, I could only bring a single processor Xserve up to $3500). Include a PPC Linux or two while you're at it. As it stands, the results will probably be at least a 6-way yawn-fest.

Include other OSes

[Quixote]

They should include other OSes, with the stipulation that all tweaks, etc. done by the engineers be well-documented and reproducible.

Recycle Bin Performance?

[Gothmolly]

"In this competition they will be allowing a community of experts in each OS to tweak their configurations to ensure maximum performance."

I can see it now, teams of KDE and GNOME developers going head to head to see who can come up with the best color scheme, antialiased fonts, and 'Are you sure you want to delete this?' dialog box. Followed by Round 2, where each group has to compile something built for the other camp's desktop, whoever can fight through the dependencies quicker wins!

Lord Linus, save us from OSNews.

What?

[blair1q]

just imagine... well you know.

What?

Just imagine what?

A Beowulf cluster of Beavers?

Hyper Threading

[SunBug]

Will they let you tweak Hyper Threading?

It'll be interesting to see how many people turn Hyper Threading OFF when doing some tests. I found that my database was 212% FASTER for read operations after I turned Hyper Threading off on the 2650.

Re:Hyper Threading

[Graelin]

If you enable HT, you are cutting your L3 cache in half per "processor." So your 2.5Ghz Xeon with 512k cache turns into two 2.5ghz chips with 256k each.

We have deployed a few 6650s here, these are Quad Xeons with 1 meg of cache, and it's amazing to look at top reporting 8 procs. But it didn't take long before we saw the same thing you have.

But it depends on your usage patterns too. If you're serving a lot of small requests - that run very quickly - HT may not be a bad idea. OTOH, running fewer and larger requests would certainly benefit from disabling HT.

The reason is that database servers can take advantage of large cache sizes more than most apps. They can move a lot of the dataset near the proc and cut down on query times dramatically. Less cache, more RAM accesses, slower queries.

About a year ago Dell was recommending that HT be disabled for better performance. Not sure if that is still the case today.

Re:Hyper Threading

[slamb]

If you enable HT, you are cutting your L3 cache in half per "processor." So your 2.5Ghz Xeon with 512k cache turns into two 2.5ghz chips with 256k each.

Interesting. That's not the reason for disabling HyperThreading that I've heard. I often hear people say it should be disabled unless you have a scheduler that supports HyperThreading well. There are lots of opportunities to go wrong when scheduling tasks on HT-enabled CPUs.

For example, if you have one real processor and are running a high-priority task and a low-priority task, the low-priority task will get 50% of the processor time with a non-HT aware scheduler, since it says "well, I've got this processor free, so I might as well use it" when that's not really true. This problem is discussed more here.

Similarly, if you've got several high-performance tasks and several real processors, you want to spread them out across as many real processors as possible to maximize parallelism.

Blatantly Rigged Contest

[Anonymous Coward]

This contest is blatantly set up so that Gentoo, the obvious victor in a fair benchmarking contest, cannot win.

They are only allowing three days to set up the OS, everyone knows that you can't get gentoo installed, much less customized in that time.

Re:Once and for all

[dilvish_the_damned]

I guess your feeling kind of angry over gentoo's general acceptance among some geek crowds. I assume you have your reasons.
But I truelly believe that gentoo is what slackware once was, i.e. a distro that was forgiving of modifications, and it gives a pretty interface for doing so.
Its tedious to set up, but once initial install is done, its almost painless for the life of the machine. To me, the theoretical (or proven) performance advantages are almost secondary, in most applications, to portages forgiving nature.. Its just so easy to administer.
You would argue against the 'Gentoo-zealots' having no discernable advantage performance wise... And then suggest that portage is the only advantage Gentoo(zeolots) have (infered as I read it) over slackware.
A fact to consider: Optimized binaries generally run faster than unopimized ( an unqualified 3% - 15%, got the charts to prove it).
By the fact that portage is being ported to slackware, I assume your chosen distro, and by the fact that you mention it here, means that portage is important in your eyes. I have long held that the defining factor of any distro is its chosen package managment system (excusing directory layouts).
So in a way, isnt slackware becoming more like gentoo in effect? I mean after all, we are all dealing with the same fucking code with some minor tweaks and major package maintainance differences.
Oh, by the way. Gentoo is faster. Its going to kick Slackewares ass.
Noted Debian and 'apt-get' fan