Slashdot Mirror
Open Source OS Benchmarking Competition
BenchmarkingFreak writes "OSnews is running a story about a new benchmarking competition: OSU Open Source Lab wanted to take the concept of benchmarking a little bit further with the Beaver Challenge 2004. In this competition they will be allowing a community of experts in each OS to tweak their configurations to ensure maximum performance. And they are running it all on wicked machines, just imagine... well you know."
* Debian GNU/Linux
* Fedora Linux
* FreeBSD
* Gentoo Linux
* NetBSD
* OpenBSD
* Red Hat Linux
* Slackware Linux
* SuSE GNU/Linux
Where's Mandrake?
First a story about screws, now a story about beavers. Apparently the /. crew had a slow weekend.
Will they be benchmarking database performance, GCC compiling speed, I took at look at the methodology page and it wasn't particularly specific.
It will be decided once and for all that Gentoo offers no discernable advantage over more conventional distros. At which point, Gentoo-zealots will only have portage to pull in converts.
For me, that's enough. Portage doesn't have *half* the dependency problems of apt, or up2date. I was a long time RedHat user, and I've tinkered with Debian here and there (can't stand it actually). But now I'm a certified Gentoo Zealot because of portage alone.
"Ignorance more frequently begets confidence than does knowledge"
- Charles Darwin
This is a really neat idea, and it's a long time coming; however, I wouldn't expect overly divergent results among Linux distributions. Afterall, they're all going to use the latest 2.4 and 2.6 kernel and comparable glibc versions(with maybe the exception of Debian), so the only speed difference should be in the compiler flags used to build the packages. I'm not trying to negate the coolness of this competition because it should give a good measure of performance between the BSD distros VS Linux distros, but don't be surprised when the Linux distros all show comparable results. As a footnote, I do expect Gentoo to come in the lead of the Linux distros having tried them all and found it the fastest in empircal testing...
A musician without the RIAA, is like a fish without a bicycle.
Mandrake is more of a "user friendly" distro then a performance optimized distribution. Someone might add Mandrake to the list, but it's not as tightly configured as say Gentoo and I can't imagine what purpose it would be to add it to the competition except for just representation.
If they really wanted to make it a good competition, they would award beaver to the winner. That would get them fired up.
Given OSNews' recent penchant for poorly-done benchmarks (e.g. 1, 2), I'm glad to see them run an article about someone else's (hopefully well-done) testing. By having expert teams who know what they're doing tweak the configurations, this should be a much more representative result. Hopefully OSNews will learn some methodology from these guys...
"You can never have too many elephants on your team."
The only way to get real world benchmark results would be to distribute the benchmarks setiathome style and then do some funky statitical analysis. But you'd also need a way to verify the actual hardware used for each test.
Otherwise, we still end up with data that needs to be heavily interpreted to get any take on what will happen in production on any given hardware.
Oh yeah, not to mention that for real world production, performance is also dependant on maintainability, uptime and a variety of other factors.
Still, this will result in a bunch of pretty graphs and nice rants and raves and dick waving for years to come, even though the data will only be good for one given point in time, for one given hardware platform, for one given configuration.
If Linux wins, the BSD snobs will have a field day babbling and flaming about how unfair the contest was.
Sure buddy. And if Santy Claus comes a-jumping down the chimney, you'll get the BB gun you always wanted.
I don't think it's all about a winner or a loser. As some others have pointed out each distro has it's quirks. This will show raw speed, as well as how it was accomplished, but there are plenty of other reasons to pick one over the other... Preference of package management, security, easy of maintaince. setup time etc and so forth. I personaly can't wait to see how this turns out...
Why not include Windows and perhaps others? I guess they wouldn't include non-open-source ones because it's a site about open source but I'd love to see the comparison. Have any other sites done that?
Comprison of various Linux distros (and of the 2.4 versus 2.6 kernel) is interesting. However, what is really lacking is an objective comparison of MS Windows Server 2003 versus Linux. I know Microsoft tries to prevent such benchmarking, but can they really enforce such a ban? It ought to be possible to find a team of Windows experts to tune Windows so the comparison is fair. Why not?
This is clearly a plot by Microsoft and SCO to destroy OSS! They couldn't beat any one of us, but if they get us to fight each other...
I have to confess to being more interested in the universality of the testing framework than any of the results. Whatever is done I hope it leads to some standards for future, lower profile but perhaps more useful benchmarks.
An accepted cross distro testing criteria would be nice.
ls
Dude, I know you're not supposed to read the story, but at least read the text you quoted.
This list is not final and if people want to ante in to try this with their favorite distro, let us know at bc2004 at osuosl dot org or in #beaverchallenge on the Freenode.net IRC network.
Note what's implied...
Please help metamoderate.
Too bad Gentoo isn't the only one with a PORTS clone.
;-)
Ever looked at Crux, Arch, or if you want "the original", BSD?
Gentoo isn't very unique in respect to portage.
Come on, if it's built by Dell in 3min the
PS/2 & USB controller will die, 3hrs the HDD controller will fail & wipe each HDD, but in 3 days they'll replace them with the equivelant Compaq or clone machine....
Flamebait I know... Am I Ashamed? No
: )
-Ted
-=-=- Quantum physics - the dreams stuff are made of.
What's with the x86- and Linux-centric approach? Do we really need to see how 6 different distributions can be tweaked to behave like one another on the same $4300 piece of hardware? I'd be extremely interested to see a G5 Xserve entered into that mix, although you'd clearly have to add some unnecessary doo-dads to the Mac to bring the price over $4000 (even with hardware RAID and the inability to drop below an 80GB HD to the 18GB like the Dell has, I could only bring a single processor Xserve up to $3500). Include a PPC Linux or two while you're at it. As it stands, the results will probably be at least a 6-way yawn-fest.
"In this competition they will be allowing a community of experts in each OS to tweak their configurations to ensure maximum performance."
I can see it now, teams of KDE and GNOME developers going head to head to see who can come up with the best color scheme, antialiased fonts, and 'Are you sure you want to delete this?' dialog box. Followed by Round 2, where each group has to compile something built for the other camp's desktop, whoever can fight through the dependencies quicker wins!
Lord Linus, save us from OSNews.
I want to delete my account but Slashdot doesn't allow it.
just imagine... well you know.
What?
Just imagine what?
A Beowulf cluster of Beavers?
Will they let you tweak Hyper Threading?
It'll be interesting to see how many people turn Hyper Threading OFF when doing some tests. I found that my database was 212% FASTER for read operations after I turned Hyper Threading off on the 2650.
This contest is blatantly set up so that Gentoo, the obvious victor in a fair benchmarking contest, cannot win.
They are only allowing three days to set up the OS, everyone knows that you can't get gentoo installed, much less customized in that time.
I guess your feeling kind of angry over gentoo's general acceptance among some geek crowds. I assume you have your reasons.
But I truelly believe that gentoo is what slackware once was, i.e. a distro that was forgiving of modifications, and it gives a pretty interface for doing so.
Its tedious to set up, but once initial install is done, its almost painless for the life of the machine. To me, the theoretical (or proven) performance advantages are almost secondary, in most applications, to portages forgiving nature.. Its just so easy to administer.
You would argue against the 'Gentoo-zealots' having no discernable advantage performance wise... And then suggest that portage is the only advantage Gentoo(zeolots) have (infered as I read it) over slackware.
A fact to consider: Optimized binaries generally run faster than unopimized ( an unqualified 3% - 15%, got the charts to prove it).
By the fact that portage is being ported to slackware, I assume your chosen distro, and by the fact that you mention it here, means that portage is important in your eyes. I have long held that the defining factor of any distro is its chosen package managment system (excusing directory layouts).
So in a way, isnt slackware becoming more like gentoo in effect? I mean after all, we are all dealing with the same fucking code with some minor tweaks and major package maintainance differences.
Oh, by the way. Gentoo is faster. Its going to kick Slackewares ass.
Noted Debian and 'apt-get' fan
I think you underestimate just how much I just dont care.
"Open source" has taken on a whole new meaning since Hustler got involved...
bytesmythe
Hypocrisy is the resin that holds the plywood of society together.
-- Scott Meyer
If one of the reasons to run the benchmark is to show people how great F/OSS is, "round 2" seems like a natural. After all, don't we all know some people who have a couple of heavy paperweights sitting around? Wouldn't more people be willing to try something other than Windows if they saw that their old crappy box ran just as fast as their new shiny WinXP box?
Huh? Part of the reason I switched over to NetBSD from Linux years ago (for the most part, there are still times when a quick-and-dirty Slackware box does a special trick or two) was because NetBSD (and FreeBSD is very similar) was FAR easier. There's a learning curve involved, but there's 'one way' that things are done and it's the classic Unix way. You can pick up an O'Reilly book from 1993 and the info in it closely applies. Linux, on the other hand, is a big snarl of forks, each distro doing each task and configuration in it's own way, everybody contending that THEIR way is BEST, and as a consequence, no clear straight-forward anything, except gui buttons in places where GUI buttons aren't needed.
/etc/ files on a NetBSD box is a converging process. You learn more and more as you work with it, and it doesn't change when Johnny volunteer coder at Distro X learns Python and gets tricky with pretty buttons on a control panel.
Learn how to configure the
Anyway, harumph. BSD is NOT less user-friendly. Perhaps it has a smaller userbase, but if you're reading and commenting on this article, you're capable of working with it.
---
Bogomips
'nuff said.
And, seeing as I managed to not read the topic of your post before I went off and posted myself, I misinterpreted what you said.
First of all, Gentoo is good.
Second of all, Gentoo isn't the easiest thing to setup.
Third of all, it is a matter of taste. And just as we shouldnt judge the general users of GNU/Linux on the mad zealots that shouts the loudest, you shouldnt judge Gentoo on the Gentoo zealots.
I use gentoo and I love it. On the other hand, Ive also used pretty much any other distro out there. IMHO, the one that was most pain-like to set up was debian. Though it is, a matter of opinion.
Now, I will go hide under a rock for a month.