Slashdot Mirror
Benchmarks of *BSD, Linux, and Solaris at LinuxTag
AnonymousCow writes "At LinuxTag, an unbiased comparison of performance of FreeBSD, NetBSD, OpenBSD, Linux, and Solaris." I'll let Tim's comment on this story stand: "Unbiased is hard to claim - all tests can be seen as biased in their formulation - but this is thorough, with 45 slides and
well-explained methodology -- BSD does very well ..."
Lots of people are saying Linux didn't have DMA turned on, and that's the reason for the low scores. You can see on Slide 41 that hdparm is mentioned with the -d flag. I think that means that he turned it on, ladies and gentleman.
----------------------------
This would kill performance in nearly every test.
The hard disk appears to max out at around 9MB/second in BSD but only around 4 in Linux, which is odd because it's been proven in other places that EXT2fs is at least a little bit faster than the BSD filesystem (even with async turned on).
Maybe they should redo these tests?
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
small update: some updates about the bad io results of linux will follow after more tests with different ide drivers (which
might be the reason for the extreme difference)
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
> FreeBSD... The choice of those, who know how to choose...
You sound like a few wine snobs I know.
FreeBSD may have it's technical merits, but I know plenty of people who run it just for the "fringe appeal". These are the people that ran Slackware way back because it seperated them from the crowd.
After the Red Hat IPO, Linux officially arrived and was no longer properly counter-culture. Once big money was involved, Linux became establishment, and that's just not chic. Then the people who ran Linux just to be different had to look for a new system -- luckily for them, the BSD's provide an easy migration. (yes, I know that BSD dates back to before Linux)
I feel some of this effect myself. I confess that part of why I use Debian is to seperate myself from the kids who purchased a RH boxed set at CompUSA, clicked their way through an install and now proudly play Solitaire in Gnome.
In the old days it *did* mean something to hack together a functional system from the disk sets that Slack provided, or that you downloaded by hand. There was a base level of proficiency that was necessary even to become a Linux user. This exclusion is rapidly breaking down in the Linux world, but lives on in the BSD world.
I'm not sure that any of the BSD's want to lower the entry barrier. Arguments of elitism aside, lowering the barrier definately lowers the mean competence of your user base, and I think that's something FreeBSD and the others would rather live without.
It seems to me that a lot of the *BSD users on Slashdot just wait around for good news about their system so that they can triumphantly reveal how superior their tastes are. It gets a little old.
FreeBSD... Because Linux just isn't leet enough anymore.
--Lenny
I think you should be more cautious in your cheerleading. 2.4 has potential, but the realization of that potential is diffcult and not gauranteed.
The patterns in the disk read performance figures look suspiciously like the sort of gaps you get with and without DMA enabled on a disk. I note that the motherboard is a VIA board; Linux is known for having problems with DMA on via boards. Also, depending on distro, some may or may not enable DMA by default on bootup.
My opinion is that the BSDs having in the range of (say) 10% improvement over competitors would be easily explained by possibly better file system and VM architecture. But when we see a difference of five to one surely there's got to be something seriously wrong there. I've gotten better bonnie figures than that on an old Pentium.
in any case, we should take these tests with a large grain of salt. there are MANY factors unaccounted for: driver quality for the hardware used on each OS, IDE settings (hdparm on Linux), choice of filesystem, sync/async mounts, softupdates or not on BSD, journaling or not, and the large can of worms that is kernel tuning (did they do any?).
at least there are a few things we can tell kind of reliably from these tests: 1) the BSDs and Linux are all great; Solaris/x86 is generally slower (but may have better SMP, which wasn't tested here), 2) Linux 2.4 is a real improvement over 2.2, and 3) nfs and network stuff is faster when the client and the server use the same OS flavor.
the I/O performance varies from development kernel to development kernel and they're still tracking issues down (ie, someone reports 200% improvement on one kernel, but says it's worse in another.. really amazing actually).
What this means is that the Linux developers are playing with black magic and don't really know the effects of their performance tuning. Each dev kernel is a "hmm, does this fix the problem? or what about this fix?".
cpeterso
Obviously, people didn't read the tests very carefully.
Solaris had big numbers for the SQL tests. Unfortunately, those were in seconds -- so Solaris actually did very POORLY on the SQL tests.
Please, everyone, read carefully. There's some good info on those slides but it is not clearly presented.
Torrey Hoffman (Azog)
Torrey Hoffman (Azog)
"HTML needs a rant tag" - Alan Cox
It looks like they used Solaris x86, which is not as finely tuned as Solaris/SPARC. It would be intersting to see what Solaris on SPARC could do against linux, assuming they both had similar hardward cost constraints.
Also, I thought an IDE hard drive an odd choice for a high performance server test - wouldn't SCSI or RAID be more appropriate?
Who am I? Subscribe and find out
I think one could attribute this to network stack idiosyncracies. For example, if the Linux stack only sets its recv buffer to some specific size that's smaller than the *BSD size, would account for the difference. IIRC, Van Jacobson pointed out a number of problems with the TCP implementation doing weird things in slow start and congestion avoidance that might also account for impedance mismatching.
A similar problem cropped up when Samba was benchmarked against NT's SMB implementation.
-scooter
There is a very thorough benchmark comparing Linux (kernel 2.2.12) to
FreeBSD (4.0). The benchmark takes time to analyse file system
performance, kernel timings such as contexts switches and use of
memeory managers and thread/process creation, all tied up with an
excellent summary.
OK, not trying to start a flamewar here (though I think that for this story, that may be an impossiblity). But was anyone really expecting *BSD to just roll over and die in the face of Linux? BSD has one badass TCP/IP stack, and it can't be much of a suprise that it did well. Keep in mind, readers and moderators, that I mostly use Linux. I'm not trying to be a BSD evangelist. I feel that FreeBSD 4s user-friendlyness is fairly low comared to, say, RH 6.2. But OTOH the *BSDs, especially FreeBSD, seem to really try and make their networking fast as hell (and the succeed).
Simliarly, the fact that Solaris cleaned up in the SQL test shouldn't be suprising.
BTW, some of those graphs were really hard to read, but in some didn't 2.2 wildly outperform 2.4? Specifically Seite 34, parellel compiling, real time. I'm confused!
I'm always wary when I see a test such as this that says "let's keep the hardware the same and compare just the software". There's a very dangerous illusion of equality which is just a fantasy.
Linux may have better disk access -- for IDE drives. But if FreeBSD handles SCSI better, is it fair to conclude that Linux is the better OS simply because you only used IDE drives in your comparison?
The reality is that you can't compare OSes in any meaningful sort of way and generalize the results to say "X is better than Y". To be scientific about it, you have to say "X is better at Y, only if run on <this_platform>, and then only at doing <this_task>, etc..."
And to be fair, the guy who did this study, pretty much said as much. His tests are basically valid for anybody with a stock K6-450 PC wanting to run a free Unix(like) system and possibly a web server.
Somehow, this gets transmogrified into "thorough" and "unbiased". =) Not that I'm disputing the results, I run FreeBSD myself. I just think it's important to take a step back and look at the whole picture.
--
It seems that the test system was a K6-400 with a VIA ide chipset. It was probably an Apollo Pro, which isn't very well supported by Linux at the moment. Examining the filesystem performance graphs shows around 3 MB/s throughput, which is exactly what I was getting out of my Apollo before I got UDMA working with it. You have to apply the IDE patches to 2.2 to even have it available, though it's in the 2.4-test series. You need to enable support for VIA82CXXX, ATA Works in Progress, Use PCI DMA by default when available, and VIA82CXXX Tuning support (a work in progress). After doing these things, I get 22 MB/s (!) out of my IBM, and 19 MB/s out of my Western Digital. If the Linux kernels had been compiled with these options, the results of the tests would have been quite different, I'd say.
Questions of bias aside, I think this benchmark makes it pretty clear that there's no such thing as a absolute "best" system--even the author says as much in his conclusions. While the *BSDs performed very well for filesystem I/O, Linux did nicely in the HTTP tests and Solaris topped everyone in SQL performance.
Also, something the holy-war people seem to forget is that even the comparatively badly-performing systems are more than sufficient for the majority of users. How many people, for example, serve more than 10 dynamic pages per second?
I guess what it boils down to is use the one you like best. Linux, *BSD, Solaris, etc. all have things to recommend them, which all appeal to different people. I personally have the most experience with Linux--I was introduced to it first--so it's what I use daily, but I've had my eye on FreeBSD for a while as well. (No spare computer to put it on yet, though...)
And think of the irony of trying to tear down a Windows monopoly only to replace it with a {Linux,FreeBSD,...} one. Competition is good, and variety is the spice of life. (^: