Slashdot Mirror
Benchmarks of *BSD, Linux, and Solaris at LinuxTag
AnonymousCow writes "At LinuxTag, an unbiased comparison of performance of FreeBSD, NetBSD, OpenBSD, Linux, and Solaris." I'll let Tim's comment on this story stand: "Unbiased is hard to claim - all tests can be seen as biased in their formulation - but this is thorough, with 45 slides and
well-explained methodology -- BSD does very well ..."
Really, then please tell me what the three servers I've been playing with today are. Well, okay, they're not quite that big, after all they're basically FTP servers to carry away the data generated by the big machine. (little ones: 1 PIII/800, 1G of RAM, 2 RAID-1 arrays. big one: 14 Ultrasparcs with 8MB cache for each, 14 gigs of ram, and a whompingly large RAID array on fibre channel).
Big machines do exist, and are quite common among the set of people who keep their load averages over 1, and don't run seti or dnetc. After all, am I supposed to tell my company 'yeah, I know we've got a couple hundred million bucks in the bank, but I really think we should run the mission critical apps on cheap PC hardware'.
----------------------------
Lots of people are saying Linux didn't have DMA turned on, and that's the reason for the low scores. You can see on Slide 41 that hdparm is mentioned with the -d flag. I think that means that he turned it on, ladies and gentleman.
----------------------------
I imagine, with DMA turned on (as it is by default, of course, with FreeBSD), Linux would've looked much better in these tests (and probably most of the others too since they all involve disk transfers to some degree.)
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
This obviously would've affected Linux's showing in nearly every test.
Maybe they should re-do this thing. And get rid of the horrid jpeg slides.
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
This would kill performance in nearly every test.
The hard disk appears to max out at around 9MB/second in BSD but only around 4 in Linux, which is odd because it's been proven in other places that EXT2fs is at least a little bit faster than the BSD filesystem (even with async turned on).
Maybe they should redo these tests?
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
small update: some updates about the bad io results of linux will follow after more tests with different ide drivers (which
might be the reason for the extreme difference)
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
> FreeBSD... The choice of those, who know how to choose...
You sound like a few wine snobs I know.
FreeBSD may have it's technical merits, but I know plenty of people who run it just for the "fringe appeal". These are the people that ran Slackware way back because it seperated them from the crowd.
After the Red Hat IPO, Linux officially arrived and was no longer properly counter-culture. Once big money was involved, Linux became establishment, and that's just not chic. Then the people who ran Linux just to be different had to look for a new system -- luckily for them, the BSD's provide an easy migration. (yes, I know that BSD dates back to before Linux)
I feel some of this effect myself. I confess that part of why I use Debian is to seperate myself from the kids who purchased a RH boxed set at CompUSA, clicked their way through an install and now proudly play Solitaire in Gnome.
In the old days it *did* mean something to hack together a functional system from the disk sets that Slack provided, or that you downloaded by hand. There was a base level of proficiency that was necessary even to become a Linux user. This exclusion is rapidly breaking down in the Linux world, but lives on in the BSD world.
I'm not sure that any of the BSD's want to lower the entry barrier. Arguments of elitism aside, lowering the barrier definately lowers the mean competence of your user base, and I think that's something FreeBSD and the others would rather live without.
It seems to me that a lot of the *BSD users on Slashdot just wait around for good news about their system so that they can triumphantly reveal how superior their tastes are. It gets a little old.
FreeBSD... Because Linux just isn't leet enough anymore.
--Lenny
DMA is part of the protected mode disk drivers, along with 32-bit operations.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
There is no such thing as a $500 Sparc, unless you are using an old Sparc 5/10, circa 1995.
That is one of the best things about this comparison: it's in crappy hardware. I am sick to death of these useless 4CPU, 4G of RAM, 12 disk RAID array tests that just have no bearing on reality. Give me $500 of intel crap, load it with pr0n, and see which one the 'net kills first. THAT would be a test!
Actually, the sysinstall is a nice, menu driven program. It's just not graphical, which means it will also work without X and Gnome. Packages can be installed via the ports (this will mean compiling, except in the case where the ports are only available in binary format, like netscape), or as binaries with pkg_add.
It would be a very good idea if you read some docs on XFree86 configuration... you can physically damage your monitor and your videocard if it's done wrong by the configuration program, as appears to have been the case with you. The autogenerated XF86Config file looks intimidating, but most of it you don't need at all. In the case of XF86-4.0 it's been made a bit easier too.
I think you should be more cautious in your cheerleading. 2.4 has potential, but the realization of that potential is diffcult and not gauranteed.
IIRC, bonnie uses the stock getc() and putc(), which under Linux are threadsafe. These functions acquire a lock in userspace, and so have more overhead than is necessary for a single-threaded application. As Doug Ledford explains on this page, bonnie performs much more realistically when modified to use the non-threadsafe versions (which is a valid thing to do in a single-threaded application).
--Joe--
Program Intellivision!
The patterns in the disk read performance figures look suspiciously like the sort of gaps you get with and without DMA enabled on a disk. I note that the motherboard is a VIA board; Linux is known for having problems with DMA on via boards. Also, depending on distro, some may or may not enable DMA by default on bootup.
My opinion is that the BSDs having in the range of (say) 10% improvement over competitors would be easily explained by possibly better file system and VM architecture. But when we see a difference of five to one surely there's got to be something seriously wrong there. I've gotten better bonnie figures than that on an old Pentium.
in any case, we should take these tests with a large grain of salt. there are MANY factors unaccounted for: driver quality for the hardware used on each OS, IDE settings (hdparm on Linux), choice of filesystem, sync/async mounts, softupdates or not on BSD, journaling or not, and the large can of worms that is kernel tuning (did they do any?).
at least there are a few things we can tell kind of reliably from these tests: 1) the BSDs and Linux are all great; Solaris/x86 is generally slower (but may have better SMP, which wasn't tested here), 2) Linux 2.4 is a real improvement over 2.2, and 3) nfs and network stuff is faster when the client and the server use the same OS flavor.
I agree with the above comment. The filesystem performance differences should be at WORST a magnitude of 100-200 % worse/better in either case. BSD by default enables DMA, (for detected/supported chipsets, in my experience), while linux tends to require specific support for the udma transfers (recompiled UDMA patched kernel, my experience). Also, the hdparm -unmask option for servicing other interrupts greatly improves the overall responsiveness and feel of the system (in X, once again, my experience).
It doesn't appear these tuning parameters were applied, so I would expect any linux i/o bound results do be dog slow relative to any of the other OS's. The solaris results are so similar to the linux results for prolly the same reason... SCSI would have been a better test for this one benchmark.
--Adrian
the I/O performance varies from development kernel to development kernel and they're still tracking issues down (ie, someone reports 200% improvement on one kernel, but says it's worse in another.. really amazing actually).
What this means is that the Linux developers are playing with black magic and don't really know the effects of their performance tuning. Each dev kernel is a "hmm, does this fix the problem? or what about this fix?".
cpeterso
Solaris really performs on sparc.
And it's not designed for 'serving pages'. THat is a computationally light task.
Try a real server, serving many users on interactive X sessions at once, all with wierd simulation packages running. Solaris reigns.
> The *BSD's perform better under heavy I/O.
I'm not so sure this is the case had they compared against 2.4 rather than 2.2. But, the goal was to compare against production kernels, so it was a fair test I guess.
2.4 does scale WAY WAY better than 2.2, though.
Obviously, people didn't read the tests very carefully.
Solaris had big numbers for the SQL tests. Unfortunately, those were in seconds -- so Solaris actually did very POORLY on the SQL tests.
Please, everyone, read carefully. There's some good info on those slides but it is not clearly presented.
Torrey Hoffman (Azog)
Torrey Hoffman (Azog)
"HTML needs a rant tag" - Alan Cox
It looks like they used Solaris x86, which is not as finely tuned as Solaris/SPARC. It would be intersting to see what Solaris on SPARC could do against linux, assuming they both had similar hardward cost constraints.
Also, I thought an IDE hard drive an odd choice for a high performance server test - wouldn't SCSI or RAID be more appropriate?
Who am I? Subscribe and find out
It shows what linux groupies always try to deny...
The *BSD's perform better under heavy I/O.
Unlike the general linux mentality of, get it out the door if it barely works, the BSD mentality has alwasy been... it only goes out, if it's ready, and mature enough to go out... Hence the different release branches...
FreeBSD... The choice of those, who know how to choose...
Flame filters... ON... you can flame all you want, but you cannot change the facts.
It took me awhile, but I finally managed to get a mirror of the site up. Hopefully this should ease the Slashdot effect. It will be up unless the author asks me to remove it.
The URL is http://www.dynamine .net/mirrors/innominate.org/projects/tuning/
The server is on a 100Mb Exodus link, running FreeBSD 4.0. Have fun!
eh? the current version of linux is 6.2? freebsd 5.0 is comparible to linux 2.3? sounds like someone else who has fallen into the "redhat=linux" myth, yet is also confused in other ways too. Geeze...how long has it been since redhat 2.3? was there ever one? (lol) I know I used to use slack 2.3...(shrug) current version of REDHAT is at 6.2. Current "version" of Linux (stable) is 2.2.16
I think one could attribute this to network stack idiosyncracies. For example, if the Linux stack only sets its recv buffer to some specific size that's smaller than the *BSD size, would account for the difference. IIRC, Van Jacobson pointed out a number of problems with the TCP implementation doing weird things in slow start and congestion avoidance that might also account for impedance mismatching.
A similar problem cropped up when Samba was benchmarked against NT's SMB implementation.
-scooter
Looks like I'll be installing some flavor of BSD on my spare machine. Last time I tried Linux, it was just so unorganized and non-standard (yes, yes, I know LSB, but still...). I'm hoping the BSDs are a bit more tame as far as organization and standardization. I would even take whatever performance hits there were with NetBSD if it is actually written and designed elegantly enough that someone as rusty as me in C and C++ might actually be able to tinker with it (I am assuming most Linux users don't even dare putzing directly with their kernel). I wish there were some force for convergence instead of divergence as far as Linux distributions are concerned. Maybe that LDPS will help. It just seems that nothing really has enough teeth to tame the wildly fragmenting Linux distributions (how about a single package standard, a single update standard, even compliance with LSB?). I don't know, maybe I'm just talking out my ass.
It's 10 PM. Do you know if you're un-American?
There is a very thorough benchmark comparing Linux (kernel 2.2.12) to
FreeBSD (4.0). The benchmark takes time to analyse file system
performance, kernel timings such as contexts switches and use of
memeory managers and thread/process creation, all tied up with an
excellent summary.
The problem is that most systems ship with DMA off. It causes problems in some configurations, and is usually a liability for the vendor.
A deep unwavering belief is a sure sign you're missing something...
This compared to the Linux crowd whose benchmarks are, "oh, I just installed this and have only used Windows before and Linux is the fastest thing I've ever seen," or "on my 386 33 with 4 MB of RAM, Linux absolutely BLOWS AWAY Windows 2000" or my favorite "I've been using Linux ever since kernel .9 and its so fast! I mean, I played around with the Win2K alpha release, and Linux is SO much faster. What WM am I using? I don't run X, what a silly question! What do you MEAN that my comparison is invalid?"
A deep unwavering belief is a sure sign you're missing something...
Actually, over the years, MaximumPC's editors have suggested all manners of ways to punish vendors who ship with DMA off. I think one involved a ferret and a wet noodle...
A deep unwavering belief is a sure sign you're missing something...
OK, not trying to start a flamewar here (though I think that for this story, that may be an impossiblity). But was anyone really expecting *BSD to just roll over and die in the face of Linux? BSD has one badass TCP/IP stack, and it can't be much of a suprise that it did well. Keep in mind, readers and moderators, that I mostly use Linux. I'm not trying to be a BSD evangelist. I feel that FreeBSD 4s user-friendlyness is fairly low comared to, say, RH 6.2. But OTOH the *BSDs, especially FreeBSD, seem to really try and make their networking fast as hell (and the succeed).
Simliarly, the fact that Solaris cleaned up in the SQL test shouldn't be suprising.
BTW, some of those graphs were really hard to read, but in some didn't 2.2 wildly outperform 2.4? Specifically Seite 34, parellel compiling, real time. I'm confused!
thats why there is meta-moderation, if you are modded down unfairly than you will get moderated up when meta-moderation steps in , and the guy that moderated you down unfairly will get - points in karma.
-- http://electronicintifada.net --
I'm always wary when I see a test such as this that says "let's keep the hardware the same and compare just the software". There's a very dangerous illusion of equality which is just a fantasy.
Linux may have better disk access -- for IDE drives. But if FreeBSD handles SCSI better, is it fair to conclude that Linux is the better OS simply because you only used IDE drives in your comparison?
The reality is that you can't compare OSes in any meaningful sort of way and generalize the results to say "X is better than Y". To be scientific about it, you have to say "X is better at Y, only if run on <this_platform>, and then only at doing <this_task>, etc..."
And to be fair, the guy who did this study, pretty much said as much. His tests are basically valid for anybody with a stock K6-450 PC wanting to run a free Unix(like) system and possibly a web server.
Somehow, this gets transmogrified into "thorough" and "unbiased". =) Not that I'm disputing the results, I run FreeBSD myself. I just think it's important to take a step back and look at the whole picture.
--
Testing? What's that? If it compiles, it is good, if it boots up, it is perfect.
-- Linus Torvalds
This has to be my favorite argument with linux tcp/ip
http://www.spatula.net/proc/linux/localhost.src
With the linux development process the users are the alpha/beta testers.
Only the State obtains its revenue by coercion. - Murray Rothbard
It seems that the test system was a K6-400 with a VIA ide chipset. It was probably an Apollo Pro, which isn't very well supported by Linux at the moment. Examining the filesystem performance graphs shows around 3 MB/s throughput, which is exactly what I was getting out of my Apollo before I got UDMA working with it. You have to apply the IDE patches to 2.2 to even have it available, though it's in the 2.4-test series. You need to enable support for VIA82CXXX, ATA Works in Progress, Use PCI DMA by default when available, and VIA82CXXX Tuning support (a work in progress). After doing these things, I get 22 MB/s (!) out of my IBM, and 19 MB/s out of my Western Digital. If the Linux kernels had been compiled with these options, the results of the tests would have been quite different, I'd say.
Why didn't they include Windows NT or 2000? Aint they sposed to replace unix in the future?
Adam's Preliminary Page of BANG~!
Adam's Preliminary Page of BANG~!
http://www.ualberta.ca/~engel
Questions of bias aside, I think this benchmark makes it pretty clear that there's no such thing as a absolute "best" system--even the author says as much in his conclusions. While the *BSDs performed very well for filesystem I/O, Linux did nicely in the HTTP tests and Solaris topped everyone in SQL performance.
Also, something the holy-war people seem to forget is that even the comparatively badly-performing systems are more than sufficient for the majority of users. How many people, for example, serve more than 10 dynamic pages per second?
I guess what it boils down to is use the one you like best. Linux, *BSD, Solaris, etc. all have things to recommend them, which all appeal to different people. I personally have the most experience with Linux--I was introduced to it first--so it's what I use daily, but I've had my eye on FreeBSD for a while as well. (No spare computer to put it on yet, though...)
And think of the irony of trying to tear down a Windows monopoly only to replace it with a {Linux,FreeBSD,...} one. Competition is good, and variety is the spice of life. (^:
1) the IO numbers look very suspect, Linux was always capable of saturating IO bandwidth up to 60MB/sec even on modest hardware. This is so basic that they should have stopped the test when they saw how far the block IO numbers are apart. It certainly does not show alot of experience in tuning Linux.
2) the 'per character' numbers of bonnie are utterly meaningless. Bonnie is not a smart benchmark tool, but it gets quoted often. The 'per char' numbers simply measure the performance of libc's "getc()" function [and both Linux and FreeBSD use glibc.]. So the effect of the 'per char' measurement only slows Bonnie down and skews the numbers with additional CPU time. In fact i use a hacked Bonnie that just does the 'block IO' numbers - looking at the 'char IO' numbers is a waste of time.
3) While i understand the deadline issues, the 2.4-ac series were seriously buggy. It's only 2.4.0-test5 that started behaving properly, VM and IO-wise.
4) Linux's name is 'Linux', not 'linux' - they got it right with 'FreeBSD', so it's not hard :-)
5) the apparent IO misconfiguration then reflects in all the 'high load' numbers, so all the slides (except maybe the network numbers) should be redone IMO.
Anyone else notice NFS performance for *BSD to to *BSD to be significantly faster than Linux to *BSD? Same goes for Linux to Linux. Hmm...
*/
(end comment) */ }
[an error occurred while processing this directive]