Slashdot Mirror
Benchmarking the Scalability of BSD and Linux
Fefe writes "I recently did some benchmarks for a talk about scalable network programming I held at Linux Kongress 2003. The benchmark results turned out to be surprising enough to present them on their own. This ought to end those pesky flame wars about whose IP stack or memory management scales better. Or maybe not."
Can anyone explain the discontinuities in the FreeBSD plots? Intuitively I would guess that something is breaking at high load, rather than getting miraculously faster. The author suggests that a clever optimization is kicking in, but I wonder if his tests were actually ensuring that the calls succeed.
Also watch out as you read the graphs - just to keep you on your toes, he changes the colors in every one!
Apparently the webserver with the results didn't scale so well. It's /.ed already.
The winner in this case is Open Source software.
The article is very fair and very well thought out. It is almost like reading a research paper. It looks like he is inviting criticism, insight, and corrections, rather than trying to force the experiments into a pre-determined outcome.
Such a thing is not possible in the proprietary world. Any study done on proprietary software has to be tainted with opinion and the experiments must be skewed. Read the EULAs. Some EULAs won't even allow you to publish the results of such tests.
Open Source software, of the BSD kind and the GPL kind, has totally changed the way we think about and work with software. One day, we will be able to scientifically determine what software we need to suit our needs. We will know ahead of time exactly what limits and what capabilities each piece of software has. IT managers will be able to sort through real facts based on real research, rather than a bunch of shallow articles and biased reports. Software will survive on its merits alone.
The whole industry is going to benefit by this, in a large, large way. The question one day will no longer be "Microsoft or Linux?" but "Which Open Source software should we use, and why?"
The radical sect of Islam would either see you dead or "reverted" to Islam.
Topic of this paper is
"Scalable Network Programming
Or: The Quest For A Good Web Server (That Survives Slashdot)"
What a coincidence!
By the way, fnord web server has at least once survived one slashdotting-event. 4 seconds of googleing result in this comment which should have let to a stream of visitors.
I hope fefe will publish the numbers of visitors and the behavior of its web server as soon as possible.
Who wants to start taking bets on when Theo takes the bait.
I have a complete copy (graphs and all) here.
I hope RMS reads the slides. They're in German at the link I used, so here's a translation of slide 13 which is page 14 of the PDF file:
"The memory required for an empty process is shockingly large on current Linux systems. However, this is not the fault of Linux, it's the fault of GNU libc.
GNU libc leads all libc implementations by a large margin in bloat and waste of memory. One day it got so painful that I wrote my own libc. With this, a static binary of 'Hello world' took only 300 bytes..."
I've long suspected that FSF stands for Fat Software Foundation.
(He doesn't say, but I assume his home-brew libc was a subset, otherwise we'd all want it).
I took three computers out in my rowboat, a Windows system, a Linux system, and a BSD system, and threw them overboard to see what would happen.
The Windows system sank like a rock, the Linux system bobbed back to the surface, and the BSD system rose to the sky, to be greated by a chorus of angels.
Then I woke up, so I don't know what the angels were singing.
Sheesh, evil *and* a jerk. -- Jade
So this guy grabs a beta version of a new tree in freebsd, and runs it against stable netbsd, openbsd and linux? Eh? Did he even compile his own kernel and take out all the debugging information in the released kernel? Did he turn off the debug info in the kernel config, on by default? I kinda doubt it somehow... If he knew enough to do that, he'd know that 5.1 has NOT been tagged stable.
Most freebsd users will look at this study and laugh. But people who don't know anything about freebsd (perhaps that includes the author?) might get the wrong idea. The study needs to be redone using the -stable tree, not the debug-riddled 5.1-RELEASE.
Not sabotage, security. In case you don't know: itojun is the guy between all the BSD's IPv6 support, and has been very active in the standarisation process.
I'm wondering, if he was going to be doing a scalability test, why didn't he test the version of FreeBSD that is actually reccomended for production (4.8)?
He had the time to test the stable and devel versions of the linux kernel, but only the new technology version of freebsd?
The reviewer does sound a bit biased towards Linux. In the same way Bill sounds a bit biased towards Microsoft.
Does everything include nothing?
There's no need for such a very technical benchmark.
Regular usage of various operating systems on the same host makes it obvious.
When it comes to speed and features (or bloat), Linux is more efficient than FreeBSD, NetBSD and OpenBSD. This is especially significant in SMP environments.
Linux users are always talking about the just-released experimental patches that will help their system to get 0.1% faster, or the most aggressive flags to optimize their Gentoo system.
BSD users just advocate their system with the generic word "robust".
Nowadays, stability is not really the key. Every Linux or BSD free operating system has basically the same stability. The software is the same, with the same bugs. The package system have equivalents (Debian works on NetBSD, Gentoo works a lot like BSD ports, etc) and support for common hardware is almost identical.
The reason to choose one OS over another is often more political than technical. People tend to use FreeBSD just to try "something else". People tend to use Linux because the Mandrake/RedHat/Conectiva/SuSE installers are beautiful or because Gentoo is fashion and a good way to learn what Unices are made of.
But if this is just to use common software like Apache and Qmail there's no real difference except speed. If this is what you need, Linux is definitely the best choice nowadays, especially since 2.6 kernels are almost ready for production use.
For other needs, your mileage may vary.
For instance I love OpenBSD for development. The compiler and the libc have very handy features to automatically detect bogus code. And the man pages are also excellent, with helpful hints.
For firewalls and trafic shaping, I wouldn't use anything but *BSD because of PF. PF is really the best thing in *BSD systems IMHO. The firewall is very easy to configure yet extremely powerful and fast. And I was fond of Iptables before.
For bridging and transparent firewalls, I would also use BSD because it seems to work better than Linux in this area.
In fact it's just like the girl of your dreams. Everyone's always looking for the perfect operating system that will perfectly fit all needs, but it just doesn't exist.
{{.sig}}
Programming can be fun again. Film at 11.
Also watch out as you read the graphs - just to keep you on your toes, he changes the colors in every one!
The author should use a graphing technique Edward Tufte calls "small multiples". In that you lay out a series of thumbmails of the each graph series. The eye can quickly scan down the thumbmails and get a rough feeling for how each series compares.
This would avoid the problem where graphs overlay each other and with the inconsistency between graphs in the plotting conventions.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
I knew when I read the post that this would lead to another FreeBSD v. Linux flamewar, despite the author's claims of 'hoping' to end those.
I don't think using 5.1-CURRENT is a problem, but the way the benchmark results were layed out was begging for a flamewar. As I explained earlier the results are not as bad as either linux fanatics, or FreeBSD fanatics would have it. It would have been simple to avoid such flamewar (or am I too optimistic?) by doing two things:
Explain the status of both (linux 2.6 and FreeBSD 5.1) development branches - as I have outlined in my earlier post. If you take into consideration what I have written above, than you would have realized that results for FreeBSD are not that bad, in fact, they are excellent.
Include results for 4.8 - or 4.9 rc3 (but I would be happier with just the production release) in the test, just as 2.4 was included on the linux side.
To sum up: I believe that these benchmarks confirms what I thought for a long time: FBSD 5.1 development is on par with Linux 2.6. Perhaps this was the reason for his last "Or may be not" remark.
In The article, The author mentions that he finds "unacceptable", or embarassing, a few things about openbsd, mainly concerning it's general speed and scalability.
ALthought not beeing an expert in bsd nor in linux, maybe i'm rong, but isn't OpenBSD made to be secure, and not the fastest operatin system ( additionally, comparing oragnges to apples, by testing release/current/stable, wasn't the best way of comparing those OSes) ????
By implementing a few feature in the OpenBSD stack and Kernell, I guess that the devellopers are delibetatelly sacrifying performance in order to get an Os less vulnerable to DOS and other vulnerabilities
It owuld be interesting to compare the same OSes in a security test to find out which OS is more secure, if speed and scalability isn't the only issue in a OS...
Felix is more than opinionated about Linux vs. BSD. Check the qmail mailing list archives for his rants with BSD users about filesystem and other performance issues. In particular, he's had a number of run-ins with those on the list who use OpenBSD--the one he labels a `stinker'.
The interesting point is that all of these operating systems seem to be getting faster. It seems to come down to how many recent developments have been integrated into the version being tested, not any inherent differences between operating systems. This is, of course, as it should be: the source for all of these operating systems is available, and there are even frequently papers describing the techniques. If a technique is, in fact, better, it should eventually be adopted by all of them, and so your results will depend on how much has been adopted in the version you're testing.
It is encouraging to see that all of these developers are competing with the real opponent, which is not each other or even Microsoft, but the slashdot effect. After all, the goal should not be simply to be better than the others, but to be sufficient for the user's purpose, which is not hampered but rather assisted by sharing all of your tricks. It can sometimes seem like there are endless wars between Linux and BSD, but, behind the scenes, the sides actually share information. Never as much as they'd like, but always more than people think.