Slashdot Mirror
FreeBSD 7.0 Bests Linux In SMP Performance
cecom writes "After major improvements in SMP support in FreeBSD 7.0, benchmarks show it performing 15% better than the latest Linux kernels (PDF, see slides 17 to 19) on 8 CPUs under PostgreSQL and MySQL. While a couple of benchmarks are not conclusive evidence, it can be assumed that FreeBSD will once again be a serious performance contender. Some posters on LWN have noted that the level of Linux performance could be related to the Completely Fair Scheduler, which was merged into the 2.6.23 Linux kernel."
Update: 03/06 21:32 GMT by KD : An anonymous reader sent in word that Linux kernel developer Nick Piggin reran the benchmark today and came to a different conclusion: In his benchmark Linux was faster than FreeBSD.
Is it any wonder that FreeBSD is better than Linux? Nothing against Linux, but it's hardly an apples to apples comparison.
I'd be interested to see results from pre-CFS kernels.
Not that FreeBSD hasn't made major performance improvements.
Also, I think that a database test isn't a complete picture. For example, some OSes like IRIX or Mac OS X perform very well on streaming of local video and audio, but I wouldn't benchmark Oracle or PostgreSQL on either.
My blog
I can finally make full use of my quad-core toaster!
That toast isn't going to serve itself!
Skiffy is Spiffy, but Ort is tort.
Maybe now we can finally declare year of the linux desktop!
Wait, what?
Does this apply to single processor machines with dual cores or just multiple processors?
If you haven't made a developer cry, you've wasted a day.
Speaking of which: are there any "distros" out there ship a combination of FreeBSD and the latest Gnome desktop? I think that would be a better combination than Ubuntu's Debian+Gnome combo, personally.
Does FreeBSD have a pre-emptable kernel? One of the things Linux has really focused on lately is desktop interactive performance, so there may be performance tradeoffs vs. a kernel which can't pre-empt itself.
I want to delete my account but Slashdot doesn't allow it.
to the enlightening and respectful conversation this article will provoke.
Since they switched to the Completely Fair Scheduler to improve performance then it means one or two things. Either they have failed, as it seems to run slower than earlier Linux versions in the BSD test, or the tests that BSD chose are "untypical", maybe selected to show a particular advantage to BSD. I don't have the expertise to tell which, but I would be happier seeing some benchmarks from an independent source rather than BSD.
Cathedral FTW!
Very, very nice scaling performance under PGSQL is evident in the PDF, and I've no reason to assume the benches aren't legit. I think part fo the reason that PG was traditionally slower than MySQL was that it did lots of complicated locking to provide better scalability across processors, whereas we see MySQL performance dropping off after we go to more than eight cores. I think this was the same philosophy Sun took with "Slowaris", which was also far more scalabe than Linux at the time the moniker was in widespread use.
.24 and .25, although it was a little sad to see the first iteration of CFS performing more poorly than its predecessor (and, if this is the case, I can see why Linus stonewalled CK's patches for so long, since they were mainly tested on desktop workloads). Are there any apples-to-apples comparisons out there that test various flavours and versions of Linux and BSD with a wide range of benchmarks? At the best review sites do a few benches with MySQL, and six months later everything has changed so it's incredibly difficult to do good performance comparisons.
:)
Still, I hope Linux can at least match this sort of superb scalability. CFS is fairly new, and I know there's optimisation work been done to it in
Even so, it's refreshing to see precious little of the "BSD fudged their benchmarks!" trollspeak in the LKML thread, and plenty of talk about how to make Linux better. Open source is hippy capitalism - it also needs healthy competition to keep it in check
Offtopic: bug linked to in the LKML pointed me at this http://www.latencytop.org/ Sounds quite nifty
Moderation Total: -1 Troll, +3 Goat
Linux is actually better than BSD because you can roast marshmallows over the schedular flamewars.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
Dig up Desktop BSD.
Quite nice, if not my cup of tea. IMHO, BSD is for servers, but if I were tempted to use BSD for desktops, it'd be Desktop BSD.
That being said, it seems VERY clean and useful, but it does use KDE (no load time on my laptop, when accessing menus, which means something has been fixed on BSD that hasn't been fixed on Linux KDE yet.)
I think one can force it to use something other than KDE and still keep the "Desktop Tools", which make Desktop BSD quite useful. (Only OS that detected my SD MMC controller without a single extra effort from me.)
" What luck for rulers that men do not think" - Adolf Hitler
Well, the next time I have an 8-core xeon processor and want to run MySQL or PreSQl I'll check it out. But in the meantime I'll wait for some more comprehensive, independent benchmarks. If Microsoft announced that based on their testing Windows Vista was capable of performing 10% better than their competitors on multi-core testing, would we jump all over that?
While I'm glad for FreeBSD they're showing good numbers again, their testing of PostgreSQL in this study is rather odd. The results are using the read-only tests from sysbench. You can see from its sourceforge page that sysbench is a MySQL benchmarking tool that has some rudimentary PostgreSQL support bolted on top. That particular code is so bad that the last time I checked, turning on the write OLTP tests deadlocked the PostgreSQL server, as it wasn't putting statements into transactions correctly (which of course the ancient MySQL versions this code targeted doesn't care about). As the sysbench tool hasn't been actively maintained in ages I doubt that has improved.
The claimed "15% faster than linux" is pretty clear in the MySQL tests; the PostgreSQL ones have a weird dip in them but are in general much closer. I'd be comfortable if the result of this study was "FreeBSD 7 has been optimized to be 15% faster running MySQL than Linux", because that matches what they did (note the specific libpthread patch for example). But the fact that they used such an awful PostgreSQL benchmarking methodology leaves me hesitant to draw a broader conclusion than that based on their tests.
but I heard it wasn't compatible with Windows or labtops.
:) I want to download the internet onto my labtop.
Can you help? Sorry, I am not good with computers.
+++ATH0
http://article.gmane.org/gmane.linux.kernel/650739
Post a comment. When faced with the two incompatibilities, slashcode keeps the comment and tosses the mods, presumably because comments might have attracted subcomments and thus be impossible to remove.
This assumes that youare still within the time frame of modability. I don't know what happens if you have used all your modpoints; does that automatically end your modability time slot?
Infuriate left and right
was actually usable with kqemu, or ran as a Xen DomU, that "extra performance" might be helpful.
Agreed, on both points. What I want to know though, is where this performance improvement, and 7.0 in general, leaves Dragonfly BSD... do they still feel that Dragonfly's choice to split off at 4x and start making radical changes is paying off? Is dragonfly making progress towards better performance, in general, or on particular workloads?
I saw what Matt Dillon did back in Amiga days. I saw what Amigas themselves could do. If Amigas inspired Dragonfly towards a more lightweight model, I'd love to see that fork making more progress.
http://www.kernel.org/pub/linux/kernel/people/npiggin/sysbench/
This problems were fixed! Take a look here in scalability of the upcoming 2.6.25 and glib 2.7
Sounds good. Presuming that there is merit in the methodology here, we may see some more competition in the market. I'd like it more if there were more code sharing going on, but maybe if we don't get tempers too hot we'll see a little more of that too.
Cool, however it would be better if software working on Linux were also working on FreeBSD.
boehm-gc is totally broken when using threads on FreeBSD SMP. And it's still totally broken on FreeBSD 7.
The Neko virtual machine is in ports, but it's unuseable due to this, I don't even understand why it's in the ports tree. Was it ever tested before being imported?
Just creating a thread:
$loader.loadprim("std@thread_create", 2)(function(z) { $print(z) }, "OK");
makes is crash with a corrupted stack. It works on every other operating system. It seems to work on an UP FreeBSD system, but on a FreeBSD 7 SMP system, it crashes, crashes, crashes.
{{.sig}}
I don't care how powerful FreeBSD is in term of SQL's performance because I still won't use it for my desktop OS.
Hi, I am the one who performed these benchmarks and I'd like to clarify a couple of things:
:-)
* The point of this benchmark is not to unilaterally declare victory over Linux, but to point out that FreeBSD is once again competitive with it on modern high-end hardware and certain workloads. Of course, we are working on other workloads too, and currently perform better than Linux on other benchmarks, and still worse on others. There will no doubt be further friendly competition between the two OSes that will work to the benefit of both. Our message to the Linux developers is that they should not expect to get away with resting on their laurels
* I benchmarked both mysql and postgresql, and FreeBSD 7.0 performs better than all Linux kernels (at least up to 2.6.23) with both databases. Incidentally postgresql is much faster than mysql, contradicting common wisdom. Other fun facts are that mysql 5.0.51 has poorer scaling than 5.0.47, and 5.1.x has *much* worse performance and scaling than 5.0.47 on my tests.
* I benchmarked several versions of Linux including 2.6.20.x, 2.6.22 and 2.6.23. 2.6.20.x has terrible performance http://people.freebsd.org/~kris/scaling/scaling.png. This graph is from Feb 2007 and the FreeBSD performance also improved after this point.
* 2.6.22 (which is pre-CFS) mostly fixed this but still performs worse than FreeBSD http://people.freebsd.org/~kris/scaling/os-mysql.png. 2.6.23 included the new scheduler and was a major performance regression. I did not yet retest with 2.6.24, so maybe they have fixed CFS by now.
* Contrary to some commenter's assertions that this is not a CPU benchmark, this benchmark is *extremely* sensitive to CPU performance and especially scheduling (in fact, as noted in the PDF, I/O performance is not a factor here). The scheduler really matters here, which is why Linux took a big hit when they switched to CFS (similarly, on FreeBSD the 4BSD scheduler performs much worse). Tuning the scheduler is critical to performance on this kind of workload. The other critical aspect is having a highly optimized kernel without concurrency bottlenecks. 2.6.20 fell over on kernel concurrency, and 2.6.23 fell over with the scheduler.
Hope this helps to clarify things.
Looks like it didn't last for long:
http://www.kernel.org/pub/linux/kernel/people/npiggin/sysbench/
I upgraded to FreeBSD 7.0 recently, and I have to wonder: Who came up with the name SCHED_ULE?
I'm not bad, I'm just drawn that way.
As Zpin wrote a few posts above, the linked PDF contains pre-CFS kernel benchmarks.
Short version:
Linux pre-CFS is faster than post-CFS, but FreeBSD still comes out ahead, by maybe 5%.
C - the footgun of programming languages
Here, and it applies to a significant number of other network servers.
Dramatic improvements in performance and SMP scalability shown by various database and other benchmarks, in some cases showing peak performance improvements as high as 350% over FreeBSD 6.X under normal loads and 1500% at high loads. When compared with the best performing Linux kernel (2.6.22 or 2.6.24) performance is 15% better.
http://people.freebsd.org/~kris/scaling/bind-pt.png
Summary:
* FreeBSD 7.0-R with 4BSD scheduler has close to ideal scaling on this test.
* The drop above 6 threads is due to limitations within BIND.
* Linux 2.6.24 has about 35% lower performance than FreeBSD, which is significantly at variance with the ISC results. It also doesn't scale above 3 CPUs.
* 7.0 with ULE has a bug on this workload (actually to do with workloads involving high interrupt rates). It is fixed in 8.0.
* Changes in progress to improve UDP performance do not help much with this particular workload (only about 5%), but with more scalable applications we see 30-40% improvement. e.g. NSD (ports/dns/nsd) is a much faster and more scalable DNS server than BIND (because it is better optimized for the smaller set of features it supports).
Interested in open source engine management for your Subaru?
If FreeBSD 7.0 can do 30k TPS with 4 CPU, why does it take 4 and a half CPU to do the same TPS load when the server has 8 CPU.
They need to retract their claim. That graph clearly shows a performance difference based on installed CPU count.
Whats the advantage of not having any distros?
What's the advantage of having multiple distros?
that everything needs to be compiled from source (ala gentoo) or that everything is precompiled for somebody else's needs (ala debian)
If you want to compile from source, you can, that's how Ports works. If you want to use precompiled versions you can install packages. You don't need to have multiple distros to be able to configure the system the way you need, you start with a core OS (which you can rebuild from source if you *really* want) and then add packages. There are people who do canned FreeBSD installs and ISOs, similar to Linux distros, but without the distro drama.
(especially as stupid license nazis stop us sharing code *shakes fist*)
What on earth does that refer to?
The arms race continues: http://jeffr-tech.livejournal.com/17426.html#cutid1
-- Sig down
There's an obvious issue I see here as to why there's this 15% difference. Freebsd 7 has jemalloc but the older Linux kernel they tested didn't. Linux will get jemalloc in the near future. It would be interesting to retest these benchmarks with both sides having the new malloc. Mark
Of course, this one is about SMP, so maybe this is really "Dawn of the Dead in 3D".
From that page:
And I do tend to agree with that. Ultimately, there are enough reasons keeping me on Linux (vs FreeBSD) that as long as there isn't that huge gap (seen in the graph linked to), I don't really care that much about whatever's left.
Nor am I particularly loyal to Linux. I understand and respect the GPL, but if there was another sufficiently open OS that beat Linux in ways I care about, I'd probably be using it, at least at home.
Don't thank God, thank a doctor!
Well, it's actually taken the Linux developers more than 12 months to get to this point, which is a little longer than 5 minutes :). I have been in contact with Nick and he had trouble replicating the older results, so it is possible that his system is still not configured properly on the FreeBSD side. Even if it turns out that the next Linux kernel fixes the performance deficit, then that's fine too. Ultimately both kernels will have to asymptote to the same performance anyway, assuming both are efficiently designed.
Is 8-way still considered SMP? I mean, 8-way is kind of consumer level now, isn't it? Even Apple produce 8-way machines SSI machines.
Get it to scale on some serious SGI kit, for example, then we'll talk.
Max.
Yes, I meant that: who cares?
Nobody living outside their parents' basement is going switch from Linux to BSD for a 15% performance increase. Somebody already using BSD might upgrade if the latest BSD kernels and environment are significantly better than past environments, but 15% is so slight as to be basically undetectable in a real-world environment!
My rule of thumb for upgrading equipment has been to not bother until we hit a full order of magnitude improvement. In other words, if 1) we can 10X the performance of a system AND 2) there have been complaints about performance, then the upgrade is probably worth it. Even then, the value is dubious. For example, in Postgres, (or any other database application) it's very typical to see 100x improvement simply by creating an index!
Maybe this is good for frail BSD egos, who have been long bruised by the mindshare success of Linux over the more historic and "more free" BSD. So be it. But it's not performance that's kept me from using BSD, it's familiarity and the pain of switching. And that's also what kept me running it yesterday, will today, and tomorrow too.
Don't get me wrong - I would hate to see BSD "die" in any meaningful way. The different cultures between Linux and BSD create a very rich, diverse environment where ideas can be tested, and the cross-feed of proven concepts and technologies (EG: Open SSH) benefits all involved!
But the benefit of a 15% performance increase is almost never going to be sufficient reason to pick one computing technology over another!
I have no problem with your religion until you decide it's reason to deprive others of the truth.
The mind boggles.
If they would have tested MySQL and Postgres with Hoard, Linux would have been faster than FreeBSD 7.
It's very odd how they tested with jemalloc with Hoard benchmarks and cited Hoard in the jemalloc paper but did not list Hoard timings.
http://www.hoard.org/
jemalloc
I run 7.0 now - I was running 6.2, then upgraded to 7.0 - a 10 minute task, by the way - and gained a noticeable performance increase. I haven't had the time or the inclination to run any benchmarks, but for a MySQL/Escapade/IMAP/squid server, I can certainly tell the difference.
-- Ed Carp, N7EKG erc@pobox.com PGP KeyID: 0x0BD32C9B What I'm up to: http://intuitives.mine.nu
I did this once. Your mod points are undone and lost.
Space game using normal deck of cards: http://BattleCards.org
dead + 15% is still dead.
I use Linux at home.
What's FreeBSD?
So... is this picture an accurate representation of FreeBSD vs Linux now?
More like this picture instead. You notice who's smiling more?
A dual core is likely to be different from a dual processor machine. With Intel's Core2 Duo machines (am only using that processor because I know it's architecture, not because it's better or worse than anything else), both cores on a chip share the L2 cache. So a Dual Core Xeon with 8MB-L2 cache, shares the caches between the two cores and is not the same as 2 processors with each having 4MB of L2 cache. Besides the ability to have 1 Core use 8MB of the cache (presuming the 2nd core is forcibly halted and left idle), there are scheduling differences and differences in migration costs. Intel's 1st Quad core chips after the Core 2, were logically 2-Core2 Processors on the same chip. Each pair shared an L2 cache with their being a total of 2-L2 cache's on the one chip.
...even partly with security (a hybrid model with some security being configurable, (LSM) and some designed not to be (the "standard", user-controlled Unix file-access bit checking isn't modularized). It's odd that CPU scheduling was thought to be a 1-size fits all model when virtually nothing else is). But because it isn't configurable, there was no way to make the CFQ cpu scheduler an optional, _testable_ scheduling module before it was chosen as the "one-and-only" model.
In some ways, that quad arrangement is like a Dual-Socket motherboard that has a Core2 Duo in each socket. Migration costs between adjacent cores (if migration includes cache loading costs) would be considerably less than between the two separate processors.
I believe the first Dual Core chips were similar to Dual processors machines in that each core had its own separate, fixed size cache. Logically -- one could achieve maximal resource usage on Processors with shared-caches, since whether your workload involved 1 active thread or multiple, the threads that are active can use all of the available core, whereas multi-core processors with each core having it's own separate cache will be limited to that cache even when other cores are idle.
At the time the Core Duo came out, AMD chips seemed to mostly (?completely?) sport per-core cache's, so the Core duo was a jump forward. Which the Quad-Core2 based chips had fully shared L2 caches -- would have been a no-brainer to upgrade to a quad-core with 8M L2 from a dual-core with 8M, but the processors on the quad core chip would be limited to 4M, max/core (or per/pair), whereas the dual-core chips could use up to 8M cache.
Of course the impact of cache size and whether it is sharable is totally dependent on what program(s) you are running, but local benchmarks between a 2GHz-8M-Core2Duo and a 3.2GHz-4M-Core2Duo showed the 2GHz beating out the 3.2GHz chip on small-medium problems with the 3.2GHZ chip taking the lead, only, in larger problems.
Supposedly, the linux kernel scheduler (pre-CFQ), recognizes the increased costs of inter-Processor switching being higher than intra-processor switching, but I've been unable to verify this. It might require some manual configuration using "CPUsets", but don't know.
FWIW, the new CFQ-cpu scheduler (which is different than the block-layer's CFQ Block-I/O scheduler) seemed awfully rushed into use as the "mainline" scheduler. I think it is because Linux has a "design choice" that it doesn't allow for modular CPU-schedulers as it does in the case of "block-i/o" (and USB I/I scheduling, and file systems, and choice of network layer, and partition type
Lies, damn lies, statistics and bench marking
=)
Linux has had topology aware CPU scheduling for nearly half a decade.
Bests or beats?
It's the BSD license which spawned all the proprietary *nix... and made a mess of it pushing the borg OS on the desktop. Moreover if you want performance: do not use a relational database. Whatever happens, THE kernel have to be GPL. Do not contribute to BSDed software, only single licensed GPLed ones.
"and mine is faster that yours !!!! - That's not even true, mine is faster than yours ........."
Lies, damned lies, and benchmarks. Some things never change.
It burns! Oh, it's just the BSD section.
:)
But seriously, where is the third test to confirm who's test is accurate and who's is a lie? Or are they different, and both Linux and BSD perform better in their own way?
Promote true freedom - support standards and interoperability.
I'm not subject to M$ lock-in, I have ten times the applications you have in your best BSD variant.
But most important of all, the last time I tried to install FreeBSD, I ended up with something that no average user could use. I'm not a BSCS, I'm a business owner looking for an easy to install desktop OS that is production ready, reliable, stable, and not Microsoft.
The BSD system is still a cousin and great for server application but it is not ready for the desktop prime time. When it has the software ported to it and is as simple to install as Debian Etch or Lenny, then you may have something to brag about and I'll give it a sincere consideration.
"Suppose you were an idiot...and suppose you were a member of Congress...but I repeat myself." Mark Twain