NetBSD 2.0 vs FreeBSD 5.3 Benchmarks

diegocgteleline.es writes "According to OSnews, Gregory McGarry benchmarked NetBSD 2.0 against FreeBSD 5.3 and found that NetBSD 2.0 surpasses FreeBSD 5.3 in most of benchmarks. The machine used for benchmarks is a 3 Ghz P4 so it doesn't reflect the improvements of FreeBSD 5.3 in the SMP arena, which is where their developers have put their efforts in the last years and where NetBSD is still using a "big-lock" model. Newsforge is also carrying a interview with some NetBSD developers about the technology behind NetBSD 2.0."

Comments

Microbenchmarks...

[kernelistic]

It is important to note that all of the tests that were performed where done on a uni-processor workstation.

The blanket statement that "NetBSD 2.0 has better threading and process management and network latency than FreeBSD 5.3" does not hold water when the test suite is run on 2 and 4-way SMP systems. FreeBSD 5.x is an amazing engineering effort in which various parts of the kernel have been locked down and decent thread concurancy can occur on multi-proc machines. Part of the latency that you see in these benchmarks are due to the mtx_lock() and mtx_unlock() overhead that is incurred.

It is also important to note that P4-systems with HyperThreading (As the one used in these tests) have been the "bastard child" on FreeBSD. For the longest of times, compiling anything with CPUTYPE=p4 would produce broken code (In all fairness, this was mostly due to a set of bugs in GCC 3.x). Significant work was put into 5.3 to ensure that the Pentium 4 lived up to the Tier-1 platform robustness standard. In short, it would be fun to see these benchmarks be run on i386/pentium3, i386/Athlon-MP, amd64/opteron and Alpha as well!

Re:Microbenchmarks...

[kernelistic]

OTOH, do you know what the advantage of FGL over BKL on 2-way systems is? Zero, absolutely nothing. So my bet is that NetBSD kicks FreeBSD's ass on 2-way systems as well.

Nothing could be further from the truth. Here's why:

Big Kernel Lock (BKL) systems can only have one processor doing something in the kernel at one time. There's only so much work that can be performed in user-land before re-entry into the kernel is necessary. At this point the application processor (AP) is spinning, waiting around for the mother-of-locks ("Giant" on FreeBSD) to be released by another AP. The more processors are present in the system, the bigger the contention on the single kernel lock becomes.

Fine Grained Locking (FGL) allows a system with multiple CPUs to enter the kernel and perform work that is at hand or read the process runqueue to perform work in user-land. Sure you do run into instances where two (or possibly even more) CPUs need to perform an operation within a given critical section, but at this point the granularity of the locking allows for atomic operations per CPU to be performed before the critical section's mutex is released. This effectively prevents race conditions that would be encountered, if no locking was present. It also allows as many CPUs as your system can hold to perform the various tasks (Fetch, store, etc) at near full-speed (The finer the locking, the less waiting around for mutexes/locks there is).

Re:Microbenchmarks...

[kernelistic]

I am not making an exception to the exception. In fact, I am not even talking about HyperThreading. What I did say is that using GCC 3.x with P4 as a target produced broken code until very recently and as such, the architecture has not gotten the amount of scrutiny as it could have.

The default install of FreeBSD doesn't do HTT or SMP without a kernel recompile. My beef with this benchmark is as follows: How about you test the benchmark on other platforms/cpus? In order to truly assertain that NetBSD 2.0 is faster than FreeBSD 5.3 latency-wise, more than just one target should be tested. Without doing so, all that has been "proven" is that NetBSD performed better on the given hardware and this tidbit is useless information as the board they used is designed for desktops in the first place. Care to indicate which chipset the Asus P4-800SE uses anyway (ASUS' site appears to be down)?

Moreover, Hyperthreading is really "less SMP than more". Sure you have an APIC with two target IDs, but you still have one CPU core, which presents two virtual cores to the operating system. To the uninitiated, this means that the two virtual CPUs share the L1 cache, which is suboptimal on lower end P4 with small caches (The number of cache flushes because of misses is scary).

If arguing over the technical merits (Or lack thereof) of a benchmark makes me a fanboy then congrats, you have me pinned down pretty good! :-)

my reply in the mailinglist

[MPHellwig]

In the freebsd mailing list there was a troll using this test to come down on a couple of highly skilled developers.
Being lazy by nature I copy/paste my respone in the mailing list for the /. folks to enjoy:

Benchmark are made to be put into perspective, although everybody has a right to say what (s)he wants to say, this doesn't mean that you have to say it.
It seems to me that FreeBSD is focusing it performance onto MP 64bit processors. As we can see in the benchmark it has in comparison to other projects a negative impact on UP system.

But just put it in the perspective of processor developments, AMD (followed by Intel) is heading towards a multi-core 64 bit systems, what probably becomes mainstream at the end of next year.
With this technology the FreeBSD model could have a winner on there hands.

Doing the same job but not having the same philosophy on it, is always inefficient, but in the real world it leads to the Darwin effect.
What means that the best solution gets there chance of survival against the test of time.
Luckily these are all BSD's, good solution will spread, just take a look at PF.
OpenBSD has a good user base but not compatible to the sum of user base of the other BSD's. Still PF has spread there wings beyond the user base of OpenBSD.

FreeBSD is just a name for an OS, if any other OS can give me more "bang for the buck" and provides a full solution, I will use it.
Be it DragonFly/Free/Open/Net, MacOsX, GNU+Linux, Windows or any of the other hundreds of OS'es out there.
I like the BSD license so I will tend to stick to "gratis" BSD OS'es.

All of the disagreements in development is a healthy process to make sure the sort "BSD" an not the specie *BSD will survive.
Sure I have my disappointments about some decision, but hey so is live, this ain't a fan club for next biggest boy band (he he BSD-Boyz), where using an OS to provide solution for our technologic problems, you favor your solution but don't blind yourself.

And when you don't blind yourself you re-evaluate your situation and move forward with the best solution for your problem.
Sure it is a pain to migrate my boxes to another OS (well that is the fun part) and do some massive rewriting of my documentation, but thats my job and I tend to like it. Just standing still and not progress has its attractiveness when you had a very rough ride, but it gets dull very soon and then you find yourself back on the dirty tracks.

But these are my opinion only, however I like to share them ;-)

Re:Benchmarks valid?

[molnarcs]

Also, it must be emphasised that these are microbenchmarks. Robert Watson explains nicely the situation:

Since the performance optimization on FreeBSD for the last few years, with features like SMPng, libpthread, and UMA, has been focussed on macro performance not micro performance, it's not surprising the micro performance requires tuning. However, there is lots of on-going work on this front so I'd expect to see continued improvement in the immediate (5.4) life time. There are a number of optimizations in 6.x that are on the merge path for 5.x that will directly impact the results in these measurements -- in particular, what is clearly a bug in the way mutexes are released on UP kernels that adds almost a hundred cycles to every mutex release operation. This was identified in my micro-benchmarking shortly after the release of 5.3, and may play a substantial part in the posted results, especially for the very micro benchmarks that involve kernel memory allocation.

It is also unfortunate, that the otherwise excellent benchmark, when mentioning the barrage of criticism leveled at freebsd (of which 90% is posted by random and anonymous trolls on both ./ and osnews I might add) he refers to the "article" of this guy. This was my reply - and his adequate asnwers seems to be putting me on his foe list (lol.)

Anyway, I don't want' to downplay the importance of this benchmark (nor does rwatson, if you read the whole of his post) - in fact, I'd like to see more of these coming. And overall, it was a good reading :)

Re:One flaw with tests

[setantae]

Yes, it does. There is no SMP option anymore, because it is always on.

Re:One flaw with tests

[molnarcs]

SMP kernel does not cause lockups on UNI machines. You may be confusing ACPI lockups that troubled pre 5.3 releases (and in some cases 5.3 as well)? FreeBSD from 5.2 to 5.3 came with "options SMP" in GENERIC enabled. You say:

but that's how it is. at least it is when I move a p6-built kernel from my dual xeon box to my single k8 box. both DO accept p6 as the ARCH type but the SMP code just causes a lockup upon boot.

Are you sure it's the SMP code itself? As I said, 5.2/5.2.1 shipped with an SMP kernel by default, and I didn't see hang problems because of SMP (ACPI was a different issue). Also, someone mentioned APIC - which might be the case on some boards, but right now I run an APIC enabled non-smp kernel on my UP box (athlon xp 2400+, crappy asrock mb). I think it is more likely that code compiled on your dual xeon box won't run smoothly on your athlon64, smp or not, even though you might have specified i686 in your make.conf. (also, did your specify p6 with CPUTYPE?=i686)?

Re:One flaw with tests

[molnarcs]

To clarify a few things: FreeBSD 5.2/5.2.1 shipped with an SMP kernel by default, and options SMP and device APIC was enabled in GENERIC.

Options are still there, see /usr/src/sys/conf/NOTES:

# SMP OPTIONS:
#
# SMP enables building of a Symmetric MultiProcessor Kernel.

# Mandatory:
options SMP # Symmetric MultiProcessor Kernel

And in /usr/src/sys/i386/conf/NOTES:

# SMP OPTIONS:
#
# The apic device enables the use of the I/O APIC for interrupt delivery.
# The apic device can be used in both UP and SMP kernels, but is required
# for SMP kernels. Thus, the apic device is not strictly an SMP option,
# but it is a prerequisite for SMP.
#
-------------snip--------------
# Mandatory:
device apic # I/O apic

What happened is this (from their errata):

(3 Nov 2004) For FreeBSD/i386 and FreeBSD/amd64, SMP support in the GENERIC kernel has been disabled by default because the SMP kernel can degrade the performance on UP machines. A kernel configuration file SMP, which can be used to enable SMP support, has been added. More details on building the custom kernel can be found at http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/h andbook/kernelconfig.html.

Re:One flaw with tests

[phoenix_rizzen]

Nope, your info is very out-of-date.

A couple weeks after the release of FreeBSD 5.2.1, the GENERIC kernel was changed to compile an SMP kernel, and SMP support was controlled via a sysctl.

Just before the release of FreeBSD 5.3, GENERIC was changed to create a UP kernel, as that is the most common install. However, you can compile an SMP kernel and run it just fine on a UP system. You will lose a bit of system performance, but it will run.

Re:As long as there are a bunch of BSD types here

[Anonymous Coward]

OpenBSD doesn't turn on softupdates by default - you have to do it manually.