NetBSD 2.0 vs FreeBSD 5.3 Benchmarks

diegocgteleline.es writes "According to OSnews, Gregory McGarry benchmarked NetBSD 2.0 against FreeBSD 5.3 and found that NetBSD 2.0 surpasses FreeBSD 5.3 in most of benchmarks. The machine used for benchmarks is a 3 Ghz P4 so it doesn't reflect the improvements of FreeBSD 5.3 in the SMP arena, which is where their developers have put their efforts in the last years and where NetBSD is still using a "big-lock" model. Newsforge is also carrying a interview with some NetBSD developers about the technology behind NetBSD 2.0."

Comments

Microbenchmarks...

[kernelistic]

It is important to note that all of the tests that were performed where done on a uni-processor workstation.

The blanket statement that "NetBSD 2.0 has better threading and process management and network latency than FreeBSD 5.3" does not hold water when the test suite is run on 2 and 4-way SMP systems. FreeBSD 5.x is an amazing engineering effort in which various parts of the kernel have been locked down and decent thread concurancy can occur on multi-proc machines. Part of the latency that you see in these benchmarks are due to the mtx_lock() and mtx_unlock() overhead that is incurred.

It is also important to note that P4-systems with HyperThreading (As the one used in these tests) have been the "bastard child" on FreeBSD. For the longest of times, compiling anything with CPUTYPE=p4 would produce broken code (In all fairness, this was mostly due to a set of bugs in GCC 3.x). Significant work was put into 5.3 to ensure that the Pentium 4 lived up to the Tier-1 platform robustness standard. In short, it would be fun to see these benchmarks be run on i386/pentium3, i386/Athlon-MP, amd64/opteron and Alpha as well!

Re:Benchmarks valid?

[molnarcs]

Also, it must be emphasised that these are microbenchmarks. Robert Watson explains nicely the situation:

Since the performance optimization on FreeBSD for the last few years, with features like SMPng, libpthread, and UMA, has been focussed on macro performance not micro performance, it's not surprising the micro performance requires tuning. However, there is lots of on-going work on this front so I'd expect to see continued improvement in the immediate (5.4) life time. There are a number of optimizations in 6.x that are on the merge path for 5.x that will directly impact the results in these measurements -- in particular, what is clearly a bug in the way mutexes are released on UP kernels that adds almost a hundred cycles to every mutex release operation. This was identified in my micro-benchmarking shortly after the release of 5.3, and may play a substantial part in the posted results, especially for the very micro benchmarks that involve kernel memory allocation.

It is also unfortunate, that the otherwise excellent benchmark, when mentioning the barrage of criticism leveled at freebsd (of which 90% is posted by random and anonymous trolls on both ./ and osnews I might add) he refers to the "article" of this guy. This was my reply - and his adequate asnwers seems to be putting me on his foe list (lol.)

Anyway, I don't want' to downplay the importance of this benchmark (nor does rwatson, if you read the whole of his post) - in fact, I'd like to see more of these coming. And overall, it was a good reading :)

Re:One flaw with tests

[molnarcs]

To clarify a few things: FreeBSD 5.2/5.2.1 shipped with an SMP kernel by default, and options SMP and device APIC was enabled in GENERIC.

Options are still there, see /usr/src/sys/conf/NOTES:

# SMP OPTIONS:
#
# SMP enables building of a Symmetric MultiProcessor Kernel.

# Mandatory:
options SMP # Symmetric MultiProcessor Kernel

And in /usr/src/sys/i386/conf/NOTES:

# SMP OPTIONS:
#
# The apic device enables the use of the I/O APIC for interrupt delivery.
# The apic device can be used in both UP and SMP kernels, but is required
# for SMP kernels. Thus, the apic device is not strictly an SMP option,
# but it is a prerequisite for SMP.
#
-------------snip--------------
# Mandatory:
device apic # I/O apic

What happened is this (from their errata):

(3 Nov 2004) For FreeBSD/i386 and FreeBSD/amd64, SMP support in the GENERIC kernel has been disabled by default because the SMP kernel can degrade the performance on UP machines. A kernel configuration file SMP, which can be used to enable SMP support, has been added. More details on building the custom kernel can be found at http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/h andbook/kernelconfig.html.