NetBSD 2.0 vs FreeBSD 5.3 Benchmarks

diegocgteleline.es writes "According to OSnews, Gregory McGarry benchmarked NetBSD 2.0 against FreeBSD 5.3 and found that NetBSD 2.0 surpasses FreeBSD 5.3 in most of benchmarks. The machine used for benchmarks is a 3 Ghz P4 so it doesn't reflect the improvements of FreeBSD 5.3 in the SMP arena, which is where their developers have put their efforts in the last years and where NetBSD is still using a "big-lock" model. Newsforge is also carrying a interview with some NetBSD developers about the technology behind NetBSD 2.0."

Comments

Benchmarks valid?

[marcovje]

FreeBSD's focus is typically on overall throughput of massive server, with responsetime as a tradeoff, while this benchmark with all of its timings of single OS operations is more something realtimish?

If NetBSD aspires to be a RT focussed distro/OS, they'd better have benchmarked it against some Linux with RT patches?

Re:Benchmarks valid?

[molnarcs]

Also, it must be emphasised that these are microbenchmarks. Robert Watson explains nicely the situation:

Since the performance optimization on FreeBSD for the last few years, with features like SMPng, libpthread, and UMA, has been focussed on macro performance not micro performance, it's not surprising the micro performance requires tuning. However, there is lots of on-going work on this front so I'd expect to see continued improvement in the immediate (5.4) life time. There are a number of optimizations in 6.x that are on the merge path for 5.x that will directly impact the results in these measurements -- in particular, what is clearly a bug in the way mutexes are released on UP kernels that adds almost a hundred cycles to every mutex release operation. This was identified in my micro-benchmarking shortly after the release of 5.3, and may play a substantial part in the posted results, especially for the very micro benchmarks that involve kernel memory allocation.

It is also unfortunate, that the otherwise excellent benchmark, when mentioning the barrage of criticism leveled at freebsd (of which 90% is posted by random and anonymous trolls on both ./ and osnews I might add) he refers to the "article" of this guy. This was my reply - and his adequate asnwers seems to be putting me on his foe list (lol.)

Anyway, I don't want' to downplay the importance of this benchmark (nor does rwatson, if you read the whole of his post) - in fact, I'd like to see more of these coming. And overall, it was a good reading :)

Re:Benchmarks valid?

[marcovje]

Correct. And this is how it went with 4.x either. 3.6 was significantly better than the first 4.xRC.

It's not even something of FreeBSD in particular. In the project we are RC'ing for 2.0, and all kinds of detail benchmarks are popping up.

However overall, for significant programs 2.0.x is generally faster than 1.0.x even when 2.0.x doesn't even use optimizations (the project is a compiler).

The failure at microbenchmarks is just that the 1.0.x branch had 6 revisions in 3-4 years, some a full year tuning apart even.

Microbenchmarks...

[kernelistic]

It is important to note that all of the tests that were performed where done on a uni-processor workstation.

The blanket statement that "NetBSD 2.0 has better threading and process management and network latency than FreeBSD 5.3" does not hold water when the test suite is run on 2 and 4-way SMP systems. FreeBSD 5.x is an amazing engineering effort in which various parts of the kernel have been locked down and decent thread concurancy can occur on multi-proc machines. Part of the latency that you see in these benchmarks are due to the mtx_lock() and mtx_unlock() overhead that is incurred.

It is also important to note that P4-systems with HyperThreading (As the one used in these tests) have been the "bastard child" on FreeBSD. For the longest of times, compiling anything with CPUTYPE=p4 would produce broken code (In all fairness, this was mostly due to a set of bugs in GCC 3.x). Significant work was put into 5.3 to ensure that the Pentium 4 lived up to the Tier-1 platform robustness standard. In short, it would be fun to see these benchmarks be run on i386/pentium3, i386/Athlon-MP, amd64/opteron and Alpha as well!

Re:Microbenchmarks...

[kernelistic]

Under load, I find that my RELENG_5 machines have higher network throughput than my RELENG_4 boxes. I am using mainstream server hardware with a nicely fine-grained set of network (nge) and SCSI RAID (amr) drivers. Would you care to share what kind of hardware you are using?

Matt and John aren't the only people that have worked on the locking of the kernel and drivers. Credit also goes out to Poul-Henning Kamp, Alan Cox, Jeff Robertson, Robert Watson and a lot of others that have reviewed, commented and submitted patches to all of sys/dev/ and sys/kern/.

As a sidenote, Dillon has also said a number of things that have been greatly unpopular, some of which have shown his unwillingness to think outside the box. I think the whole episode where core removed his bit, shows just how hot-headed he could be at times. That said, I have great respect for him as a programmer -- His stuff works.

Re:Microbenchmarks...

[Brandybuck]

Hyperthreading is *NOT* SMP! It is a single CPU that can only do the work of a single CPU. It pretends to be two CPUs but it is not. If you were talking about multiple core CPUs, then you would be right, but p4 HTT processors are single core.

Treating HTT at SMP is, to quote from the FreeBSD Handbook, "naive". Intel agrees, though they don't use that word. Here's a quote from the article you mentionL "but further performance gains can be realized by specifically tuning software for Hyper-Threading Technology." In other words, you tune software for HTT *differently* than you do for SMP.

Re:Microbenchmarks...

[idiotnot]

Except that FGL isn't supported still on a good amount of hardware. Things as ubiquitous as a at keyboard, and last time I checked, the realtek 8139 network device give you the nice [GIANT LOCKED] message in dmesg.

If you have blessed hardware, you get better SMP support.

I will say that FreeBSD 5.x is better than 4.x in some areas (background fsck, smp on supported hardware, better threading support), however, as a total system, it's really not finished, and shouldn't have been released in its current state. Maybe Matt Dillon was right -- I don't know. But what they've put out isn't up-to-par quality-wise, when you compare it to 4.x.

NetBSD, on the other hand, really has come a long way with the 2.0 release. I would say that it's a better choice on i386 now than either FreeBSD 4 (which is deprecated, although there will be a 4.11 release), or FreeBSD 5. That they've gotten to that point is really a testiment to the amazing design work they do. It's a very good system that doesn't get enough attention.

Re:One flaw with tests

[molnarcs]

To clarify a few things: FreeBSD 5.2/5.2.1 shipped with an SMP kernel by default, and options SMP and device APIC was enabled in GENERIC.

Options are still there, see /usr/src/sys/conf/NOTES:

# SMP OPTIONS:
#
# SMP enables building of a Symmetric MultiProcessor Kernel.

# Mandatory:
options SMP # Symmetric MultiProcessor Kernel

And in /usr/src/sys/i386/conf/NOTES:

# SMP OPTIONS:
#
# The apic device enables the use of the I/O APIC for interrupt delivery.
# The apic device can be used in both UP and SMP kernels, but is required
# for SMP kernels. Thus, the apic device is not strictly an SMP option,
# but it is a prerequisite for SMP.
#
-------------snip--------------
# Mandatory:
device apic # I/O apic

What happened is this (from their errata):

(3 Nov 2004) For FreeBSD/i386 and FreeBSD/amd64, SMP support in the GENERIC kernel has been disabled by default because the SMP kernel can degrade the performance on UP machines. A kernel configuration file SMP, which can be used to enable SMP support, has been added. More details on building the custom kernel can be found at http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/h andbook/kernelconfig.html.