NetBSD 2.0 vs FreeBSD 5.3 Benchmarks

diegocgteleline.es writes "According to OSnews, Gregory McGarry benchmarked NetBSD 2.0 against FreeBSD 5.3 and found that NetBSD 2.0 surpasses FreeBSD 5.3 in most of benchmarks. The machine used for benchmarks is a 3 Ghz P4 so it doesn't reflect the improvements of FreeBSD 5.3 in the SMP arena, which is where their developers have put their efforts in the last years and where NetBSD is still using a "big-lock" model. Newsforge is also carrying a interview with some NetBSD developers about the technology behind NetBSD 2.0."

Comments

Benchmarks valid?

[marcovje]

FreeBSD's focus is typically on overall throughput of massive server, with responsetime as a tradeoff, while this benchmark with all of its timings of single OS operations is more something realtimish?

If NetBSD aspires to be a RT focussed distro/OS, they'd better have benchmarked it against some Linux with RT patches?

Re:Benchmarks valid?

[molnarcs]

Also, it must be emphasised that these are microbenchmarks. Robert Watson explains nicely the situation:

Since the performance optimization on FreeBSD for the last few years, with features like SMPng, libpthread, and UMA, has been focussed on macro performance not micro performance, it's not surprising the micro performance requires tuning. However, there is lots of on-going work on this front so I'd expect to see continued improvement in the immediate (5.4) life time. There are a number of optimizations in 6.x that are on the merge path for 5.x that will directly impact the results in these measurements -- in particular, what is clearly a bug in the way mutexes are released on UP kernels that adds almost a hundred cycles to every mutex release operation. This was identified in my micro-benchmarking shortly after the release of 5.3, and may play a substantial part in the posted results, especially for the very micro benchmarks that involve kernel memory allocation.

It is also unfortunate, that the otherwise excellent benchmark, when mentioning the barrage of criticism leveled at freebsd (of which 90% is posted by random and anonymous trolls on both ./ and osnews I might add) he refers to the "article" of this guy. This was my reply - and his adequate asnwers seems to be putting me on his foe list (lol.)

Anyway, I don't want' to downplay the importance of this benchmark (nor does rwatson, if you read the whole of his post) - in fact, I'd like to see more of these coming. And overall, it was a good reading :)

Re:Benchmarks valid?

[marcovje]

Correct. And this is how it went with 4.x either. 3.6 was significantly better than the first 4.xRC.

It's not even something of FreeBSD in particular. In the project we are RC'ing for 2.0, and all kinds of detail benchmarks are popping up.

However overall, for significant programs 2.0.x is generally faster than 1.0.x even when 2.0.x doesn't even use optimizations (the project is a compiler).

The failure at microbenchmarks is just that the 1.0.x branch had 6 revisions in 3-4 years, some a full year tuning apart even.

I use them

[brilinux]

As an OS junky, I have used both of these BSDs, and am very impressed with them. I use FreeBSD on my P4 3GHz laptop, as Project Evil lets me use the WiFi card nicely, and ACPI kind of works. I use NetBSD on my Sparc Station 5 and my Athlon desktop, and I have found both to be wonderful for desktop use and as servers. I salute both of the groups of people who make these OSes, and wish that I had more time to contribute.

Microbenchmarks...

[kernelistic]

It is important to note that all of the tests that were performed where done on a uni-processor workstation.

The blanket statement that "NetBSD 2.0 has better threading and process management and network latency than FreeBSD 5.3" does not hold water when the test suite is run on 2 and 4-way SMP systems. FreeBSD 5.x is an amazing engineering effort in which various parts of the kernel have been locked down and decent thread concurancy can occur on multi-proc machines. Part of the latency that you see in these benchmarks are due to the mtx_lock() and mtx_unlock() overhead that is incurred.

It is also important to note that P4-systems with HyperThreading (As the one used in these tests) have been the "bastard child" on FreeBSD. For the longest of times, compiling anything with CPUTYPE=p4 would produce broken code (In all fairness, this was mostly due to a set of bugs in GCC 3.x). Significant work was put into 5.3 to ensure that the Pentium 4 lived up to the Tier-1 platform robustness standard. In short, it would be fun to see these benchmarks be run on i386/pentium3, i386/Athlon-MP, amd64/opteron and Alpha as well!

Re:Microbenchmarks...

[kernelistic]

OTOH, do you know what the advantage of FGL over BKL on 2-way systems is? Zero, absolutely nothing. So my bet is that NetBSD kicks FreeBSD's ass on 2-way systems as well.

Nothing could be further from the truth. Here's why:

Big Kernel Lock (BKL) systems can only have one processor doing something in the kernel at one time. There's only so much work that can be performed in user-land before re-entry into the kernel is necessary. At this point the application processor (AP) is spinning, waiting around for the mother-of-locks ("Giant" on FreeBSD) to be released by another AP. The more processors are present in the system, the bigger the contention on the single kernel lock becomes.

Fine Grained Locking (FGL) allows a system with multiple CPUs to enter the kernel and perform work that is at hand or read the process runqueue to perform work in user-land. Sure you do run into instances where two (or possibly even more) CPUs need to perform an operation within a given critical section, but at this point the granularity of the locking allows for atomic operations per CPU to be performed before the critical section's mutex is released. This effectively prevents race conditions that would be encountered, if no locking was present. It also allows as many CPUs as your system can hold to perform the various tasks (Fetch, store, etc) at near full-speed (The finer the locking, the less waiting around for mutexes/locks there is).

Re:Microbenchmarks...

[kernelistic]

Under load, I find that my RELENG_5 machines have higher network throughput than my RELENG_4 boxes. I am using mainstream server hardware with a nicely fine-grained set of network (nge) and SCSI RAID (amr) drivers. Would you care to share what kind of hardware you are using?

Matt and John aren't the only people that have worked on the locking of the kernel and drivers. Credit also goes out to Poul-Henning Kamp, Alan Cox, Jeff Robertson, Robert Watson and a lot of others that have reviewed, commented and submitted patches to all of sys/dev/ and sys/kern/.

As a sidenote, Dillon has also said a number of things that have been greatly unpopular, some of which have shown his unwillingness to think outside the box. I think the whole episode where core removed his bit, shows just how hot-headed he could be at times. That said, I have great respect for him as a programmer -- His stuff works.

Re:Microbenchmarks...

[Brandybuck]

Hyperthreading is *NOT* SMP! It is a single CPU that can only do the work of a single CPU. It pretends to be two CPUs but it is not. If you were talking about multiple core CPUs, then you would be right, but p4 HTT processors are single core.

Treating HTT at SMP is, to quote from the FreeBSD Handbook, "naive". Intel agrees, though they don't use that word. Here's a quote from the article you mentionL "but further performance gains can be realized by specifically tuning software for Hyper-Threading Technology." In other words, you tune software for HTT *differently* than you do for SMP.

Re:Microbenchmarks...

[idiotnot]

Except that FGL isn't supported still on a good amount of hardware. Things as ubiquitous as a at keyboard, and last time I checked, the realtek 8139 network device give you the nice [GIANT LOCKED] message in dmesg.

If you have blessed hardware, you get better SMP support.

I will say that FreeBSD 5.x is better than 4.x in some areas (background fsck, smp on supported hardware, better threading support), however, as a total system, it's really not finished, and shouldn't have been released in its current state. Maybe Matt Dillon was right -- I don't know. But what they've put out isn't up-to-par quality-wise, when you compare it to 4.x.

NetBSD, on the other hand, really has come a long way with the 2.0 release. I would say that it's a better choice on i386 now than either FreeBSD 4 (which is deprecated, although there will be a 4.11 release), or FreeBSD 5. That they've gotten to that point is really a testiment to the amazing design work they do. It's a very good system that doesn't get enough attention.

my reply in the mailinglist

[MPHellwig]

In the freebsd mailing list there was a troll using this test to come down on a couple of highly skilled developers.
Being lazy by nature I copy/paste my respone in the mailing list for the /. folks to enjoy:

Benchmark are made to be put into perspective, although everybody has a right to say what (s)he wants to say, this doesn't mean that you have to say it.
It seems to me that FreeBSD is focusing it performance onto MP 64bit processors. As we can see in the benchmark it has in comparison to other projects a negative impact on UP system.

But just put it in the perspective of processor developments, AMD (followed by Intel) is heading towards a multi-core 64 bit systems, what probably becomes mainstream at the end of next year.
With this technology the FreeBSD model could have a winner on there hands.

Doing the same job but not having the same philosophy on it, is always inefficient, but in the real world it leads to the Darwin effect.
What means that the best solution gets there chance of survival against the test of time.
Luckily these are all BSD's, good solution will spread, just take a look at PF.
OpenBSD has a good user base but not compatible to the sum of user base of the other BSD's. Still PF has spread there wings beyond the user base of OpenBSD.

FreeBSD is just a name for an OS, if any other OS can give me more "bang for the buck" and provides a full solution, I will use it.
Be it DragonFly/Free/Open/Net, MacOsX, GNU+Linux, Windows or any of the other hundreds of OS'es out there.
I like the BSD license so I will tend to stick to "gratis" BSD OS'es.

All of the disagreements in development is a healthy process to make sure the sort "BSD" an not the specie *BSD will survive.
Sure I have my disappointments about some decision, but hey so is live, this ain't a fan club for next biggest boy band (he he BSD-Boyz), where using an OS to provide solution for our technologic problems, you favor your solution but don't blind yourself.

And when you don't blind yourself you re-evaluate your situation and move forward with the best solution for your problem.
Sure it is a pain to migrate my boxes to another OS (well that is the fun part) and do some massive rewriting of my documentation, but thats my job and I tend to like it. Just standing still and not progress has its attractiveness when you had a very rough ride, but it gets dull very soon and then you find yourself back on the dirty tracks.

But these are my opinion only, however I like to share them ;-)

Re:One flaw with tests

[TheGratefulNet]

no.

in freebsd (I know this for a fact, I've gotton burned by it before) - if you compile a kernel for SMP, its NOT like linux in that it will fallback to UNIprocessing. it does NOT fallback! it hangs!

when moving an SMP custom compiled kernel to a non-smp box, you MUST rebuild the kernel or boot from an alternate non-smp kernel.

that sucks - but that's how it is. at least it is when I move a p6-built kernel from my dual xeon box to my single k8 box. both DO accept p6 as the ARCH type but the SMP code just causes a lockup upon boot.

(been running freebsd at home and at work for about 3 yrs now; and linux for about 10)

fyi

Re:One flaw with tests

[molnarcs]

SMP kernel does not cause lockups on UNI machines. You may be confusing ACPI lockups that troubled pre 5.3 releases (and in some cases 5.3 as well)? FreeBSD from 5.2 to 5.3 came with "options SMP" in GENERIC enabled. You say:

but that's how it is. at least it is when I move a p6-built kernel from my dual xeon box to my single k8 box. both DO accept p6 as the ARCH type but the SMP code just causes a lockup upon boot.

Are you sure it's the SMP code itself? As I said, 5.2/5.2.1 shipped with an SMP kernel by default, and I didn't see hang problems because of SMP (ACPI was a different issue). Also, someone mentioned APIC - which might be the case on some boards, but right now I run an APIC enabled non-smp kernel on my UP box (athlon xp 2400+, crappy asrock mb). I think it is more likely that code compiled on your dual xeon box won't run smoothly on your athlon64, smp or not, even though you might have specified i686 in your make.conf. (also, did your specify p6 with CPUTYPE?=i686)?

Re:One flaw with tests

[molnarcs]

To clarify a few things: FreeBSD 5.2/5.2.1 shipped with an SMP kernel by default, and options SMP and device APIC was enabled in GENERIC.

Options are still there, see /usr/src/sys/conf/NOTES:

# SMP OPTIONS:
#
# SMP enables building of a Symmetric MultiProcessor Kernel.

# Mandatory:
options SMP # Symmetric MultiProcessor Kernel

And in /usr/src/sys/i386/conf/NOTES:

# SMP OPTIONS:
#
# The apic device enables the use of the I/O APIC for interrupt delivery.
# The apic device can be used in both UP and SMP kernels, but is required
# for SMP kernels. Thus, the apic device is not strictly an SMP option,
# but it is a prerequisite for SMP.
#
-------------snip--------------
# Mandatory:
device apic # I/O apic

What happened is this (from their errata):

(3 Nov 2004) For FreeBSD/i386 and FreeBSD/amd64, SMP support in the GENERIC kernel has been disabled by default because the SMP kernel can degrade the performance on UP machines. A kernel configuration file SMP, which can be used to enable SMP support, has been added. More details on building the custom kernel can be found at http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/h andbook/kernelconfig.html.