FreeBSD 7.0 Bests Linux In SMP Performance

cecom writes "After major improvements in SMP support in FreeBSD 7.0, benchmarks show it performing 15% better than the latest Linux kernels (PDF, see slides 17 to 19) on 8 CPUs under PostgreSQL and MySQL. While a couple of benchmarks are not conclusive evidence, it can be assumed that FreeBSD will once again be a serious performance contender. Some posters on LWN have noted that the level of Linux performance could be related to the Completely Fair Scheduler, which was merged into the 2.6.23 Linux kernel." Update: 03/06 21:32 GMT by KD : An anonymous reader sent in word that Linux kernel developer Nick Piggin reran the benchmark today and came to a different conclusion: In his benchmark Linux was faster than FreeBSD.

Comments

Well

[morgan_greywolf]

I'd be interested to see results from pre-CFS kernels.

Not that FreeBSD hasn't made major performance improvements.

Also, I think that a database test isn't a complete picture. For example, some OSes like IRIX or Mac OS X perform very well on streaming of local video and audio, but I wouldn't benchmark Oracle or PostgreSQL on either.

Re:Well

[archen]

And I like the article summary stating that FreeBSD may now be considered "a serious performance contender". Like FreeBSD was 1000% slower than Linux? Most servers spend their time spinning their wheels anyway, generally I'd rather look at security, how it handles under load and other metrics than whatever "performance" is considered in this instance. Linux is good for some things, BSD for others.

About the only really good news here is that MySQL performance is actually adequate. As MySQL has always been a dog (usable, but a dog) on FreeBSD, the general rule of thumb was that if you needed MySQL you should stick to Linux; all other factors being equal. So now at least we can get down to other factors that are important instead of one database that performs poorly on one system.

Re:Dual Core

[morgan_greywolf]

Does this apply to single processor machines with dual cores or just multiple processors? Interesting point, but from the OS kernel's point-of-view a single-processor, dual-core machine looks exactly like a machine with two processors. So it runs the same code to support SMP whether we're talking single-processor dual-core or multiple processors.

IOW, if there is a performance difference, I would expect it to show up exactly the same in both FreeBSD and Linux (as well as any other OS that supports SMP).

Re:You don't have to be Kreskin

[oscartheduck]

You think so? I dunno, it seems to me that FreeBSD suits the desktop role really well; I use it for preference. Especially when you consider that the only OS with more packages is Debian, it makes sense that it can fit a desktop role extremely nicely.

In short, wow

[MrNemesis]

Very, very nice scaling performance under PGSQL is evident in the PDF, and I've no reason to assume the benches aren't legit. I think part fo the reason that PG was traditionally slower than MySQL was that it did lots of complicated locking to provide better scalability across processors, whereas we see MySQL performance dropping off after we go to more than eight cores. I think this was the same philosophy Sun took with "Slowaris", which was also far more scalabe than Linux at the time the moniker was in widespread use.

Still, I hope Linux can at least match this sort of superb scalability. CFS is fairly new, and I know there's optimisation work been done to it in .24 and .25, although it was a little sad to see the first iteration of CFS performing more poorly than its predecessor (and, if this is the case, I can see why Linus stonewalled CK's patches for so long, since they were mainly tested on desktop workloads). Are there any apples-to-apples comparisons out there that test various flavours and versions of Linux and BSD with a wide range of benchmarks? At the best review sites do a few benches with MySQL, and six months later everything has changed so it's incredibly difficult to do good performance comparisons.

Even so, it's refreshing to see precious little of the "BSD fudged their benchmarks!" trollspeak in the LKML thread, and plenty of talk about how to make Linux better. Open source is hippy capitalism - it also needs healthy competition to keep it in check :)

Offtopic: bug linked to in the LKML pointed me at this http://www.latencytop.org/ Sounds quite nifty

Re:You don't have to be Kreskin

[oscartheduck]

Well, I don't think I've ever installed any package from anything other than the ports system. Lots? I know I've installed everything from Gnome, XFCE and KDE, through OpenOffice and a bunch of stuff in between.

You're right that mere numbers of packages is a weird metric, but what else can we offer? FreeBSD has great performance, and has everything necessary to be either a good server *or* a good desktop. It's much like Gentoo that way -- it doesn't focus on being either one or the other, it focuses on being a solid basis. What you put on top of that basis is your choice. It honestly seems to me that the distinction between server OS and desktop OS is its own entire discussion; if we can come to a good notion of what either means, we can reach a nice conclusion. If we take the current crop of Linux desktop OSes, though, I don't see any more integration between, say, Fedora and Gnome and FreeBSD and Gnome, or Ubuntu and Gnome and FreeBSD and Gnome.

If I think about it, it does seem that Ubuntu starting with a GUI interface and letting you find the command line by yourself is more friendly to the average user; I haven't installed FreeBSD using anything other than minimal-install for so long that I don't know whether you can have a GUI start up by default. And FreeBSD's installer, whilst excellent for its audience, is less friendly to a first timer. If we take those metrics, the idea of "can I sit down and first time use it without documentation?" then a lot of the linux crop are friendlier, yeah. But the documentation *is* very hand-holdy, and very very thorough for FreeBSD. And nicely available online.

Bogus PostgreSQL tests

[greg1104]

While I'm glad for FreeBSD they're showing good numbers again, their testing of PostgreSQL in this study is rather odd. The results are using the read-only tests from sysbench. You can see from its sourceforge page that sysbench is a MySQL benchmarking tool that has some rudimentary PostgreSQL support bolted on top. That particular code is so bad that the last time I checked, turning on the write OLTP tests deadlocked the PostgreSQL server, as it wasn't putting statements into transactions correctly (which of course the ancient MySQL versions this code targeted doesn't care about). As the sysbench tool hasn't been actively maintained in ages I doubt that has improved.

The claimed "15% faster than linux" is pretty clear in the MySQL tests; the PostgreSQL ones have a weird dip in them but are in general much closer. I'd be comfortable if the result of this study was "FreeBSD 7 has been optimized to be 15% faster running MySQL than Linux", because that matches what they did (note the specific libpthread patch for example). But the fact that they used such an awful PostgreSQL benchmarking methodology leaves me hesitant to draw a broader conclusion than that based on their tests.

Re:Dual Core

[Wdomburg]

Have you ever looked at a block diagram of the predominant dual core designs? They're not simply "two processors on one piece of silicon". Both Intel and AMD used a shared cache design with a single connection to the system bus (FSB and HT, respectively). In the case of AMD, it also means a shared memory controller. It's a real difference with real performance and power implications, not a "silly marketing term".

Now if you complained about Intel shoving two dies into a multi-chip package and calling that quad-core, I'd agree with you. All the reduced bandwidth of a shared connection to the FSB with none of shared cache! Sign me up!

Seriously: who cares?

[mcrbids]

Yes, I meant that: who cares?

Nobody living outside their parents' basement is going switch from Linux to BSD for a 15% performance increase. Somebody already using BSD might upgrade if the latest BSD kernels and environment are significantly better than past environments, but 15% is so slight as to be basically undetectable in a real-world environment!

My rule of thumb for upgrading equipment has been to not bother until we hit a full order of magnitude improvement. In other words, if 1) we can 10X the performance of a system AND 2) there have been complaints about performance, then the upgrade is probably worth it. Even then, the value is dubious. For example, in Postgres, (or any other database application) it's very typical to see 100x improvement simply by creating an index!

Maybe this is good for frail BSD egos, who have been long bruised by the mindshare success of Linux over the more historic and "more free" BSD. So be it. But it's not performance that's kept me from using BSD, it's familiarity and the pain of switching. And that's also what kept me running it yesterday, will today, and tomorrow too.

Don't get me wrong - I would hate to see BSD "die" in any meaningful way. The different cultures between Linux and BSD create a very rich, diverse environment where ideas can be tested, and the cross-feed of proven concepts and technologies (EG: Open SSH) benefits all involved!

But the benefit of a 15% performance increase is almost never going to be sufficient reason to pick one computing technology over another!

Re:Seriously: who cares?

[Pharmboy]

But the benefit of a 15% performance increase is almost never going to be sufficient reason to pick one computing technology over another!

So if you are google, and your software will all 100% run with Linux or BSD, you don't see the idea that 15% better performance means the same work with 15% less machines means something? In certain cases, 15% can mean thousands or millions of dollars, all for changing to an operating system that will basically run on the exact same hardware and run the exact same software, after a recompile.

No for most it isn't a big deal and may not make people CHANGE operating systems on existing hardware. We use both Linux and BSD, so it *might* make me consider BSD instead of Linux on the next new box. I'm likely not alone in this.