OK, so basically you're just going on marketing ("it says it's completely fair, that has GOT to be better than a BIASED scheduler!") and your own prior beliefs, instead of technical insight. Thanks for clarifying:)
Yes, indeed. So far Nick has not been able to replicate our results, so there is clearly something different about his setup that needs to be understood. One hypothesis is that he is using a self-compiled glibc, and maybe something in the fedora 8 glibc is either slowing performance, or was fixed in the version Nick is using. Or maybe it is something non-default in his kernel. We need a baseline comparison of the same configuration (fedora 8 + updates) on his hardware before we can determine that some other Linux configuration improves performance.
Why do you "expect" this? The ULE scheduler is also designed with interactive (e.g. desktop) performance in mind, so your expectation seems unmotivated by prior probability and more by a prejudice that Linux must be better:-)
Yes, I agree. We in fact track a spectrum of workloads, and try to optimize for all of them simultaneously. Sometimes there are trade-offs but this is rare.
The main point you should take away from this is that FreeBSD 7.0 does not run into scalability bottlenecks on this configuration, so it can make efficient use of 8 CPUs where FreeBSD 6.x (and older Linux kernels) could not.
The scalability of the architecture is more important than a few % variation in absolute numbers, which as you say will be wiped out by the march of technology within months.
The database is putting load on the kernel, and the performance of the kernel is what is being tested here. If you like, sysbench+pgsql is just a fancy concurrent syscall load generator and the underlying workload is irrelevant as long as the pattern of syscalls models the kind of thing that a reasonable application could be doing. The sysbench output numbers are just a function of the number and type of syscalls performed, i.e. they measure kernel throughput.
Analysis of the syscalls being performed indeed shows that it is not doing anything bogus that a reasonable application should not be doing, so in this sense it is a reasonable kernel benchmark target. If your contention is that the sysbench throughput is not an accurate model of what an *SQL* application is doing because it is generating sequences of commands that are non-representative when interpreted as SQL commands, then I can't comment. It still doesn't invalidate the benchmark, because if the kernel has good performance on a certain workload of syscalls then it will perform well no matter what application is executing them.
My RW patch indeed just detects the deadlock so I don't resolve the underlying problem. I agree that this makes sysbench not so useful for testing postgresql rw performance.
The "0.0.0.0 client address" bug was only present in the 7.0 development versions and was fixed prior to 7.0-RELEASE. If you are running IRC servers on unreleased OS code, you should expect problems.
Well, it's actually taken the Linux developers more than 12 months to get to this point, which is a little longer than 5 minutes:). I have been in contact with Nick and he had trouble replicating the older results, so it is possible that his system is still not configured properly on the FreeBSD side. Even if it turns out that the next Linux kernel fixes the performance deficit, then that's fine too. Ultimately both kernels will have to asymptote to the same performance anyway, assuming both are efficiently designed.
You are correct that the pgsql driver in sysbench is broken for read-write loads. It needs a small patch to get it to detect deadlocks properly. However your objection is irrelevant to this benchmark, which is for read-only loads. Furthermore, the goal is not to evaluate mysql vs postgresql, it is to evaluate FreeBSD vs Linux.
Yes, there can be different challenges in how to optimally schedule processes depending on what resources (e.g. cache) are shared between multiple CPU cores. The ULE scheduler does a good job in all cases though. In 8.0 Jeff has just committed patches that make use of explicit knowledge about CPU topology, which improves performance in some configurations (e.g. a lot on 16-core xeon systems).
Dragonfly has still made no progress towards SMP performance with even 2 CPUs (they are still "competitive" with FreeBSD 4.x performance, and sometimes much slower), but this somehow doesn't deter their advocates from believing that Matt Dillon's -- still largely unimplemented -- ideas are still vastly superior.
As we know, unwritten code has no bugs and runs at twice the speed of thought, so maybe so:)
Hi, I am the one who performed these benchmarks and I'd like to clarify a couple of things:
* The point of this benchmark is not to unilaterally declare victory over Linux, but to point out that FreeBSD is once again competitive with it on modern high-end hardware and certain workloads. Of course, we are working on other workloads too, and currently perform better than Linux on other benchmarks, and still worse on others. There will no doubt be further friendly competition between the two OSes that will work to the benefit of both. Our message to the Linux developers is that they should not expect to get away with resting on their laurels:-)
* I benchmarked both mysql and postgresql, and FreeBSD 7.0 performs better than all Linux kernels (at least up to 2.6.23) with both databases. Incidentally postgresql is much faster than mysql, contradicting common wisdom. Other fun facts are that mysql 5.0.51 has poorer scaling than 5.0.47, and 5.1.x has *much* worse performance and scaling than 5.0.47 on my tests.
* I benchmarked several versions of Linux including 2.6.20.x, 2.6.22 and 2.6.23. 2.6.20.x has terrible performance http://people.freebsd.org/~kris/scaling/scaling.png. This graph is from Feb 2007 and the FreeBSD performance also improved after this point.
* 2.6.22 (which is pre-CFS) mostly fixed this but still performs worse than FreeBSD http://people.freebsd.org/~kris/scaling/os-mysql.png. 2.6.23 included the new scheduler and was a major performance regression. I did not yet retest with 2.6.24, so maybe they have fixed CFS by now.
* Contrary to some commenter's assertions that this is not a CPU benchmark, this benchmark is *extremely* sensitive to CPU performance and especially scheduling (in fact, as noted in the PDF, I/O performance is not a factor here). The scheduler really matters here, which is why Linux took a big hit when they switched to CFS (similarly, on FreeBSD the 4BSD scheduler performs much worse). Tuning the scheduler is critical to performance on this kind of workload. The other critical aspect is having a highly optimized kernel without concurrency bottlenecks. 2.6.20 fell over on kernel concurrency, and 2.6.23 fell over with the scheduler.
2) The ports collection is not branched, so there's no possibility for "several ports downgraded" in the "4.x series". The only situation in which ports are downgraded is if there are serious problems with the newer version, and a reversion to the previous version is a net gain.
You're overstating the case. Most of these broken ports affect non-i386 architectures, or only particular versions of FreeBSD. And most of the ports that have problems are the less popular ports that have no maintainer and have often been abandonded by their original developers, i.e. the software that few people care about. The actual percentage of broken ports on i386 5.x is very low, about 4% the last time I counted it.
There have actually been a number of local and remote root holes in the default install of OpenBSD during that time frame..the only sense in which their claim is true is that they don't count root holes except in the head of the CVS tree. If a release from a year ago had the hole, but the current tree does not, they don't count it.
For example, a couple of years ago there was a telnetd exploit discovered after OpenBSD had disabled telnetd by default in OpenBSD-current, but a recent prior release had shipped with telnetd enabled. That allowed them to rationalize not counting it as a remote hole. There are a number of other similar examples.
Sounds like you misconfigured XFree86 to run with 8-bit color depth.
Re:Additional packaging systems for FreeBSD?
on
FreeBSD 5.2 Review
·
· Score: 2, Informative
What do you mean by "lack of a large, centralized resource for FreeBSD binary packages"? The FreeBSD FTP site contains binary packages for all supported branches and architectures (almost 9000 binary packages for FreeBSD 4.x on i386 at the present time). They're indexed at http://www.freebsd.org/ports and you can download them easily with pkg_add -r.
Re:FreeBSD 5 works fine in production, here
on
FreeBSD 5.2 Released
·
· Score: 1
Check around, it's probably still on some mirrors.
Also, I hope you submitted a Problem Report about your problems with 5.1, but you should definitely try 5.2 (and submit/update your PR based on whether or not it works).
Sounds a bit unlikely to me..it might be stable now because relatively few major changes have occurred, but when you track a development version of an OS you will run into serious problems from time to time, especially as it diverges more from FreeBSD-STABLE.
Slashdot Mirror
OK, so basically you're just going on marketing ("it says it's completely fair, that has GOT to be better than a BIASED scheduler!") and your own prior beliefs, instead of technical insight. Thanks for clarifying :)
Yes, indeed. So far Nick has not been able to replicate our results, so there is clearly something different about his setup that needs to be understood. One hypothesis is that he is using a self-compiled glibc, and maybe something in the fedora 8 glibc is either slowing performance, or was fixed in the version Nick is using. Or maybe it is something non-default in his kernel. We need a baseline comparison of the same configuration (fedora 8 + updates) on his hardware before we can determine that some other Linux configuration improves performance.
Why do you "expect" this? The ULE scheduler is also designed with interactive (e.g. desktop) performance in mind, so your expectation seems unmotivated by prior probability and more by a prejudice that Linux must be better :-)
Yes, this was indeed the latest kernel distributed by Fedora at the time those slides were produced in October 2007.
Just to clarify, 7.0 is now released, so we're not comparing "prerelease FreeBSD" to Linux.
Yes, I agree. We in fact track a spectrum of workloads, and try to optimize for all of them simultaneously. Sometimes there are trade-offs but this is rare.
The main point you should take away from this is that FreeBSD 7.0 does not run into scalability bottlenecks on this configuration, so it can make efficient use of 8 CPUs where FreeBSD 6.x (and older Linux kernels) could not.
The scalability of the architecture is more important than a few % variation in absolute numbers, which as you say will be wiped out by the march of technology within months.
The database is putting load on the kernel, and the performance of the kernel is what is being tested here. If you like, sysbench+pgsql is just a fancy concurrent syscall load generator and the underlying workload is irrelevant as long as the pattern of syscalls models the kind of thing that a reasonable application could be doing. The sysbench output numbers are just a function of the number and type of syscalls performed, i.e. they measure kernel throughput.
Analysis of the syscalls being performed indeed shows that it is not doing anything bogus that a reasonable application should not be doing, so in this sense it is a reasonable kernel benchmark target. If your contention is that the sysbench throughput is not an accurate model of what an *SQL* application is doing because it is generating sequences of commands that are non-representative when interpreted as SQL commands, then I can't comment. It still doesn't invalidate the benchmark, because if the kernel has good performance on a certain workload of syscalls then it will perform well no matter what application is executing them.
My RW patch indeed just detects the deadlock so I don't resolve the underlying problem. I agree that this makes sysbench not so useful for testing postgresql rw performance.
The "0.0.0.0 client address" bug was only present in the 7.0 development versions and was fixed prior to 7.0-RELEASE. If you are running IRC servers on unreleased OS code, you should expect problems.
Well, it's actually taken the Linux developers more than 12 months to get to this point, which is a little longer than 5 minutes :). I have been in contact with Nick and he had trouble replicating the older results, so it is possible that his system is still not configured properly on the FreeBSD side. Even if it turns out that the next Linux kernel fixes the performance deficit, then that's fine too. Ultimately both kernels will have to asymptote to the same performance anyway, assuming both are efficiently designed.
AFAICR it was a suggestion made on IRC that happened to stick. ULE does not in fact stand for anything.
You are correct that the pgsql driver in sysbench is broken for read-write loads. It needs a small patch to get it to detect deadlocks properly. However your objection is irrelevant to this benchmark, which is for read-only loads. Furthermore, the goal is not to evaluate mysql vs postgresql, it is to evaluate FreeBSD vs Linux.
Yes, there can be different challenges in how to optimally schedule processes depending on what resources (e.g. cache) are shared between multiple CPU cores. The ULE scheduler does a good job in all cases though. In 8.0 Jeff has just committed patches that make use of explicit knowledge about CPU topology, which improves performance in some configurations (e.g. a lot on 16-core xeon systems).
Thanks, it's useful to know. As a "consumer" of mysql it's kind of disappointing they don't do this kind of performance analysis in-house.
Dragonfly has still made no progress towards SMP performance with even 2 CPUs (they are still "competitive" with FreeBSD 4.x performance, and sometimes much slower), but this somehow doesn't deter their advocates from believing that Matt Dillon's -- still largely unimplemented -- ideas are still vastly superior.
:)
As we know, unwritten code has no bugs and runs at twice the speed of thought, so maybe so
Hi, I am the one who performed these benchmarks and I'd like to clarify a couple of things:
:-)
* The point of this benchmark is not to unilaterally declare victory over Linux, but to point out that FreeBSD is once again competitive with it on modern high-end hardware and certain workloads. Of course, we are working on other workloads too, and currently perform better than Linux on other benchmarks, and still worse on others. There will no doubt be further friendly competition between the two OSes that will work to the benefit of both. Our message to the Linux developers is that they should not expect to get away with resting on their laurels
* I benchmarked both mysql and postgresql, and FreeBSD 7.0 performs better than all Linux kernels (at least up to 2.6.23) with both databases. Incidentally postgresql is much faster than mysql, contradicting common wisdom. Other fun facts are that mysql 5.0.51 has poorer scaling than 5.0.47, and 5.1.x has *much* worse performance and scaling than 5.0.47 on my tests.
* I benchmarked several versions of Linux including 2.6.20.x, 2.6.22 and 2.6.23. 2.6.20.x has terrible performance http://people.freebsd.org/~kris/scaling/scaling.png. This graph is from Feb 2007 and the FreeBSD performance also improved after this point.
* 2.6.22 (which is pre-CFS) mostly fixed this but still performs worse than FreeBSD http://people.freebsd.org/~kris/scaling/os-mysql.png. 2.6.23 included the new scheduler and was a major performance regression. I did not yet retest with 2.6.24, so maybe they have fixed CFS by now.
* Contrary to some commenter's assertions that this is not a CPU benchmark, this benchmark is *extremely* sensitive to CPU performance and especially scheduling (in fact, as noted in the PDF, I/O performance is not a factor here). The scheduler really matters here, which is why Linux took a big hit when they switched to CFS (similarly, on FreeBSD the 4BSD scheduler performs much worse). Tuning the scheduler is critical to performance on this kind of workload. The other critical aspect is having a highly optimized kernel without concurrency bottlenecks. 2.6.20 fell over on kernel concurrency, and 2.6.23 fell over with the scheduler.
Hope this helps to clarify things.
"Predicting particle masses" was the big claim of the Heim believers...although they were never able to explain the derivation to physicists.
Finally, http://groups.google.com/group/sci.physics.researc h/browse_thread/thread/f72785ee1bb126a9/bc0ec8393f 3f3b6f#bc0ec8393f3f3b6f someone dug back into Heim's book, translated it from German, and discovered the secret of the incredible particle mass predictions: they were put in explicitly by hand. The Heim disciples apparently took his equations that they didn't understand, fumbled them around a bit and extracted the same numbers that Heim put into them...and called it a prediction!
So much for that space drive...
Wrong on both counts:
1) The stable branch does include security fixes
2) The ports collection is not branched, so there's no possibility for "several ports downgraded" in the "4.x series". The only situation in which ports are downgraded is if there are serious problems with the newer version, and a reversion to the previous version is a net gain.
You're overstating the case. Most of these broken ports affect non-i386 architectures, or only particular versions of FreeBSD. And most of the ports that have problems are the less popular ports that have no maintainer and have often been abandonded by their original developers, i.e. the software that few people care about. The actual percentage of broken ports on i386 5.x is very low, about 4% the last time I counted it.
There have actually been a number of local and remote root holes in the default install of OpenBSD during that time frame..the only sense in which their claim is true is that they don't count root holes except in the head of the CVS tree. If a release from a year ago had the hole, but the current tree does not, they don't count it.
For example, a couple of years ago there was a telnetd exploit discovered after OpenBSD had disabled telnetd by default in OpenBSD-current, but a recent prior release had shipped with telnetd enabled. That allowed them to rationalize not counting it as a remote hole. There are a number of other similar examples.
Can you provide a pointer to your PR with details of the problems you encountered?
Sounds like you misconfigured XFree86 to run with 8-bit color depth.
What do you mean by "lack of a large, centralized resource for FreeBSD binary packages"? The FreeBSD FTP site contains binary packages for all supported branches and architectures (almost 9000 binary packages for FreeBSD 4.x on i386 at the present time). They're indexed at http://www.freebsd.org/ports and you can download them easily with pkg_add -r.
FreeBSD supports ext2fs..what do you mean?
Check around, it's probably still on some mirrors.
Also, I hope you submitted a Problem Report about your problems with 5.1, but you should definitely try 5.2 (and submit/update your PR based on whether or not it works).
Sounds a bit unlikely to me..it might be stable now because relatively few major changes have occurred, but when you track a development version of an OS you will run into serious problems from time to time, especially as it diverges more from FreeBSD-STABLE.