Slashdot Mirror

Actually, you can. The Linux CPU scheduler is pluggable by design, and several schedulers are available to choose from in the stock kernel. Con Kolivas was an active kernel developer for many years with a vested interest in improving kernel latency. He was one of the prime motivators behind a redesign of the scheduler to make it more suited for desktop use. And even after he stopped working on the main kernel, he maintained a few of his own out-of-tree patches, including the infamous Brain Fuck Scheduler (BFS). You can read about it here,
http://ck.kolivas.org/patches/...

The main argument between CK and the rest of the kernel team was that CK wanted extremely good latency in the linux kernel, and he wanted to set it up in a way to be configurable, so you could purpose a machine for a specific task (desktop or server) with different priorities. The main kernel team (Linus, Alan Cox, Greg Kroah-Hartman, Ingo Molnar, etc) wanted a more elegant solution. They wanted an intelligent CPU scheduler that could scale well depending on the load and hardware capabilities, with the idea being that a sysadmin should not have to think too much about it. A mainframe has very different hardware, but it also is used for a very different purpose, from a desktop machine, so the kernel should be able to realize that and figure out what it needed to prioritize. There were a few tuning variables exposed, but it was not supposed to require an extensive configuration. Also, they specifically had the developer workstation in mind when they were working on the scheduler. That is, they were considering situations where a user likely wanted both interactivity and throughput performance. The best of both worlds as best they could manage. They went back and forth a few times until CK finally got frustrated and left, because he didn't think it was possible.

So, there is BFS, and it probably still can be used to patch the stock kernel, but CK didn't design it to be pluggable. So if you patch your kernel with it, you will lose the other options, like the Completely Fair Scheduler (CFS). However, this thread on Stack Overflow might be a good resource if you are either interested in writing a new scheduler, or you want to modify BFS to be pluggable.
https://stackoverflow.com/ques...

And, I was Googling a bit and found that CK is apparently working on another scheduler (MuQSS) which he is intending to be the successor to BFS,
http://ck-hack.blogspot.pt/201...

In the end, though, while CFS is not as good as BFS at latency, it strikes a pretty good balance. And GKH has been working on soft realtime patches to the kernel for a number of years that are slowly getting incorporated into the main branch. So Linux is pretty good in general; much better than it was when CK first started his work. Here is an old, but nice, comparison of the two schedulers,
http://cs.unm.edu/~eschulte/cl...

As others have said, the headache you will have if you do want to come back (potentially years later) to that one email you know you had only to find your attachment-stripping program has foobar'd the whole archive up (or that you need the attachment after all) probably isn't worth the hassle for saving 500MB per year this year (even taking into account reasonable growth rates - I'd note that bandwidth per $, which will be the factor limiting your email size, has been growing rather more slowly than storage capacity per $ over the past decade and things are likely to continue that way).

If the problem is that you have significant duplication between emails (e.g. the same attachment being emailed several times), gzip and bzip2 may well miss the opportunity to de-dupe this because the distance between duplicated sections is large. One solution to consider if this is an issue may be to use something which is better at compressing over long distances. I would suggest trying something like lrzip to compress tarballs of the annual sets of mbox files before archiving those.

Of course, if you just have lots of attachments which *aren't* duplicated (which is probably more likely), that won't really help much.

I guess the lesson is: if you want to get a change made, don't be a whiny dick about it and throw your toys out of the pram when you don't get your own way on the first attempt. Bless.

Its explained in the BFS FAQ http://ck.kolivas.org/patches/bfs/bfs-faq.txt

Why "Brain Fuck"?

Because it throws out everything about what we know is good about how to design a modern scheduler in scalability.
Because it's so ridiculously simple.
Because it performs so ridiculously well on what it's good at despite being that simple.
Because it's designed in such a way that mainline would never be interested in adopting it, which is how I like it.
Because it will make people sit up and take notice of where the problems are in the current design.
Because it throws out the philosophy that one scheduler fits all and shows that you can do a -lot- better with a scheduler designed for a particular purpose. I don't want to use a steamroller to crack nuts.
Because it actually means that more CPUs means better latencies.
Because I must be fucked in the head to be working on this again.
I'll think of some more becauses later.

Con Kolivas is still actively working on BFS, it's not unsupported. He's even got a patch for 2.6.36, which was only released on the 20th.
http://ck.kolivas.org/patches/bfs/

He's also got a patchset out that I use on all my desktops which includes a bunch of tweaks for desktop use.
http://www.kernel.org/pub/linux/kernel/people/ck/patches/2.6/

No kidding. Talk about misinformation. An editor should really fix the summary.

Con Kolivas is still actively working on BFS, it's not unsupported. He's even got a patch for 2.6.36, which was only released on the 20th. http://ck.kolivas.org/patches/bfs/ He's also got a patchset out that I use on all my desktops which includes a bunch of tweaks for desktop use. http://www.kernel.org/pub/linux/kernel/people/ck/patches/2.6/

Since it's all text, I'm surprised that 40GB only compressed down to 15GB. I wonder how small it would be if he used lrzip with max settings instead... I didn't see mention of which type of compression was used in the short article.

Yet Linus reaction was to create his own kernel.

While Kolivas reaction was to give up kernel development.

Wrong .He still does kernel development, with his own patches, just like Linus made his own kernel.

http://linux.slashdot.org/story/09/09/06/0433209/Con-Kolivas-Returns-With-a-Desktop-Oriented-Linux-Scheduler?from=rss

http://ck.kolivas.org/patches/bfs/bfs-faq.txt

Glad you enjoyed it. The initial flamebait moderation made me think it would be buried. Thanks to your sibling poster.

And Con Kolivas didn't completely quit. He made a comeback of sorts with the Brain Fsck Scheduler. http://ck.kolivas.org/patches/bfs/bfs-faq.txt

I guess kernel development is too addictive to quit.

The upshot is that this is huge for Linux in certain business areas (and other RT OSes are currently quite pricey), but totally useless for your desktop or home server.

I don't think so. If this means we'll get a pluggable scheduler architecture in Linux, i'm all for it.

I thought it was all hogwash until i tried Ken Colivas' BFS patch. The difference it makes on a desktop system is notable, and it clearly demonstrates that having a single scheduling solution for a kernel oriented for everything from embedded systems to desktops to 4096-CPUs servers is insane.

I really like the design of his website. Even IE renders it correctly... That's what I call a clean design, UI experts and gnome devs take note!!

For those that didn't catch it CK is back with the Brain Fuck scheduler, which improves desktop interaction, by ignoring the past. I CBA to recompile but the patches are here and while the chances of it getting into mainline are "LOL", it is being adopted by android.

The content of http://ck.kolivas.org/patches/bfs/bfs-faq.txt below, geeks wanna know

FAQS about BFS. v0.209

Why did I write it?

After years of using my old kernel and numerous hardware upgrades, I finally
had hardware that needed a newer kernel for drivers and to try out the newer
filesystems. Booting the mainline kernel was relatively reassuring in that
the scheduler behaviour was much better than what was in earlier kernels.
However, it didn't take long before I started being disappointed in that too.
Random stalls in mouse movements, keypresses, strange cpu distribution in
various workloads and unpredictable behaviour all around were exactly what I
was hoping had gone away. So I did what I vowed never to do, looked at the
code. After seeing it had grown into a monster of epic proportions I sat down
and thought about what was wrong. One of the key features of fairness and
interactivity that I always argued for were very simple semantics for how
cpu should be distributed, with guaranteed low latencies so that interactivity
was assured by design instead of bolted on. CFS in essence does that, but it
does something else too. It varies timeslice length to try and preserve some
deadline list and it determines cpu distribution based on a run/sleep
relationship. It also is designed to scale to monster proportion hardware
that the common man will never see. The whole sleep calculation thing is
exactly what I found was responsible for making varied behaviour under
different loads and relative starvation and unfairness. It's not a profound
effect in CFS and that's admirable. It just doesn't behave the way I feel
the scheduler should being forward looking only (not calculating sleep) and
it doesn't really make the most of a relatively lightly loaded machine without
many many cpus. So I threw it all out and wrote exactly the opposite.

What is it?

BFS is the Brain Fuck Scheduler. It was designed to be forward looking only,
make the most of lower spec machines, and not scale to massive hardware. ie
it is a desktop orientated scheduler, with extremely low latencies for
excellent interactivity by design rather than "calculated", with rigid
fairness, nice priority distribution and extreme scalability within normal
load levels.

Extreme scalability within normal load levels? Isn't that a contradiction?

For years we've been doing our workloads on linux to have more work than we
had CPUs because we thought that the "jobservers" were limited in their
ability to utilise the CPUs effectively (so we did make -j6 or more on a
quad core machine for example). This scheduler proves that the jobservers
weren't at fault at all, because make -j4 on a quad core machine with BFS
is faster than *any* choice of job numbers on CFS. See reverse scalability
graph courtesy of Serge Belyshev showing various job numbers on a kernel build
on a quad core machine. The problem has always been that the mainline
scheduler can't keep the CPUs busy enough; ie it doesn't make the most of
your hardware in the most common situations on a desktop! Note that the
reverse scalability graph is old; the scalability has improved since then.

Why "Brain Fuck"?

Because it throws out everything about what we know is good about how to
design a modern scheduler in scalability.
Because it's so ridiculously simple.
Because it performs so ridiculously well on what it's good at despite being
that simple.
Because it's designed in such a way that mainline would never be interested
in adopting it, which is how I like it.
Because it will make people sit up and take notice of where the problems are
in the current design.
Because it throws out the philosophy that one scheduler fits all and shows
that you can do a -lot- better with a scheduler designed for a particular
purpose. I don't want to use a steamroller to crack nuts.
Because i

Con actually put up a scheduler related patch last month, though he placed it in the crap category:

sched-weight-io.patch

I was hoping that he will come back and fix bug 12309, i.e. the bug that appeared on slashdot 6 months ago and still has no resolution. However, after reading this, I will just wait for FreeBSD 8 and make the switch.

Shake. Con Colivas defrag. ext4 will ship with a defrag tool.

You shouldn't say "someone did", it was Kolivas himself that first offered the pluggable scheduler patch so that his patch could be used along side any new future schedulers and offer a concrete way to benchmark the changes caused by scheduling. And this was done years ago, circa 2004: http://ck.kolivas.org/patches/plugsched/

Of course, Linus and Ingo rejected those patches as well.

Yes, LZMA is good, and more importantly it's free (would you really trust your data to some binary blob implementing a secret algorithm?). On Windows there's the excellent 7-zip (also free) and on Unix you can use LZMA Utils to get a gzip-style single file compressor, though it's still a bit developmental and it doesn't have gzip or bzip2's advantage of being well-known and installed everywhere.

However, the very best lossless compression, not mentioned in the article, is probably lrzip which combines LZMA compression with a pre-compression stage of shuffling around the data somehow (a bit technical I know, but bear with me). It likes to gobble memory but it tends to be either much smaller than bzip2, much faster than bzip2, or both.

Personally, I think they shouldn't have tested on a laptop with a 5400 RPM HDD.

I'd go with a system that spouts pretty fast dual-channel memory and a fast HDD and focus heavily on I/O tests.

Wasn't http://members.optusnet.com.au/ckolivas/interbench / designed with this in mind?

Or http://ck.kolivas.org/kernbench/

I'd rather see interbench scores than FPS.

Renicing X is not recommended with 2.6 kernels (run "uname -r" to check the kernel version):

"A common trick to improve desktop performance...was to renice X to a negative number. The tuning in the 2.6 kernel scheduler...[is] specifically designed to not need [negative] nice levels for any normal userspace tasks... If X is reniced to a negative number it is very easy and in fact likely that audio playback will suffer under heavy use of the desktop... To see what nice level X is currently running at, start the 'top' utility and look in the column NI. 0 is ideal, and any negative number is bad. You can fix the situation instantly (as root) with 'renice 0 pidof X'..."

However, Con Kolivas maintains a patchset for desktop users which incorporates a fix that allows the nVidia drivers to work at his kernel patch page. If you don't want the other stuff and just the nVidia fix, you can find the patch split out, and instructions on which patches to apply in his announcement of his patchset release. Check out the -ck patch though, it has a lot of cool stuff.

(yay, I actually got a story submission in...hi mom!)

i run Gentoo and had no trouble getting Cedega working.

that said, i also use Con Kolivas' kernel patchset. initially i had problems, but we came up with a nice list of audio tips to help get things working right.

i'm waiting right now for some work Ingo Molnar has indicated he's going to do that could help Wine out dramatically. be prepared to recompile your kernel several times in the near future.

Sorry, the Con Kolivas link should be kernel.kolivas.org, but you can find the patches on the other one ;-)

I've been using the -mm patchset since 2.5.4x and have been very happy with it. Since it includes the Interactivity patches from Con Kolivas it kicks ass on your desktop, too. Even moreso than the 2.4.x-ck series of patches, which are intended for desktop use. Do note however that it is sometimes more experimental in nature than the mainline kernel, since new functionality is often tested out there first.

If you know how to patch your kernel already, you don't need to read the article, get the patch for your kernel here: http://ftp.kernel.org/pub/linux/kernel/people/akpm /patches/2.6/

If you want the new scheduler, preemption and other goodies on the stable 2.4.21 kernel, try the CK patch.

I think it requires the CK patch to change it. The patch also includes other low latency features which can be quite useful.