Debating the Linux Process Scheduler

An anonymous reader writes "The Linux 2.6.23 kernel is expected around the end of the month, and will be the first to include Ingo Molnar's much debated rewrite of the process scheduler called the Completely Fair Scheduler. In another Linux kernel mailing list thread one more developer is complaining about Molnar and his new code. However, according to KernelTrap a number of other Linux developers have stood up to defend Molnar and call into question the motives of the complaints. It will be interesting to see how the new processor really performs when the 2.6.23 kernel is released."

Comments

THIS FP WAS POSTED BY THE LINUX PROCESS SCHEDULER.

[Anonymous Coward]

Can someone provide some insight?

[Opportunist]

Is someone who does understand the differences able to explain, in non-kernel-developer terms, what the big differences will be for the average user, developer or administrator? I mean, I'd love to discuss it, but first of all I'd want to know what we're discussing.

Re:Can someone provide some insight?

[Jeff DeMaagd]

I'd love to discuss it, but first of all I'd want to know what we're discussing.

That's not the Slashdot way. We're supposed to have an unfounded opinion based on insufficient facts and preexisting prejudices.

Re:Can someone provide some insight?

[Aladrin]

Average? Probably nothing. But for devs/admins that are worried about certain processes taking more time than others, it -should- be more fair and keep things running smoother.

It's possible for programs right now to exploit how the current schedule dishes out time. As far as I know, they currently only do so out of ignorance, rather than malice. The new scheduler just corrects the problem.

It's not something a user can really see unless they know exactly what they are looking for, and unless a dev/admin has a program that's behaving unfairly, it's not really going to matter to them, either.

There is another invisible effect as well... Kolivas apparently publicly announced his decision to stop working on the kernel, which would include the current scheduler. That means finding another maintainer for his code, should any problems surface. If you've got 2 pieces of code that test the same in speed (as they do according to some), and 1 has a dev that's willing to keep working on it, and the other doesn't... Which would you pick?

The new code also has the added advantage of being a really really neat idea, which encourages people to work on it as well.

Re:Can someone provide some insight?

[Nosklo]

what the big differences will be for the average user, developer or administrator? I guess we must wait until it is out so we can test and benchmark both options.

Re:Can someone provide some insight?

[poetmatt]

From what I understand (correct me if I'm wrong), its a rewrite to how the kernel handles process thread priority and resources, something that was an issue before. Not intentionally but certain programs would put themselves far above where was needed or something.
Made little to no effect on anything because it wasn't a huge change, it was a huge forum debate but more of a small thing (like deciding to use 91 octane instead of 89 in a car - this is not even remotely a life or death situation). Obviously changing this priority could improve or degreade performance randomly depending on the processor and program run, but it's unlikely that it's going to make a huge difference on the performance of a program in the first place. This is in comparison to the previous scheduler

Re:Can someone provide some insight?

[Zephiris]

Essentially, the difference is how well processor resources are divided up, how evenly, and how big the pieces are each process or task gets. Most anyone who has used Linux has had the dreaded moment where you're trying to multitask a bit, and are compiling a program while listening to music, or waching a video, and then...that's terrific, video frames are dropped, or the audio skips. Even if intermitant, it's quite annoying, at the very least. The 'Completely Fair Scheduler' is an attempt to have more fair, sane, and generally less complex scheduling. This also happens to reduces the worst case latencies, averaging from (at least on the tests with my computer) 120+ms on vanilla 2.6.22 scheduler, to ~2.6ms with CFS.

It's largely a drastic improvement over the old scheduling mechanisms that Linux has relied on, although other OSes have largely worked through such problems some time ago.

While it's not exactly THE most scientific, I had a few rounds of testing over which did better on load vs. things still behaving exactly the way they should. I ran all of them with audio playing through KDE artsd, video player, glxgears, etc, loaded, plus inducing a CPU load via 'stress'. Linux, even with CFS, it's still fairly easy to 'upset' it by just producing a fairly large (2-4) amount of load. Solaris did notably better. While it seemed to have a few quirks with scheduling in general, it could sustain a load of around 8-12 without producing video/audio frame drops. FreeBSD, with the experimental SCHED_ULE 2.0 scheduler (as of March 2007) could sustain a load of over 80 with no problems, frame drops, or even glxgears slowing down to a complete crawl (although you wouldn't want to especially use OpenGL at such, it was still getting the speed of software glxgears), and even at 120+ load, the mouse wouldn't respond, while everything else kept going fine. This seems purely useless, but it really comes in handy if trying to do one or more KDE compiles while watching video, on Linux, this tends to be prevented. For the uninitiated, load averages like that are basically a multiplier vs. how much actual work your computer can do in real time. Eg, a 0.5 load would mean you're doing 50% of what you could in realtime. A 2.0 load means you're trying to handle twice what you can do in realtime, it is weighted against how many processors you have (I have one), but other things like disk access can also contribute to the load average, depending on OS.

So, longer story short, a superior CPU scheduler can make a world of difference in how things behave when your system's something else with the CPU(s) at the same time.

Re:Can someone provide some insight?

[FauxPasIII]

Each process running on Linux has a "niceness" value which you as the user can set. The value indicates which ones you want to have more access to CPU power. The numbers range from 19, meaning roughly "only use the CPU when noone else needs it", to -20 meaning "all your CPU are belong to it".

The new scheduler will make those values behave more like they're supposed to relative to one another, and hopefully use fewer resources for itself in doing so.

Re:Can someone provide some insight?

[Anonymous Coward]

...it could sustain a load of around 8-12 without producing video/audio frame drops. FreeBSD, with the experimental SCHED_ULE 2.0 scheduler (as of March 2007) could sustain a load of over 80 with no problems, frame drops, or even glxgears slowing down to a complete crawl (although you wouldn't want to especially use OpenGL at such, it was still getting the speed of software glxgears), and even at 120+ load, the mouse wouldn't respond, while everything else kept going fine... I'm calling bullshit on that one.

Re:Can someone provide some insight?

[Anonymous Coward]

Typical Linux response. Usually it's Linus that calls bullshit, then other people discover it's true, then Linux appropriates it, and acts like it invented it.

Object lesson: /dev/poll. Linus ranted about how Solaris must have "cheated", then mysteriously the same architecture appears in Linux less than a year later.

Re:Can someone provide some insight?

[Opportunist]

That sounds sensible. With the increase of Linux boxes run by "average" people (who will not directly notice a difference), the threat of malware for Linux is going to be on the rise, too. And those people usually know how to exploit even the smallest flaws in a system.

Re:Can someone provide some insight?

[Hatta]

Linux, even with CFS, it's still fairly easy to 'upset' it by just producing a fairly large (2-4) amount of load. Solaris did notably better. While it seemed to have a few quirks with scheduling in general, it could sustain a load of around 8-12 without producing video/audio frame drops. FreeBSD, with the experimental SCHED_ULE 2.0 scheduler (as of March 2007) could sustain a load of over 80 with no problems, frame drops, or even glxgears slowing down to a complete crawl (although you wouldn't want to especially use OpenGL at such, it was still getting the speed of software glxgears), and even at 120+ load, the mouse wouldn't respond, while everything else kept going fine. This seems purely useless, but it really comes in handy if trying to do one or more KDE compiles while watching video, on Linux, this tends to be prevented.

Well damn. That's just... embarassing. Why's everyone using linux if it sucks so much?

Re:Can someone provide some insight?

[heelrod]

Bullshit on you dude. You got anything to back that up?

Re:Can someone provide some insight?

[Anonymous Coward]

Okay, a load of 1-2 locks up the Linux kernel, Solaris, can do 8-10, and FreeBSD, an open source, lets see how this works, can copy into linux, etc. can pull off 80 to 120? That makes complete sense. So sorry for doubting an unattributed "study" posted on Slashdot.

I just ran a test on my machine, and I can model the entire atmosphere at the particle level, with complete environmental data, as well as random occurrences of something that I like to call "The God Effect", Break RSA encryption, and watch "Cars" DVD in realtime.

BULLSHIT!

Re:Can someone provide some insight?

[MajinBlayze]

The interesting thing that seems to be missing from the distribution packagers is that everythin comes defaulted to the same nice level, even if you are running a "desktop" distro or "server" distro. It seems that it would make sense to set up some default nice values (or even a profile-like setup) that could assign priority to where it would be noticed the most (i.e. media, desktop interactiveness).

Re:Can someone provide some insight?

[Anonymous Coward]

Apparently as much as the O/P does...

Re:Can someone provide some insight?

[dpilot]

Do you want your media to play without skips or drops while you're compiling your new kernel?

There have long been tricks like "interactive priority boost" or "nice -10 X" that attempt to make the desktop more responsive, and media play smoothly. But others believe those are just tricks, bound to misbehave in corner cases, and that a good scheduler and well implemented priority scheme will do just as well without the drawbacks. That's where CFS is trying to be. In particular most desktop responsiveness is of the sort, "I need a little CPU, and I need it NOW!" while compiles and such are "I need lots of CPU, and I'll take it whenever I can get it." The CFS keeps track of not just who's using a timeslice, but how much time they're using. That way, those short bursts of CPU keep their priority intact, while more CPU-intensive processes tend to get some priority degradation.

This goes back a little farther than Ingo Molnar's current involvement. A while back, Con Kolivas began putting in a bunch of work on the scheduler trying to get desktop response to work right, essentially he wanted his media, and his compiles, too. He did a lot of work and attracted a lot of users and fanbois along the way. More recently, Ingo Molnar get interested too, and came up with the "Completely Fair Scheduler." When it came time to pick one, Linus saw the CFS doing pretty well, still under heavy and active development. CK's scheduler was also pretty good, but the fanbois poisoned the waters, insisting that it was perfect as it was, and didn't need fixing. Linux chose an active development model over "perfection." Unfortunately Con Kolivas felt slighted in the process, and left. IIRC, he may have been absent during the decision window, and his fanbois did him in.

Add to all of this the fact that the kernel can now run tickless, so that laptops can really scale back their power in between keystrokes or while you're reading the screen. There has been quite a bit of interesting work on scheduling, lately.

Re:Can someone provide some insight?

[the_B0fh]

I don't know, and don't care about the schedulers, but that last bit is a bit... disingenuous. The only reason Kolivas stopped working on his scheduler is because his was kicked out.

Re:Can someone provide some insight?

[berashith]

But then no one would be willing to pay me to do it.

Re:Can someone provide some insight?

[Anonymous Coward]

I just ran a test on my machine, and I can model the entire atmosphere at the particle level, with complete environmental data, as well as random occurrences of something that I like to call "The God Effect", Break RSA encryption, and watch "Cars" DVD in realtime. And I can watch "Cars" being rendered in realtime while doing all this.

Re:Can someone provide some insight?

[ciroknight]

Kolivas apparently publicly announced his decision to stop working on the kernel, which would include the current scheduler. That means finding another maintainer for his code, should any problems surface. If you've got 2 pieces of code that test the same in speed (as they do according to some), and 1 has a dev that's willing to keep working on it, and the other doesn't... Which would you pick?

Wow, not even a full year has past and we're already getting revisionist historians trying to change the situation.

Kolivas quit because of the scheduler debacle, because nobody would listen to Kolivas but were apt to follow Linus and his cronie Ingo around when they drum up more-or-less the exact same thing. Instead of critically listening to Kolivas' points, Linus and Ingo attacked Kolivas' merits. Under that kind of personal attack, I couldn't say I wouldn't have quit just to shut them up. Not all of us are stubborn mules and jackasses.

Re:Can someone provide some insight?

[Anonymous Coward]

The average person with a below average PC will have a slightly better experience. This is work to improve interactivity (Mouse jumpiness).... I know with old hardware and windows, music can get choppy because extra packets arrive on the ethernet. That sort of thing is pretty ugly.

The scheduler is important and I think in general people will have a slightly better desktop experience. On fast hardware, you probably wouldn't really notice.

Re:Can someone provide some insight?

[mamer]

Another question: I've been waiting for OpenMP support in gcc, it seems to be coming soon. In the meantime, I tried to parallelize my code by hard coding my threads using pthreads. I tested it by running several matrix-matrix multiplications "in parallel" on a multi-core CPU, and what I've got was all threads running on the same processor. Only after increasing my number of threads to a couple of dozens, I get some of them to run on the second processor. So basically, I am not getting any performance gain. I asked a number of people an they tell me "this is an old problem, basically the Linux kernel scheduler is stupid and nobody has bothered to fix it". Now, is Ingo's new scheduler fixing this? if gcc-openmp relies on the kernel scheduler, should we expect that open-mp will basically work-but-not-really-work on shared memory multi processor machines? I think this is an important issue to address, especially in an era where high-performace computing has become the driving force behind the hardware. BTW, how are otehr commercial compilers overcoming this scheduling problem?

Re:Can someone provide some insight?

[Ant P.]

I haven't bothered testing it with numbers, but I've definitely noticed the improvements since 2.6.0.
Back then, skipping sound happened just by dragging windows around. In 2.6.22, the only time I've had that happen is when something else in my system went berserk and ate all my ram+swap up, or a full-on crash.

Re:Can someone provide some insight?

[Anonymous Coward]

25% Funny, 25% Insightful, 25% Informative, 25% Troll and 25% incorrect math.

Re:Can someone provide some insight?

[WhiteWolf666]

Average developer or administrator? Your system will be more "stable" under heavy loads, with fewer/no processes starved for CPU cycled. The new scheduler (building on Kovilas work in an unfriendly fashion) better divides up processor time among multiple tasks.

Average user? Multimedia tasks will not skip or stutter while the system is under load. The opposite of Vista's network performance taking a nose dive while playing MP3s, Linux systems with the new scheduler will see little/no impact from background/normal operations on their gaming, music, and video. Your mouse won't skip around while the system is loaded, and responsiveness will remain high except in situations involving super-heavy I/O usage (I/O starvation is more difficult to solve than CPU starvation).

It actually makes a substantial difference, and the system is much more fun to use.

There are some informal test results (LKML) from kernel trap:

here's an update: checking whether Wine could be a factor in your
problem i just tested latest CFS against latest SD with a 3D game
running under Wine: v2.6.22-ck1 versus v2.6.22-cfsv19 (to get the
most comparable kernel), using Quake 3 Arena Demo under Wine (0.9.41).
Here are the results in a pretty graph:
      http://people.redhat.com/mingo/misc/cfs-vs-sd-wine-quake.jpg
or, in text

2.6.22-ck1 2.6.22-cfs-v19
quake + 0 loops 41 fps quake + 0 loops 41 fps
quake + 1 loop 3 fps quake + 1 loop 41 fps
quake + 2 loops 2 fps quake + 2 loops 32 fps
quake + 3 loops 1 fps quake + 3 loops 24 fps
quake + 4 loops 0 fps quake + 4 loops 20 fps
quake + 5 loops 0 fps quake + 5 loops 16 fps

Quake3-under-Wine behavior under SD/-ck: framerate breaks down massively
during any kind of load. The game is completely unusable with 1 CPU loop
running already!

Quake3-under-Wine behavior under CFS: framerate goes down gently with
load, gameplay remains smooth. Framerate is still pretty acceptable and
the game is playable even with a 500% CPU overload. The graph looks good
and the framerate reduction goes roughly along the expected 1/n
'fairness curve' - so it all looks pretty healthy. [Note: quake3 keeps
its fully 41 fps even with 1 competing loop running on the CPU due to
"sleeper fairness".]

Re:Can someone provide some insight?

[TheRaven64]

Well damn. That's just... embarassing It's more embarrassing than the grandparent states, actually. ULE 2 has been improved a lot in the last few months, and the newer ULE 3 performs a lot better, particularly on SMP systems. From what I've seen, the new Linux scheduler has roughly the same shape (and size) performance curves on 1-8 CPU systems as the old 4BSD scheduler that ULE replaces.

Why's everyone using linux if it sucks so much? Because Linux sounds cool, while BSD sounds geeky. I was recently reminded of how badly Linux sucks when I went over some old code I'd written to get the CPU name and speed. The FreeBSD and OpenBSD implementations of this code each called a single sysctl for each result. The Linux version had to read /proc/cpuinfo and parse it. Actually, it had to parse it in two different ways, because it turns out the format of cpuinfo is different on x86 and all other platforms. Reading the battery life was even worse. On FreeBSD it's just a matter of reading the hw.acpi.battery.life sysctl (one line of code). For Linux, it involves parsing some messy procfs stuff with a format that has a habit of changing between releases. I don't understand how a developer could prefer Linux to any other UNIX.

Re:Can someone provide some insight?

[peragrin]

What goes around comes around.

Revisionist history is working both ways I see. Whenever Linux or another kernel developer would bring up a point of failure in Kolvias's scheduler instead of Fixing the problem Kolvias would lash out and say it wasn't broken.

CFS won not because it was a better scheduler at the time, but because Inglo worked with the developers to make it better, instead of fighting everyone who questioned anything about it. FOSS projects are about helping everyone, and listening to new Ideas. Something Kolvias was having a hard time doing.

That is at least how i read the whole debate.

Re:Can someone provide some insight?

[Anonymous Coward]

Speaking of revising history, you're doing a pretty good job of that yourself.

Re:Can someone provide some insight?

[dysfunct]

I wouldn't be so fast to call bullshit. FreeBSD has always been known to scale better than many other operating systems, although I must have to admit that I haven't yet tried SCHED_ULE. On the regular scheduler (SCHED_4BSD or similar) everything will *still* crawl to a halt at some load level, be it does so pretty gracefully.

I mean, why shouldn't it? Things like audio skipping while compiling simply happen because one or a few processes are given more time for the sake of throughput while "starving" other processes. A load of 80 over 5 minutes only means that on average there were 80 processes waiting for CPU time. If you gave each of them enough CPU time (and didn't have many other distractions like IRQs) why shouldn't they run pretty smoothly assuming the bottleneck wasn't the CPU itself?

Re:Can someone provide some insight?

[Wdomburg]

Kolivas maintained the SD scheduler, which never made it into mainline.

Re:Can someone provide some insight?

[SEAL]

Whenever Linux or another kernel developer would bring up a point of failure in Kolvias's scheduler instead of Fixing the problem Kolvias would lash out and say it wasn't broken.

CFS won not because it was a better scheduler at the time, but because Inglo worked with the developers to make it better, instead of fighting everyone who questioned anything about it.

I believe Linus was the claiming Ingo worked better with the developers (or at least I saw him write to that effect on the LKML). By contrast, Kolivas had many individuals helping out with his branch who were quite pleased with his progress.

The problem is that Kolivas was working to help desktop, and particularly 3D game users. He'd say it wasn't broken if it was optimizing for those particular platforms at a cost of a fraction of a percent Oracle performance. Most of the kernel devs don't care 2 cents about 3D or desktop users, and many are employed by large enterprise businesses, so you can see who won.

Personally I think Kolivas should've been given full access to merge his code. Many of us found his work very useful and it's sad to see him driven away by a few people who are oblivious to what everyday users need. Even if his scheduler was not the default one, it still would've been nice to have as a mainline kernel option.

- SEAL

Re:Can someone provide some insight?

[recoiledsnake]

CFS won not because it was a better scheduler at the time, but because Inglo worked with the developers to make it better, instead of fighting everyone who questioned anything about it.

That was certainly not how I read it when it happened. I came off with the impression that Kolivas was ripped off of his ideas by Inglo and that Linus was wrong in brushing Inglo off and going with Inglo.

The LKML email linked reinforces my suspicion. Habits and attitudes don't die easily. We see ANOTHER guy complaining that his patch was half bakedly ripped off with no explanations whatsoever and a tangential acknowledgement.

FOSS projects are about helping everyone, and listening to new Ideas. Something Kolvias was having a hard time doing.

Did you even read the linked email? Inglo seems to be talking in patches and with no discussion. If something were to happen to him or if he quits due to any reason, there would be no one who would understand the code and the features behind it enough to maintain it.

Re:Can someone provide some insight?

[Opportunist]

Folks? Could we concentrate on creating a good system instead of egos?

You know, this really has a lot of the Life of Brian. "This is his gourd", "No, we'll follow his sandal".

The nice thing about Linux is that we can actually choose which one we prefer. Put both into the distributions and let the people sort it out. The better one will prevail.

Re:Can someone provide some insight?

[GooberToo]

Average? Probably nothing.

I don't think that's really fair. Average these days actually has a large spread. Average web user? Average game player? Average video watcher? Average musician? Averaging what? And that's the point of the CFS. Each of those users have different expectations and each reflect a different type of scheduler workload. The old scheduler has lots and lots of code to handle various performance problems for corner cases for each type of "average user" depicted above. This in turn means the code is hard to read, hard to understand, and even harder to maintain. Worse, some code which help some corner cases actually make things worse for others. This in turn means more special case coding and even more complex testing.

The CFS is designed to simplify most everything the O(1) schedule does without tons of complex heuristics and special code to address various corner cases. In other words, in stead of trying to guess what you really want, the CFS simply tries to be as fair as possible for everyone, thusly ensuring many categories of corner cases are simply no longer an issue with the CFS.

This does not mean CFS is always better than O(1), but early results indicate they are traveling down a good road. In fact, the last round of patches I read about, actually establish CFS ~12% faster then the old O(1) schedule. Of course, it's yet to be seen how well it will be received and how quickly it will prove it self with a large user base. The devil is in the details with schedulers and sometimes real world user workloads don't reflect anything which has been tested/validating during development.

Nonetheless, development on CFS continues and it certainly is providing hope it will be smaller, faster, provide lower latency, be easier to maintain, and address a wider range of workloads, including many corner cases, without requiring complex code to track and analyze heuristics.

Long story short, yes, if all goes well, even the average user may notice an improvement depending on the particulars of their workload. Wouldn't you notice a more responsive desktop? ;)

Re:Can someone provide some insight?

[GooberToo]

Well damn. That's just... embarassing. Why's everyone using linux if it sucks so much?

Take his comments with a grain of salt. You can safely and completely ignore his comments about worst case latency because they are meaningless. In the Linux kernel, worst case latency almost always is caused by device drivers. Besides, most people have no idea how to properly measure worst case latency...and the measurement method may not reflect a real world workload in any meaningful way.

Depending on the devices you have, using otherwise identical hardware, worst case latency can vary based on an order of magnitude and the workload at hand. This is why realtime Linux vendors won't provide a guaranteed, worst case latency. Which is to say, they have no idea what your hardware will be so they don't know. Despite this, making some assumptions, worst case latency with good hardware and proper device drivers and reasonable CPU will usually be measured in single to double digit ms. Anything outside of this is enough to make one question the method of measurement, the workload use, and/or the machine/hardware specs involved.

Re:Can someone provide some insight?

[EvanED]

The opposite of Vista's network performance taking a nose dive while playing MP3s, Linux systems with the new scheduler will see little/no impact from background/normal operations on their gaming, music, and video.

It's not quite the opposite, because the reason Vista's network performance drops is to achieve that same result.

Now, whether there was a better way that MS could have gone about it is an entirely different matter. But, if they indeed found that they needed to drop network traffic to get short latencies for multimedia, my vote is that this was a worthwhile tradeoff.

Re:Can someone provide some insight?

[m50d]

Well damn. That's just... embarassing. Why's everyone using linux if it sucks so much?

The same reason as windows: hardware support.

Re:Can someone provide some insight?

[fastest fascist]

You know, it's stuff like this that shows Linux is NOT meant for consumer use, and why attempts at making user-friendly desktop distros still feel rather patched together. A -20 to 19 scale? What the hell? I'm sure there's some valid reason why that particular scale is used, but I'm also sure that reason has very little to do with ease of use. To someone without arcane knowledge, that is a completely arbitrary range, as so much about Linux systems seems to be arbitrary. And when something appears arbitrary to the user, they tend to find it hard to learn and/or remember.

Re:Can someone provide some insight?

[SL Baur]

When it came time to pick one, Linus saw the CFS doing pretty well, still under heavy and active development. No, that's not really what happened. Based on my reading of the list at time, Linus had already decided to use CFS from the time it first was published.

Two things happened that were very misleading. The first was the initial CFS announcement. It appeared at the time, that at the same time that Ingo was being highly critical of SD, he was implementing a variant behind everyone's back. This wasn't true; I have no reason to doubt his words that he did throw together the first version in a day or so of hacking. The second was the manner in which SD was pulled out of the -mm tree. It was in a thread relating to an IDE bug that appeared to be scheduler-related and Andrew posted something to the effect "I guess this is where the experiment stops, I've pulled SD out of -mm". A few days later they did track down the root cause of this bug and it was something in the block layer, I believe - nothing to do with the scheduler.

Of course, what really happened is that Linus wasn't satisfied with how Con was handling bug reports and had a lot more confidence in Ingo as a long-term maintainer. I cannot fault that logic either.

Unfortunately Con Kolivas felt slighted in the process, and left. IIRC, he may have been absent during the decision window, and his fanbois did him in. True, true and true. His fanbois did him no favors using Microsoft Windows XP as the ideal of process scheduling. Besides the long slow automatic Alt-TAB of death, there's the Leeroy Jenkins "I can't move! I can't move!" lockup whenever a battle gets "interesting" (neither of which scheduling artifacts appear in OS X, btw). Yes, this is important because improving the gaming experience was one of the rationales behind changing the process scheduler.

Re:Can someone provide some insight?

[RockoTDF]

Err...only slightly. Unix based operating systems separate user space from the rest of the system. Nothing much can be done without root access. Installing Malware on a Unix system is, for this reason virtually impossible. MA large user base is often used as a rationalization for windows malware, but in reality it is just an incorrect excuse.

Re:Can someone provide some insight?

[dpilot]

I tend to skim LKML once in the morning, and the site I'm using to get there doesn't have a thread view, either. So it's pretty easy for me to miss depth and details. Usually if something is really active or controversial, it'll endure long enough for my sampling to see it. I did follow the scheduler discussion, including Linus' announcement deciding for CFS, but obviously missed a bit. I didn't catch where he'd pre-decided, only that the SD fanbois had rubbed him the wrong way with their cries of perfection, and obviously ongoing support and development would win the day. Too bad CK wasn't there to speak for himself. These days I'm not sure if I like Roman Zippel's simplifications or if I'd just like him to get over feeling slighted.

Re:Can someone provide some insight?

[cartman]

I was recently reminded of how badly Linux sucks when I went over some old code I'd written to get the CPU name and speed. The FreeBSD and OpenBSD implementations of this code each called a single sysctl for each result.

There are good reasons for exposing cpu info (and other kinds of OS data) as files, rather than gathering that data through system calls. Exposing the data as files makes that data usable to the range of tools people use on an OS (Unix) that supposedly treats everything as files. The data can be grepped, piped, opened, run through 'sed', cut, awked, etc. Exposing OS-internal data as files is done for the same reason that RDBMSes expose internal data as tables, and for the same reason that XML schema are XML documents themselves (this is why people complained about DTDs--they weren't XML).

Linux did not invent this idea. The inventors of Unix (Thompson and Ritchie) came up with the idea of exposing internal OS data as files, which they considered a natural extension of the Unix philosophy of treating everything as files.

You may disagree with it, or you may not be familiar with all the arguments for and against it. However you should refrain from just concluding that "LINUX SUCKS" because something was not the way you expected.

Whether the data should be exposed through files is an arguable point. It's not an area in which BSD is clearly factually superior. My own opinion is that gathering that data through files is better.

Re:Can someone provide some insight?

[Spoke]

Well damn. That's just... embarassing. Why's everyone using linux if it sucks so much?

I suspect that the Solaris/FreeBSD schedules aren't completely fair and instead are boosting the priority of certain processes to achieve interactivity... but I could be wrong, I haven't actually tested them. Not to mention, how many people actually load up their machines to loads over 10?

The latest versions of CFS are much better than earlier revisions (not sure what version exactly the grandparent tested exactly, 2.6.22 was pretty early for CFS the latest 2.6.22.5 has accumulated a number of fixes/improvements) so the situation has likely improved already.

Re:Can someone provide some insight?

[Anonymous Coward]

Would it be so hard to expose the data as files and also provide sys calls to retrieve the information as well. In the worse case the sys calls could just read and parse the files for you. It would make changes to those files be considered more judiciously and also document how to parse them.

Re:Can someone provide some insight?

[rbanffy]

The scheduler has nothing to do with security exploits. This will only distribute cycles more evenly. Additionally the ideal malware should take steps to prevent it from getting too much CPU time in order to avoid detection.

Re:Can someone provide some insight?

[rbanffy]

"The problem is that Kolivas was working to help desktop, and particularly 3D game users."

I don't know for sure, but I think it should be trivial to have a kernel switch activated on boot to set the preferred scheduler. This way, 3D gamers would be happy to set the -ks (I fail to remember its correct name) while the rest of us would be happy to leave it alone and get the CFS. Maybe by having a modular scheduler architecture would allow to have kernels with the -openvz or -oraclesbest or whatever other schedulers one may want (and could get support from the kernel developers).

A slightly more clever approach would be to let one switch schedulers on the fly, but I think it's asking too much.

Re:Can someone provide some insight?

[rbanffy]

"Why's everyone using linux if it sucks so much?"

Well... I am using because it has all the GNU userland tools I love, APT (wouldn't live without it) and because all my hardware mostly works. The rest can be attributed to inertia, because I am satisfied with Ubuntu and don't feel any urge to change.

But claiming BSD performs smoothly under a load of 80 sounds rather suspicious... I would love to see some real benchmarks comparing Linux to BSD to Solaris on that one.

BTW, my quite modest backup notebook sustains a load of 5 (linux 2.6.22, Ubuntu Gutsy) before making me want to log-out.

Re:Can someone provide some insight?

[piojo]

Err...only slightly. Unix based operating systems separate user space from the rest of the system. Nothing much can be done without root access. On a properly configured system, you don't need to be root to send spam, nor to install a keylogger as a firefox extension. On an improperly configured system, it's easy to get root access from sudo by waiting for the real user to type it on a different terminal. (I have never tried writing a script or program that did that, but it seems possible based on my understanding of the semantics of sudo.)

Re: Can someone provide some insight?

[Dolda2000]

I was recently reminded of how badly Linux sucks when I went over some old code I'd written to get the CPU name and speed. The FreeBSD and OpenBSD implementations of this code each called a single sysctl for each result. The Linux version had to read /proc/cpuinfo and parse it. While I agree with you that Linux seems almost pathetic compared to BSD when under heavy load, I have to object to that reason for calling the *BSDs superior to Linux. In fact, the sysctl interface is the single most bothering thing about the *BSDs to me. Isn't the treatment of everything as a file the very core of the Unix philosophy? It may be true that files like /proc/cpuinfo could very well be made easier to parse, but I find nothing to complain about with e.g. the /proc/sys/ tree, or, for that matter, sysfs. Requiring a special syscall just to fetch kernel data that could just as well be made available through the file system is, in my mind, just ugly for a Unix system, and I can only imagine that it is done that way for historical reasons.

In almost any other way, however, I would prefer the *BSDs over Linux. I still run Linux, though, but that's mainly because I want to keep running Kerberized NFS. :) When FreeBSD implements that, I may well switch.

Re:Can someone provide some insight?

[piojo]

You know, it's stuff like this that shows Linux is NOT meant for consumer use, and why attempts at making user-friendly desktop distros still feel rather patched together. A -20 to 19 scale? What the hell? Yes, but it works fine without any of that stuff. Normal users don't need to know about nice. It's more useful for those of us that want to compile two software projects at the same time (one going as fast as possible, while the other can starve for all we care), while listening to audio and running firefox, thunderbird, and one huge momma of a text editor. (On a pretty old laptop.)

Re:Can someone provide some insight?

[AnyoneEB]

You are right. It would be easy. In fact, someone wrote a patch for it called plugsched. It was not accepted into the kernel due to the fact that it would supposedly discourage the idea of simply making a scheduler which worked well for everyone.

Re:Can someone provide some insight?

[ciroknight]

You shouldn't say "someone did", it was Kolivas himself that first offered the pluggable scheduler patch so that his patch could be used along side any new future schedulers and offer a concrete way to benchmark the changes caused by scheduling. And this was done years ago, circa 2004: http://ck.kolivas.org/patches/plugsched/

Of course, Linus and Ingo rejected those patches as well.

Re:Can someone provide some insight?

[ciroknight]

PlugSched would have let us do this in a big way, letting us change at boot or even while the system is running what cpu scheduler is working. But, most distributions pull most of their code from the Mainline and Linus' blessed code, so it's unlikely any distribution will ever see Kolivas' scheduler in action to know whether or not it could do better than Ingo's bastard-clone.

Now we get even more people questioning Ingo's code, and instead of examining the code, we get people starting to question Roman's merits too. The personal attacks over this need to stop, and we need to look at the actual problem instead of looking at the developers who are questioning the choices. I personally hope Kolivas will come back to the kernel and hasn't moved on for good, just needing some time away from the hot-heads and their corporate Linux eccentricity. Desktop Linux needs to start at the Kernel and end at the user, no component between left behind.

Re:Can someone provide some insight?

[X0563511]

Sudo is great, when used right. Ubuntu doesn't use it right.

Basically sudo allows specific users and groups the ability to run specific commands on/from specific hosts... either with or without a password. Useful for instance when one user is responsible for working the print server, and is given root access only for print-related commands on the print server itself - and if they require a password to do so, you don't need to give the super-user password, they can use their own.

It's damage control and compartmenting.

The way Ubuntu does it is makes it easier for a cracker, while the intention is to make it easier for the user. Crackers guess your password or obtain it from elsewhere (average user... easy passwords combined with using the same password elswhere) and bang, they have root. To be fair though, on a single-user system all that compartmentalization is overkill and a hurdle.

Re:Can someone provide some insight?

[ciroknight]

So the worst case is just doing it the way the UNIX designers intended it to be done in the first place. So why not just do it the way the UNIX designers designed it to be done in the first place? Adding system calls for every little thing just means more things you need to port from system to system, having a file with a well-defined interface is more than enough.

If the user that posted his tirade against Linux's UNIX-style design had coded his application correctly to begin with, it'd be a non-issue, all of the code would be hidden away inside of a (rather clean) module and wouldn't have affected his application at all. In fact, there are already existing modules/libraries/pieces of code to do exactly what he wanted to do, so he wasn't "lazy enough" when coding and instead had his "Do it Myself" hat on, completely ignoring the groundwork that people had placed for him to use.

Re:Can someone provide some insight?

[Cajal]

Of course, the parent poster wasn't just complaining that Linux exposes this information as files rather than via sysctl. The real problem is that the format of the information changes frequently, and varies between platforms. /proc/cpuinfo, for example, varies widely between x86, ppc and sparc.

It's exactly this sort of inattention to detail that makes me not enjoy developing on Linux.

Re:Can someone provide some insight?

[rg3]

I'm not trying to debunk your claims or anything, but I tried to reproduce that behaviour in my 2.6.22.6 kernel (no CFS, that will be in 2.6.23), and I couldn't make mplayer drop any frames. I launched mplayer to play an MPEG4 movie with MP3 audio, launched 8 infinite loops and ran updatedb so as to keep a good amount of IO too. mplayer didn't drop any frames and the audio didn't skip at all. The load average was 10 point something. And I set the scheduler governor to powersave, so I was effectively running at 800 MHz instead of the full 1800 Mhz.

Maybe my setup is different. Preemptive kernel, 1000 Hz, CFS IO scheduler. So I'm not sure how a load average of 2 or 4 can disturb the Linux kernel at all. Can you please elaborate on your setup? Maybe it's a bug in artsd or your video player. I have no idea. My mplayer uses ALSA.

Re:Can someone provide some insight?

[cartman]

The real problem is that the format of the information changes frequently, and varies between platforms. /proc/cpuinfo, for example, varies widely between x86, ppc and sparc... It's exactly this sort of inattention to detail that makes me not enjoy developing on Linux.

I think that's just a consequence of the freewheeling, patchwork nature of OSS in general. Granted, BSD may be better than Linux with regard to the kernel (I understand they have a more traditional, hierarchical development approach). But FreeBSD is still distributed with all the OSS tools which are made by different programmers all over the world. We must expect things like different conf file formats, different command-line option formats, different doc standards, etc. BSD may not have that in the kernel, but they have it everywhere else.

Re: Can someone provide some insight?

[TheRaven64]

In fact, the sysctl interface is the single most bothering thing about the *BSDs to me. Isn't the treatment of everything as a file the very core of the Unix philosophy?

To be honest, the 'everything is a file' thing is the thing that bugs me most about UNIX. In principle, I am in favour of an 'everything is a {something}' interface, but files are a horrible abstraction for persistent data storage, and are much worse for other things. Most things that look like files but aren't have a load of custom ioctls associated with them, and so don't really behave like files. They have a convenient shared namespace, but not a shared interface (sure, you can open and stat them, but if you want to get any real work done it's all ioctl manipulation).

Sysctls are nice because they present a single namespace for exposing both text and binary information in the contexts they are most useful. If you want binary information in a C program, you use the sysctl() system call. If you want a text version in a shell script, you use the standard sysctl program. You could put all of the sysctls in /sys (or whatever), but then you'd probably want to make them text-based, which would require you to add a load of fscanf things to your C code. You'd also have the temptation to bloat them up, as happens with Linux, and require more complicated parsing, rather than just a single value in each file. And you would need a minimum of three system calls (open(), read(), close()) to read them, instead of just one (sysctl()).

Re:Can someone provide some insight?

[Anonymous Coward]

If Linux were simply "freewheeling," someone would have unified the formats. Unfortunately, it's both careless *and* incredibly conservative. I was trying to get info out of /proc/n/mem, checked the kernel documentation for my current kernel, still couldn't make any sense of it, and googled out of frustration. I found some casual discussion on the kernel mailing list about how half the fields had never worked (at least on x86), and it was better to create an entirely new way of exposing that information rather than simply implement the interface as it was documented and force developers to fix programs that depend on those fields being zero. What????

How in the world can Linux be freewheeling enough to release broken stuff but too conservative to fix it? How the hell does that make sense?

And how does it make sense to document something as working while intentionally leaving it broken, and perversely believe that this is for the convenience of developers?

Re:Can someone provide some insight?

[Anonymous Coward]

You're speaking of a single individual who was not happy with Kolivas' best efforts to please him. The unaddressed "failures," from him and from others, were not real bug reports but simply cases of people demanding Con to explain how his scheduler could possibly work because of X and Y and Z. When the answers are already in the work you've written up, there's not much you can do in those cases except try to improve your explanations. It looks like Roman has the same problem; he has taken an extremely rigorous approach to scheduler work, and Ingo is simply hacking with a (relatively) vague plan. Everyone wants to hack instead of understanding the theoretical frameworks that both Con and Roman have taken a lot of time to develop and *thoroughly* write up. People just whine and say they don't understand.

People should have at least enough respect to read and understand the work that Con and Roman did before they start ignoring them or raising spurious objections. I'm not saying the kernel team should read every theoretical paper anyone sends them with a patch attached. Not everyone has earned that level of consideration. Roman and Con HAVE earned it. Because Ingo evidently doesn't think the scheduler is important enough to bother understanding the work of two very capable people, which is actually relatively simple (unless you compare it the academic models studied in CS classes,) he looks like an amateur by comparison. It sounds like he doesn't understand how his schedule is similar or different from Con's and Roman's, and doesn't even understand his own scheduler well enough to answer Roman's questions. It doesn't inspire confidence at all. Why doesn't he hand off control to someone who has the confidence to engage with other developers and have a possibly very fruitful exchange of ideas?

Re: Can someone provide some insight?

[Dolda2000]

To be honest, the 'everything is a file' thing is the thing that bugs me most about UNIX. In principle, I am in favour of an 'everything is a {something}' interface, but files are a horrible abstraction for persistent data storage, and are much worse for other things. Most things that look like files but aren't have a load of custom ioctls associated with them, and so don't really behave like files. They have a convenient shared namespace, but not a shared interface (sure, you can open and stat them, but if you want to get any real work done it's all ioctl manipulation).

But the namespace part is the whole point of it. Sure, it would be better without the ioctl part, but it's not as if there is such a thing as an interface which can incorporate anything while still being meaningful on its lowest level. I don't even know what you would argue in favor of instead -- some kind of smalltalk-like system where you can send messages instead (for some meaning of "message")? In that case, it would still be equivalent, since any message can be encoded as a standard Unix bytestream.

In fact, I think that, too, is one of the great things with Unix, as long as one manages to avoid ioctls. Not only can anything be encoded as a bytestream, a bytestream is also the only thing which can be handled by anything -- it can be stored on disk, transmitted over a network, input from a keyboard, sent over an RS-232 link, or hexdumped, printed and put on a carrier pigeon, if you fancy that. There may be some parsing involved, indeed, but I cannot think as anything else so universal.

And the original genius of Unix was, of course, to put all these bytestream end-points into one namespace. Sadly, over time that has become more and more diluted, first with the Berkeley sockets API, and then with the sysctl interface. (I get the distinct feeling that there are other examples as well, but, frustratingly, I cannot think of them)

Of course, I don't claim that Unix is the end of all interface abstractions -- I have quite a few ideas of my own, and Ken Thompson has shown his own ideas with Plan9 -- but it is far better than almost anything currently available today, at least. Certainly better than anything used commercially.

Sysctls are nice because they present a single namespace for exposing both text and binary information in the contexts they are most useful.

But then you have two namespaces; one for sysctls, and the other for everything else. Surely, at least you cannot deny that it would be more elegant to make them one? Furthermore, the sysctl namespace follows dissimilar rules -- It has different path conventions. It has no standardized interface for enumerating a directory, getting/setting permissions, and getting other stat() data. You cannot have a sysctl node as your cwd in a process. &c. (Admittedly, procfs in Linux also violates a few useful rules, like the ability to chmod files in /proc/sys and such things, but that can just be considered a simple bug)

You could put all of the sysctls in /sys (or whatever), but then you'd probably want to make them text-based, which would require you to add a load of fscanf things to your C code.

OTOH, a call to atoi() probably takes less than a percent of the time it takes to context-switch into the kernel, so that would probably be the least concern. As for code elegance, everything needed would be to just add an abstraction function to libc for it.

You'd also have the temptation to bloat them up, as happens with Linux, and require more complicated parsing, rather than just a single value in each file.

As for temptations, all you'd have to do is to resist it. I'd have every confidence in the BSD developers to do that. However, I for one wouldn't even be tempted, and to be honest, I don't think most Linux developers are either. I'd guess files like /proc/cpuinfo is just a historical artifact that just can

Re:Can someone provide some insight?

[SL Baur]

I did follow the scheduler discussion, including Linus' announcement deciding for CFS, but obviously missed a bit. I read lkml in gmail which does threading very badly and exact chronology can be quite difficult to figure out at times. The first Linus post after Andrew removed SD from -mm was one flaming Con & SD and showing strong support for Ingo & CFS - i.e. he had already made up his mind. CFS wasn't in -mm at this time.

It is indeed a pity that Con's health deteriorated when it did. Also, to be fair, I don't think Ingo gets any particular special treatment. He appears to be sitting on a mountain of unmerged patches that probably will never make it into mainline and Linus did flame him pretty hard over the state of the high resolution timer patches that caused so much trouble a couple of releases ago.

Of all the people who have the right to feel slighted, I think Andrew Morton is right at the top of the list. He has a hateful and largely thankless role in kernel development (merging lightly or untested patches is never fun), but still manages to track all kinds of bugs, deliver insightful and sometimes frighteningly humorous code reviews and keep a mind-blowing amount of patches flowing towards Linus. I don't know how he does it, but I'm glad he does it.

Re:Can someone provide some insight?

[simcop2387]

thats just it, the old scheduler has a flaw that allows a program to take all of the power but sleep JUST before the stats are gathered making the scheduler think its not doing anything so that i can use 99% of the power without being seen as the culprit

Re:Can someone provide some insight?

[try_anything]

You don't have to know the whole OS to use a Linux system. Everybody understands that the approach with Grandma is not to educate her about niceness levels but to make sure that she never has to use them. In fact, she already doesn't need to use them. They're just a performance tweak for enthusiasts and professionals who want to get around the ordinary performance limitations. Sure, learning some obscure tricks will allow Grandma to get a little bit more out of her hardware, but the same is true under Windows. Normal users just accept the limits of a stock, untweaked system, and experts squeeze out a bit more performance using special tricks.

I'm not sure what problem you have with the situation -- do you want to eliminate the special tweaks so Grandma isn't jealous of the performance her Linux guru friends get out of their systems? Even if you eliminated all flexibility and tweakability from the OS, there would still be overclocking and compiling applications from source and other tricks employed by people with too much time on their hands. This is all bull, of course; it doesn't matter whether there are complicated things in the system that granny will never understand. She doesn't care. She only cares whether it performs well enough for her to do what she wants.

Re:Can someone provide some insight?

[clayne]

I'm going to call bullshit on most of this, except that dysfunct's post seems the one with the most actual insight - that being the point that not all processes demand CPU as 100% of their running time.

Not all aformentioned OS' even calculate load using the exact same algorithm. Some may choose to include processes waiting on I/O and some may include only processes waiting on CPU.

I've personally had a Solaris 9 box up to a load of 1100+ doing testing with thread capacity handling, etc. and while the host took around 20 seconds to even handle my SIGINT to stop the main load generating process. While I highly doubt anything interactive would function fluidly, I pretty much interpreted the load average as a rough indicator rather than anything concrete.

Re:Can someone provide some insight?

[rbanffy]

"Of course, Linus and Ingo rejected those patches as well."

You make it sound like a conspiracy to suppress Kolivas' work. I am sure it's not.

The explanation that this is a way to steer the development toward a single scheduler that should function well enough for everyone, desktop, embedded or server, works for me. If Linus had issues with the way CK responded to bug reports or doesn't like the idea of supporting two or more different schedulers or if he feels more comfortable to work with Ingo, it's his call. I will continue using the Linux kernel for as long as it is the best solution to my problem.

If someone else wants to publish a different kernel with the plugsched patch applied and a range of different schedulers, CFS, CK or a variation of the BSD ones that got so much praise in another thread of this discussion, it's very simple: Anyone can do it.

Heck... Even Microsoft could release a fork of the Linux kernel with a custom scheduler. It's more or less what all distros do.

Re:Can someone provide some insight?

[martin-boundary]

Indeed. There are known discussions. These are discussions we know that we know. There are known undiscussions. That is to say, there are discussions that we know that we don't know. But there are also unknown undiscussions. These are discussions we don't know we don't know. They occur a lot on slashdot.

Re:Can someone provide some insight?

[bcrowell]

Why's everyone using linux if it sucks so much?
Because Linux sounds cool, while BSD sounds geeky.

I use both Linux and BSD every day. I think there are some sane reasons to prefer BSD, and some sane reasons to prefer Linux. For a desktop user, one really good reason to prefer Linux is that there's no ubuntu for BSD. I've gotten spoiled by the very high quality of work that goes into packaging desktop apps for ubuntu. I used to run BSD on my desktop, and I was always running into hassles getting all my apps to work. There are also, however, some perfectly good reasons for picking BSD. One of the biggest, IMO, is that an OS like FreeBSD is a single operating system, not a variety of distributions like Linux, so you can buy a book about BSD, and it's documenting the exact system you're using. None of this stuff about "do this if you're using Debian, or this if it's Mandriva, or ..." As far as the schedulers, I've had problems with some of the Linux schedulers, and problems with some of the FreeBSD schedulers. Both have been in a very rapid state of flux in the last 5-10 years. MS has taught us how bad a monoculture can be, so I think we should all be happy that both Linux and the BSDs exist and are in good health.

Re:Can someone provide some insight?

[bh_doc]

"One size fits all" has never been a good approach to anything. Why is it being entertained here?

Re:Can someone provide some insight?

[Anonymous Coward]

Adding system calls for every little thing just means more things you need to port from system to system, having a file with a well-defined interface is more than enough.

...that you still need to port from system to system.

Re:Can someone provide some insight?

[clayne]

Apparently you haven't been using unix that long have you? -20-to-20 has been around for ages and is by no means relegated to Linux.

Re:Can someone provide some insight?

[try_anything]

So the worst case is just doing it the way the UNIX designers intended it to be done in the first place.

The UNIX designers made good use of the "everything is a file" idea, but they didn't apply it dogmatically. They used the idea as a way of generating design ideas for UNIX and discriminating between ideas when one wasn't obviously better by another measure. The results were brilliant, which is why the idea is so valued today, and people continue to apply it. HOWEVER, the original UNIX designers never justified anything obviously lame by saying, "This has a bunch of glaring flaws and seems obviously lame, but it makes everything a file, so we're going to do it anyway." There are lots of things in UNIX that aren't files, because the designers knew that files don't work for everything.

Just to be fair, I should admit that maybe there IS a good way to use files for absolutely everything; still, in the cases where we can't figure out how, it's better to use a non-file interface than to do something we know is stupid. Exposing process data as parseable text through the /proc filesystem is a pretty awesome idea that is very handy for scripting and sysadmin work, but forcing systems programmers to use that interface is awful. It's inconvenient and about as robust as a wet paper bag.

Re:Can someone provide some insight?

[bgspence]

Basicly it gives priority to mp3s, but your network speed drops 90%

Whoops, no, thats Vista.

Re: Can someone provide some insight?

[daveinthesky]

Everything is a fs. Get used to it.
It's the future, and it's not going away ;-)

http://cm.bell-labs.com/plan9/

Re:Can someone provide some insight?

[Jah-Wren Ryel]

I think it should be trivial to have a kernel switch activated on boot to set the preferred scheduler. This way, 3D gamers would be happy to set the -ks (I fail to remember its correct name) while the rest of us would be happy to leave it alone and get the CFS. Maybe by having a modular scheduler architecture would allow to have kernels with the -openvz or -oraclesbest or whatever other schedulers one may want (and could get support from the kernel developers). I don't know one thing about how PlugSched was implemented, but other unices like HPUX have the ability to designate that certain cpus will run under specific schedulers. So you can run straight FIFO scheduler on one pair of cpus, a real-time scheduler on another set of cpus and the normal unix scheduler on the rest. You need to combine that with a way to bind processes to those cpus to get the benefit of the different schedulers, but that almost goes without saying.

I'm kind of surprised that linux does not support that sort of thing already, with SGI's heavy use of linux on their NUMA altix systems, I would have thought that support for multiple simultaneous schedulers would already be in the mainline.

Re:Can someone provide some insight?

[Opportunist]

Since most machines are single-user anyway, because there is only one person using it, whether malware infects the whole machine or just one user doesn't really matter.

Re:Can someone provide some insight?

[makomk]

As I understand it, there was exactly one time Con Kolvias responded to a complaint with "it wasn't broken", and that was because it was working as intended. (Specifically, it was giving some process - I think it was X - a smaller share of the CPU time than before, because it gave processes CPU slices fairly in line with their niceness rather than using heuristics to try and boost interactive processes.)

Also, I think Ingo's patch was basically in line for merging from the time it was first posted on -lkml (out of nowhere with no previous discussion, as far as I could tell).

Re:Can someone provide some insight?

[drsmithy]

Err...only slightly. Unix based operating systems separate user space from the rest of the system. Nothing much can be done without root access.

Pretty much everything malware might want to do, it can do without elevated privileges.

Installing Malware on a Unix system is, for this reason virtually impossible.

It's as simple as convincing an end user to execute a program with elevated privileges. Which is only marginally more difficult than convincing them to run a program at all. Which isn't particularly difficult.

MA large user base is often used as a rationalization for windows malware, but in reality it is just an incorrect excuse.

It's not an "excuse", it's an inescapable aspect of the "malware problem".

Re:Can someone provide some insight?

[drsmithy]

Maybe my setup is different. Preemptive kernel, 1000 Hz, CFS IO scheduler. So I'm not sure how a load average of 2 or 4 can disturb the Linux kernel at all.

Because "load average" in and of itself is a nearly meaningless number for comparing performance. Without knowing anything else about what's going on, a load average of <some number> tells you nothing about how "utilised" the system actually is.

Re:Can someone provide some insight?

[AnyoneEB]

Thank you for clarifying that. I had forgotten that it had been Kolivas's patch.

Re:Can someone provide some insight?

[marcosdumay]

"The explanation that this is a way to steer the development toward a single scheduler that should function well enough for everyone, desktop, embedded or server, works for me."

Ok, so the scheduler must be perfectly fair, priorizing no process, so servers work well. Must give priority to IO bound processes, so desktop users feel the system smoot. And shouldn't be configurable, to not penalize the resource constrained embebed systems.

Let me add that it should give priority entirely for time constrained processes and never reschedule processes, so it can be used for real time. Of course, it should reschedule processes periodicaly, so desktop users won't need to wait CPU bound processes to end before they are able to move their mouse.

There you have, your ultimate process shceduler. Seems quite doable.

Re:Can someone provide some insight?

[rbanffy]

It is useful for a starting point. It keeps confusion down and avoids duplication of efforts. While usually "one size fits all" is not a good approach, it can be useful in a lot of situations. The Linux kernel is still evolving a lot and, perhaps, this is the case.

Re:Can someone provide some insight?

[rbanffy]

Keep in mind "well enough" and "perfectly" are not the same. ;-)

Re:Can someone provide some insight?

[Cajal]

Agreed.

I've largely given up on Linux for precisely this reason. Barely anything is documented (are there man pages for anything in /proc or /sys ?). Coming from a BSD and Solaris background, I find this appalling on Linux.

Re:Can someone provide some insight?

[Viol8]

"One of the biggest, IMO, is that an OS like FreeBSD is a single operating system, not a variety of distributions like Linux, so you can buy a book about BSD, and it's documenting the exact system you're using. None of this stuff about "do this if you're using Debian, or this if it's Mandriva, or ..." "

Agreed. Also downloading a "linux" app only to find the damn thing will only work on RedHat 6.4 or Suse 10.1 or whatever because some idiot thought it would be good to rely on some foo.so only shipped with a few distributions. And if you try and find it yourself its a fscking paper chase aruond 101 web sites only to file foo.so requires baa.so and moo.so which are elsewhere.

Re:Can someone provide some insight?

[Anonymous Coward]

You are describing a problem of "processor affinity". This developerWorks article should be useful to you. In summary, since you are planing to launch one thread/process per cpu, you can directly tie each thread to a unique cpu. You don't need to depend on the scheduler's ability to balance them.

Re:Can someone provide some insight?

[Xabraxas]

You do know what a package manager is don't you? Who downloads random packages from the web anymore? If your distro doesn't have an app in their official repository then most likely there is a third party repository that has what you are looking for, if not the application you are trying to install is probably some small, replaceable app. At least this has been my experience.

Re:Can someone provide some insight?

[Viol8]

"Who downloads random packages from the web anymore? "

People who want a package not in the distributions package list you idiot. Who do you think?

Expected debate

[Miltazar]

Expected Debate: For-Molnar: Its better! Against: No its not, its totally unfair. For: No, its a fair scheduler, and its better! Against: Nuh-uh! Your a liar*hits him* For: I'm telling Linus!

Re:THIS FP WAS POSTED BY THE LINUX PROCESS SCHEDUL

[Anonymous Coward]

Don't you mean "processor"?

In a blind taste test....

[budword]

I doubt any of us could tell the difference. Storm in a tea cup.

Re:In a blind taste test....

[Anonymous Coward]

And besides, what is this crap? A story on Slashdot actually related to something significant in Geekdom? NO WAY! I want to here more justification for IP piracy!

Re:In a blind taste test....

[MyLongNickName]

As a windows user who has very little experience with Windows: This is one of the strengths of open source. if you have a large enough base of contributors, these "little" details are brought out into the open, and you can really understand how things work. I've read a bit on the subject, and it is interesting to see the different approaches that can be taken to something that most of us do not even think about.

With Windows, how does this work? I will never know for sure. if MS doesn't choose to make it known, it isn't known. If they choose to make it known, then I just have to trust they are telling the truth (Windows Update anyone).

With a project like this, you are much more likely to get the best approach to the situation.

Re:In a blind taste test....

[MyLongNickName]

As a windows user who has very little experience with Windows

Should be "very little experience with Linux". You would think Slashdot would have a 'preview' to help with this :) Sheesh :)

Re:In a blind taste test....

[peterpi]

It could help stop you forgetting to close your 'i' tag too! :p

Re:In a blind taste test....

[Anonymous Coward]

Someone stole your IP Address?

Re:In a blind taste test....

[morgan_greywolf]

I doubt any of us could tell the difference. Storm in a tea cup. I probably could. But then again, I do a lot of realtime audio recording and editing and therefore I make use of Ingo Molnar's REALTIME and PREEMPT kernel scheduler patches.

The standard scheduler, without those patches, is just about completely useless for realtime audio recording and editing, even with nothing more than the necessary apps, JACK, X, a lightweight window manager (openbox), HAL, syslogd, anacron, and 6 gettys running. Even taken anacron out of the situation didn't help.

Re:In a blind taste test....

[samkass]

I see your argument, but the facts don't seem to actually back it up. By many benchmarks, Linux is currently worse than almost all the closed-source schedulers at handling loads greater than 2.0 on an interactive system and has been for years. The CFS patches seem to mostly close the gap, but I haven't seen many argue it will make Linux better than the alternatives, just let it at least play in the big league ballpark.

Re:In a blind taste test....

[Anonymous Coward]

Hot dog in a hallway.

Re:In a blind taste test....

[DohnJoe]

actually, whenever I load amazon.com in firefox, my mp3 player (xmms) will temporarily stop playing (2.6.18 on debian etch).
h So I, for one, would welcome our new completely fair scheduling overlords, if it can fix this annoyance!

I might even be tempted to trying my own kernel again, it's been a while since I've done that...

Re:In a blind taste test....

[dysfunct]

Indeed, and on top of that I have the feeling that instead of resolving the scheduling issues many other "fixes" were proposed and/or implemented like preemptive locks and 1000Hz ticks, etc. We'll see how all those together will affect performance.

On the other hand, I have always liked the scheduler when it came to high-load and not too latency critical situations, even though IMHO the FreeBSD scheduler still scales better.

Re:In a blind taste test....

[oojah]

> 1000Hz ticks

Don't forget that the "tickless" option is available now.

Cheers,

Roger

Re:In a blind taste test....

[piojo]

actually, whenever I load amazon.com in firefox, my mp3 player (xmms) will temporarily stop playing (2.6.18 on debian etch).

So I, for one, would welcome our new completely fair scheduling overlords, if it can fix this annoyance! Maybe it will help, maybe not. What it will do is make sure that firefox and xmms each get 50% of your processor if nothing else is asking for it. If that's not enough for xmms, you need to use "nice" to change its priority. If you don't know how, the easiest way is probably to use the process manager "top" and press "r", enter the process ID of xmms, and give it a niceness of -5 or -7. (Tweak to taste. and you need to run "top" as root to do this. And yes, this is all done on the command line.)

Re:In a blind taste test....

[clayne]

Bucket of water.

Re:In a blind taste test....

[DohnJoe]

thanks, I already tried that but to make sure I tried it again just now, I even set xmms to -20 and both X and firefox to 19, didn't make any difference. So I'm really hoping this scheduler will help.

Re:In a blind taste test....

[aug24]

With Windows, how does this work? I will never know for sure.

Something to do with slowing down the network, I think ;-)

J.

I think there was a TV show about this

[RightSaidFred99]

When Nerds Attack 3: The Nerdening.

Re:I think there was a TV show about this

[Azarael]

What I wonder, is which scheduler they will use to pick a programming slot for the soap opera that someone is apparently trying to produce..

On next weeks show, Linus' returned from the dead, evil twin hatches a devious plot to turn kernel design discussions into a nerd-culture flame war. Tune in to see what happens next!

Re:I think there was a TV show about this

[Anonymous Coward]

In my best Jedi VOICE and hand wave 'This is not the site you are looking for. I think the site you are looking for is called Digg'

Typo?

[Anonymous Coward]

New process scheduler? Not processor.

let this flamewar die already

[Anonymous Coward]

It's been beaten to death everywhere else already.
Some improvements have been identified for 2.6.24

more important things

[LM741N]

I know its not easy getting info on wireless chips, but time would be better spent working on something like that. Just look at all the live CD's out there and how many can connect to wifi? Ubuntu and not much else. (note to self- take wifi chip developers out to strip club and get them drunk next time they are in town)

Re:more important things

[Chirs]

Have you considered the fact that different people work on different areas of the kernel? The people that work on the scheduler generally have very little to do with hardware drivers.

Re:more important things

[0xABADC0DA]

Basically they are replacing the spark plugs when they could be switching to an electric engine. A huge amount of overhead comes from system calls, context switches, and data copying -- three things that effectively cannot be eliminated if the system runs unsafe code. You can *sometimes* eliminate data copying in specific cases using shared memory or complex operations (sendfile), but not normally.

I recently found this paper which claims:
  * 20%-80% performance by the compiler placing parameters in a shared memory so the kernel does not have to copy data
  * 60-80% from doing multiple calls as one (ie, readdir-stat, open-read-close)
  * 40-90% from putting some parts of the app logic in the kernel even with safety checks inserted into the code

This suggests that the average overhead of say 1.5x from Java code running in the kernel over C code would be dwarfed by the inefficiencies of actually running that C code safely. This is the kind of thing they should be working on.

Re:more important things

[GreatBunzinni]

I know its not easy getting info on wireless chips, but time would be better spent working on something like that.

Apples and oranges. The expertise and information needed to write a process scheduler has absolutely nothing in common with the expertise and information needed to write a device driver, specially if the problem pertaining to the lack of device drivers for that particular class of hardware is, as you know, the complete absence of information on the hardware itself or even how the hardware was done in the first place.

Re:more important things

[phantomlord]

Basically they are replacing the spark plugs when they could be switching to an electric engine. A huge amount of overhead comes from system calls, context switches, and data copying -- three things that effectively cannot be eliminated if the system runs unsafe code. You can *sometimes* eliminate data copying in specific cases using shared memory or complex operations (sendfile), but not normally. I just changed the spark plugs, oil/oil filter and air filter in my truck. Mind giving me and installing an electric engine to put in it? After all, I know how to work on gasoline engines but I've never done work with electric engines so you're forcing me to deprecate my current abilities and knowledge to spend a considerable amount of time and money changing to a new paradigm for something that has more long term potential but will, at least in the short term, be less maintainable and will cause me significant pain (by not having a vehicle to drive) until I get it all installed just right.

Someone who is good writing a compiler may not be good at creating a new kernel design. Someone who is great at writing a window manager may not be great at creating a new cryptography algorithm.

Last time I saw, Linus was pretty open to taking patches... the bar you have to meet is that the patches need to be understandable by him (and thus maintainable) and that he thinks it will improve the kernel more than hurt it. If you have patches, submit them (or convince the person who did write them to submit them) and persuade Linus. If you don't have patches, pay someone to write them for you or at least convince (but don't berate) a kernel developer of the merits of your idea so they will write them for you. Don't, however, expect a kernel dev to go off on a tangent that is going to require large amounts of the kernel, some of which he may not have any interest in, to be replaced just because you think it might be a good idea. That's like expecting a government to completely dissolve its defense system because someone got the idea that everyone would be more relaxed if they got a free massage every week.

Re:more important things

[Hatta]

There isn't much more important work for a kernel than improving performance under load. They could probably do better by focusing on I/O scheduling than CPU scheduling. My CPU spends an inordinate amount of time waiting for IO. But kernel performance is of the utmost importance to kernel developers. What does it matter if it runs on your hardware if it doesn't run well?

Re:more important things

[LM741N]

Yes, I understand that people have different interests. Actually I should have been more specific. My frustration comes out as I don't understand why all these distros expect you to go in and hack the system just to get wpa-supplicant working.
Maybe it would take a kernel guy 5 min to write the code?

Re:more important things

[0xABADC0DA]

Don't, however, expect a kernel dev to go off on a tangent that is going to require large amounts of the kernel, some of which he may not have any interest in, to be replaced just because you think it might be a good idea. I linked to the paper to show that other people have demonstrated that it is a good idea. Kernel developers shouldn't care what I think, they should care about facts and about improving their system. Instead they dithering over registers and cache lines when the whole approach is grossly inefficient to begin with in many cases.

Programs need to take ~6000 cycles between system calls to have only a small 10% overhead just for making the calls, not even to mention the copying of data and setting up fault handlers and such. This is a crazy amount of overhead for some types of programs. And if you can switch a process in 100 cycles versus 2000 then the scheduler doesn't even need to be as fair.

Mind giving me and installing an electric engine to put in [my truck] Here you go. It's your truck, you put it in.

You act as if the entire kernel would have to be rewritten and maybe even userspace, but that is not so. They would need to make some changes like being able to do anything from an in-kernel thread that an application can, which is not too terribly far from the current Linux (as opposed to say FreeBSD which would require a *lot* of work to do this). Hell even just start with something simple like inotify (the design of which is absurd) and make it so programs can process events inside the kernel, making it actually be useful (see BPF).

Re:more important things

[phantomlord]

I linked to the paper to show that other people have demonstrated that it is a good idea. Kernel developers shouldn't care what I think, they should care about facts and about improving their system. Instead they dithering over registers and cache lines when the whole approach is grossly inefficient to begin with in many cases. Were they submitted? Are they clean enough to be maintainable and don't cause any regressions or is it just proof of concept code?

Mind giving me and installing an electric engine to put in [my truck] Here you go. It's your truck, you put it in. Way to completely miss the point. You're telling me that I need to spend MY time and money to make my truck conform to YOUR ideals. Not only that, but you need me to completely drop all of my current knowledge to work on something I know nothing about because you say it is going to be better. The onus is on you to prove that your concept is valid, not to expect me to stop working on something I enjoy/find useful to work on something that might not interest me at all.

You act as if the entire kernel would have to be rewritten and maybe even userspace, but that is not so. They would need to make some changes like being able to do anything from an in-kernel thread that an application can, which is not too terribly far from the current Linux (as opposed to say FreeBSD which would require a *lot* of work to do this). Hell even just start with something simple like inotify (the design of which is absurd) and make it so programs can process events inside the kernel, making it actually be useful (see BPF). Again, if you're so confident that it is trivial, why don't you code up some patches and send them to Linus (well, Andrew) with benchmarks? There's no short supply of people mailing LKML to tell them that they should implement some idea (quite a few of which are totally hare-brained)

These two recent examples come to mind
Thinking outside the box on file systems
The vi editor causes brain damage
or the dozens of threads requesting the kernel should use C++, glibc internally, etc.

Re:more important things

[Anonymous Coward]

You're telling me that I need to spend MY time and money to make my truck conform to YOUR ideals. I claimed that 'electric engines' are better in many ways and showed you evidence.

You replied with 'well come over and put it in my truck for me then'.

To which I said 'here it is, if you want it do it yourself'.

I an NOT telling you what to do. I am saying that 'electric engines' in many cases are better and for what reasons. I even linked you one to mess around with. If you choose to ignore or mock that information then that's your business.

I really like how you dismiss facts and measures by saying it's just like this or that loony idea or troll (vi causes brain damage? come on). That's classy. It looks like you may be somehow related to work on the linux kernel -- that would be unfortunate because this is exactly the kind of closed mentality and ignoring of facts that people are complaining about re: the scheduler. Honestly, you sound pretty whiney when you say things like 'typesafe code and vms are too hard boo hoo do it for me'. If you only know how to rearrange the deck chairs then maybe you should get off the ship entirely.

This here is why the cool stuff like say dtrace and zfs get done on other systems not linux.

Again, if you're so confident that it is trivial, why don't you code up some patches and send them to Linus (well, Andrew) with benchmarks? I did NOT say it was trivial, I said it was not terrible far from the current state and especially not compared to bsd. Again you are purposely misinterpreting what I said seemingly because you are some sort of jackass. And yes I would do this if my coding time was not already spent on things I consider more important than the linux kernel.

Re:more important things

[phantomlord]

This here is why the cool stuff like say dtrace and zfs get done on other systems not linux. Versus Linux getting stuff like ReiserFS, ALSA, etc. Solaris got dtrace and zfs because the corporation that owns Solaris paid developers to create them. FreeBSD gets to use them because the CDDL and BSD license doesn't have the incompatibility problems that CDDL and GPL have. Can you say strawman?

I did NOT say it was trivial, I said it was not terrible far from the current state and especially not compared to bsd. Again you are purposely misinterpreting what I said seemingly because you are some sort of jackass. And yes I would do this if my coding time was not already spent on things I consider more important than the linux kernel. And Ingo considers his scheduler more important than implementing the code you've discussed. It's ok for you to work on something else but not for him or another kernel dev to work on what they consider important? Oh, and calling me a jackass clearly proves how little work there is in converting Linux to use the model you suggest. Thanks for the tip.

Re:more important things

[Anonymous Coward]

Versus Linux getting stuff like ReiserFS, ALSA, etc. A mostly classic filesystem few people use anymore by a probable murderer and a sound system that is so complicated that pretty much any thread you google on it has exasperated people at wit's end. I'm still running 2.6.20 because 2.6.22 broke the sound driver for Intel hda. Thanks ALSA.

Great examples.

And Ingo considers his scheduler more important than implementing the code you've discussed. It's ok for you to work on something else but not for him or another kernel dev to work on what they consider important? Seriously what is your problem. You're acting like a thirteen-year-old or worse. 'Oh they *have* to do this-and-that that you didn't actually say but *you* can do whatever you want?' You clearly have something emotional going on that is not healthy.

Re:more important things

[try_anything]

Dude, just because the Linux kernel guys are rolling out regular upgrades to a real, mature, working kernel that is the basis of a usable, feature-rich operating system does not mean that throwing them at some promising theoretical ideas will magically bring those ideas to your desktop. It might not even speed up the process, and it certainly would result in massive short-term disruption. Linux couldn't accomodate those ideas without a massive ground-up rewrite. Besides, we don't want Linux to live forever, right? How will we shake off our generation's cruft if not by the introduction of a new system whose virtues can't be practically assimilated into Linux?

Re:more important things

[Anonymous Coward]

It might not even speed up the process, and it certainly would result in massive short-term disruption. Linux couldn't accomodate [sic] those ideas without a massive ground-up rewrite.

    * 20%-80% performance by the compiler placing parameters in a shared memory so the kernel does not have to copy data

The kernel could generally do this anytime it knows there are only one thread sharing the memory. This is not a massive or disruptive change. I believe in some cases linux can already do something like this where the page becomes read only by the process during a system call (to prevent modification). It may require some coordination between the compiler and the OS to get performance benefits from it.

    * 60-80% from doing multiple calls as one (ie, readdir-stat, open-read-close)

Mac OS X already does this for reading folders, and can return in one go even more information than readdir-stat. In any case these are simply new system calls and a tiny stub in the stdlib that does the fast one if it exists or issues the multiple calls otherwise. This is clearly not massive change or disruptive.

    * 40-90% from putting some parts of the app logic in the kernel even with safety checks inserted into the code

The BSD Berkley Packet Filter already does this to filter packets, only its safety checks are limitations of the expressiveness of the safe language. For something like a decent inotify this could be a self-contained vm module -- neither a massive change or disruptive, since it has no effect on anything else. More useful would be to be able to simply call syscalls from within the kernel (currently you can't really do this since they expect parameters to be in userspace memory). At least in the first case doing this does not need to be a massive change or disruptive as normal programs would continue to run exactly as before.

What are you guys saying that any of these changes would need a "massive ground-up rewrite". What is so horrible about these ideas that kernel developers should run in fear of them so and mod down the information? Rather than just saying 'its too hard trust me' how about actually explaining why it is so hard. After all it has been done by researchers. These are demonstrated benefits. If you are seriously claiming that any of these ideas cannot be made solid for including in the main kernel then by all means explain why.

Or you can just blindly pretend that the kernel developers know everything, so if they haven't done it already it must not a good idea. I am starting to believe that the OSS kernel developers are really only good at copying after all.

Re:more important things

[try_anything]

Ad-hoc optimizations of the sort you're describing are exactly what the kernel developers should be doing. Exploiting those techniques consistently and pervasively means linguistic and security model work. Think of the effort required to generalize the use of a DSL for the packet filter to all the configurable aspects of the kernel. (Or you could do it in a pervasive AND ad-hoc way -- maybe create a different DSL for every possible programmable part of the kernel -- but that would hardly be worth it.)

For another example, you can combine multiple system calls into one by defining lots of "combo" system calls. That's kind of an atrocity, but it might be worth it in a few especially critical places. Evidently Linux is getting along okay without it. Alternatively, you could hide the functionality in the platform somehow at the linguistic or linking level. That's what I'm talking about when I say a massive reworking would be required.

Or you can just blindly pretend that the kernel developers know everything, so if they haven't done it already it must not a good idea. I am starting to believe that the OSS kernel developers are really only good at copying after all.

They're "only good at copying" because they refuse to copy the "demonstrated benefits" achieved by researchers? I guess that would be insulting if it were coherent.

Demonstrating a performance improvement is only the first step in development. Researchers don't put their products through QA and deploy them to hundreds of thousands of users on dozens of platforms. Nor do they take on responsibility for maintaining their hacks through further changes and enhancements. Taking on that responsibility is not simply "copying."

The Linux kernel is open-source and is often used for kernel research. Anyone who makes improvements that are suitable for inclusion in the kernel are free to do so. If the kernel team doesn't accept them, then they're free to hire themselves out to Oracle or an embedded Linux company or any one of the hundreds of companies that roll kernels for their own use and could use a performance boost. (The mainline kernel is NOT where you should look for these improvements first, by the way -- there's a much lower effort required, both in implementation in testing, in special-purpose kernels that need to support a very narrow range of usage. In that sense, looking to "the Linux kernel developers" (by whom I assume you mean the developers who are direct contributors and/or gatekeepers to Linus' and Andrew Morton's kernels) to drive breakthroughs is misguided in most cases.)

The researchers who don't want to do the grunt work required to make a kernel usable for hundreds of thousands of users on dozens of exotic hardware platforms, well, they're doing their job. A different job, that allows them the freedom to break things in dramatic ways on the way to making them dramatically better.

Re:more important things

[Anonymous Coward]

For another example, you can combine multiple system calls into one by defining lots of "combo" system calls. That's kind of an atrocity, but it might be worth it in a few especially critical places. Evidently Linux is getting along okay without it. Linux was getting away with a pretty bad scheduler for a long time too and it was 'ok' -- that's a bs line of reasoning. You could just as well say that getting a list of filenames in a folder then having to get their info one at a time is a kind of atrocity -- except that's how POSIX says it must be done. You are seriously going to argue that introducing a race condition between name and info is a good thing? It's annoying -and- slow.

Demonstrating a performance improvement is only the first step in development. Researchers don't put their products through QA and deploy them to hundreds of thousands of users on dozens of platforms. Exactly -- that's what kernel developers do. They take the good idea and they implement it well. except they don't seem to be doing that very much.

The Linux kernel is open-source and is often used for kernel research. Anyone who makes improvements that are suitable for inclusion in the kernel are free to do so. If the kernel team doesn't accept them, then they're free to hire themselves out to Oracle or an embedded Linux company [...] In that sense, looking to "the Linux kernel developers" [...] to drive breakthroughs is misguided in most cases.) So... who does this work? These researchers already created their research kernel and demonstrated significant benefits ... but you need a company to turn that into quality code that is fit to put into a standard kernel. There's nobody in OSS that does this work... what exactly do kernel developers do then, just twiddle some bits until a corporation turns the good ideas into reliable code?

They're "only good at copying" because they refuse to copy the "demonstrated benefits" achieved by researchers? I guess that would be insulting if it were coherent. What I mean is, they sure don't seem to be developing or incorporating new ideas on their own. Linux is a really good implementation of POSIX, SuS, whatever. It's 25+ years old. That's the reason why the things I mentioned earlier, zfs and dtrace, were not created for Linux .. because the companies behind them were trying to solve actual problems, not just do what everybody else has done only a little bit better.

Think of the effort required to generalize the use of a DSL for the packet filter to all the configurable aspects of the kernel. Sure, why would you create a DSL for every thing when you could use a GL one instead? The interface is already there, just have it use system calls like a process. That's not rocket science... or for that matter, why not create a DSL for something like inotify? Every other major system can monitor all file activity from userspace... windows can, mac os x can, even freebsd mostly can. Linux can't -- and on the second major attempt to do so (dnotify was even worse!). Something is seriously messed up about that, and from where I'm sitting it's the Linux kernel development culture itself.

Re:more important things

[try_anything]

You are seriously going to argue that introducing a race condition between name and info is a good thing?

I'm making a distinction between piecework hacks and systematic exploitation of these ideas. You're thinking on a much smaller scale than your first earlier implied (since you first referred the ideas in their most general form), and I think if I understand you now I don't necessarily disagree. Also, with respect to this particular example you're talking about correctness vs. cruft, which is an entirely separate argument.

Sure, why would you create a DSL for every thing when you could use a GL one instead? The interface is already there, just have it use system calls like a process. That's not rocket science

That is rocket science, in the sense that some kind of security model would have to be enforced to prevent malicious or careless users from borking the kernel. To avoid the overhead of constant checking, the security check would have to be done on the code itself when it is presented by the user, rather than on the code's runtime activity. The rocket science comparison here is particularly apt, since it's an old idea, has been demonstrated in practice, is routine for some uses (rockets take satellites into orbit; Java checks loaded bytecode to make sure it's safe for the VM), yet it is still only practical (so far) in a narrow range of applications. A DSL is much easier, because it can include a very limited set of abstractions and control flow facilities (or no flow control facilities at all.)

Re:more important things

[0xABADC0DA]

You're thinking on a much smaller scale than your first earlier implied (since you first referred the ideas in their most general form), and I think if I understand you now I don't necessarily disagree. I listed out three lines of optimization that were not being done:

    1) using shared memory segments that the userspace and kernel use to avoid copying (ie mark read-only while in the kernel)
    2) combining operations that take many syscalls into a single operation
    3) running some user codes withing the kernel itself

and evidence showing their utility. I am not even the source of these ideas, and I only restrict anything down to a smaller scale because the 'larger' ideas have already been shot down as 'too hard' without any real reason why. I mean if a person honestly believes that readdir+stat (the moral equivalent of OS X getdirentries) is too much work or would require a complete rewrite then convincing them of any optimization at all that isn't say spelled out in POSIX is going to be a major accomplishment.

Sure, why would you create a DSL for every thing when you could use a GL one instead? The interface is already there, just have it use system calls like a process. That's not rocket science That is rocket science, in the sense that some kind of security model would have to be enforced to prevent malicious or careless users from borking the kernel. To avoid the overhead of constant checking, the security check would have to be done on the code itself when it is presented by the user, rather than on the code's runtime activity. The rocket science comparison here is particularly apt, since it's an old idea, has been demonstrated in practice, is routine for some uses (rockets take satellites into orbit; Java checks loaded bytecode to make sure it's safe for the VM), yet it is still only practical (so far) in a narrow range of applications. A DSL is much easier, because it can include a very limited set of abstractions and control flow facilities (or no flow control facilities at all.) The same could be said for a lot of kernel work. I would argue that a bytecode verifier is simple compared to many parts of the kernel -- real filesystems and memory management are particularly dense. Getting the complex interactions of mmap, read/write, disk cache, etc correct among all the different types of files and filesystems is far from simple and there have been many actual security flaws in this area in linux (the number is at least an order of magnitude more than flaws in the Java bytecode verifier for instance). It's not like just because there is potential for error it isn't worth doing.

Apparently there aren't kernel developers interested in doing these kinds of things, but I never said there were -- I just said that it would be a lot more fruitful than say getting a slightly better scheduler.

Re:more important things

[Anonymous Coward]

the moral equivalent of OS X getdirentries I meant to say getdireentriesattr ...

More like...

[Anonymous Coward]

Nerd Attack 2: Electric Boogaloo

Re:Snooze

[Reason58]

Seriously has Slashdot got such a boner for Linux that they'll post absolutely every fucking little boring thing they find? You must be new here.

just give me round robin....

[Anonymous Coward]

.... and call it a day.

That was actually quite fun. Thanks.

[ishmalius]

That was a very entertaining read. I love it when strong personalities squabble, and egos collide. Open Source is Fun!

Still don't understand the fixation

[Anonymous Coward]

Another CFS flamewar within 2 weeks of the last slashdot article on it.
http://linux.slashdot.org/article.pl?sid=07/09/01/1853228

And yet the most important performance bugs in the kernel haven't had any updates.
http://bugzilla.kernel.org/show_bug.cgi?id=7372
http://bugzilla.kernel.org/show_bug.cgi?id=8636

I do not understand the fixation on CPU scheduling when there are so many other things that need attention. [Heck, if disk IO performance is so broken, I certainly don't have the guts to try out the new firewire code in 2.6.22 as well and add another variable into my life.]

Re:Still don't understand the fixation

[Aladrin]

Because they care. What else do they need? An individual developer doesn't care what you, or anyone else wants. They work on what they think is

A) Important.
and
B) Possible for them to fix.

Why you'd want a scheduler guy to work on IO performance is completely beyond me. Might as well have a janitor cook dinner for you.

Re:Still don't understand the fixation

[Billly Gates]

The flamewar is based on drama between 2 waring linux developers. One is accusing the other of being Linuses favorite and stealing algorithms and ideas with the different schedulers. I think the other developer quit linux alltogether as a result.

That is why its a big deal. Many developers have loyalities to one of the process scheduler develoepers.

So in other news its drama

Re:Still don't understand the fixation

[gmack]

For 7372 the fix is to use a better designed filesystem such as XFS. As far as I know there are people dedicated to trying to fix ext3's inefficiancies but for the most part people who want better performance are just switching to other filesystems.

Re:Still don't understand the fixation

[Ant P.]

No, that was Con Kolivas. The new guy this time around seems to have appeared out of nowhere claiming to have some far better scheduler implementation than the CFS one that's been in development for months, but is being a complete dick to everyone else.

Re:Still don't understand the fixation

[DirkGently]

Did you actaully read through those bug reports? The problem is CFQ. Choose a different one. Unlike the CPU schedulers, the IO schedulers *are* pluggable. Try anticipatory or deadline.

Re:Still don't understand the fixation

[phantomlord]

Roman is a long time kernel dev and is the maintainer of AFFS, HFS, M68K and kconfig. He's hardly new to the scene. New to the scheduler code, perhaps.

All of this started with Roman doing a code review of CFS a month or two ago. Roman asked some questions to clarify what certain parts of the code were doing, Ingo asked Roman to provide more info so he could see where CFS was falling short on Roman's test cases. Both sides kept trying to talk passed each other. Eventually, Roman got frustrated and provided a new scheduler/patch with some mathematical proofs behind it to make the scheduling better. Roman and Ingo continued to talk passed each other. Ingo picked up some of Roman's patches. Roman feels slighted like he's being ignored but was just as guilty in ignoring Ingo (and other CFS devs) along the way. Factor in the taste in people's mouth from earlier this year with Con and the lines were already drawn before Roman got involved.

It's a clash of egos all around and nothing atypical for a large open source project.

Re:Still don't understand the fixation

[masterzen666]

There is absolutely no evidence that the bug #7372 lies in ext3 code.
Some kernel developers thinks it's a write back issue (but I'm still not convinced).

Re:Still don't understand the fixation

[Anonymous Coward]

The problem is CFQ. Choose a different one. Unlike the CPU schedulers, the IO schedulers *are* pluggable. Try anticipatory or deadline.

And what should I do if I want to run a big database system, which works best with the CFQ IO scheduler? Do I have to choose between "good file copy performance" and "good DB performance", on the same big RAID block device?

I think there should be one really good IO scheduler, not multiple specialized ones. In that sense, making them pluggable was a mistake.

I'm also glad that there's only one CPU scheduler, that I dont have to configure one CPU for best Firefox latency while having another CPU configured for best Java development performance.

Pluggable schedulers might be an excellent idea for development, but they are a disaster to the user. (Except perhaps if you have a really specialized system that only runs one type of workload, but that is rarely the case in practice.)

Re:Still don't understand the fixation

[TJamieson]

I certainly don't have the guts to try out the new firewire code in 2.6.22

YMMV of course, but when I tried the new 1394 code it sent my kernel down in flames. I therefore advise avoiding it.

Re:Still don't understand the fixation

[Spoke]

And yet the most important performance bugs in the kernel haven't had any updates.
http://bugzilla.kernel.org/show_bug.cgi?id=7372
http://bugzilla.kernel.org/show_bug.cgi?id=8636

I do not understand the fixation on CPU scheduling when there are so many other things that need attention. [Heck, if disk IO performance is so broken, I certainly don't have the guts to try out the new firewire code in 2.6.22 as well and add another variable into my life.]

Check out the "per device write throttling" patches which improve this behavior a significant amount. The patchset is currently on the 10th revision.

For more information, check out this LWN.net article: http://lwn.net/Articles/245600/

Re:Still don't understand the fixation

[masterzen666]

At least for #7372, CFQ is out of cause, since the OP said he's using deadline.

Esoteric Discussions

[Archangel Michael]

More importantly, if there are more than one Scheduler, and if someone could tell the difference, why isn't s/he using the ALTERNATE Scheduler and compiling their own custom, tweaked and totally tuned kernel?

Seriously, most people aren't going to notice, and those that do notice, ought to be able to compile their own kernel, and ought to do exactly that. This is nothing short of an esoteric discussion and shouldn't extend beyond kernel developers. Most people don't know, and don't care which scheduler is implemented.

I'm one of those somewhere between caring and not. I only care about the supposed differences in approach to scheduling, and quite frankly, from what little I understand, the various schemes to scheduling have their advantages and disadvantages. I seriously doubt that ONE is better in all circumstances compared to all the others.

Re:Esoteric Discussions

[LizardKing]

if there are more than one Scheduler, and if someone could tell the difference, why isn't s/he using the ALTERNATE Scheduler and compiling their own custom, tweaked and totally tuned kernel?

Someone (Con Kolivas?) suggested a "pluggable" scheduler API. I think this was even backed up by patches to provide this functionality. Linus Torvalds rejected the proposal - I think he said that the benefits would be outweighed by the need to maintain multiple schedulers. My opinion is that the kernel could have included a single scheduler in the kernel.org tarballs, but by providing the pluggable API it would lower the bar for those who wish to develop or play with different schedulers.

Re:Esoteric Discussions

[huckamania]

"most people aren't going to notice"

I think most people notice when their computer starts to lag, skip frames and audio tracks, etc. They may not be able to identify this as a scheduling problem, but they do notice.

Re:Esoteric Discussions

[zojas]

the big concern is that the current scheduler is great for big servers, but totally sucks for desktop systems. using the old scheduler on my core2 duo system with 3 gb ram, I get crappy response from gui apps if heavy disk io is running in the background. that to me seems completely ridiculous, and in fact the kernel developers should be downright embarrassed by that situation!! I'm hoping the new scheduler will be better. (to be fair, part of the blame lies with the developers of the bloated gui apps, but still! my system should be ridiculously overpowered, and it doesn't seem to be)

Re:Esoteric Discussions

[tinkerghost]

Someone (Con Kolivas?) suggested a "pluggable" scheduler API. [snip] Linus Torvalds rejected the proposal - I think he said that the benefits would be outweighed by the need to maintain multiple schedulers.

ISTR, the actual issue was the overhead needed to put the scheduler in an API instead of in the kernel was unacceptably high. That said, there are modules for compiling in the different schedulers.

It's been stated a couple of places that the different schedulers have different strengths & weaknesses and the differences are mostly seen on fringe cases where your specific need will probably dictate which one is best for you anyway.

Re:Esoteric Discussions

[larien]

Hrm, wonder how Solaris does it? You can have multiple schedulers running under a single OS instance, although using different ones in the same processor set means you might not get the results you're after. Source is in Open Solaris, so there may be scope to re-use it elsewhere, dependant on licensing restrictions.

For those who wonder, default schedulers under Solaris 10 include TS (Timeshare, the default), IA (Interactive), FSS (Fair share scheduler, workload management), RT (Real Time), FX (Fixed) and SYS (System). You can, if you really want, run processes under all of these on the same processor, but it wouldn't be recommended :P

Re:Esoteric Discussions

[GooberToo]

Well, those same people have admitted that plugable schedulers are not a good idea outside of scheduler development and research. Since maybe a dozen people need/want to do that, the overhead imposed simply didn't make sense.

Re:Esoteric Discussions

[Anonymous Coward]

actually yes, there is ;)
pick the best scheduler for any given situation and run it when its needed ;)

Re:Esoteric Discussions

[strikethree]

"Seriously, most people aren't going to notice..."

Truly, I am not most people. That being said. *I* notice. The 2.0 kernel always felt pretty snappy to me. The 2.2 kernel seemed a little less snappy but it worked very well for me. The 2.4 kernel was where it all seemed to go downhill from a desktop users perspective and 2.6 made me think I was back in Windows with all of the waiting I had to do. You might think that a millisecond or two means nothing to a desktop user, but it means a lot to me. Surely the kernel devs think that way. Honestly, I do not even use Linux as a desktop OS anymore because of how bad it is.

Maybe it is because I play Counter Strike a lot, but I _really_ notice when a mouse movement or a window adjustment lags even a little. I suppose it could be KDE and Gnome being so bloated that even if the kernel were perfect, the cpu still would not have enough horsepower to handle those ungodly beasts, but I doubt it, since even Blackbox is not really snappy. The kernel is for servers only anymore.

It really saddens me that Linux, X, and Enlightenment with all of its eye candy going was snappier on a 600mhz computer with 512mb ram than it is on a 2.4ghz Athlon with 2 gigs of ram running Blackbox. It is a shame that I never tried out Mr. Kolivas' scheduler but I had already given up on Linux by then. If I will not have a snappy OS, I would rather use something less painful like OS X. Linux has lost me as a supporter because of this crap (there were not desktop optimized kernels in early 2.6). I do not buy new hardware and expect my experience to be less snappy than it was. I wish the kernel devs would respect the desktop users at least a little.

strike

Impressive...

[Anonymous Coward]

That was quite a round robin way to do it...

Re:That was actually quite fun. Thanks.

[HoboCop]

I was not entertained. What a bunch of babies.

Just fork it

[zymano]

And find out which ones better.

Re:Just fork it

[Anonymous Coward]

Con's SD scheduler started like that, separately from the main linux development. He was heavily criticized, until a scheduler based on the same basic concepts was written from scratch (~72h) and merged, despite SD's years of testing.

Linus himself criticized him because his development went on on a separate mailing list.

It seems to me that forking would mean condemning his work to irrelevance.

This is how OSS works. In the end, the maintainer decides what's merged, what's not merged and what he himself rewrites to merge it as his work - inspired by others.

Misleading title....

[Anonymous Coward]

Although they mention improvements and the like, it seems like "Flamewar involving Linux Process Scheduler personalities +BONUS shreds of Linux Process Scheduler Goodness" would be much more appropriate.

Re:Snooze

[monkeySauce]

Well, we've got to get boners for something, and girls are too elusive.

Re:Snooze

[EricR86]

If Microsoft or Apple were more open with regards to their kernel development, I'm pretty sure issues like this would be posted here on slashdot - regardless of a favored OS.

Re:Snooze

[Anonymous Coward]

the upper management of slashdot is such a bunch of losers. we know. i wish they'd start thinning out the herd and get some real talent in here.

fairness and interactivity

[Chirs]

The big user-visible difference are improved fairness and interactivity. If you have multiple tasks sharing a cpu, the amount of cpu time allocated to each task is better regulated.

Also, nice levels are more predictable. In general, decreasing the nice level by one means that the task gets 1.25 times as much cpu. This means that a nice -19 task gets approximately 50x the cpu time as a nice 20 task.

Ugh bring back 2.7 please

[maelstrom]

I know they've changed the model of development for the kernel, but how many new schedulers have we gone through between 2.4 and 2.6 now? Maybe it is just me, but the scheduler seems like a pretty important piece of the kernel.... Ripping it out every 6 months and calling it "stable" seems a bit off to me.

Oh well. I guess I'm just getting cranky in my "old" age.

Re:Ugh bring back 2.7 please

[Billly Gates]

The reason behind this according to an interview with Linus is that he wanted to stabilize a common codebase.

Many commerical developers have steered clear of linux and supported Windows and Solaris for this reason. Oracle even has a script that will make their rdms refuse to run if any modifications are made on a stock rhles installation. Binary and abi compatibility is important as Microsoft knows.

Yes there are changes like this but it wont break apps or a huge way linux works. 2.6 is likely to be permanent unless a real overhaul or change is needed (like maybe rewriting for gpl v3).

Small incremental improvements serve everyone better and make less development time. It took years for a total switch from 2.4 to 2.6.

Re:Ugh bring back 2.7 please

[luciofm]

There was the 2.4 schuduler, the old O(1) 2.6 scheduler and now the new 2.6 CFS scheduler...
This doenst seem to me to be ripped every 6 months, unless the 2.6 tree is just about 6 months older...

Re:Ugh bring back 2.7 please

[maelstrom]

I seem to recall uni-processor preempting in 2.4 as well...

Re:Ugh bring back 2.7 please

[Anonymous Coward]

KernelTrap had an article about how many process schedulers the Linux kernel has had. Quoting from the article:

"In a June of 1992 posting to the linux-activists mailing list, Linus Torvalds described the original Linux scheduler noting, 'the scheduler in linux is pretty simple, but does a reasonably good job at giving good IO response while not being too unfair against cpu-bound processes.' A year later, Linus posted a more detailed description of the scheduler noting, 'the linux scheduling algorithm is one of the simplest ones possible'. Comments in the original 254 line sched.c file read, '"schedule()" is the scheduler function. This is GOOD CODE! There probably won't be any reason to change this, as it should work well in all circumstances (ie gives IO-bound processes good response etc). The one thing you might take a look at is the signal-handler code here.'

"Comments in the current 6,709 line sched.c file show the first changes being made in 1996 by Dave Grothe, 'to fix bugs in semaphores and make semaphores SMP safe'. Two years later Andrea Arcangeli is credited with implementing 'schedule_timeout() and related stuff'. It was not until 2002, ten years after Linus' original code was written, that the scheduler received a complete rewrite, 'new ultra-scalable O(1) scheduler by Ingo Molnar: hybrid priority-list and round-robin design with an array-switch method of distributing timeslices and per-CPU runqueues.' Con Kolivas is credited with 'interactivity tuning' in 2003, and Nick Piggin added 'scheduler domains' in 2004. A more recent rewrite of the scheduler happened in April, again by Ingo Molnar, this time with his Completely Fair Scheduler."

Re:Ugh bring back 2.7 please

[Spoke]

Ripping it out [the scheduler] every 6 months and calling it "stable" seems a bit off to me.

If you want a stable kernel, don't use the vanilla kernel. Linus has repeated stated that a perfect kernel is not the goal with his tree. His goal is to promote the development of the kernel which maintains some sense of stability but he pretty much guarantees that it will not be perfect.

The process of creating a bugless kernel requires that all new development stop for an extended period of time. Which is what happened (mostly) with the old style odd/even kernel development and as a result, you end up with massive upheaval between the so called stable releases (which never were all that stable in the first place).

If you want something rock solid and stable, use a vendor's kernel with long release cycles. Try RHEL/CentOS, Ubuntu LTS or Debian.

Re:Ugh bring back 2.7 please

[ChilyWily]

I second that... I develop applications on Linux in a commercial environment that is pretty sensitive to scheduler changes. Scheduling priority and what not, I understand the technical reasons, but to the army of project manglers surrounding me, it's just an excuse to point to the constant "varability" of the Linux kernel that causes them much churn. Unexpected side-effects, context switching issues... the proposition is very difficult to defend when staying with the "current" scheduler results in custom mods that they are scared of, and going with the latest put at risk, many 'system' applications to unforseen side-effects.

I wish there was better choice here.

Well, if you can't win an argument on merit...

[Anonymous Coward]

a number of other Linux developers have stood up to defend Molnar and call into question the motives of the complaints In other words, a number of other Linux developers have resorted to ad hominem attacks on the voices of dissent.

Did the Open Source is about choise 6

[denisbergeron]

Why not having the possibility to choose the sheduler ? What about a modular kernel sheduler, so everyone will be happy.

Re:Did the Open Source is about choise 6

[Chirs]

Because then every function call becomes an indirect call, causing performance penalties.

More importantly, however, this requires a stable "scheduler API". Not all schedulers want to hook the same things, which means that you need to add (and maintain) a superset of the hooks required by all the various schedulers.

Finally, anyone who cares enough to be replacing their scheduler should be technically advanced enough to apply a patch and recompile the kernel.

Re:Did the Open Source is about choise 6

[TheRaven64]

Depends on how it's done. FreeBSD supports two schedulers, but you have to make the choice at compile time. There is no overhead, because it's just a matter of compiling one set of scheduler functions instead of the other. The linker then sorts it all out for you. Xen, in contrast, uses an extra layer of indirection, since it allows schedulers to be chosen at boot time. I believe Solaris uses a mechanism similar to Xen, although it might do some runtime patching.

Re:Did the Open Source is about choise 6

[EvanED]

Because then every function call becomes an indirect call, causing performance penalties.

You could also patch the code, though whether this is a good idea or not is questionable at best. No performance penalty though except at switching time.

More importantly, however, this requires a stable "scheduler API". Not all schedulers want to hook the same things, which means that you need to add (and maintain) a superset of the hooks required by all the various schedulers.

Finally, anyone who cares enough to be replacing their scheduler should be technically advanced enough to apply a patch and recompile the kernel.

You're (somewhat) contradicting yourself. You seem to suggest that patching the kernel source is a viable option. But if there isn't a stable "scheduler API", then what is the patch you apply to the kernel source going to use?

Finally, the other big point is that the kernel isn't even including multiple schedulers you can choose at compile time. I know I have rather higher faith in general of code that comes with the mainline kernel than someone's website. (There are, of course, exceptions to this; I would be willing to install Reiser4, I have installed the KDB patches, vendor patches (gentoo-kernel instead of vanilla-kernel or whatever it is), etc.)

Re:Did the Open Source is about choise 6

[Chirs]

"You're (somewhat) contradicting yourself. You seem to suggest that patching the kernel source is a viable option. But if there isn't a stable "scheduler API", then what is the patch you apply to the kernel source going to use?"

Most patch maintainers keep their patches up-to-date with the kernels. So you apply whatever patch is appropriate for your kernel version.

Re:Did the Open Source is about choise 6

[EvanED]

Most patch maintainers keep their patches up-to-date with the kernels.

If they are keeping their schedulers up-to-date with the kernel so it can be compiled in, they are already doing a large part of the maintenance that would be required to keep a runtime plugin system up-to-date. That's my point.

Flamewars are usual in developer websites

[Spy der Mann]

For example, ReactOS had a flamewar regarding the "stolen code from Windows", and it was nearly identical. There was this obsessive guy that got fed up over nothing just because his pride as a person was hurt. In the end he was just misinterpreting stuff. The other guy tried to be calm and understanding, but it didn't work.

In the end, it's just about one thing: Some developers, no matter how high their IQ is, are too full of themselves because they have a stupid complex and a low self-esteem.

Re:That was actually quite fun. Thanks.

[jeevesbond]

What a bunch of babies.

Maybe to you. To me all the flaming and arguing, over a change that will not--apparently--have much of an effect upon the average user, means that the kernel developers are passionate about what they do. It means that, once the dust settles, we'll get a superior product. Maybe if the developers in Microsoft were this passionate Windows would be as good as GNU/Linux.

If you're not entertained by this you don't have to read the stories, just apt-get upgrade and enjoy the software. This is the way things work in the world of Free software, as Linus says: 'May the best code win!' If you want a kernel developed by boring, outsourced workers: choose Microsoft.

Why wait?

[ruiner13]

It will be interesting to see how the new processor really performs when the 2.6.23 kernel is released." Why would you have to wait? Couldn't someone just grab the source, compile it, and benchmark it? Yes, it may have bugs, but it should at least give an indication of overall speed.

Too much backing

[Anonymous Coward]

Let's just hope Roman Zippel does not throw the towel like Con Kolivas just because everyone thinks Ingo Molnar's scheduler math is always the best to back him.

BEOS scheduler?

[Danathar]

Does anybody know what kind of scheduler BEOS used before it's demise? I seem to recall it ran circles around other OS's at the time when it came to multitasking multimedia.

Re:BEOS scheduler?

[norkakn]

Yeah, but how was it as a LAMP stack?

Re:BEOS scheduler?

[Danathar]

I don't know...was it bad?

Re:BEOS scheduler?

[norkakn]

My guess is that it'd be horrible. It isn't that hard to create a scheduler for a specific task. I wouldn't be suprised if dynebolic were to create a custom scheduler, perhaps even one based on BeOS's, but I wouldn't want to use it on my desktop, and certainly not on my server.

I go back in forth, but I think I agree with Linus that there should be one scheduler that is ordained and 'good' everywhere. If someone wants to make a custom one, they can, but it'd probably do more damage to have them be common.

No, there aren't. This is what a kernel DOES.

[Valdrax]

I know its not easy getting info on wireless chips, but time would be better spent working on something like that.

I'll ignore for a moment the fact that you're essentially making the same argument as "Why aren't all scientists (from solid-state physicists to cognitive neuroscientists) working on a cure for cancer instead of [perceived frivolous research in the news]?" You're ignoring the different kinds of expertise that go into a complex field of work like kernel development.

Instead, I'm just going to focus on your assertion that support for a few more wireless chipsets than the abundant choices we have today is more important than fixing problems in the most central and fundamental task of the kernel -- a task that even the most minimalist microkernels consider necessary to put into the microkernel.

This is simply hogwash. Scheduling affects every single part of the system, and it's a major factor in the perceived and real performance of a system. Fixes to the scheduler will affect how a user enjoys their system over the entire life of the system whereas a missing wireless driver affects them once -- at purchase time.

Furthermore, not all Linux systems have wireless networking. Adding more wireless drivers is going to be useless in nearly all server and most embedded uses. You seem to be under the mistaken impression that the purpose of Linux is to provide a good desktop or laptop experience. There are considerably more application domains that Linux operates in.

And frankly...

Just look at all the live CD's out there and how many can connect to wifi? Ubuntu and not much else.

This is not the kernel developers' problem. They've provided the functionality as evidenced by the fact that Ubuntu can do it. This is up to the distro developers to work on. Again, you make the mistake of assuming that all developers are equal and interchangeable and that they all have the same responsibilities in bringing the product to you, the unpaying customer.

There is no debate

[SpaceLifeForm]

Real world benchmarks will decide.

Re:There is no debate

[Anonymous Coward]

the way i see it, the debate is weither the benchmarks should be for server or personal use?

Re:That was actually quite fun. Thanks.

[Climate Shill]

To me all the flaming and arguing, [...] means that the kernel developers are passionate about what they do.

They're passionate alright. Passionate about preventing anyone outside the inner clique from touching "their" kernel.

It means that, once the dust settles, we'll get a superior product.

...by making it clear that anyone who has a superior idea will be told to fuck off, and have their idea reimplemented badly by someone pretending it was theirs.

as Linus says 'May the best code win!'

What staggering hypocricy.

Already suggested

[tinkerghost]

back when the scheduler issue first popped up there were a few suggestions about developing a method to preset nice levels for software - IE, kphone & skype could be niced to -10 & most daemons to 10 with a start rather than having to reset them after launch with 'renice'. The advantage of that is that launching with preset negative nice numbers could be done by anyone rather than only by the superuser. That would show some benefits for interactive/latency sensative software without creating any problems for software that doesn't report a need - they would still default to 0.

Re:Already suggested

[phantomlord]

The advantage of that is that launching with preset negative nice numbers could be done by anyone rather than only by the superuser. That would show some benefits for interactive/latency sensative software without creating any problems for software that doesn't report a need - they would still default to 0. Potentially dangerous to allow "anyone" to do it... it would be much better to have some type of ACL that allows specific users/group and/or perhaps specific programs to be reniced outside of root's control. Back when I was in college, it was typical to PPP in and then telnet to some machine in the CS lab there to get work done. Someone would be sitting at the physical terminal and we might have 3 or 4 other users logged in on the machine doing some coding or whatever. If those other 3 users could renice some buggy, or perhaps deliberately malicious, code and deny everyone else the ability to do anything significant with the machine, it would be a problem. Factor in, as well, at one point, we were learning about distributed computing and were running things like distributed fractals across the entire lab. I can only imagine the fun as 2-3 people (or maybe 30) attempt to run their distributed fractal at -20 across the entire lab .

Re:Already suggested

[tinkerghost]

Potentially dangerous to allow "anyone" to do it... it would be much better to have some type of ACL that allows specific users/group and/or perhaps specific programs to be reniced outside of root's control.

The principle is that Skype/Kphone/FPS register themselves with a 'loader' type function requesting a nice level of -10 at install. Daemons that don't care about latency can register themselves as +10. Anything that doesn't care, doesn't register.

Since only Root should be able to perform a systemwide install & consequentially have access to the 'loader', Root would be determining the runlevel, but any user can use it. If the program were installed locally by a non-Root user, then it wouldn't be able to access the 'loader' to configure it & would run normally at nice 0.

The current model, limits any normal user to nice 0 and above, with everything defaulting to 0. Lets face it, most daemons aren't time sensative enough to require a nice of 0 - the 10mS difference in mounting the CD is totally overwhelmed by the 2 seconds it takes the tray to close & the disk to spin up anyway - a perfect candidate for a nice of 10.

On the other side of the coin, dropping Skype or Kphone into nice 0 is bad. As audio systems, pretty much any delay is a recipee for poor audio quality. They need control when they need it - not 10mS later. The same with GUIs, because they directly interact with people, they need to have the snappy response of a negative nice program - not as negative as multimedia, but better than the baseline.

The Geek wars have a new topic

[drolli]

-Emacs vs. vi
-Reiderfs vs. ext3 (obsolete)
-GPLv3 vs GPLv2
-GPL vs BSD ....

and now:

Scheduler wars!

Keep pushing upward

[athloi]

It may be time for Linux development to split. One fork will focus on stable code that works like a UNIX, and the other in forging new boundaries. I think the FreeBSD developers did something simple.

There is a good reason for this. When you want to make something stable, you want to take proven ideas and refine them so you can make guarantees.

But for our hacker souls, and our inner adventurers, we also need something that is determined to break new ground and make no guarantees. The CFS is being justified by performance, or administrative reasons, but why not focus instead on the real reason we'd like to see it happen?

It's a cool idea to play with.

We're primates. Play is how we learn and invent. Keep pushing upward, making the code do new things and taking on new challenges, because the hacker spirit is play. That playfulness has brought us most of our thinking outside the box, good inventions. Keep it up. Aim for the stars.

Re:Keep pushing upward

[Anonymous Coward]

Yeah! They should have two branches of development, one stable and one where all the cool new toys are built. And there should probably be a quick and easy way to tell them apart... maybe if the second component of the version number is prime it means stable, and if it's composite it means development. I wonder why no-one ever thought of this before?

Re:Snooze

[TheRaven64]

Maybe I missed them, but where were all of the Slashdot articles about the ULE 2 and ULE 3 FreeBSD schedulers? From all the benchmarks I've seen, they make the Linux scheduler look embarrassingly antiquated (performance characteristics matching the 4BSD scheduler that ULE was originally designed to replace).

MOD PARENT UP!

[Anonymous Coward]

Interesting question!

OSS - FAME is Important

[tjstork]

You know, its funny, but, OSS people, and the academic world in general, since they work for peanuts, if not free, are really big into having their names attached to things that they did.

I wonder how much of a troll it would be, if one were to make a math book that took out all the names of all the people, renamed everything based on descriptive terminology, and just made some part of science a faceless API, like so much a Windows SDK reference or C# help.

Re:OSS - FAME is Important

[Solra Bizna]

Only half true.

The names of the people who didn't care about their names being attached to the things they did didn't get attached to the things they did, so they disappeared. Therefore, the only names you see attached to things are the names of the people who did care about their names being attached to the things they did.

That paragraph was begging for LISPification.

-:sigma.SB

Fairness is relative...

[fitten]

On a server, you want the different server tasks to have fair access (usually, this isn't always the case) to the processor so that all the server stuff can go on smoothly.

On a personal workstation, however, an interactive user doesn't necessarily want all the programs to have fair access... we typically would like to have what we're concentrating on currently to be more responsive (have potentially unfair access) or else we may see dropped frames, stuttering music, or the like because the scheduler is trying to be fair to applications that aren't interactive with the console user.

This is the battle that was going on... one claims that Linux process scheduling methodologies were being swayed too far by the interests of admin for servers and this was compromising the single-user in front of a desktop experience.

What probably needs to happen is to have multiple schedulers or, at least, one where you can set the behaviour at runtime to be either more fair (server type machine) or more responsive to user tasks (desktop type machine).

Re:Fairness is relative...

[ultranova]

On a personal workstation, however, an interactive user doesn't necessarily want all the programs to have fair access... we typically would like to have what we're concentrating on currently to be more responsive (have potentially unfair access) or else we may see dropped frames, stuttering music, or the like because the scheduler is trying to be fair to applications that aren't interactive with the console user.

This is where priority levels, or "nice", come in. They allow the user to tell the computer which stuff should be preferred/unpreferred, and how much.

Unfortunately, some people insist that the scheduler should guesstimate the priorities by itself, rather than depending on the user's judgement; since the scheduler can't read the mind of the user, such guesstimating (or "heuristic") scheduler frequently guesses wrong, leading to performance problems. What's even worse is that the constantly changing "dynamic priority" for every program can easily lead to temporary high latency, which in turn leads to audio/video skips and unresponsive system.

A totally fair scheduler will give each process CPU time according to their nice level, without trying to guesstimate. Con Kolivas came up with such a design (the Staircase Deadline scheduler), after which Ingo Molnar wrote a similar design (the CFS) based on the SD which then got accepted into the kernel. Con wasn't happy about this and quit kernel development altogether, which is unfortunate since the "-ck" patchset maintained by him was/is a huge improvement over the vanilla kernel.

THIS FP WAS POSTED BY THE OLDLINUX PROCESS SCHEDUL

[Daimanta]

FP!

Re:Snooze

[revengebomber]

No, I'm New Here.


Damn, why is no one laughing?

Scheduler Question for the /. Experts

[Cassini2]

One way to cause dramatic performance problems on a Windows machine is to simply write a program that accesses lots of files. Performing a network backup with the Windows Networking API is a good example of this. Windows responds by fetching the files from disk and using system memory as a cache. In the process, the working set of programs running on the computer is paged out. The result is that low-priority activities can dramatically slow down potentially important activities on the computer. A good example of this is doing a network backup or a background virus scan on a Windows computer while trying to do any foreground activity (like browsing the web or using Microsoft Word).

So far, in my experience, Linux seems pretty immune to these priority inversions. Will the new scheduling algorithms allow low-priority processes to cause priority inversions by abusing non-processor resources like the network or disk drives?

My name is Ingo Molnar

[Anonymous Coward]

Hello. My name is Ingo Molnar. You killed my scheduler. Prepare to die.

CFS issues - some bugs to iron out before 2.6.23

[tota]

The discussion is not as bad as it sounds (almost normal for LKML!), it's just that Roman wants to talk about the maths and Ingo works with patches... as Willy Tarreau pointed out "I know for sure that the common language here on LKML is patches".

Beyond the heated discussion with Roman Zippel, there are still a few workloads which can trigger regressions, one of which I found running some unit tests.

This is covered in this thread, and although there is now a version of CFS which does not exhibit the problem (see graph of combo3-yield patch) it is not the one that is meant to be merged in 2.6.23 (these patches are 2.6.24 material) so Ingo is getting me to test patches until this regression can be solved.

One slightly annoying thing is that the current fix involves using sysctl to switch back (at least partially) to the old scheduler mechanism!

not sure but maybe possible?

[Anonymous Coward]

not a programmer here, but it seems people are trying to
program a schedular here ... PROGRAM a shedular, get it?
that is like asking a calculator "2+2=?" please?

i honestly think a shedular is magic. and like magic it means,
we should KNOW the outcome before we start processing.
that said, we shounldn't ASK the CPU to tell us how to shedule,
which we obviously ARE.

we need to take a step back, like 2 meter, look at the computer and
say "HEY, i know who you work", then ... do magic.

im sure in a serially processing computer, there's a magic question/combo
with build-in into CPU instructions we can get a "live pulse" running for
a Operating system ... :)

Re:not sure but maybe possible?

[Anonymous Coward]

"HEY, i know who you work"
should be:
"HEY i know HOW you work"

sry.
is just want to add, for more visual explanation.
there're are many ways to make a meal.
like a cake ... you can buy some ready made cake powder, then add
the ingridients they list on the cake powder pack. etc ...
or you can start from scratch ...

trying to make shedular with C or c++ or what not is
just going with the cake powder.

it think a project like linux needs a garden, love and care.
after all everything in a good cake is eatable. if it's eatable
it's growable. and as they say, dirty is oh well ... dirt cheap :D

so let's start from scratch. not FROM scratch, but the idea about
wanting to eat a cake, needs to start in the dirt, not some ready mixed
C, C++ whatnot pre-made (API) super market cake mixture.

we don't use the factories that provide the ingredients for the cake,
we need to MAKE the FACTORIES .... our cake. >:D

(if you want to dicusss this, then the
ulimatifly is: "can a (best) shedular be ported (like to different CPU) or is a
efficient shedular CPU specific." i'll stand with "depends on the
MAKE script" (i guess there's a shedular involved there too) ; )

Linus, it's second time

[12357bd]

that Ingo takes the work/ideas of others about the scheduler and presents his own implementation without proper reconnaisance or collaboration.

As the main responsable of the kernel, don't you think that something has to be done? At the current state I (as a developer) would not try to contribute to such a personally 'monopolized' project, that's sad.

Re:Linus, it's second time

[Bill, Shooter of Bul]

First, he was credited with the inspiration behind the cfs changes(RTFA), second its the freaking GPL. No attribution legally required.

If you want your name to be forever attached with code you wrote, choose a different license that requires that. Then you can be upset and do something if someone doesn't give you credit.

Introducing the EGPL (Egotistical Gnu Public License). just like GPL3, but attribution required for every line & change to file including comments. If you remove code, that too must be attributed in the file. Attributes will be represented in comments, but they can not be removed. They can be moved, however, provided sufficient attribution is made for the person moving the attributions.

Re:No, there aren't. This is what a kernel DOES.

[Bearhouse]

Indeed. Mind you, if we could get it work better 'everywhere', maybe some people would be *gasp* prepared to pay for it.

I know, chicken and egg, but if there was a larger user base, perhaps there would be more room and resources for - as many people are suggesting here - different schedulers, for example.

Re:Roman Zippel's picture

[Anonymous Coward]

You're very bitter about something, aren't you?

Gross mis-characterizations of Molnar

[l4m3z0r]

As I started reading the comments on here I noticed that many were quick to down Ingo for his transgressions and its quite obvious from the comments that no one has bothered to read the exchange on LKML in order to become familiar with what is going on. I have read it, I have 0 bias for either Zippel or Molnar and I can say without any reservation that Zippel is a wank and Molnar is borderline saintly.

A recap of what I have read and understood about the entire situation:

• Zippel shows up with some patches out of the blue and makes some claims about them.
• Molnar takes time out of his day to look at and understand the patches and points out a few things such as lack of comments and missing functionality, also points out that some of the things Zippel wants to do have been done since the last time Zippel looked at the code and that he should look into the latest they've done.
• At this point Zippel decides to try to get into a pissing contest with Molnar and makes more claims about the superiority of his math and how the patches aren't the point.
• They argue back and forth all the while Molnar very seriously considering and understanding Zippels concerns. Molnar asks Zippel to split up his giant patch into smaller pieces that are commented and cleaned in order to bring the useful ones into the kernel more easily.
• Zippel essentially whines and repeats the same arguments over and over for a bunch of threads.
• Molnar basically pleads for him to cooperate so CFS can get the benefits Zippel has to offer.
• Zippel refuses, acts like a child.
• Molnar reworks a few of Zippel's ideas into smaller patches and incorporated them into CFS and attributes this to Zippel.
• Zippel is very rude about this and accuses Molnar of stealing from Zippel.
• Molnar again asks him to cooperate.
• Other people get annoyed start criticizing Zippel.

Ultimately I think Zippel is purposefully trying to provoke Molnar throughout all of this. His wild accusations are nothing more than games that he is playing, the guy has a chip on his shoulder and if Linux was my toy, I would have blocked him from the mailing lists.

Re:Gross mis-characterizations of Molnar

[jobsagoodun]

Insightful summary, ta.

Re:Gross mis-characterizations of Molnar

[12357bd]

I also RTFA, your resume is quite biased, Zippel is asking Ingo for explanations about his scheduler code, trying to understand how it is suppoused to work, Ingo doesn't care to answer about his code, but implements the Zippel ideas on his own.

The base question si simple, if Zippel (and previously Kon) ideas were ok (they were re-implemented), why Ingo was unable to work with them? IMO Ingo wants to monopolize the part of the kernel under his control, that's sad.

Re:Snooze

[oojah]

I guess articles on the FreeBSD schedulers are either not being submitted or are being rejected. It's a shame either way. I don't use BSD (for no particular reason) but I'd still very much like to hear about what's going on there.

It doesn't even matter

[Anonymous Coward]

This scheduler will be obsolete almost before it gets out the door. I'm waiting for the Utterly Completely Fair Scheduler (thought I might wait for the Totally Utterly Completely Fair Scheduler).

Don't they have anything more important to say

[Mutyournaps10987]

Guess the scheduler issue should be important... and the debate about what's the best scheduler may be fascinating tough... But wtf sometimes seems to me that people spends lots of time doin' things just because they don't have anything more interesting to do? ... duh.

Re:THIS FP WAS POSTED BY THE LINUX PROCESS SCHEDUL

[Anonymous Coward]

That wasn't completely fair, you know

Given you both can't seem to get Ingo's name right

[Chuck Chunder]

That was certainly not how I read it when it happened

Perhaps we shouldn't trust either of your readings.

Always been possible...

[Sits]

I'm no expert I'm afraid but I'll let I'll share with you what I know. The Linux I/O scheduler (which I believe turned up some time in the 2.6 era) is somewhat separate from the CPU scheduler (although there is a link in priority and timeslices). Thus it was always possible to drag a process doing I/O down by having something else perform enough disk I/O, especially if you are the root user. If you can make a system swap that will hurt things even more (and that's a form of I/O).

The thing to remember is that servicing certain types of I/O need not necessarily use up much CPU time. If a device is capable of doing DMA it will need comparatively little time from the CPU to be serviced (bigger transfers can happen while the CPU is off doing other things as opposed to have the device generating interrupts all the time that the CPU has to service before any more data is transferred because some temporary buffer is full and needs moving). Additionally, certain network drivers are able to do NAPI which can reduce CPU load during heavy transfers. The way that Linux handles interrupts (which have a top and bottom "half") allows the bottom half to happen in a process context (so the heavier part of the processing counts towards a process's timeslice). This is touched upon in on Robert Love's MMCSS entry. However, if you have an important process's I/O queued up behind something less important (and the low priority task is able to generate enough a big enough request for I/O on its timeslice) then the important processes may appear to go slower (effectively its latency will go up due to having to be passed over because its I/O isn't ready) despite having more CPU time assuming the hardware can't satisfy all the requests for I/O quickly enough (imagine big writes to a slow disk with deep queues and the task needing acknowledgement all the data made it to disk).

Depending on which I/O scheduler you picked and your hardware you may be able to alleviate this problem but that's not to say that things can't be improved (or that no one is complaining about the problem).

In short I would imagine the new CPU scheduler impact would be marginal improvement on I/O performance or latency under I/O load. If it was OK before it should still be OK (but bear in mind in memory virus scanning is a special case that I would imagine would make any OS go slower).

Re:Always been possible...

[alvieboy]

If a device is capable of doing DMA it will need comparatively little time from the CPU to be serviced (bigger transfers can happen while the CPU is off doing other things as opposed to have the device generating interrupts all the time that the CPU has to service before any more data is transferred because some temporary buffer is full and needs moving).

True, but, as you don't have multiple port RAM, you won't be able to access RAM (or you will interrupt the DMA process). Hence, you will rely on the I/D cache - so bigger caches (and presumabilly filled) will help you on the DMA process.

I am not sure on x86 whether the arbiter gives priority to CPU or DMA bursts.

There are drawbacks for all the systems. Once we have dual port SDRAM (if we will ever) there *might* be possible to have the CPU reading the memory (maybe paged/bank based) while the DMA writes on another one.

Re:CFS issues - some bugs to iron out before 2.6.2

[12357bd]

One slightly annoying thing is that the current fix involves using sysctl to switch back (at least partially) to the old scheduler mechanism!

Is that's true?

Are we making a 'new' scheduler that under some circumtances reverts to the old one?. If that's true, please why not try to learn a little form the BSD code/people? It seems they already have-it right.

Re:No, there aren't. This is what a kernel DOES.

[datadigger]

... a task that even the most minimalist microkernels consider necessary to put into the microkernel.

Naah, only the dispatcher should in the kernel. The ordering of the processor queue (scheduler) can be delegated to a process outside the kernel.