Slashdot Mirror
Torvalds Explains Scheduler Decision
Firedog writes "There's been a lot of recent debate over why Linus Torvalds chose the new CFS process scheduler written by Ingo Molnar over the SD process scheduler written by Con Kolivas, ranging from discussing the quality of the code to favoritism and outright conspiracy theories. KernelTrap is now reporting Linus Torvalds' official stance as to why he chose the code that he did. 'People who think SD was "perfect" were simply ignoring reality,' Linus is quoted as saying. He goes on to explain that he selected the Completely Fair Scheduler because it had a maintainer who has proven himself willing and able to address problems as they are discovered. In the end, the relevance to normal Linux users is twofold: one is the question as to whether or not the Linux development model is working, and the other is the question as to whether the recently released 2.6.23 kernel will deliver an improved desktop experience."
I know this is news for nerds, but I'm just curious as to how many people here actually have a clue as to what this article is about.
Religion for nerds. Stuff that really matters
Oh Yeah Fr15t p05t!
nobody cares about the scheduler. I do care about the lack of ZFS support. Get over your "not invented here" syndrome and integrate it already.
As it seems many others don't agree to his opinions of Con, http://lkml.org/lkml/2007/7/27/426
Yeah, but does linux run it?
What do CFS and SD stand for in this case? The summary and linked articles do not describe this.
Having had my fair share disagreements with customers over technical issues, it just isn't worth trying to "win." The damage to your working relationships is still there even if you are shown to be 100% right. Try and help them address their problems as much as you possibly can, while trying to compromise as little as possible of the design. It's called diplomacy, and it's the difference between being given huge amounts of responsibility, and wanting to quit. You don't even have to agree with them, you just have to make them think you care.
Finally, it is common for programmers to try to avoid a subset of the problems in an area because it gives them the ability to write something "correct." Certainly a very satisfying experience for a programmer. However, that is exactly why it can be a bad idea to let a programmer rewrite a messy module. Very soon you can find the users of that module asking why a laundry list of things don't work anymore and an idealist developer trying to argue that they shouldn't... And it is exactly those idealists that like to rewrite working code. Not that major rewrites are bad, just that they have to be approached by someone mature enough to both expect a list of things they overlooked, and be willing to work with customers to resolve them.
-- http://thegirlorthecar.com funny dating game for guys
The only important points to me are:
1) Games are basic to linux desktops, we need linux games, so if Con scheduler was best on games, please, incorporate those changes.
2) EGO's are a pain in the ass, it seems that Con has been refused by Linus , because he didn't take the whole history into acount, too bad, that kind of things happend in any not trivial project management. Don't let EGO's rule the ball, that's the most basic point a manager (Linus) has to respect to keep people on the project.
What's in a sig?
The guy walked away. It's like quitting the Internet. Obviously if your reaction is to take your ball and go home (and I know, the ball is everybody's in this situation, but go with it) then you aren't mature enough to handle the responsibility Linus requires.
Slashdot - where whining about luck is the new way to make the world you want.
This Linus character showcases how much we (nerds) lack tact. Either we better learn some, or grow super thick skins as pre-requisite.
Oh, this post? No tact? I'm an AC, asshole.
which kernel scheduler is pretty low on the list of factors affecting what the Linux desktop experience is all about...
frankly, really high quality experiences take organizational planning and leveraging the expereince of huge groups in way that the "bazzaar" model of software developemnt in open source does not do well. Would someone please just build a mutual benefit corporation for open source users and maintainers? Let's start paying for project managers and the other experienced professionals required to make a "desktop experience" and you will see Linux take over.
The official kernel site says 2.6.23 is only on release candidate 1.
Shouldn't that be "2.6.22(.1)"?
Having lurked on http://www.lkml.org/ for several years, I find Linus to be rather rude. May be it's because English is not his first language...so words are not well chosen. I must say though, that I excuse him because he produces, [or helps produce] a very useful product on the world today. That is the Linux kernel.
I find Linus' comments very interesting, and very honest. He has good arguments, and to me it seems like he did the right thing. He is even open to change scheduler if someone can prove that SD does a better job than CFS, and get someone reliable to maintain it.
As a computer geek who first started using linux as a learning tool, and then used it as a server alternative back in the days when SCO and Unixware were the only "unix" options on Intel/AMD CPUs, I'm starting to hate the new desktop direction/hype Linux is taking. And it has nothing to do with the fact that under Centos5 x86_64, my new 2xE5320 Xeon, 8G of memory seems to be just as fast as Centos5 on Dual Xeons (2.4Ghz), 4G :(
Enough alerady with Linux and the desktop. Let the desktopers use MS or Mac.
Can someone explain to me why it can't be a kernel option to choose between these two schedulers?
It's an interesting set of emails. In addition to admitting that he actually didn't have any problem with the SD code, Linus points out that he made a gut call that Con is difficult to deal with without even looking into it because, apparently on near religious grounds, he doesn't believe in reading "specialized" mailing lists. What i find of concern is that he'd express such strong opinions about people basically without having even spent an hour or two browsing some list archives. Further, he seems perfectly aware that he may have heard just one side of the story, and yet he STILL doesn't feel he needs to look into it further or to soften his view? WTF ?
... arbitrary ?
Has Linux kernel development always been this
From TFA (actually form the quoted emails) after several mails where Linus has been bashing this Con Kolivas guy for not taking feedback and being argumentative, and then offers some statements about the virtues of a good maintainer some guy "Kasper Sandberg" asks him:
"Okay, i wasn't going to ask, but i'll do it anyway, did you even read the
threads about SD?"
to which Linus responds:
"I don't _ever_ go on specialty mailing lists. I don't read -mm, and I
don't read the -fs mailing lists. I don't think they are interesting.
And I tried to explain why: people who concentrate on one thing tend to
become this self-selecting group that never looks at anything else, and
then rejects outside input from people who hadn't become part of the 'mind
meld'.
That's what I think I saw - I saw the reactions from where external people
were talking and cc'ing me.
And yes, it's quite possible that I also got a very one-sided picture of
it. I'm not disputing that.
"
You can be pretty damn right that the desktop experience is improved with Ingo Molnar's scheduler! If you have done any serious audio work on any platform you know that Linux kernel + Ingo Molnar's IO scheduler = the best platform for serious audio work. This combination has the lowest latencies. Linux kernel+Ingo Molnar's IO scheduler+Ardour offers currently lowest latencies and the best of all - it's completely free! It is pretty amazing - really. Every true professional audio engineer will agree with me.
Whatever way you want to paint it, that's exactly what he did.
He believes that Ingo is a more reliable maintainer, so he chose Ingo's few-day old hack instead of Con's very mature and well tested scheduler.
Personally, I think that the person who is at fault here is Ingo, because he has a "Not Invented Here / By Me" mentality, and instead of developing Con's scheduler further, he totally objected to Con's work for ages (which prevented it from getting into mainline), and then suddenly saw the light and wrote his own quick hack based on the same design.
Ingo may be a good developer and maintainer, but he sure as hell isn't a friendly co-developer.
the Linux desktop responsiveness definitely improved (in my subjective experience) in the past two years or so.
If that was Con's scheduler, then thanks.
At this point I blame GNOME over the kernel for any remaining sluggishness, but then maybe my 1.2GHz athlon is just obso1ee7.
(Disclaimer - I have never met Linus or heard him speak on anything )
After reading over the article, I came away with the impression that Linus appears to be an angry person. I have also noticed this in past topics where emails from Linus have been mentioned. So I am asking the question as to why he appears this way. Is it because I always see such references at particular time like now with hot button issues? Or perhaps because English is not his native tongue? Or is it because he really does have an anger management issue?
Based on his own comments in those emails (and the standards that he seems to hold everyone to) it seems to me that Linus would not work with himself as he himself appears very pig-headed and stubborn.
Anyone care to explain?
I am Slashdot. Are you Slashdot as well?
Why can't there be a flag that determines what scheduler is used at runtime, with both schedulers built into the kernel? I thought the whole point of Linux is that it is customizable and modular - I know this doesn't necessarily apply to the kernel, but why not?
;)
I know very little about operating systems, schedulers, and maintaining large projects, so please excuse any ignorance in my post
Come on, it's quite obvious that Linus's secret superhero alter-ego has done battle with Kolivas' supervillain identity before (I mean, Con Kolivas? Do the writers even try to make these things sound authentic?) and is now trying to thwart his evil plans of global domination.
In Debian Etch the basic scheduler could have some serious improvements. At first things were fine and I expected to be happy with Debian Etch and use klik to run cutting edge Debian packages in user space but as things were made ready for the server target market plug and play Wacom use was removed and smooth audio play simply dissapeared, fades in Amarok stutter at start and end. klik was never made to work on Etch, so it seems server client deployments are the focus and the individual desktop is left behind. I won't be switching until this new kernel has become in use across distros.
... I find a bit of hypocrisy.
.... consideration about inclusion of code they wrote.. Uhhh did I forget to say its "Open Source"?
Three is right, as its Con and an email exchange between Linus and another.
Whooo hooo..
That settles it.... everybody is accounted for..... right??
Its open source but with all the talk about having a maintainer of certain character as a part of the consideration of
Its not uncommon for pioneers to be forgotten as what comes next, takes over...
Or does this mean that when a maintainer dies, so does what they were maintaining?
The general message Con seemed to be expressing was more interesting as a general observation than of specific code.
The response from Linus suggest that although Linus does not frequent specific topic lists because of inherent bias, he has his bias none the less.
There is a general across the board bias, proprietary and open source, and it is one of exclusion of the end users.
And it comes across as arrogance motivated by money and/or ego.
To explain, programming is an act that includes creating functionality that is then accessible thru an easier to use interface such as a function call with arguements and expected return value. The general concept is common knowledge in coding regardless of what programming language you are using,
However, this concept is not typically provided to the end user, but instead kept away from the user and certainly not provided to the user, when some distortion of it is provided the user, in any sort of easy common consistent manner.
To clarify, users access programs typically via a command line or GUI. Neither of which are so conducive to allowing teh users to put things together for themselves. All the functionality if available to the user via the programs GUI or command line. But the same functionality is not available in an mode that allows user to call the functionality in the program and make use of the results outside of teh programs command line or GUI interface.
Con mentioned the Amiga. The Amiga had all three user interfaces. The command Line, The GUI and the missing user interface an every other system today, The IPC port interface, most commonly known as the AREXX port but did not need AREXX running in order to use the port for "user putting things together for themselves".
SO YOU DON'T LET USER PUT THINGS TOGETHER FOR THEMSELVES! WHY NOT?
Cause you can dumb down the user and get ideas from them and sell those ideas implemented, back to them.
But when you take away from the users, the ability to put things together for themselves, then that makes you a hypocrite when you then call them ignorant, armchair coders or any other demeaning term. As it is you who have created that self supported dependency of trying to justify your lack of inclusion of others.
Con outright stated how he got started.
Maybe You linus and a whole world of other coders, need to pull you head out of your asses and SERIOUSLY realize, THERE ARE OTHERS you are not considering.
Ultimately, if people want to optimize their system for their needs, they should be able to. But there is serious prevention of that across the board.
Is the Linux development model more like Surf's Up or Happy Feet?
Have you improved it to the point that when the system is borderline out of main memory or has a moderately high load average it actually *works* as a desktop system?
E.g. when Firefox is consuming 65-70% of main memory and slower than #%#$ and you know it is waiting on swapped out pages and your swap rate is measured in the dozens to hundreds rather than hundreds to thousands (on vmstat)? (I mean really, how can one take an operating system seriously when only memory is at 100% and not CPU + memory + Disk I/O?)
The real issue, for those who have read comments that Con has made in interviews, seems to be the lack of concern on the part of most of the "in-crowd" Linux developers for performance on the desktop. In part this seems driven by the fact that the people who actually get paid to maintain Linux, benchmark it and "improve" it only care about its performance in server farms and *not* on the individual desktop. I will weigh in on the side of desktop user out there (that wants the Linux sitting beneath their desk to devote its every waking minute to making *them* happy) by saying that if my mplayer "hangs" in the middle of a song (only to continue with a loud burst of noise 10 seconds later) when the CPU is busy with "nice -19" processes, my Firefox browser takes half a minute to scroll a page or open a screen) when memory is tight, and it takes minutes to bring up a tab or minimized program I haven't touched in 3 days and return them to a functional state then the operating system *Has a PROBLEM*.
Con was very clear in his interviews that the problem is the lack of caring about *desktop* performance. Given my comments in the previous paragraph -- some of these areas may be very difficult to benchmark and as a result one is left with nothing but handwaving and loud voices when it comes to discussions about whether the problem exists and how it should be fixed.
I will say this, in the mid-'90s I used X-windows under Unixware on *Pentium 1s* as a desktop machine. I now use X-windows under Linux on a Pentium 4 (with 5-10x more main memory) as a desktop machine. I would argue that my desktop user experience is as problematic now as it was then *despite* the hardware improvements. That IMO is what Con felt was the problem he was trying to address. That is what it would appear the core Linux developers may be failing to address. Con's points raised my awareness level to the extent that I actually went investigating to see whether there were open source distributions of the BeOS and/or Darwin (which is based on Mach) available since they are based on different OS architecture models and might be more end-user friendly [1]. I was hoping to find something I could run in a VM under Linux on my current hardware without major file system surgery. But I have little confidence that such an approach would fix core problems with the Linux scheduling and paging systems. I would *love* to see a real side-by-side comparison of Linux vs. FreeBSD for desktop users with an emphasis on how BSD scheduling, paging and swapping may be different (better?).
(And as a side note, I could care *zero* about the performance of Linux in file server applications!)
1. I did use both Nextstep (on Pentiums) and IRIX (on a R4000) for a while and found both to provide better end-user experience than Unixware (X) or Windows (95-98). I am disappointed that Linux barely manages to match those experiences given the hardware available nearly a decade later.
The scheduling debate is the lost prophecy of John Titor!
I am not a kernel developer and do not follow the mailing list. I was under the impression that a new scheduler that was supposed to drastically improve performance went into the kernel a couple years ago? I know there has been a huge difference in desktop performance from that time.
Any "reasonable" reason why they couldn't integrate both?
Upward mobility is a slippery slope - the higher you climb the more you show your ass.
"Schedulers are actually not at all that important
.. i'm dying from laughter.
... dude, i'm watching your ...
in the end: they are a very very small detail in the kernel."
lol. crips
seems like somebody has been in the usa too long
ship sink. hilerious. honestly.
freakin' americans, sheesh
Whether Con was aware of it, when he tried to integrate into mainline Ingo was his main customer. Specifically the person he was trying to deliver work to. And Con committed the cardinal sin of telling a customer that the customer was wrong about what he wanted. Even if Ingo were too coked up to operate a keyboard reliably and had it all wrong, trying that never seems to work.
Did Con gain anything by refusing to re-introduce the hack to get X working the way it had previously under load? Even if he'd just put in a #define that allowed it, and then spent the next year arguing to take it out, there wouldn't have been this breakdown.
-- http://thegirlorthecar.com funny dating game for guys
Start profiling the damn thing! Write performance tests, good ones, write really evil stress tests, stress the crap out of it, and then you will know what is *wrong*. Might not be the kernel at all, in fact, I think that a lot will be because of applications (always a huge source for problems), the UI/Graphics subsystem (again, huge source for trouble, X11, drivers, UI toolkits, they all tend to be far from perfect) and such.
:(
But pissing and moaning won't do you any good. At least Con did try to write stuff, but not being a professional software engineer hurt his efforts I'm assuming. The guy would probably make a good technical manager though. It is a shame he felt he had to quit, it would have been much better if he could have gotten a few other kernel hackers on with him to go on. I think he ended up with a lot of users backing him, but no coders
Not that I am a Linux person, but I always find it sad to see people who are really into something quitting for bad reasons (bad in the sense that the shouldn't have to, not that he did something unwise).
I am with Linus on this one. For the life of me I can't understand what this sucking up to RMS is about. Linus himself does not think GPLv3 is a good thing. So why do people keep adopting it.
Linux without Linus cannot succeed. A lot of young folks don't realize this. RMS can't advance the cause of Linux like Linus does.
And you still wonder about linux not dominating the desktop?
Bill is a thousand times better dictator than this scandinavian junk.
More like Surf's Up, but with emphasis on kernel development, instead of surfing. By the way, stay tuned for Dreamworks' Pictures "Tux". An animated comedy, starring yet even more penguins, and the lovable little guy himself. Tux is an ordinary penguin, working away at his operating system all day, like good penguins should. A chance meeting with the BSD Daemon and Gnu marks the start of a grand adventure that takes Tux all over the internet (including a brief visit to Slashdot itself). Tux makes friends with a host of colourful characters, meets the girl of his dreams, and learns an important lesson in process prioritization and task scheduling.
Unlike porn, which yada yada rimshot hey-ooh!
So, in other words, he considers the desktop to be the most import part because developers have to use a desktop in order to develop?
And what's with the massive ego? It's as if suddenly Linus thinks he invented compilers or something. I think he needs to take a vacation and regain some perspective.
This post expresses my opinion, not that of my employer. And yes, IAAL.
The CDDL was designed to be GPL incompatible, Solaris didn't want its crown jewels to be in Linux.
hearing yourself talk. I assure you, you are the only one.
Now stfu, kokgobbler!
Too many security holes because of the bloat and there is no standard for software to run on all these different operating systems.
Linux os is a pipedream.
Try Haiku or another operating system built for the desktop.
Quoting someone on ck mailing list, i think this is worth reading:
---8----
I don't really want to keep all that -ck flamewar going but this sum-up is
a little strange for me:
If Con was thinking SD was "perfect" why he released 30+ versions of it?
And who knows how many versions of his previous scheduler?
Besides Con always tried to help people and improve his code if some bugs
or problems were reported. Archives of this list prove that. I reported
several problems (on list and privately) and all were fixed very fast and
with very kind responses. I had run -ck for months and years and it was
always very stable (I remember one broken "stable" version).
I don't know what exactly are you refering to when you say about those
unaddressed reports but maybe it depends on who was asking, how and to do
what (for example - purely theoretical one, I don't remember exact emails
you refering to so I am not saying it happened - stating at the beginning
that the whole design is unacceptable and interactivity hacks are a
must-have won't make a friend from any maintainer and for sure lowers his
desire to get anything fixed for that guy). Or maybe Con had some bad day
or was depressed. Happens. But I really don't remember Con ignoring too
many valuable user reports in last 3 years...
And no - I am not thinking that SD was "perfect". Nothing is perfect,
especially not software. But it was based on months and years of Con's
experience with desktop and gaming workloads and extensively tested in
similar uses by _many_ others. In nearly all possible desktop
configurations, with most games and all video drivers. This is why it was
perfectly designed and tuned for such workloads while still being general
enough and without any ugly hacks. And because of these tests and Con's
believe that the desktop is very (most?) important all bugs and problems
in this area were probably killed long ago. I think even design was
changed and tuned a little at the early stages to help solve such
interactivity/dekstop/gaming problems.
So it does not surprise me that CFS is worse in such workloads (at least
for some people) because I strongly suspect that the number of people who
played games with current version of CFS is limited to about 5, maybe 10.
And I also suspect that you (and Ingo) will get many regression reports
when 2.6.23 is released (and months later too... or maybe you won't
because users will be to "scared" to report such hard to mensure and
reproduce "unimportant" bugs). Hopefully such problems when reported will
be addressed as soon as they can. And hopefully they will be easy enough
to solve without rewriting or redesigning CFS and causing that way even
more regressions in other areas. If not people will probably be patching
O(1) scheduler back privately...
Software projects need a benevolent dictatorship. If anything, there is too little in Linux. Look at MS, even a poor standard is better than none. Too many GUIs lead to people not liking to use Linux. I just want my OS to work, I don't want to have to make 8,000 choices. I want 1 answer to thc question of how do I do 'X', not a dissertation on the 8,000 ways that may apply depending on the settings I have chosen.
Your ad here. Ask me how!
Thread through many mails in the thread archived at lkml.org.
Overall, it looks like almost everyone feels that Linus's
portrayal of Con is absolutely unfair. Linus seems to have
made a decision & is then doing character attacks on Con
to justify his decision. There is even one mail where he
talks about Con's illness & says that it didn't help.
My personal opinion is that Linus comes out looking worse
in this Linus vs Con debate.
I barely know the first thing about Linux, but as a developer, I can say three things. First, I hate Windows' hegemony; two, I think anything that forces M$ to improve their offerings is a Good Thing; three, I feel sorry of Linus Torvalds. Freedom of speech is also a Good Thing, and people should say what they think. But the constant invective directed at Linus, while he tries to keep the wheels on the Linux Express, must be incredibly wearing. Somebody has to call the shots, somebody has to make the final decisions. I wonder if Linus would have made his source public in the first place, if he knew what he was getting into.
That's it! I'm switching to this new OS called Vista...
over MS ones any day.
Sounds to me either scheduler will do the job just fine.
The decision between two good alternatives is always a difficult decision - someone, no matter how good the ideas, will feel slighted.
Microsoft has certainly made their share of mistakes but they have been very successful over the years giving their users exactly what they want.
In this case, Microsoft has given users a very responsive desktop full of whiz bang graphics and a (mostly) unified way of installing, configuring, using, and uninstalling applications which very rarely requires anything more than clicking "next, next, next, done".
Linux has a long way to go before it can be taken seriously on the desktop as a general use OS and you're just fooling yourself and drinking too much of your own koolaid if you think otherwise.
Not read the comments, so refrain from marking down as redundant... that option should be removed anyway, except for abuses (e.g direct copies of other posts).
The statement "one is the question as to whether or not the Linux development model is working"
is ill thought.
The Linux development model is directly based up evolutionary theory - if it fails it will die, something else will take it's place.
THINK
Go ahead buy an apple g5+ system and pay mega bucks for pro tools audio just so you can do everything in realtime without having latency issues. However if you are a cheapo home studio freak like me then you will want to know the difference between a realtime kernel and a server like setup.
The beauty of the the 2.6 line of kernels is that you can, with a simple recompile run both by choosing which kernel to run from a boot loader. What would be really nifty is if there was a way to change realtime settings between different applications...for example if you run an audio app like jack then the kernel timing will automatically reset to 1000hrz then if you shift to something like the Apache web server the timing will automatically switch back to a default 250.
It would sure simplify things for the average user if kernel timing and realtime schedule settings could be addressed through an interface layer rather than having to do everything at the boot core level. I know I am just blowing smoke out my ass but is this not how the Apple OS treats realtime scheduling and timing?
A few months before Ingo wrote the O(1) scheduler, he flamed anyone who dared to suggest that an O(n) scheduler is a bad idea. He was *very* aggressive about it, going on and on about why O(n) is best and how O(1) would be worthless. Using Linus's words (about Con), Ingo "ended up arguing against people who reported problems [scheduler linearity], rather than trying to work with them". It therefore seems a bit strange that Linus uses this statement to describe Con, arguing this is why he favors Ingo...
Importantly, Ingo was dead wrong back then (indeed, this is why months after, Ingo came up with the O(1), announcing it as if it was his idea and as if nothing ever happened, not *ever* saying something like "I was wrong, sorry for the flames").
In contrast, Con was right in refusing to pollute the design of SD with Ingo's unfairness discipline. (This is what Linus referred to when he made the "arguing against" statement.) And what do you know? A few years after, Ingo comes up with a "Completely Fair Scheduler"...
I'm in scheduling research for many years. I followed the long Linux scheduling saga, which actually started way before Con was in business. Please believe when I tell you: Linus comments about Con are ludicrous, and petty. This is not Linus's finest hour.
Note however that this does not mean that Linus made the wrong decision: Even though SD is somewhat better than CFS, Ingo is orders of magnitude a better programmer than Con, orders of magnitude more knowledgeable, he gets paid to do the work, has gotten along with Linus for years, and will eventually make CFS as good as SD and even better. This is the real reason for Linus's decision. (Or at least, it should be.)
But the stuff Linus said about Con... well, that's just Linus being small.
"if there have been any"
Genuflecting fanboy, much? Also, how about having a fair outlook on the world, rather than relying on some internal pre-conceived set of values? Wait, this is Slashdot, why am I bothering?
I hate printers.
One only has to look at the reaction of a programmer who quits because his pet project didn't get into the mainline kernel. He obviously was not the best choice for a long term maintainer.
Arguements over which code was superior are irrelevant at this point. The decision was made and will be made again when next revisited, after extensive testing and reporting. Right now the independent reports recieved by Linus were that they were pretty much equal. That only left him with the decision of who would be the better maintainer. He is the guy who gets to choose and I think he chose wisely. No offense to Con. I wish him all the best and hope he reconsiders as he matures.
I play games under Linux and I appreciate Con's efforts, but I understand that it is not the OS only focus. He didn't, Ingo does.
All Hail the Benovolent Dictator Linus. May his pragmatic glory shine upon thee.
OSGGFG - Open Source Gamers Guide to Free Games
A few months before Ingo wrote the O(1) scheduler, he flamed anyone who dared to suggest that an O(n) scheduler is a bad idea. He was *very* aggressive about it, going on and on about why O(n) is best and how O(1) would be worthless. Using Linus's words (about Con), Ingo "ended up arguing against people who reported problems [scheduler linearity], rather than trying to work with them". It therefore seems a bit strange that Linus uses this statement to describe Con, arguing this is why he favors Ingo...
Importantly, Ingo was dead wrong back then (indeed, this is why months after, Ingo came up with the O(1), announcing it as if it was his idea and as if nothing ever happened, not *ever* saying something like "I was wrong, sorry for the flames").
In contrast, Con was right in refusing to pollute the design of SD with Ingo's unfairness discipline. (This is what Linus referred to when he made the "arguing against" statement.) And what do you know? A few years after, Ingo comes up with a "Completely Fair Scheduler"...
I'm in scheduling research for many years. I followed the long Linux scheduling saga, which actually started way before Con was in business. Please believe when I tell you: Linus comments about Con are ludicrous, and petty. This is not Linus's finest hour.
Note however that this does not mean that Linus made the wrong decision: Even though SD is somewhat better than CFS, Ingo is orders of magnitude a better programmer than Con, orders of magnitude more knowledgeable, he gets paid to do the work, has gotten along with Linus for years, and will eventually make CFS as good as SD and even better. This is the real reason for Linus's decision. (Or at least, it should be.)
But the stuff Linus said about Con... well, that's just Linus being small.
Don't worry, there are enough non-knowledgeable people here to mod you down. I must admit that this debate really boils down to Linus and Ingo taking a long time to reach Con's conclusions, and then not being as gracious about it as they could be. Con, of course, was insulted by this (who wouldn't be?) and all the while we had two major camps of Linux users: the pro-CK one (holy crap! My system works with this patchset!) and the anti-CK one (damn! My server doesn't need this, wtf?).
In the end this just demonstrates that having egotistical bastards running the show isn't always going to yield the best results. Linus and Ingo do fantastic work, and they do sub-par work. But when someone points the sub-par stuff out, head for the hills! They will always have a legion of fans defending them, more than willing to dumb-down the real issue in favor of pretending their infallible leaders are never wrong.
Thank you! I may not know what "staircase deadline" means yet (though the name gives some idea), but it's a lot easier to Google than "SD".
I read TFA, but I was still puzzled.
Agree 100%
Cheers
Ben
This is written by Linus in one of his emails to LKML:
> That was where the SD patches fell down. They didn't have a maintainer
> that I could trust to actually care about any other issues than his own.
I have been on Con's SD mailing list for >2 years now, and not once ever have i seen this to be the case.
He has always (time, etc permitting) followed up on user reported problems.
So... what happens if Ingo suddenly dissapears?
Im sure others will continue his work.
The same should be said for Con's scheduler also, which makes Linus's comments even stranger.
> Has Linux kernel development always been this ... arbitrary ?
Once upon a time... When "Linux" came on a stack of floppies.
There was just this one kernel developer. And whatever the fuck he felt like that day, shipped.
It's LESS arbitrary these days by a significant margin.
Do daemons dream of electric sleep()?
This whole situation reminds me a lot of devfs. The developer of devfs (Richard Gooch) was maybe a bit of an outsider with good ideas and strong opinions that sometimes clashed with those of kernel developers that were more insiders or closer to Linus. The story is a little different in that devfs had actually made it into the mainline kernel, but then later was replaced by udev (the first draft of which was by Greg KH in a day or so... again very much like Ingo's whipping up the first draft of CFS).
/lot/ of energy into their work. But that energy was not lost to the rest of us, as without the experience of their groundbreaking we would not have gotten the solutions we eventually did. Even if Greg KH's udev and Ingo's CFS share no appreciable amount of code or algorithms with devfs and SD, if they are honorable, I'm sure they would admit that they could not have so quickly whipped up their solutions without the example and inspiration of RG and CK's work.
/Maybe/ instead of losing them in the long run we could have gotten some more innovation from talented developers like RG and CK. The problem, I think, wasn't the decision adopt CFS instead of SD. Rather, regardless of whether or not is true, the problem is Linus's public judgement that CK is not a "responsive maintainer". I didn't follow the CK or the lkml lists in that time frame enough to know if he is right... but a really good leader would not have made such a judgement public even if hed believed it, and instead would have worked to find a way to keep such talented persons contributing.
Then as now with CK, eventually Richard stopped doing linux kernel work altogether. I thought it was a sad loss of a talented kernel hacker, and I had been a devfs user, but I must say that in retrospect I do think udev is a better solution. It is simpler, has less impact on the rest of the kernel, but has proven itself to solve all the problems devfs tried to solve that actually mattered.
What's the moral of the story? That both sides are right... on the one hand, there's something sad here, because at least several times in linux history an outsider had to fight for innovation and in the end was pushed away even as his innovation was grudgingly adopted by reinventing it. On the other hand the actual results do seem to indicate that linux is NOT resistant to change, and maybe that the better, more maintainable solution tends to win out.
There's also another thing to keep in mind... it is a pattern in this history of technology that the first attempt to solve a problem is rarely the one that becomes dominant. Both Con Kolivas and Richard Gooch should be recognized for the innovators that they are... and if they were wise they should also not begrudge the fact that it wasn't their exact solution which eventually got adopted by the mainstream. I know this is difficult... they both put a
Finally, I would like to add that although the way I see all this, it has little if anything to do with Linus's personality, nevertheless I think Linus could have handled these cases better.
And so it becomes obvious what has happened here. There were two schedulers competing for inclusion in the mainline Linus kernel. Linus evaluated both and selected one. Having lost this race, Con declared Linux a failure and stormed off in a huff.
In other words: nothing to see here, move along, the increase in desktop Linux adoption will continue its slow but steady pace.
Tired of FB/Google censorship? Visit UNCENSORED!
You have to give Google some context. If you use con kolivas sd as your search terms you can just hit "I'm Feeling Lucky" and go right to this.
He lives in San Jose these days . . .
Actually, he moved to Oregon in 2004. I thought he lived in West Linn, but on at least one website, Dunthorpe claimed him. They're both suburbs of Portland.
Even though SD is somewhat better than CFS, Ingo is orders of magnitude a better programmer than Con, orders of magnitude more knowledgeable, he gets paid to do the work, has gotten along with Linus for years, and will eventually make CFS as good as SD and even better.
Well, that's to be expected. Ingo is a programmer, whereas Con is a medical doctor who hacks the kernel in his spare time.
While Ingo might be a better coder, I think it would be a tragedy if Con gives up altogether. He has been right most of the time, and he has a rare instinctive skill in determining what works and what doesn't.
He has been right most of the time, and he has a rare instinctive skill in determining what works and what doesn't.
That's called raw talent and is the obvious reason for Linus to be afraid to work with him (Linus simply never could have taken on kernel programming without years of training and a book of BSD source in front of him), that and the fact that Linus and the rest of the kernel developers having a gang-mentality where they beat on people who try to join up.
Violent thug gangs here in NZ have the same process where you must be a "prospect" which entails being a scapegoat and being shat on for a few years from a great height, and maybe getting a patch at the end, after an "initiation".
In Con's case apparently, according to Linus. In order to pass the initiation he was supposed to sit in silent reverence while a "patched" gang member steals his ideas.
Sorry Con, but no "bro" status for you in the Linux gang if you can't keep your mouth shut.
Do you also go as "Charles Goodwin" or "Free Gamer". The parent was also posted to http://kerneltrap.org/node/14008 , apparently 8 hours earlier.
If the answer is war, you are asking the wrong question
Somebody directing a large effort does not have the ability to fully investigate every subsidiary dispute fairly without the effort grinding to a halt. Shorthand criteria are a necessity, as are rules of thumb as to what sources to bother investigating. This will inevitably lead to a number of less-than-optimum technical decisions along the way, but the results will be markedly superior to those where the manager stops everything to thoroughly investigate every aspect of every decision.
Con Kolivas's reaction to "losing" was not to continue to maintain SD and try to get it in later, or to try to improve CFS, but to quit kernel-hacking entirely. Which means he is not of a temperament that can accept that large projects will have arbitrary decisions that go against him, which means he would be a bad choice for the maintainer of a major kernel system. His actions in retrospect justify Torvalds's judgment that he couldn't trust him as a maintainer. Kolivas proved Torvalds correct on the management question, even if Torvalds is wrong on the technical one.
I fully agree with the first part of your post, but I don't buy this 'Con's reaction justify Linus judgment' argument:
1) It seems clear on the mailing lists that Con was really a good maintainer (only one 'problem' reported).
2) Con's reaction seems quite understandable, after a very long time working on a project you see it not only refused by the maintainers (perfecty ok on that), but suddenly 'copied/inspired/wahtereryouwant' by that very same maintainer (not quite right).
EGO leads to sectarism, that's all. The problem is that it seems that Con's scheduler was very good at gamming, and it's a shame that Linux dimissed a good piece of code on a specially sensible area for personal motives.
What's in a sig?
It would be? It was! C.f. http://linux.slashdot.org/article.pl?sid=07/07/24/ 1432245, and if you search on lkml.org you can find the actual email where ck submits his last contribution and asks not to be "contacted regarding this or any other kernel development" issues. Really rather disappointing. And by the way, the interview linked from the article is very much worth reading.
To summarize this situation in a few points:
Cross-pollination is beneficial to everyone - plugsched (allowing the user to select between SD and CFS) is not a good idea for the following reasons:
Linus doesn't care about politics and narrow-minded viewpoints - he raises the following very important points:
Linus isn't abusive or showing a large ego
"But the stuff Linus said about Con... well, that's just Linus being small."
Well, no.
I mean, was Linus as abrasive as usual here? Yeah. But did you read Con's messages sent to the LKML (rather than just those sent to Con's private ML)? Con really did get quite whiny and argumentative at times, in bad ways. He seemed to be prone to taking criticism from certain people much too personally when he should have just focused on discussing the technical issues. Even when he did have good counterpoints to make, he often ruined them by combining his response with obvious drama bomb material.
After Ingo wrote CFS, Con's attitude got worse; he more or less trolled several CFS threads. Not the deliberate and conscious form of trolling; he just couldn't resist the temptation to toss out bitter and sarcastic remarks when there were regression reports against early versions of CFS. He should've said something constructive or nothing at all; instead he came across as jealous and resentful of competition.
I do have sympathy for Con, but I also believe he could have helped his cause a lot by growing a thicker skin. The Linux kernel community has had problems in the past with subsystem maintainers who were drama queens. Richard Gooch and Andre the IDE guy come to mind. It's not too surprising that core kernel developers react negatively to similar behavior, even if it's much milder than the examples I gave.
Yours is a very good post. Shame that you did so anonymously. I would like to add that I think the overall sentiment here on slashdot (well, mine anyway) is that it's a shame that _nowhere_ on these lists does Linus come off as anywhere near apologetic. As others have pointed out: a valuable kernel developer was lost through rudeness and incomplete information gathering and apparent cronyism and petty politics, and in the development process we all support (that of open source over the internet), that is something that should be avoidable and avoided. Why wasn't it ?
Oh and don't come back to me and say he can fork the kernel - he can make his own distribution - he can come back when there is a specialized distro which does allow for pluggable scheduling. The fact remains that a superior solution at this point in time was rejected over an inferior one, that the developer of the inferior one suffered from NIH-syndrome and made vague claims of other people's hard work and twisted and turned which made it look almost like he was trying to revise history, and that the developer of the superior solution walked away frustrated.
Open source lost a little bit - the process needs revision IMHO. And Linus should issue some kind of statement that he is at least a little bit sorry.
Religion is what happens when nature strikes and groupthink goes wrong.
For those who want a Free/Open Source OS aimed specifically at the desktop, and ignoring all the server side code that forces the kernel to be less efficient at desktop tasks, take a look at http://haiku-os.org/
If you like Con scheduler apply the patch and STFU. The Desktop user experience is subjective. A scheduler design has little to with hour glasses and smooth scrolling. The scheduler is blind to weather you are handling a http request or pushing a msg into the sound driver.
Should it be?
Monolithic design anyone?
Poor product integration? oops! Linux is not a product. Kolivas nails the price the PC architecture
has paid by de-coupling software and hardware. Hardware on steroids and the half ass software that goes with it.
But I deride those who whine about poor desktop experience and demand the OS community to do something about.
In Spanish we have a phrase for it:
"Quieren mamar y dar topes"
- these are not the droids you are looking for -
linus has lost this PR war for once. now give us the #ifdef DESKTOP so that we can compile our own desktop-optimized kernels and see for ourselves if anything breaks like hell. the one size fits all principle is stupid is so very much not taking into account the power of the C preprocessor. the cpp is a tool for allowing choice. use it. you can even call it #ifdef DESKTOP_UNSUPPORTED if you want to make it clear, that users are going to go for potentially unsupported or broken code.
The problem isn't the scheduler. The reasons Linux (or any other OS, for that matter) is slow are:
1. processors (a computer with a Core Duo is faster then one with a Celeron)
2. programs (write faster code, and use gcc -O3)
3. the kernel (M$ isn't the only one who makes bloated OSs)
4. the hardware (get more RAM and/or a bigger hard drive)
Wow, you posted the same thing twice and hot +5 Informative for one and +5 Interesting for other.
:)
Nice trick... will remember next time
If he refused to merge any of Ingo's code for 2 years and Ingo kept plugging away in his own branch then you could say that.
Tis no troll, tis +3, funny!
from a psychological point of view...
SD = staircase deadline
sounds like an algorhythm.
CFS = completely fair scheduler
sounds like a hopefully self-fulfilling promise, not an algorhythm.
this of course says nothing about the code.. just noticing the detail...
we just try to get up as fast as possible, proceed with whatever we were doing acting like nothing happened, and hope that nobody noticed this embarrassing situation
Doesn't this lead to people being paranoid about what others may be feeling about them? I.e. that they're stupid or clumsy, etc? You have no indication of what the other person thought except that they noticed your mistake. The only reinforcement in that situation is that other people are aware of your faults. Doesn't this make everyone paranoid?
My read is different, but it'll take a few paragraphs to explain why, so please bare with me. The O(1) so called "improved interactivity" was just a hack upon a hack upon a hack that never worked. If you ever read the code you know what I'm talking about. It was a complete mess. The mess was the result of trying to obtain a goal that is unobtainable: very roughly speaking, Ingo (and others) believed that it is possible to deduce how "important" a process is, based on the frequency of the process' sleep events (not the duration of the sleep). I can point you to a few research papers that show that this can simply *not* be done. Briefly, it has been shown that for each "important" application you can find an "unimportant" application such that the CPU usage pattern of the two is very similar; Hence, you would never be able to distinguish the important from the unimportant (if you are only basing you decision on CPU usage). All you can do is (1) divide the CPU equally, and (2) provide good response time to applications that sleep for long periods of time (like editors). According to several of his emails, Ingo is not reading scheduling papers, and proud of it.
I suspect Con doesn't read such papers either. But he knew this is the case nevertheless. For years Ingo and his followers attacked Con on this. Note that this is a purely technical issue, and Con was *completely* right, e.g. take a look at CFS, or at Linus's original scheduler that lasted until 2.2, or, Solaris, *BSD (with the exception of ULE), HPUX, etc. They are all much closer to the SD's philosophy than to the O(1)'s. (BTW, the only other OS that tried to do what Ingo wanted is... the Windows family).
The discussion between Con and others regarding this issue was not about minor details. Ingo and his followers view towards Con was plain and simple: "your design is crap, it doesn't do what schedulers are supposed to do". Con resisted. And I agree, after years in the business, he was occasionally emotional about it. But this is really understandable and reasonable. Especially when you compare it to the way Ingo and Linus express themselves from time to time (talk about "drama queens"). For example, notice how Linus talks to Ingo when Ingo doesn't understand something immediately.
The bottom line is that most of them are "drama queens" from time to time. Nobody is perfect. In absolute terms, however, I think Con usually expresses himself in a calm, quiet, and relatively humble manner. Certainly in comparison to Ingo/Linus.
>Even though SD is somewhat better than CFS
Ah ok a solution is better... We use that solution, right? Because after all is that not what Open Source is all about? (http://en.wikipedia.org/wiki/Meritocracy)
Meritocracy is a system of government or other organization based on demonstrated ability (merit) and talent rather than by wealth (plutocracy), family connections (nepotism), class privilege, cronyism, popularity (as in democracy) or other historical determinants of social position and political power.
>Ingo is orders of magnitude a better programmer than Con, orders of magnitude more knowledgeable, he gets paid to do the work, has gotten along with Linus for years, and will eventually make CFS as good as SD and even better.
Ok so demonstrated was a scheduler that was better, but chosen was somebody who is perceived as being more knowledgeable, and gets along with Linus and EVENTUALLY will make CFS as good or better... And until EVENTUALLY hits I should wait around and suffer the problems? And how is this different from say Microsoft who refuses to fix bugs?
Let's call this what it is! Corporatism at the open source level with Linus's nepotism!
"You can't make a race horse of a pig"
"No," said Samuel, "but you can make very fast pig"
Torvald and his followers are disticntly second rate in both their development model and technical skills when compared to McCusick, de Raadt etc...Slashdot Linux fanboys will whine, but if you want a decent Unix implementation get a BSD.
I just wish someone else had the brains to actually start another one. It seems like a fiendishly difficult process.
However, Linus is apparently massively overworked, his ego has gone berserk, there are apparently any number of technical problems, and then there's been the whole brouhaha about whether or not the thing is going to be pushed to the GPL 3 over the heads of people who don't actually want that. In other words, it's a mess.
Have you investigated the CFS code? There is an interesting discussion about CFS internals over at LWN that seems to contradict what you say:
http://lwn.net/Articles/242992/
There Ingo explains that the "sleep events" mechanism you mentioned survives in CFS too, so you get both "fairness" and "sleeper fairness".
I read the post. There's a big difference between sleep-duration (to which you refer) and sleep-frequency (to which I refer). Basing decisions about sleep-duration is fine, and is equivalent in every respect to basing decision on runtime (as the two are complementary). The O(1) was largely based on sleep-frequency. In contrast to what the lwn post claims, the two schedulers have a lot in common.
Well for one thing, I know this whole history has scared me away from ever considering to contribute to the Linux Kernel.
RogerWilco the Adventurous Janitor
How can a person be "orders of magnitude a better programmer" if they repeatedly make significant errors of judgement?
Ingo wrote the bad scheduler that Con dramatically improved! Con only had work to do because Ingo dug Linux into this poor scheduler design to begin with.
Maybe you could argue that Ingo writes cleaner code or something, but to me good judgement is what makes a programmer good.
Hmm... I didn't realise that he had made a final decision in this regard.
That's terrible for Linux.
I guess I would say that everybody that is actually doing something, makes mistakes. But I do agree that the O(1) was a big mistake, which no doubt reflects badly on the Linux development process. And Con's work has indeed managed to put the scheduling subsystem back on track. Plus, I think it's safe to say that Con is orders of magnitude a nicer guy than Ingo.
But I'm being realistic here: Please consider that even though Ingo implemented the O(1), Linus was the one that decided that all this messy interactivity mambo jumbo would make it into the mainline and replace the scheduler Linus implemented more than a decade ago. At the bottom line, it's Linus's mistake and Linus's responsibility. Also, a lot of really sharp guys didn't fight against the O(1). So there's something of a collective responsibility here.
In light of this, if you were Linus, who would you choose? I'm not at all suggesting that the answer is carved in stone. I'm merely saying that the Ingo alternative has significant benefits. He is after all a very talented professional (Con used the work "brilliant" to describe him) that gets paid to do the work. I argue that these pros shouldn't be dismissed.
As i understand it...
There is the scheduler made by CK, which has been developed for months, maybe even years in the -ck patchsets...
Then there is the scheduler made by Ingo, which performs about the same and is designed in much the same way, but which was developed in under a week...
Does this not strike you as wrong?
Firstly, that something would be merged into the mainline kernel with so little testing...
Second that Ingo would develop a new scheduler, rather than modifying the existing and fairly similar one by ck? Or is that one somehow so flawed it needed to be junked?
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
This is pure speculation, as the real work will be in comparing the two schedulers and if one really is better than the other, adopting the better one, but I'd say that if one took a very long time to make, and the other a very short amount of time to make, that perhaps the shorter one has less code. If it does the same thing, then I'd pick the one with the less code, unless the other one provided extra features, including perhaps modularity.
:)
Even though you may not know what exactly all went down with CFS and CK's version, you can look at the code and compare them yourself. There are lots and lots and lots of people with an interest in making Linux a good OS, so if Linus made the wrong choice, he should be made aware.
Promote true freedom - support standards and interoperability.
FYI: The word you probably meant is "bear", as in to go a certain direction ("bear north") or to labor under a burden ("bear up under pressure"). So "bear with me" is a request to have the reader follow the direction your reasoning. On the other hand, "bare" means to "expose" or "make naked", so "bare with me" is an entirely different type of proposition.