Slashdot Mirror
Linus Does Not Scale
EmilEifrem writes: "Seems like everybody's getting more and more frustrated by Linus' (in-) ability to handle patches. Rob Landley just wrote an "RFC on Penguin Patch Management" wherein he proposes a "Penguin Patch Lieutenant" system that he believes would scale better. The full discussion can be found on the Linux kernel mailing list. Linus seems to dislike it, as usual, source code maintenance tools/organization are for wimps!, but a lot of others find it a good idea. Anyway, it's a very good read."
You know... If we had a Beowulf Cluster of Linuses...
*rimshot*
How about a RAIKA? A redundant array of Kernel Admins? Maybe keep them hot-swapable? That way, if one goes out to the pub, the other one can keep things going....
OK, I'll just shut up now...
Objects in the blog are closer then they ap
not integrated into the kernel? That is FRIGHTENING!!!
It wont be long before the MS camp picks up on this and starts to use it as a weapon - not that we havnt had to wait a year or two for bug fixes from them in the past. But Mr Torvalds isn't doing himself or us any favours on this one.
David U.
# Hack the planet, it's important.
Before everyone flames Linus for his dislike of CVS and other management tools ....
From Kernel Trap:
His general point is summarized with this statement, "One 'patch penguin' scales no better than I do. In fact, I will claim that most of them scale a whole lot worse". He goes on to explain this statement quite thoroughly, adding: " In short: don't try to come up with a 'patch penguin'. Instead try to help existing maintainers, or maybe help grow new ones. THAT is the way to scalability".
Say what you want about Linus' attitude - Having a kernel lieutenant probably wouldn't help either. Either way, there's only so much one person can do.
It all worked before because (1) Linus could wrap his head around the entire mass of kernel code, and (2) the kernel itself was much simpler, and (3) there were far fewer people submitting code to Linux.
This is all out the window now. The kernel is very large, and continues to grow, and Linus can no longer track the project just using his head.
In short, Linux is growing up. What it grows up to be I guess we'll see in a year. Either it starts using professional tools to manage this increasingly professional project, or the bloat (yes, the kernel is bloating) leads to unmanageable chaos, and the bloom goes off the Linux rose.
It simply won't help the problem... source management isn't the problem. Making sure useless crap doesn't make it into the tree is the problem. CVS doesn't have any sort of means to make sure that what goes in is quality code.
Note that things like CVS do not help the fundamental problem at all. They allow automatic acceptance of patches, and positively _encourage_ people to "dump" their patches on other people, and not act as real maintainers.
We've seen this several times in Linux - David (Miller), for example, used to maintain his CVS tree, and he ended up being rather frustrated about having to then maintain it all and clean up the bad parts because I didn't want to apply them (and he didn't really want me to) and he couldn't make people clean up themselves because "once it was in, it was in".
I know that source control advocates say that using source control makes it easy to revert bad stuff, but that's simply not TRUE. It's not easy to revert bad stuff. The only way to handle bad stuff is to make people responsible for their own sh*t, and have them maintain it themselves.
"The good die first." "Most of us are morally ambiguous, which explains our random dying patterns." --- MST3K
Slashdot angers me. Long story short, it's a bad solution for a problem already being fixed properly.
For those who don't care to read the discussion, Linus essentially feels that this is a bad idea because no general patch manager is going to scale better than he does or get burnt out less quickly than Alan Cox.
He then goes on to say that the solution to the problem of the scalability of one maintainer is to partition the different subsystems of the kernel to such an extent that there would be precious few patches that actually require a knowledge of the entire kernel source.
For example, there was a bug in the ne2000 driver that Alan Cox points out here. According to Mr. Cox, "this is one tiny example of maybe thousands of other similar flaws lurking. There is no obvious automated way to find them either."
Lots of corporate/management types have the negative impression of Linux as an OS that has no professional control over kernel development. It's seen as a souped-up hotrod modified in the garage that runs like a dream but could fall apart at any minute. Hell, even a lot of BSD people see it this way. If (*if*) a goal of the Linux community is to gain wider acceptance or be taken more seriously, one way to do that might be to give more than one person final say over the kernel.
Yeah, it's Linus' baby -- but once IBM is advertising your OS during the superbowl, it might be ok to expand the development structure a little bit.
The basic premise he makes is one that many developers seem to miss. To quote Linus:
From that, he goes on to point out that people tend to have a small group of people they work well with and trust; perhaps 5 - 10 people. So, in essence, if Linus appoints 5 - 10 maintainers of major subsystems (which he has), and each of them has 5 - 10 people the trust to maintain specific aspects of their subsystem, then there is no problem.We're not there yet. But that is the only direction I can see that will work, long term. It just takes time for the appropriate people to move into their spots; trust takes time to build, and the structure has taken time to modularize enough to allow this to work.
And, though I would prefer to see it in use just to make the maintainers' jobs easier, CVS will do nothing to solve the problem. It is a tool, not a process. We need a process. Actually we have a process, it just isn't fully implemented yet.
No, strike that. We have two processes. The first, used by the bulk of the kernel hackers, is not fully implemented yet. The second, used by a minority of the kernel hackers and a large part of the pretenders, is simple: whine alot, and, when that doesn't work, whine some more.
Linus says it much better, and in his own unique ... er ...
Idium, sir?
Yes, in his own, unique idium.
When ideals and/or ego enter the fray you have to be careful. This could alienate the very people that are needed at this time and cause Linux to loose steam. Yes, it's his toy, yes he can take his ball and go home. But hopefully this won't cause a major split similar to what happend to Unix in the 60's/70's.
Personally, I hope I'm wrong.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
CVS... does not check for quality or if it breaks anything.
If you've been following Linux kernel development the last year or so, you could say the same about Linus.
There is absolutely nothing wrong with using CVS. Your question raises a larger issue, however. As with any large project, it is difficult if not impossible for one person to have knowledge of the entire scope. Linux has become bigger than Linus.
From its very beginning, Linus has kept a tight rein on Linux, and everybody in general let it go if for no other reason than he's a smart guy and by God, he wrote the thing! Don't get me wrong. That was a Good Thing (tm). Now, however, we are finding that it is becoming less and less practical for Linus to handle all of the patches.
Linux is becoming more and more like a cathedral and less and less like the bazaar. Whether he likes it or not, for Linux to continue to thrive (perhaps even to continue as we know it) Linus is going to have to decentralize the way patches are brought into Linux. I don't claim to have all the answers, but there must be a way to make a CVS-a-like system work and keep Linus' ability to make the final say if he wants to. Another alternative is for Linus to put more trust in his maintainers and let them accept patches for their respective subsystems.
The problem here is the star network topology that we have with respect to patches. If Linus is not willing to release his hold on the center of that star, then Linux could have a MAJOR problem!
Ben
Speed things up by forming a committee?? HAHAHAHA!
Have you ever worked at a real job before?
Conformity is the jailer of freedom and enemy of growth. -JFK
It may seems so, to people coming from the corporate culture that uses language as a bludgeon (e.g. "issue" as a euphemism for "problem")--but having worked in both worlds I'm convinced that "independence" and "freedom" are better words for it than "anarchy" and "chaos."
I think the folks at IBM and Oracle ought to seriously have a LONG talk with Linux Torvalds himself and convince him to create a true clearinghouse where every improvement is approved by a committee. That way, Linux improvements happen in an organized fashion, which makes things way easier for developers and IT managers.
Convince him how? Work him over? Impress him with their org charts? Offer him the chance to work on a bigger project? Imply darkly that it won't look good on his annual review? If a variety of very smat people haven't been able to convince him with their best rational arguments, what exactly are the corporations supposed to do? And more to the point, do we really want to "convince" people of things that way?
And do you honestly believe that committees speed things up, or make them more organized, or easier to deal with? (This isn't a rhetorical question; for all I know you could be in full agreement with me, but writing in heavy sarcasm mode here and I'm just not getting it.)
Having managed a number of large projects I certainly find Linus's arguments more convincing; I'd much rather get on a ship run by one good captain than a ship where a variety of gizmos and rubber stamp committees had been put in charge for the express purpose of making the ship go faster than the captain thought prudent.
-- MarkusQ
I'm reading lots of comments on this list about "the patch penguin will scale only as well as Linus, or maybe worse", or "Linus is right". But I'm not so sure that I agree. Having read the original proposal and the entire thread (as of last night around 1am EDT) I think that the proposal is a way of getting Linus to do what he does best (architecture) and relieving him from the mundane duties like trying to make sure that a submitted patch actually applies properly. Once the patch penguine does that kind of stuff, then Linus can review it.
Of course the advantage of this is that the mundane stuff can be done in parallel to the architectural stuff.. thus allowing more than one thing to get done at a time. Where as Linus himself can only do this serially.
The reason that this is important is that there is a very large backlog of patches that are simply not getting applied. Allegedly the whole VM issue could have been resolved if Linus had enough time to look at and apply RvR's patches. What this means is that Linux, with the inability of kernel maintainers to effectively submit patches, is turning into Minix. Remember that Linux was born out of Linus & others reaction to the fact that patches submitted to Minix couldn't get applied and redistributed (for copyright reasons). Which meant that in order to get a reasonable system, you first had to download the unpatched code, then go through a little patch-o-rama. It was ineffective. The current patch log reminds some of that situation, and has some fearing a potential kernel fork.
Of course, the biggest question that I have for the proposal is that Rob wants to just "formalize" a position that already exists and is being done, to some extent, already. If the position is already in place and already operating, then why is there still a patch backlog? How does formalizing it actually improve the situation?
Key to financial independence: Spend less than you earn. Save and invest the difference. Do it for a long time.
Disclaimer: I'm a FreeBSD fan and generally consider Linux et al to be one of the most choatic and sloppy large-scale development projects ever created. Just being a Linux user requires dealing with an absolutely insane about of pointless choas; I fear to imagine what trying to be a maintainer must be like...
That said...
"Just use CVS" isn't anything remotely close to an answer. SCM, release control, whatever you'd like to call it is 99% about the PROCESS, not the tools. CVS is a tool. It's a great tool and one I highly recommend, but at best it's 1% of a solution. It seems quite obvious that Linus's current process is incredibly flawed. It would seem a drastically different process needs to be devised. Only once you have figured out what your process is going to be can you even start shopping for tools (CVS, whatever) to help you create it (and even then...the tools are only 1%, if you even have tools. The PROCESS is what's important).
Of course "CVS forks" of Linux failed. Anyone with even the slightest understanding of CM could have easily predicted such an end. You've got a handful of people, not even the top people, trying to constantly fold in the efforts of thousands of contributers all using a completely different process (ad-hoc patches emailed to devnull@linus.org). What did you expect?
You have two choices. Devise a workable process which Linus will agree to (with or without CVS, whatever). Or, you can devise a workable process that everyone else is happy with...and simply throw Linus off his paper throne. The first option will likely never, ever happen...at least if every single word Linus has ever uttered on the subject isn't true. The second option might actually welcome the dawn of a non-choatic Linux, one which wouldn't be so easily cast aside as a cheap toy made by a small man with a big ego.
Linux/Linus Flame (-1k Karma)
My
If you read Linus' reply to this you'll see he argues that the 'patch penguin' merely replaces him with someone else, and does not actually help distribute the work. He also comments that it may be better to submit patches to subsystem maintainers.
This being the case, why not just automate that? Subsystem maintainers are effectively maintainers for a known group of source files. Its not hard to figure out (automatically) from a patch which files are affected - and from that, which maintainer(s) the patch is relevant to. Given how Linus describes his 'trust network' it looks like he wants a hierarchy of subsystem managers who can merge patches for him to accept upstream. I can see that scaling a lot better than the proposal in the article.
For this to work of course, you'd have to hope the patches had reasonably local scope - if most patches affect >1 subsystem this isnt worth doing at all.
- Baz
I'm sure Linus is healthy and a good driver, but as misfortune befell a former colleague at another job (her car was parked at a light on the off-ramp which was below the highway, a driver from the other direction suffered a cardiac arrest, crossed the median and opposite lanes and went airborne, landing on her car as she waited for the light to change. Gone, just like that,) unfortunate things happen. It would be very tragic for his wife and child to lose a father. It would be a disaster for Linux, as the unifying person would be gone and in the aftermath someone would have to take control. I imagine Linus has already considered this, but his tight grip on the kernel is a bit worrying. He should delegate more.
A feeling of having made the same mistake before: Deja Foobar
"...source code maintenance tools/organization are for wimps!"
I almost don't believe what I'm hearing. How do you manage development branches? How do you roll back your code to undo an enhancement you thought would work? How do know which bug fixes are tied to a particular instance of code? Wimp? More like "anyone who ISN'T using source code maintenance is a fool."
A lot of people have said CVS could not work, agreeing with Linus. Yet FreeBSD uses a CVS tree with multiple committers, and has had no serious problems that I am aware of. Infact, I would say they have had less problems than linux has had in the last few years.
cjk
Toothpaste likes you
You will forget this sig before you next see it
This is a basic principle of leadership/management that the Army calls "span of control".
It has been proven (through all kinds of research) that any one person has difficulty controlling much more than 10 individuals, with 8 being a practical maximum.
This is why military units are organized in strict tree structures - at no point does any one leader exceed his span of control.
So an 8-man infantry section is controlled by a Section Commander (which can be further broken down to 2 4-man fire teams, but rarely). 4 sections report to a Platoon Commander. 4 Platoons report to a Company Commander. 4 Companies report to a Battalion Commander, and so on through Regiments, Brigades, Divsions, and so forth.
In this manner, the several thousand men in a large formation like a Division can be commanded by one man.
Further aiding the General in charge of the Division is that each sub-unit is granted a certain amount of autonomy within certain boundries. As you go up the chain, orders become more general (2 Battalion is to attack objective Oscar) and as you go down the chain, orders become more specific (Bob, you're carrying the machine gun)
While Linux development does not require this level of structure (and indeed, would probably suffer if this many authority-resistant cats were forced into such a regimented structure) the basic principle of "span of control" is still applicable.
The obvious correct solution is to modularize the kernel into subsections with clearly defined areas of responsibility, with a "mini-Linus" (who Linus trusts) granted control over each module.
Not suprisingly, this is what Linus has suggested.
The trick (and the catch) is two-fold. Firstly, you have to find people who are both responsible enough to be able to act as a "mini-Linus" without dropping the ball (which means Linus has to trust them too) and secondly, the kernel has to be modular enough so that changes in Module A do not step on or otherwise negatively impact code in Module B.
That's a hard row to hoe.
What I find a little distressing is that there appears to be a whiff of revolution in the air; of people talking about overthrowing Linus (and Alan Cox seems to be among them) This will not solve the problem. It is not that **Linus** doesn't scale, but rather that any single maintainer doesn't scale. Any revolution will just be "same shit, different name"
CVS doesn't solve the problem either, as anyone with commit privilages can jsut dump anything they want in there, and it becomes canon. Linus' function as a shit filter is very very important. every patch *must* be audited by someone with very high standards before it goes in.
I hope the crew rallies around Linus - as usual, he's right.
DG
Want to learn about race cars? Read my Book
Of course you need to test patches and have others review them before you commit, but having things in CVS makes it very much easier to manage code.
How about we set up a fund so that Linus can work on Linux full-time, instead of needing a day job? That's not a permanent fix, but it would help for a while.
As "president" of FreeBSD. Then he had the courage to step down and become a "normal" member of the "core-group", a collective in which every member has the main responsability for a part of the FreeBSD code. Apart from the core group there are other committers but the core group decides who these are and can revoke commit-rights in cases of abuse.
This is a nice distributed system that continues to work very well when the load gets higher; also noone is indispensible, noone has to be afraid what would happen to FreeBSD if a certain person would somehow drop out.
Of course Linus has every right in the world to remain the status quo, even if it damages Linux. After all Linux is his baby and he can do with it what he likes. Whether it is a good idea to rely on an OS with this kind of a leadership structure is another matter however. But noone can force Linus to change, since he doesn't force anyone to use Linux (take it or leave it).
Imagine the terror of a 50 foot tall Finnish programmer wandering the streets.
Now let's just hope RMS doesn't scale either.
Microsoft must be laughing
Well, I can't claim any insider knowledge, but I bet that all kind of crap goes on within the MS development teams that we never hear about.
When you've got a lot of bright people working hard on a very big, complex thing, something would be very wrong if you didn't get these kind of problems arising occasionally.
Personally, I think Linus is right, and all those people who are bitching should sit down and think for a moment, and perhaps think of ways they can help to make Linus's very hard task a little easier, rather than just complaining.
Here's the Linux version:
-- @rjamestaylor on Ello
Did ANYBODY on /. actually read the ML Thread on this problem the kernelpeople have?
/.ers should read the actual thread before they post every instant and absurd mindfart that comes to them. Be it about jelousy amongst kernelhackers or Linus supposedly bossing around or whatever other sorts of utter bullshit.
Rob Landley and Linux Torwalds aren't bashing their heads in on one another nor is ANYBODY of the Kernel Team about to 'dethrone' Linus. Whatever that may be.
In fact Landley suggest a "Patch Penguin' to actually EMPOWER Linus in his actuall job as an arcitect of Linux.
Linus in turn says that officially shifting patchin jobs to somebody else (the said Patch Penguin) won't save the problem of, for instance, people being to lazy to clean up their code dependencies.
Linus wants to see a sort of 'web of maintainers' where everyone knows and works with a overseeable amount of others (just like he does) rather than a big patcheritis boiling around a main single/pair/group of developers.
It may, IMHO of a absolute non-kernel savy guy, kinda boil down to the monolitic/modular kernel discussion that comes up every know and then.
Then again, on the other hand I gather the impression that Rob Langley and Linus Torwals aren't that far apart in seeing the issue that needs to be addresses rather than seeing different ways of aproaching a solution to it.
Anyhow,
We suffer more in our imagination than in reality. - Seneca
...although I concede that trust comes from competence. But what does this have to do with anything?
If you're going to overhaul (or in this case, overthrow) an established system, you need to do it with much fear & trembling. Look, I see this all the time as a manager. A new project is given to a manager. That manager hires someone competent and works with that person to build something from nothing. If the project takes off, you can end up hiring a lot of people to help support that project. It is possible for all those new hires to be more competent than the original, lead developer. But that original, lead developer now has management's trust. That lead developer, who got the damn thing off the ground in the first place, has an opinion that -- like it or not -- counts for more than the other developers. You can call that unfair, but certain people are just better rounded than others -- they have social skills, they can explain things well, they understand the market pressures, or whatever. Not to mention that it's their baby and you don't fuck around with stuff like that. As a manager, if I stomp all over the lead developer's project, that lead developer may not want to lead the next project! Sure, I can call on one of the other people brought in to help, but those other people may not be leaders.
I'm not saying that such an effort is doomed. But I am suggesting that, if you want to propose that Linus relinquish some authority, you damn well better have a value proposition for him. Shouting "this needs to be a democracy" is what you want, not what Linus wants. He's looking at that as a management nightmare, a removal of a power structure that is in place because, whatever it's faults, it was the structure that worked. If it doesn't work well anymore, you either have to convince Linus, or you have to convince everyone else to stop trusting the person in authority. That is a massive undertaking.
My Greasemonkey scripts for Digg &
The notion of trust seems central to this debate -- who can be trusted to apply patches that will make the kernel better? And it reminds me of structures for establishing trust in the world of cryptography -- who do I trust to tell me that this public key really belongs to the person to whom it's supposed to belong?
The ideas about kernel patches being discussed remind me of the distinction between a "web of trust" and "certification authorities" in the world of cryptography. Certification authorities are the conventional means, which are used by the most popular web browsers. Some authority, like Verisign or Thawte, is universally accepted as trustworty, and if they sign a key, then it can be trusted. The web of trust was advocated by Phil Zimmermann and built into PGP, and AFAIK, PGP is the only context in which it's ever been applied (or even proposed). It's an attempt to reflect our everyday notion of trust -- I decide who I trust, on whatever basis I choose, and if someone I trust has signed a key, then I trust it. I can also decide if I trust people I don't know, if they are trusted by people I trust. There's no need for a central authority in a web of trust (although I might choose to include CAs in my web and trust them).
Rob Landley is suggesting that we entrust the patch penguin with the job of filtering patches towards Linus, but Linus sees the PP as a kind of central authority, and prefers to have a web of individuals around hom who he can trust.
Like Phil Zimmermann, Linus is arguing that his model is much more similar to the way that people decide to trust each other in real life. In the world of crypto, certification authorities have always been regarded with suspicion for reasons just like this -- why should I trust Verisign more than my friends? In the kernel world, Linus is saying, isn't it better and more intuitive to trust a group of people I know well, who can distribute the work amongst each other?
The analogy breaks here a bit. CAs in the crypto world are suspicious because we don't if they're corrupt -- maybe they're full of Enron managers. In the kernel world, we need to trust maintainers with their skill and above all, ability to handle the workload. Corruption is not the problem (I hope not, anyway.)
But, to take the analogy further, one should also note that PGP web of trust never really worked well as a practical matter. Most people using PGP had an island of people around them whom they trusted, with few connections outside of their immediate clique. I don't recall *ever* receiving a key from a stranger that was evaluated as trustworthy because of its position in the web topology.
So I'd say that Linus has a point in saying that the web model fits our common-sense intuitions about trust. But if Rob Landley says that it's not functioning well, the experience of PGP gives some support to his argument.
Always keep a sapphire in your mind