Slashdot Mirror
Linus Does Not Scale
EmilEifrem writes: "Seems like everybody's getting more and more frustrated by Linus' (in-) ability to handle patches. Rob Landley just wrote an "RFC on Penguin Patch Management" wherein he proposes a "Penguin Patch Lieutenant" system that he believes would scale better. The full discussion can be found on the Linux kernel mailing list. Linus seems to dislike it, as usual, source code maintenance tools/organization are for wimps!, but a lot of others find it a good idea. Anyway, it's a very good read."
not integrated into the kernel? That is FRIGHTENING!!!
It wont be long before the MS camp picks up on this and starts to use it as a weapon - not that we havnt had to wait a year or two for bug fixes from them in the past. But Mr Torvalds isn't doing himself or us any favours on this one.
It all worked before because (1) Linus could wrap his head around the entire mass of kernel code, and (2) the kernel itself was much simpler, and (3) there were far fewer people submitting code to Linux.
This is all out the window now. The kernel is very large, and continues to grow, and Linus can no longer track the project just using his head.
In short, Linux is growing up. What it grows up to be I guess we'll see in a year. Either it starts using professional tools to manage this increasingly professional project, or the bloat (yes, the kernel is bloating) leads to unmanageable chaos, and the bloom goes off the Linux rose.
It simply won't help the problem... source management isn't the problem. Making sure useless crap doesn't make it into the tree is the problem. CVS doesn't have any sort of means to make sure that what goes in is quality code.
Slashdot angers me. Long story short, it's a bad solution for a problem already being fixed properly.
For those who don't care to read the discussion, Linus essentially feels that this is a bad idea because no general patch manager is going to scale better than he does or get burnt out less quickly than Alan Cox.
He then goes on to say that the solution to the problem of the scalability of one maintainer is to partition the different subsystems of the kernel to such an extent that there would be precious few patches that actually require a knowledge of the entire kernel source.
When ideals and/or ego enter the fray you have to be careful. This could alienate the very people that are needed at this time and cause Linux to loose steam. Yes, it's his toy, yes he can take his ball and go home. But hopefully this won't cause a major split similar to what happend to Unix in the 60's/70's.
Personally, I hope I'm wrong.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
There is absolutely nothing wrong with using CVS. Your question raises a larger issue, however. As with any large project, it is difficult if not impossible for one person to have knowledge of the entire scope. Linux has become bigger than Linus.
From its very beginning, Linus has kept a tight rein on Linux, and everybody in general let it go if for no other reason than he's a smart guy and by God, he wrote the thing! Don't get me wrong. That was a Good Thing (tm). Now, however, we are finding that it is becoming less and less practical for Linus to handle all of the patches.
Linux is becoming more and more like a cathedral and less and less like the bazaar. Whether he likes it or not, for Linux to continue to thrive (perhaps even to continue as we know it) Linus is going to have to decentralize the way patches are brought into Linux. I don't claim to have all the answers, but there must be a way to make a CVS-a-like system work and keep Linus' ability to make the final say if he wants to. Another alternative is for Linus to put more trust in his maintainers and let them accept patches for their respective subsystems.
The problem here is the star network topology that we have with respect to patches. If Linus is not willing to release his hold on the center of that star, then Linux could have a MAJOR problem!
Ben
It doesn't matter how many maintainers there are: Linus can only work through a specific amount of code/patches per day/week/month. More maintainers result in more patches being dropped, and subsequently in more efforts to keep their patches up-to-date with the kernel.
This is really hurting development, and is frustrating a lot of people who are involved in the kernel with the risk of losing them.
Anyhow: Linus does what he likes, but it's exactly that kind of attitude that will hurt Linux in the longer run - both in technological as reputational (?) way.
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
Speed things up by forming a committee?? HAHAHAHA!
Have you ever worked at a real job before?
Conformity is the jailer of freedom and enemy of growth. -JFK
It may seems so, to people coming from the corporate culture that uses language as a bludgeon (e.g. "issue" as a euphemism for "problem")--but having worked in both worlds I'm convinced that "independence" and "freedom" are better words for it than "anarchy" and "chaos."
I think the folks at IBM and Oracle ought to seriously have a LONG talk with Linux Torvalds himself and convince him to create a true clearinghouse where every improvement is approved by a committee. That way, Linux improvements happen in an organized fashion, which makes things way easier for developers and IT managers.
Convince him how? Work him over? Impress him with their org charts? Offer him the chance to work on a bigger project? Imply darkly that it won't look good on his annual review? If a variety of very smat people haven't been able to convince him with their best rational arguments, what exactly are the corporations supposed to do? And more to the point, do we really want to "convince" people of things that way?
And do you honestly believe that committees speed things up, or make them more organized, or easier to deal with? (This isn't a rhetorical question; for all I know you could be in full agreement with me, but writing in heavy sarcasm mode here and I'm just not getting it.)
Having managed a number of large projects I certainly find Linus's arguments more convincing; I'd much rather get on a ship run by one good captain than a ship where a variety of gizmos and rubber stamp committees had been put in charge for the express purpose of making the ship go faster than the captain thought prudent.
-- MarkusQ
The problem is that Linus is getting lots of other patches as well. Not enough filtering is going on.
Allowing maintainers to check in their own patches will not happen as long as Linus is interested in Linux. The buck stops with him. Rightly or wrongly he prefers not to have to back out patches (roll back checkins, if you prefer). At least that is where we stand, now. I would love it if Linus would eventually trust a small group of people enough to allow a CVS system to be usefull.
I'm reading lots of comments on this list about "the patch penguin will scale only as well as Linus, or maybe worse", or "Linus is right". But I'm not so sure that I agree. Having read the original proposal and the entire thread (as of last night around 1am EDT) I think that the proposal is a way of getting Linus to do what he does best (architecture) and relieving him from the mundane duties like trying to make sure that a submitted patch actually applies properly. Once the patch penguine does that kind of stuff, then Linus can review it.
Of course the advantage of this is that the mundane stuff can be done in parallel to the architectural stuff.. thus allowing more than one thing to get done at a time. Where as Linus himself can only do this serially.
The reason that this is important is that there is a very large backlog of patches that are simply not getting applied. Allegedly the whole VM issue could have been resolved if Linus had enough time to look at and apply RvR's patches. What this means is that Linux, with the inability of kernel maintainers to effectively submit patches, is turning into Minix. Remember that Linux was born out of Linus & others reaction to the fact that patches submitted to Minix couldn't get applied and redistributed (for copyright reasons). Which meant that in order to get a reasonable system, you first had to download the unpatched code, then go through a little patch-o-rama. It was ineffective. The current patch log reminds some of that situation, and has some fearing a potential kernel fork.
Of course, the biggest question that I have for the proposal is that Rob wants to just "formalize" a position that already exists and is being done, to some extent, already. If the position is already in place and already operating, then why is there still a patch backlog? How does formalizing it actually improve the situation?
Key to financial independence: Spend less than you earn. Save and invest the difference. Do it for a long time.
Disclaimer: I'm a FreeBSD fan and generally consider Linux et al to be one of the most choatic and sloppy large-scale development projects ever created. Just being a Linux user requires dealing with an absolutely insane about of pointless choas; I fear to imagine what trying to be a maintainer must be like...
That said...
"Just use CVS" isn't anything remotely close to an answer. SCM, release control, whatever you'd like to call it is 99% about the PROCESS, not the tools. CVS is a tool. It's a great tool and one I highly recommend, but at best it's 1% of a solution. It seems quite obvious that Linus's current process is incredibly flawed. It would seem a drastically different process needs to be devised. Only once you have figured out what your process is going to be can you even start shopping for tools (CVS, whatever) to help you create it (and even then...the tools are only 1%, if you even have tools. The PROCESS is what's important).
Of course "CVS forks" of Linux failed. Anyone with even the slightest understanding of CM could have easily predicted such an end. You've got a handful of people, not even the top people, trying to constantly fold in the efforts of thousands of contributers all using a completely different process (ad-hoc patches emailed to devnull@linus.org). What did you expect?
You have two choices. Devise a workable process which Linus will agree to (with or without CVS, whatever). Or, you can devise a workable process that everyone else is happy with...and simply throw Linus off his paper throne. The first option will likely never, ever happen...at least if every single word Linus has ever uttered on the subject isn't true. The second option might actually welcome the dawn of a non-choatic Linux, one which wouldn't be so easily cast aside as a cheap toy made by a small man with a big ego.
Linux/Linus Flame (-1k Karma)
My
If you read Linus' reply to this you'll see he argues that the 'patch penguin' merely replaces him with someone else, and does not actually help distribute the work. He also comments that it may be better to submit patches to subsystem maintainers.
This being the case, why not just automate that? Subsystem maintainers are effectively maintainers for a known group of source files. Its not hard to figure out (automatically) from a patch which files are affected - and from that, which maintainer(s) the patch is relevant to. Given how Linus describes his 'trust network' it looks like he wants a hierarchy of subsystem managers who can merge patches for him to accept upstream. I can see that scaling a lot better than the proposal in the article.
For this to work of course, you'd have to hope the patches had reasonably local scope - if most patches affect >1 subsystem this isnt worth doing at all.
- Baz
I'm sure Linus is healthy and a good driver, but as misfortune befell a former colleague at another job (her car was parked at a light on the off-ramp which was below the highway, a driver from the other direction suffered a cardiac arrest, crossed the median and opposite lanes and went airborne, landing on her car as she waited for the light to change. Gone, just like that,) unfortunate things happen. It would be very tragic for his wife and child to lose a father. It would be a disaster for Linux, as the unifying person would be gone and in the aftermath someone would have to take control. I imagine Linus has already considered this, but his tight grip on the kernel is a bit worrying. He should delegate more.
A feeling of having made the same mistake before: Deja Foobar
"...source code maintenance tools/organization are for wimps!"
I almost don't believe what I'm hearing. How do you manage development branches? How do you roll back your code to undo an enhancement you thought would work? How do know which bug fixes are tied to a particular instance of code? Wimp? More like "anyone who ISN'T using source code maintenance is a fool."
> and is frustrating a lot of people who are
> involved in the kernel with the risk of losing
> them.
The thing I don't get is why these people are goofing around with the kernel to begin with. Isn't one of the biggest virtues of kernel modules the ability to plug 'em in to a running system. New version? Build a new one, unuse & unload the current, and load the new one.
Why can't people who write drivers distribute them under their OWN energy? Why must the kernel be monolithic? Why not trim down the kernel to little more than a core, let linus distribute/patch THAT.
It's worked for windows for a decade now; maybe M$'s method isn't ideal either but being able to go to a vendor website, getting a driver, and dropping it into the system seems a hell of a lot more convenient than waiting for a lone man to include something in a source tree.
Seems to me the kernel is pushing into the future in all the wrong areas, when some of it's features are still living in the dark ages.
Most systems use a mere fraction of the drivers included in the kernel. This wastes bandwidth, configuration time, possibly memory, definetly disk space.
This is a basic principle of leadership/management that the Army calls "span of control".
It has been proven (through all kinds of research) that any one person has difficulty controlling much more than 10 individuals, with 8 being a practical maximum.
This is why military units are organized in strict tree structures - at no point does any one leader exceed his span of control.
So an 8-man infantry section is controlled by a Section Commander (which can be further broken down to 2 4-man fire teams, but rarely). 4 sections report to a Platoon Commander. 4 Platoons report to a Company Commander. 4 Companies report to a Battalion Commander, and so on through Regiments, Brigades, Divsions, and so forth.
In this manner, the several thousand men in a large formation like a Division can be commanded by one man.
Further aiding the General in charge of the Division is that each sub-unit is granted a certain amount of autonomy within certain boundries. As you go up the chain, orders become more general (2 Battalion is to attack objective Oscar) and as you go down the chain, orders become more specific (Bob, you're carrying the machine gun)
While Linux development does not require this level of structure (and indeed, would probably suffer if this many authority-resistant cats were forced into such a regimented structure) the basic principle of "span of control" is still applicable.
The obvious correct solution is to modularize the kernel into subsections with clearly defined areas of responsibility, with a "mini-Linus" (who Linus trusts) granted control over each module.
Not suprisingly, this is what Linus has suggested.
The trick (and the catch) is two-fold. Firstly, you have to find people who are both responsible enough to be able to act as a "mini-Linus" without dropping the ball (which means Linus has to trust them too) and secondly, the kernel has to be modular enough so that changes in Module A do not step on or otherwise negatively impact code in Module B.
That's a hard row to hoe.
What I find a little distressing is that there appears to be a whiff of revolution in the air; of people talking about overthrowing Linus (and Alan Cox seems to be among them) This will not solve the problem. It is not that **Linus** doesn't scale, but rather that any single maintainer doesn't scale. Any revolution will just be "same shit, different name"
CVS doesn't solve the problem either, as anyone with commit privilages can jsut dump anything they want in there, and it becomes canon. Linus' function as a shit filter is very very important. every patch *must* be audited by someone with very high standards before it goes in.
I hope the crew rallies around Linus - as usual, he's right.
DG
Want to learn about race cars? Read my Book
Of course you need to test patches and have others review them before you commit, but having things in CVS makes it very much easier to manage code.
And we could get a gigolo so the wife stops bugging him to perform...
I got it! Then, we can force him to take ever-increasing amounts of speed, so he doesn't have to sleep!
Then he can dedicate his life to linux 24/7!!! That would be great!
</sarcasm>
Make sure he actually wants to dedicate that much of his life to writing one program.
I'm a leaf on the wind. Watch how I soar.
...although I concede that trust comes from competence. But what does this have to do with anything?
If you're going to overhaul (or in this case, overthrow) an established system, you need to do it with much fear & trembling. Look, I see this all the time as a manager. A new project is given to a manager. That manager hires someone competent and works with that person to build something from nothing. If the project takes off, you can end up hiring a lot of people to help support that project. It is possible for all those new hires to be more competent than the original, lead developer. But that original, lead developer now has management's trust. That lead developer, who got the damn thing off the ground in the first place, has an opinion that -- like it or not -- counts for more than the other developers. You can call that unfair, but certain people are just better rounded than others -- they have social skills, they can explain things well, they understand the market pressures, or whatever. Not to mention that it's their baby and you don't fuck around with stuff like that. As a manager, if I stomp all over the lead developer's project, that lead developer may not want to lead the next project! Sure, I can call on one of the other people brought in to help, but those other people may not be leaders.
I'm not saying that such an effort is doomed. But I am suggesting that, if you want to propose that Linus relinquish some authority, you damn well better have a value proposition for him. Shouting "this needs to be a democracy" is what you want, not what Linus wants. He's looking at that as a management nightmare, a removal of a power structure that is in place because, whatever it's faults, it was the structure that worked. If it doesn't work well anymore, you either have to convince Linus, or you have to convince everyone else to stop trusting the person in authority. That is a massive undertaking.
My Greasemonkey scripts for Digg &
The notion of trust seems central to this debate -- who can be trusted to apply patches that will make the kernel better? And it reminds me of structures for establishing trust in the world of cryptography -- who do I trust to tell me that this public key really belongs to the person to whom it's supposed to belong?
The ideas about kernel patches being discussed remind me of the distinction between a "web of trust" and "certification authorities" in the world of cryptography. Certification authorities are the conventional means, which are used by the most popular web browsers. Some authority, like Verisign or Thawte, is universally accepted as trustworty, and if they sign a key, then it can be trusted. The web of trust was advocated by Phil Zimmermann and built into PGP, and AFAIK, PGP is the only context in which it's ever been applied (or even proposed). It's an attempt to reflect our everyday notion of trust -- I decide who I trust, on whatever basis I choose, and if someone I trust has signed a key, then I trust it. I can also decide if I trust people I don't know, if they are trusted by people I trust. There's no need for a central authority in a web of trust (although I might choose to include CAs in my web and trust them).
Rob Landley is suggesting that we entrust the patch penguin with the job of filtering patches towards Linus, but Linus sees the PP as a kind of central authority, and prefers to have a web of individuals around hom who he can trust.
Like Phil Zimmermann, Linus is arguing that his model is much more similar to the way that people decide to trust each other in real life. In the world of crypto, certification authorities have always been regarded with suspicion for reasons just like this -- why should I trust Verisign more than my friends? In the kernel world, Linus is saying, isn't it better and more intuitive to trust a group of people I know well, who can distribute the work amongst each other?
The analogy breaks here a bit. CAs in the crypto world are suspicious because we don't if they're corrupt -- maybe they're full of Enron managers. In the kernel world, we need to trust maintainers with their skill and above all, ability to handle the workload. Corruption is not the problem (I hope not, anyway.)
But, to take the analogy further, one should also note that PGP web of trust never really worked well as a practical matter. Most people using PGP had an island of people around them whom they trusted, with few connections outside of their immediate clique. I don't recall *ever* receiving a key from a stranger that was evaluated as trustworthy because of its position in the web topology.
So I'd say that Linus has a point in saying that the web model fits our common-sense intuitions about trust. But if Rob Landley says that it's not functioning well, the experience of PGP gives some support to his argument.
Always keep a sapphire in your mind