Slashdot Mirror
Linus Does Not Scale
EmilEifrem writes: "Seems like everybody's getting more and more frustrated by Linus' (in-) ability to handle patches. Rob Landley just wrote an "RFC on Penguin Patch Management" wherein he proposes a "Penguin Patch Lieutenant" system that he believes would scale better. The full discussion can be found on the Linux kernel mailing list. Linus seems to dislike it, as usual, source code maintenance tools/organization are for wimps!, but a lot of others find it a good idea. Anyway, it's a very good read."
Why is there not a CVS for the linux kernel?
I can see where Linus is coming from; Linux and Linus Torvalds are inextricably linked in the public perception, so any huge blunders that occur with the Linux kernel will sully his reputation. If it's his reputation on the line, he probably wants to keep control of potential disasters close to him.
Better to get blamed for your own mistakes than for someone elses.
I thoroughly agree. Like many people I hold a fair bit of respect for Linus. However, Linux is starting to get to the stage where one man cannot possibly keep up with what's going on.
With this proposed system he could still maintain control, yet pass on the mundane task of merging. Seems like a great idea.
-- main(s){printf(s="main(s){printf(s=%c%s%c,34,s,34
not integrated into the kernel? That is FRIGHTENING!!!
It wont be long before the MS camp picks up on this and starts to use it as a weapon - not that we havnt had to wait a year or two for bug fixes from them in the past. But Mr Torvalds isn't doing himself or us any favours on this one.
How long before the Linux kernel is forked by someone that actually does version management with CVS? Linus' Linux versions could be patched back into the CVS/Linux code, while developers could focus on the actual problems at hand instead of messing with patches, submissions and the general frustration of getting you patchest dropped by Linus.
It all worked before because (1) Linus could wrap his head around the entire mass of kernel code, and (2) the kernel itself was much simpler, and (3) there were far fewer people submitting code to Linux.
This is all out the window now. The kernel is very large, and continues to grow, and Linus can no longer track the project just using his head.
In short, Linux is growing up. What it grows up to be I guess we'll see in a year. Either it starts using professional tools to manage this increasingly professional project, or the bloat (yes, the kernel is bloating) leads to unmanageable chaos, and the bloom goes off the Linux rose.
It simply won't help the problem... source management isn't the problem. Making sure useless crap doesn't make it into the tree is the problem. CVS doesn't have any sort of means to make sure that what goes in is quality code.
That's no attitude to take. What we need is cooperation and competition, each in its place, where people discuss the issues and the best ideas/implementations win. If each person who contributes to the linux kernel would have started their own kernel instead of contributing, linux would be much worse off today.
If you something to contribute, then attempt, through polite reason and good code, to do so.
*during the war...*
my 0.02 Eu:
I used to work in an environment (nuclear power) where we had a robust and comprehensive engineering change system, with umpteen layers of peer review, administrative interlocks and independent assessments for any changes to the System. The root of it all was the Quality Assurance Policy, under which were the Management Control Procedures and then the people doing the work. It may be overkill for and OS kernel, but maybe these guys need to think of using such a system (but cut down) on their kernel?
I'm out of my tree just now but please feel free to leave a banana.
Slashdot angers me. Long story short, it's a bad solution for a problem already being fixed properly.
For those who don't care to read the discussion, Linus essentially feels that this is a bad idea because no general patch manager is going to scale better than he does or get burnt out less quickly than Alan Cox.
He then goes on to say that the solution to the problem of the scalability of one maintainer is to partition the different subsystems of the kernel to such an extent that there would be precious few patches that actually require a knowledge of the entire kernel source.
this same subject pops up. It's the same on Debian lists: every now and then people say how frustrated they are with the slow pace of releases and propose some new scheme which usually doesn't even differ from the old one.
Software problems are solved by coding. The core kernel people, like core Debian people, have thought about these things before, and are doing their best. All these blatherings do is waste everyone's time.
I thought Linux belonged to everyone. Linux wouldn't be anything at all if Linus had run the show from top to bottom.
Now, how to move forward and expand the circle without diluting the value that he brings. Something like CVS can help, but the base problem isn't technical and the solution won't be technical, either.
The real problem is factoring out that part of the Linus load that Linus does better than anybody else can do, and offload other parts to those who can do them better than a loaded-down Linus.
The hard part for everyone, Linus included, is to understand how much the "other" part feeds into Linus's ability to do the "Linus" part.
Sometimes, even mundane tasks help you to maintain the vital connection needed to do the great things well.
I'm just glad I'm not the one having to work all of this out!
When ideals and/or ego enter the fray you have to be careful. This could alienate the very people that are needed at this time and cause Linux to loose steam. Yes, it's his toy, yes he can take his ball and go home. But hopefully this won't cause a major split similar to what happend to Unix in the 60's/70's.
Personally, I hope I'm wrong.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
From the article (RTFA!):
Okay everybody, this is getting rediculous. Patches FROM MAINTAINERS are getting dropped on the floor on a regular basis. This is burning out
maintainers and is increasing the number of different kernel trees (not yet a major fork, but a lot of cracks and fragmentation are showing under the stress).
The maintainers should be kept happy. As linus states:
he fact is, we've had "patch penguins" pretty much forever, and they are called subsystem maintainers. They maintain their own subsystem, ie people like David Miller (networking), Kai Germaschewski (ISDN), Greg KH (USB), Ben Collins (firewire), Al Viro (VFS), Andrew Morton (ext3), Ingo Molnar (scheduler), Jeff Garzik (network drivers) etc etc.
Rik van Riel already suggested a mailing bot that says what version it is agianst (and how many times it is dropped....). But just mailing linux only helps you to be added to the spamfiler.
I gues in some time this will all stop if linus find a toy project he wants to focus on and some kernel distributer steps in with a team to take over. (Linus will say I dont care. real men....)
That is why the CVS tree must have a code review process before is accepted. CVS is only part of the solution, and when you are only using 1/10th of the solution, you should expect to get 9/10ths more confusion.
Having one person own the entire "tree" of code slows the development process to a crawl. Linux will never be able to bring functionality to the table as fast as Windows if everything has to go through one man. Do you think Bill Gates is sitting at his desk and reviewing Windows XP code? No. He has better things to do. Linus should realize this and work on implementing a development environment that will enable broader management.
I propose CVS is a part of this solution. I would also recommend code reviews by three independant parties before a branched segment of code becomes part of the main tree. Design reviews/requirements/UML should be designed before coding should ever take place. Yes, its a lot of overhead, but it allows for new people to fill your shoes when you grow weary of development. It also allows more people to have ownership of the entire project and enable faster development, with a broader level of quality control.
There is absolutely nothing wrong with using CVS. Your question raises a larger issue, however. As with any large project, it is difficult if not impossible for one person to have knowledge of the entire scope. Linux has become bigger than Linus.
From its very beginning, Linus has kept a tight rein on Linux, and everybody in general let it go if for no other reason than he's a smart guy and by God, he wrote the thing! Don't get me wrong. That was a Good Thing (tm). Now, however, we are finding that it is becoming less and less practical for Linus to handle all of the patches.
Linux is becoming more and more like a cathedral and less and less like the bazaar. Whether he likes it or not, for Linux to continue to thrive (perhaps even to continue as we know it) Linus is going to have to decentralize the way patches are brought into Linux. I don't claim to have all the answers, but there must be a way to make a CVS-a-like system work and keep Linus' ability to make the final say if he wants to. Another alternative is for Linus to put more trust in his maintainers and let them accept patches for their respective subsystems.
The problem here is the star network topology that we have with respect to patches. If Linus is not willing to release his hold on the center of that star, then Linux could have a MAJOR problem!
Ben
So a month later hes backlogged in unmerged branges that hes not had a chance to look at yet? How exactly does that help?
It doesn't matter how many maintainers there are: Linus can only work through a specific amount of code/patches per day/week/month. More maintainers result in more patches being dropped, and subsequently in more efforts to keep their patches up-to-date with the kernel.
This is really hurting development, and is frustrating a lot of people who are involved in the kernel with the risk of losing them.
Anyhow: Linus does what he likes, but it's exactly that kind of attitude that will hurt Linux in the longer run - both in technological as reputational (?) way.
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
Speed things up by forming a committee?? HAHAHAHA!
Have you ever worked at a real job before?
Conformity is the jailer of freedom and enemy of growth. -JFK
It may seems so, to people coming from the corporate culture that uses language as a bludgeon (e.g. "issue" as a euphemism for "problem")--but having worked in both worlds I'm convinced that "independence" and "freedom" are better words for it than "anarchy" and "chaos."
I think the folks at IBM and Oracle ought to seriously have a LONG talk with Linux Torvalds himself and convince him to create a true clearinghouse where every improvement is approved by a committee. That way, Linux improvements happen in an organized fashion, which makes things way easier for developers and IT managers.
Convince him how? Work him over? Impress him with their org charts? Offer him the chance to work on a bigger project? Imply darkly that it won't look good on his annual review? If a variety of very smat people haven't been able to convince him with their best rational arguments, what exactly are the corporations supposed to do? And more to the point, do we really want to "convince" people of things that way?
And do you honestly believe that committees speed things up, or make them more organized, or easier to deal with? (This isn't a rhetorical question; for all I know you could be in full agreement with me, but writing in heavy sarcasm mode here and I'm just not getting it.)
Having managed a number of large projects I certainly find Linus's arguments more convincing; I'd much rather get on a ship run by one good captain than a ship where a variety of gizmos and rubber stamp committees had been put in charge for the express purpose of making the ship go faster than the captain thought prudent.
-- MarkusQ
The "hierarchy" goes exactly one level deep. Hardly worth talking about.
The maintainers that "report" directly to Linus are the second level of that hierarchy. From there on, the "levels" begin to blur, as many of the "lower" levels of maintainers would probably eventually be in the circle of trust of more than one other maintainer.
In effect, this method becomes a web, not a tree. Well, it could be argued, I suppose, that the web looks pretty damn tree like in parts. Or that the tree has large portions that are quite web-like.
Whatever.
The main things, as s Linus originally pointed out here is to:
The problem is that Linus is getting lots of other patches as well. Not enough filtering is going on.
Allowing maintainers to check in their own patches will not happen as long as Linus is interested in Linux. The buck stops with him. Rightly or wrongly he prefers not to have to back out patches (roll back checkins, if you prefer). At least that is where we stand, now. I would love it if Linus would eventually trust a small group of people enough to allow a CVS system to be usefull.
I'm reading lots of comments on this list about "the patch penguin will scale only as well as Linus, or maybe worse", or "Linus is right". But I'm not so sure that I agree. Having read the original proposal and the entire thread (as of last night around 1am EDT) I think that the proposal is a way of getting Linus to do what he does best (architecture) and relieving him from the mundane duties like trying to make sure that a submitted patch actually applies properly. Once the patch penguine does that kind of stuff, then Linus can review it.
Of course the advantage of this is that the mundane stuff can be done in parallel to the architectural stuff.. thus allowing more than one thing to get done at a time. Where as Linus himself can only do this serially.
The reason that this is important is that there is a very large backlog of patches that are simply not getting applied. Allegedly the whole VM issue could have been resolved if Linus had enough time to look at and apply RvR's patches. What this means is that Linux, with the inability of kernel maintainers to effectively submit patches, is turning into Minix. Remember that Linux was born out of Linus & others reaction to the fact that patches submitted to Minix couldn't get applied and redistributed (for copyright reasons). Which meant that in order to get a reasonable system, you first had to download the unpatched code, then go through a little patch-o-rama. It was ineffective. The current patch log reminds some of that situation, and has some fearing a potential kernel fork.
Of course, the biggest question that I have for the proposal is that Rob wants to just "formalize" a position that already exists and is being done, to some extent, already. If the position is already in place and already operating, then why is there still a patch backlog? How does formalizing it actually improve the situation?
Key to financial independence: Spend less than you earn. Save and invest the difference. Do it for a long time.
Disclaimer: I'm a FreeBSD fan and generally consider Linux et al to be one of the most choatic and sloppy large-scale development projects ever created. Just being a Linux user requires dealing with an absolutely insane about of pointless choas; I fear to imagine what trying to be a maintainer must be like...
That said...
"Just use CVS" isn't anything remotely close to an answer. SCM, release control, whatever you'd like to call it is 99% about the PROCESS, not the tools. CVS is a tool. It's a great tool and one I highly recommend, but at best it's 1% of a solution. It seems quite obvious that Linus's current process is incredibly flawed. It would seem a drastically different process needs to be devised. Only once you have figured out what your process is going to be can you even start shopping for tools (CVS, whatever) to help you create it (and even then...the tools are only 1%, if you even have tools. The PROCESS is what's important).
Of course "CVS forks" of Linux failed. Anyone with even the slightest understanding of CM could have easily predicted such an end. You've got a handful of people, not even the top people, trying to constantly fold in the efforts of thousands of contributers all using a completely different process (ad-hoc patches emailed to devnull@linus.org). What did you expect?
You have two choices. Devise a workable process which Linus will agree to (with or without CVS, whatever). Or, you can devise a workable process that everyone else is happy with...and simply throw Linus off his paper throne. The first option will likely never, ever happen...at least if every single word Linus has ever uttered on the subject isn't true. The second option might actually welcome the dawn of a non-choatic Linux, one which wouldn't be so easily cast aside as a cheap toy made by a small man with a big ego.
Linux/Linus Flame (-1k Karma)
My
If you read Linus' reply to this you'll see he argues that the 'patch penguin' merely replaces him with someone else, and does not actually help distribute the work. He also comments that it may be better to submit patches to subsystem maintainers.
This being the case, why not just automate that? Subsystem maintainers are effectively maintainers for a known group of source files. Its not hard to figure out (automatically) from a patch which files are affected - and from that, which maintainer(s) the patch is relevant to. Given how Linus describes his 'trust network' it looks like he wants a hierarchy of subsystem managers who can merge patches for him to accept upstream. I can see that scaling a lot better than the proposal in the article.
For this to work of course, you'd have to hope the patches had reasonably local scope - if most patches affect >1 subsystem this isnt worth doing at all.
- Baz
The whole kernel thing has to stop being like the Holy Roman Church and more like a democracy. Although commercial, closed software benefits from a hierarchical leadership system where a small number of people have the final say on where a product is going, an open project that is so visible as Linux needs a more "for the people, by the people" approach. I understand the need for one person to maintain a semblance of control, the entire decision process as far as Linux is concerned is hampered by the autocratic style under which it has operated since it began to grow so explosively.
Linux doesn't need a "patch manager", it needs a House Of Commons or a Congress or whatever you want to call it. It can't do away with the president and the chiefs of staff, but it has to have a heck of a lot more checks and balances than it does now. And it definitely needs to be a lot less autocratic.
Eventually, we're going to see more and more forks as people begin to understand that the OS is no longer Alan and Linus' own private playground but a heritage of the people who use it every day. While they bicker and fight over who gets to play king on a given week, someone is going to pull the rug from under them. Royalty has always been tolerated while the populace is enamored with its trappings, but revolutions do happen once in a while.
All the talk about how amazing it is that Linux is done by amateurs in their spare time, etc. (to the extent it is true), just points out how susceptible it is to hitting this ceiling. I've worked on commercial OS projects for a Really Big Company, and we did bigger releases, quicker, and with fewer problems than we're seeing in Linux lately. The group I worked with was not necessarily smarter or better programmers than the Linux developers, but we were all full-time employees who had careers on the line. Few things motivate programmers (or anyone else) to do a good job like that kind of pressure.
I'm sure Linus is healthy and a good driver, but as misfortune befell a former colleague at another job (her car was parked at a light on the off-ramp which was below the highway, a driver from the other direction suffered a cardiac arrest, crossed the median and opposite lanes and went airborne, landing on her car as she waited for the light to change. Gone, just like that,) unfortunate things happen. It would be very tragic for his wife and child to lose a father. It would be a disaster for Linux, as the unifying person would be gone and in the aftermath someone would have to take control. I imagine Linus has already considered this, but his tight grip on the kernel is a bit worrying. He should delegate more.
A feeling of having made the same mistake before: Deja Foobar
The part that makes *ALL* of these arguements stupid is that the Linux kernel is Open Source. If you or anyone else doesn't like the way that Linus is hadling patch management, or anything else for that matter, they are *free* to do their own. Go ahead, and fork your own kernel. Hell, build your own from scratch if you like. If your way is truely better it will be the defacto choice simply because it is better.
But, if you don't want to do it on your own and you instead *choose* to follow Linus and the direction *he* chooses to take, then you ought not complain about the way he does his work. You are not forced to follow Linus! You choose to follow Linus so if you don't like what Linus is doing, STFU!!!!!!!!!!!
"...source code maintenance tools/organization are for wimps!"
I almost don't believe what I'm hearing. How do you manage development branches? How do you roll back your code to undo an enhancement you thought would work? How do know which bug fixes are tied to a particular instance of code? Wimp? More like "anyone who ISN'T using source code maintenance is a fool."
> and is frustrating a lot of people who are
> involved in the kernel with the risk of losing
> them.
The thing I don't get is why these people are goofing around with the kernel to begin with. Isn't one of the biggest virtues of kernel modules the ability to plug 'em in to a running system. New version? Build a new one, unuse & unload the current, and load the new one.
Why can't people who write drivers distribute them under their OWN energy? Why must the kernel be monolithic? Why not trim down the kernel to little more than a core, let linus distribute/patch THAT.
It's worked for windows for a decade now; maybe M$'s method isn't ideal either but being able to go to a vendor website, getting a driver, and dropping it into the system seems a hell of a lot more convenient than waiting for a lone man to include something in a source tree.
Seems to me the kernel is pushing into the future in all the wrong areas, when some of it's features are still living in the dark ages.
Most systems use a mere fraction of the drivers included in the kernel. This wastes bandwidth, configuration time, possibly memory, definetly disk space.
This is a basic principle of leadership/management that the Army calls "span of control".
It has been proven (through all kinds of research) that any one person has difficulty controlling much more than 10 individuals, with 8 being a practical maximum.
This is why military units are organized in strict tree structures - at no point does any one leader exceed his span of control.
So an 8-man infantry section is controlled by a Section Commander (which can be further broken down to 2 4-man fire teams, but rarely). 4 sections report to a Platoon Commander. 4 Platoons report to a Company Commander. 4 Companies report to a Battalion Commander, and so on through Regiments, Brigades, Divsions, and so forth.
In this manner, the several thousand men in a large formation like a Division can be commanded by one man.
Further aiding the General in charge of the Division is that each sub-unit is granted a certain amount of autonomy within certain boundries. As you go up the chain, orders become more general (2 Battalion is to attack objective Oscar) and as you go down the chain, orders become more specific (Bob, you're carrying the machine gun)
While Linux development does not require this level of structure (and indeed, would probably suffer if this many authority-resistant cats were forced into such a regimented structure) the basic principle of "span of control" is still applicable.
The obvious correct solution is to modularize the kernel into subsections with clearly defined areas of responsibility, with a "mini-Linus" (who Linus trusts) granted control over each module.
Not suprisingly, this is what Linus has suggested.
The trick (and the catch) is two-fold. Firstly, you have to find people who are both responsible enough to be able to act as a "mini-Linus" without dropping the ball (which means Linus has to trust them too) and secondly, the kernel has to be modular enough so that changes in Module A do not step on or otherwise negatively impact code in Module B.
That's a hard row to hoe.
What I find a little distressing is that there appears to be a whiff of revolution in the air; of people talking about overthrowing Linus (and Alan Cox seems to be among them) This will not solve the problem. It is not that **Linus** doesn't scale, but rather that any single maintainer doesn't scale. Any revolution will just be "same shit, different name"
CVS doesn't solve the problem either, as anyone with commit privilages can jsut dump anything they want in there, and it becomes canon. Linus' function as a shit filter is very very important. every patch *must* be audited by someone with very high standards before it goes in.
I hope the crew rallies around Linus - as usual, he's right.
DG
Want to learn about race cars? Read my Book
I agree to a certain degree that not all drivers should always be included in the kernel, but what's kept in, and what has to go? For example: filesystems - way too many of them, but you can't boot without support compiled in, or changing some real stuff on the booting process.
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
In many of the posts (yes I read the whole thread) people state things like this:
"I perceive this to be true. Any objections? Now, since we've agreed this is a global constraint, here is a consequence of the aformentioned thing that I believe to be obvious and thus not requiring any objective support"
Then the other kernel hackers argue about the conclusions (which are typically quite logical) rather than the premises.
Here's an example from Linus: Now, I personally have worked with well over a hundred people, and I know that there are other humans who can do it (granted, it's a somewhat rare ability - like programming or musical talent). But nobody bothers to point out that Linus is committing the fundamental logical fallacy of ascribing his own attributes to all other humans.
More from the same post:and again:
Linus, I have tremendous respect for you (and I still regret the conversation we had at LinuxWorld that I'm sure you don't remember) but I think that these "obvious truths" are not obvious at all, and that more "thinking things through" may be required on your part. You do not have the same social or organizational abilities as all other humans.
Alan is one of the very few people on the LKML (as far as I can see) with the balls to say that important stuff is being dropped because the authors don't have name-recognition with Linus. Ingo has that name-recognition, so his patches get respect, and consequently he finds ways to excuse the (mis)handling of others' patches. Sorry, Ingo, but that's what I see.
--Charlie
I think Professor Tanenbaum is laughing.
I think some of you are forgetting that Linux is open source. If you want a version of Linux that has a different style of upgrading features and providing patches than what Linus does, then start your own version of Linux so you can do it the way you see fit. Isn't this one reason it's open source?
And we could get a gigolo so the wife stops bugging him to perform...
I got it! Then, we can force him to take ever-increasing amounts of speed, so he doesn't have to sleep!
Then he can dedicate his life to linux 24/7!!! That would be great!
</sarcasm>
Make sure he actually wants to dedicate that much of his life to writing one program.
I'm a leaf on the wind. Watch how I soar.
for software precisely because the "general to specific" heiearchy breaks down.
It's not enough for Linus to wave his hands and say "we need a new VM", and then Joe hacker writes one.
Joe hacker, on the bottom, is fixing typos and writing documentation. Actually, Joe Hacker is probably working on apps.
The brigade commander, in your analogy, is writing the VM in bits and pieces. Linus has to go through it carefully and apply the diffs or not. Software is all about details. There are ways (e.g. modularity and interfaces) of splitting up the work, but only to a certain extent.
Imagine a war fought almost entirely by the general staff, with their subordinates serving as assistants and sopport staff -- it's the reverse of your analogy: A small group of people at the very top write most of the code.
When in doubt, have a man come through a door with a gun in his hand.
the Bazaar is getting crowded. To put it mildly.
Lars T.
To the guy who modded me down from perfect to terrible Karma - Apple haters still suck
...although I concede that trust comes from competence. But what does this have to do with anything?
If you're going to overhaul (or in this case, overthrow) an established system, you need to do it with much fear & trembling. Look, I see this all the time as a manager. A new project is given to a manager. That manager hires someone competent and works with that person to build something from nothing. If the project takes off, you can end up hiring a lot of people to help support that project. It is possible for all those new hires to be more competent than the original, lead developer. But that original, lead developer now has management's trust. That lead developer, who got the damn thing off the ground in the first place, has an opinion that -- like it or not -- counts for more than the other developers. You can call that unfair, but certain people are just better rounded than others -- they have social skills, they can explain things well, they understand the market pressures, or whatever. Not to mention that it's their baby and you don't fuck around with stuff like that. As a manager, if I stomp all over the lead developer's project, that lead developer may not want to lead the next project! Sure, I can call on one of the other people brought in to help, but those other people may not be leaders.
I'm not saying that such an effort is doomed. But I am suggesting that, if you want to propose that Linus relinquish some authority, you damn well better have a value proposition for him. Shouting "this needs to be a democracy" is what you want, not what Linus wants. He's looking at that as a management nightmare, a removal of a power structure that is in place because, whatever it's faults, it was the structure that worked. If it doesn't work well anymore, you either have to convince Linus, or you have to convince everyone else to stop trusting the person in authority. That is a massive undertaking.
My Greasemonkey scripts for Digg &
If this rain of complaints and mishaps carries through and people realize more and more that a better solution is /badly/ needed, two things can happen:
1) Someone clones Linux and turns it into a better system by including all the features that people want and Linus drops.
2) Linus is forced to turn Linux into a pure microkernel, so subsystems are clearly defined, and he can still dictate over it.
These are Linus' two worst nightmares. (Think Minix and Hurd.)
Haven't you ever noticed that?
The notion of trust seems central to this debate -- who can be trusted to apply patches that will make the kernel better? And it reminds me of structures for establishing trust in the world of cryptography -- who do I trust to tell me that this public key really belongs to the person to whom it's supposed to belong?
The ideas about kernel patches being discussed remind me of the distinction between a "web of trust" and "certification authorities" in the world of cryptography. Certification authorities are the conventional means, which are used by the most popular web browsers. Some authority, like Verisign or Thawte, is universally accepted as trustworty, and if they sign a key, then it can be trusted. The web of trust was advocated by Phil Zimmermann and built into PGP, and AFAIK, PGP is the only context in which it's ever been applied (or even proposed). It's an attempt to reflect our everyday notion of trust -- I decide who I trust, on whatever basis I choose, and if someone I trust has signed a key, then I trust it. I can also decide if I trust people I don't know, if they are trusted by people I trust. There's no need for a central authority in a web of trust (although I might choose to include CAs in my web and trust them).
Rob Landley is suggesting that we entrust the patch penguin with the job of filtering patches towards Linus, but Linus sees the PP as a kind of central authority, and prefers to have a web of individuals around hom who he can trust.
Like Phil Zimmermann, Linus is arguing that his model is much more similar to the way that people decide to trust each other in real life. In the world of crypto, certification authorities have always been regarded with suspicion for reasons just like this -- why should I trust Verisign more than my friends? In the kernel world, Linus is saying, isn't it better and more intuitive to trust a group of people I know well, who can distribute the work amongst each other?
The analogy breaks here a bit. CAs in the crypto world are suspicious because we don't if they're corrupt -- maybe they're full of Enron managers. In the kernel world, we need to trust maintainers with their skill and above all, ability to handle the workload. Corruption is not the problem (I hope not, anyway.)
But, to take the analogy further, one should also note that PGP web of trust never really worked well as a practical matter. Most people using PGP had an island of people around them whom they trusted, with few connections outside of their immediate clique. I don't recall *ever* receiving a key from a stranger that was evaluated as trustworthy because of its position in the web topology.
So I'd say that Linus has a point in saying that the web model fits our common-sense intuitions about trust. But if Rob Landley says that it's not functioning well, the experience of PGP gives some support to his argument.
Always keep a sapphire in your mind