Slashdot Mirror
Linus Does Not Scale
EmilEifrem writes: "Seems like everybody's getting more and more frustrated by Linus' (in-) ability to handle patches. Rob Landley just wrote an "RFC on Penguin Patch Management" wherein he proposes a "Penguin Patch Lieutenant" system that he believes would scale better. The full discussion can be found on the Linux kernel mailing list. Linus seems to dislike it, as usual, source code maintenance tools/organization are for wimps!, but a lot of others find it a good idea. Anyway, it's a very good read."
It's got to be a pretty small group of people who are annoyed by Linus' in-ability to handle patches... specifically those writing the code! I'd think it pretty important to keep those folks happy, since without them our beloved little OS wouldn't progress as quickly as it does.
"This above all, to thine own self be true"
but is there really anything wrong with using CVS?
Tarsnap: Online backups for the truly paranoid
I personally think this is a good idea. Coordination would speed up development and improve the linux kernel
You know, with Linus's many comments about kernel latency and pre-emptive kernels, you'd think that he'd be all for imrpoving kernel development latency!
Moderation: Put your hand inside the puppet head!
Using technology to improve kernel maintenance is so sensible that sooner or later somebody will come up with a solution that Linus likes. One improvement that has already been made is the use of other people as maintainers for older kernel versions.
Lots of corporate/management types have the negative impression of Linux as an OS that has no professional control over kernel development. It's seen as a souped-up hotrod modified in the garage that runs like a dream but could fall apart at any minute. Hell, even a lot of BSD people see it this way. If (*if*) a goal of the Linux community is to gain wider acceptance or be taken more seriously, one way to do that might be to give more than one person final say over the kernel.
Yeah, it's Linus' baby -- but once IBM is advertising your OS during the superbowl, it might be ok to expand the development structure a little bit.
I thought somewhere along the way, the developers that be, agreed to accept a star method. Where people submit their patches to the Lieutennants (sp?) and then they do most of the work, and then just pass them on to Linus as appropriate?
Maybe we just need more Lieutennants?
Maybe we just need fewer patches!
A prediction, pulled from my ass: Here's what I think will happen.
i) A respected developer (Alan Cox, perhaps) who already has their own patch tree will open it up via some source management system
ii) Other developers, including those frustrated ones, will begin to use it, including keeping it roughly in sync with the official tree.
iii) The forked tree will develop as fast/faster than Linus' tree (assuming the pro-CVS people are right)
iv) LT will realise that its not a bad idea, and essentially adopt the forked tree as his own, complete with CVS.
In short, theoretical arguments for CVS will continue to cut little slack with LT. A demonstration of its effacacy will be very persuasive.
If (iii) is untrue, the argument will be moot anyway (why have source control if it doesn't help?)
Athletic Scholarships to universities make as much sense as academic scholarships to sports teams.
It wont be long before the MS camp picks up on this and starts to use it as a weapon - not that we havnt had to wait a year or two for bug fixes from them in the past.
Oh, I don't know about that. After all, I got three Exchange server patches within a couple days last fall! Mind you, the first to broke more than it fixed, but they were timely....
Seriously, some bug fixes can be tricky - and sometimes the patch creates tons of other problems. I know I can not cast the first stone here when I've coded around an issue, only later to have the problem fixed and my work-a-round hork things up. I can believe some issues took a year before all the code was in alignment...
+++ UGUCAUCGUAUUUCU
The basic premise he makes is one that many developers seem to miss. To quote Linus:
From that, he goes on to point out that people tend to have a small group of people they work well with and trust; perhaps 5 - 10 people. So, in essence, if Linus appoints 5 - 10 maintainers of major subsystems (which he has), and each of them has 5 - 10 people the trust to maintain specific aspects of their subsystem, then there is no problem.We're not there yet. But that is the only direction I can see that will work, long term. It just takes time for the appropriate people to move into their spots; trust takes time to build, and the structure has taken time to modularize enough to allow this to work.
And, though I would prefer to see it in use just to make the maintainers' jobs easier, CVS will do nothing to solve the problem. It is a tool, not a process. We need a process. Actually we have a process, it just isn't fully implemented yet.
No, strike that. We have two processes. The first, used by the bulk of the kernel hackers, is not fully implemented yet. The second, used by a minority of the kernel hackers and a large part of the pretenders, is simple: whine alot, and, when that doesn't work, whine some more.
Linus says it much better, and in his own unique ... er ...
Idium, sir?
Yes, in his own, unique idium.
Its very simple.
CVS does only accept patches, it does not check for quality or if it breaks anything.
This has still be done by hand. And if its not done at patch time, people will forget it.
I'm by no means a Linux fan, but bashing Linus because of that just sucks.
ONE has to check the submitted patches.
If there are 10 people checking the patches, no one knows what the other one has really done, that means even if you have people with highly communication skills (99% of the coder lack that) you still would sometimes forget to tell something and its going to break everything.
/wave
Before you email me, remember: "There is no god!"
He then goes on to say that the solution to the problem of the scalability of one maintainer is to partition the different subsystems of the kernel to such an extent that there would be precious few patches that actually require a knowledge of the entire kernel source.
Which is the proper way to design a complex system anyway. Partition the system and then present well defined interfaces, versus making a complex, monolithic system more complex and monolithic.
Linus is correct. CVS just gives you an extremely efficient filing cabinet. Without a someone to file things properly, the filing cabinet is no better than a shoebox. Running a CVS tree makes you the equivalent of a secretary. People will just come by, throw things on your desk, and yell "File this" as they run home for the day. Linux chooses to be an engineer. He refuses to just accept what it thrown at him. He wants explanations and reasoning as to WHY a patch should be included. Not using CVS slows the development system down so that it is comprehensible by a single person. The kernel isn't just a collection of source files collected in a cabinet. It's a fine masterpiece of artwork on Linus' desk.
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
Correct me if I wrong, but I don't see any good reason why a very big project (Linux kernel) still has one maintainer for active development.
Now I'm no kernel hacker, but I do think the current system could use some improvement. Wouldn't it make sense for the kernel to be split into sections (device drivers, memory/thread manager, etc) with each section having a maintainer and one "integration manager"? (Applicable areas that could be subdivided, like drivers, could divide by type again.) If the system were properly modular, the integration manager would just have to make sure that the interfaces between the systems were well designed, don't change frequently, are thourally examined between changes, and that when they must change they are reimplemented nearly simultaniously in all affected subsystems. (Hopefully, any given subsystem could also undergo rewrites without affecting other subsytem code.) Again, I'm not seriously looked into the kernel code (yet!) so I have no idea how things work right now... but the scale IMHO requires a division of the labor. (Formatting the code to match would serve to further reduce the labor involved in maintainance.)
Do you like Japanese imports?
Despite Linus's comments, CVS or other version control would make sense. *However*, Linus should be the only one with write permission. Being able to review the history of file changes or to obtain the source from a particular date or release is useful in and of itself.
It would also be useful if a split like Linus suggests might be done (modularizing things and then a "Linus" for each mode), as the "Linii" would each only have check-in permissions for their own module.
Ooh, a sarcasm detector. Oh, that's a real useful invention.
I think I'm not alone when I say that, when I came to linux a few years ago, I was very impressed. Stability, speed, tons of functions, and a lot of configuration...
Well, that was all well and good. What was not all well and good is that, like so many others, I didn't actually write anything for the kernel. I assumed it was in good, and (as I have no real OS-level programming experience) more capable hands. I know that was what I was thinking when I explained to others why linux was so cool, and would always keep getting better. People would keep on sending in improvements, and these improvements, when approved for absolute stabiity (Alan Cox's old job mentioned in the leter, wasn't it?), they would be integrated into the sacred source.
Sigh. Well, that assumes the improvements get accepted or even seen in the first place, doesn't it? Linux is a huge development project (I mean, come on, it's an OS), and anything made in last I don't know how many years of development of any significant size had ought to have a tighter system of management; human or not. I'm a little worried that Linus has any opposition to these sorts of things. I'm actually amazed that linux got this far without more- I always assumed there were some definite mechanisms of the sort involved (hence my regret for never messing with the "sacred source"). I always just assumed there were.
Now I am surprised there wasn't a call to action before this. In a project of this nature, with so many people, projects, and significant economic forces putting their investments of time, money, and effort into this, you'd think that something would have been done when good patches started getting dropped. If that is the case something has to be done now.
Like I said, I never messed with the source. I never fixed anything; I never submitted patches; therefore I never had this problem. In short, I never knew. I was never involved, and never felt I had to be- I mean, better heads than mine were on the job, right (I said the source was "sacred")? Well, not all heads are better at everything, and I think that the more heads invoved, the better off this would be (at the very least, we would have reached this crisis point sooner and come to some decision on how to handle it). Maybe if some of the right people got in on it earlier, an appropriate management system would have been advocated and introduced- and these problems would have been minimized.
Possibly not. But more has to be done on this. Linus' counterpoints are well argued, but if lots of valuable code patches are really being (uneccesarily) lost due to the current system, are they good enough?
That's right! To help humans and the activities of these humans. Now, here we have a complex organisation-related problem (software versioning), which takes a lot of effort and energy from the human(s) involved. Instead of using computers and specialized software to HELP the humans involved and thus make the process a lot simpler, the human(s) involved disagree and try to keep the process non-automated and without computers.
Hello? What if a big company says: "To hell with computers, we do all our business-directing by hand, without decision support systems etc..", is that a clever move? No! Humans are not capable of processing a lot of complex data CORRECTLY. Computers do. Use them, whenever they can help. To avoid using computers in software versioning is IMHO one of the most stupid things you can do. What do you think, that Microsoft used Bill Gates' brain to organise the 35 million lines of sourcecode of Windows 2000?
Never underestimate the relief of true separation of Religion and State.
Equally, get IBM/RHAT/other sponsor to donate a whole load of different hardware. Write a autobuilding process that runs lots of little tests -- puts up ethernet connections, pings the framebuffer, etcetc. If the machine is down, then the most recent patches broke it.
Mozilla development does this, but is obviously easier as it's not the Operating System they're testing, just a single application.
-- Azaroth
A lot of people have said CVS could not work, agreeing with Linus. Yet FreeBSD uses a CVS tree with multiple committers, and has had no serious problems that I am aware of. Infact, I would say they have had less problems than linux has had in the last few years.
cjk
Toothpaste likes you
You will forget this sig before you next see it
The rule of thumb is: if it takes you more than 20 minutes to fix a bug, either you are incompetent, go read the fucking manual, or the code you are working on is a cryptic piece of goat shit, rebuild from scratch.
First off, get an account...
Secondly, Yikes! Have you ever worked on a project that was bigger than yourself? Most apps I tangle with are 1) large and diverse enough it will take a long time to know the codebase, 2) old enough that it was mutated far from its original problem, or 3) marketing drives a deadline that required corner cutting.
Just the fact you expected documentation for the codebase made me chuckle.....
From personal experience, I had a project that took almost 8 months. We were adding security to our application (before the JAAS 1.3 was out). We had to touch almost every class file when porting to a better way of doing authorization and authentication. We could not freeze the code, so not only did we have to patch everything, but also try to fold in all the other updates that were being done in CVS. Learned a thing or two about how not to merge branches too (grin). Fluid things are hard. The Kernel sounds just as bad.
So it depends... a year may not be long at all.
+++ UGUCAUCGUAUUUCU
Of course, I was spoiled with Clearcase when I was at Teradyne, but, at $1000 a seat, it isn't exactly cheap (and is dangerous in the wrong hands). Working at a startup now, and living with CVS, has tought me that it doesn't provide the kind of isolation, yet, tracking that Linus needs.
However, perhaps all is not lost. If Linus doesn't like CVS, let him not build out of CVS, but still provide an incomming CVS repository for certain trusted leutenants that filter patches. This provides the following:
1. A history of patches received.
2. A mechanism for delivering those patches to testers.
3. A source Linus can diff against his private tree.
Of course, when Linus makes changes outside the CVS repository, it will be out of sync, but the suggested Patch Penguin can be charged with syncing it up with Linus' latest changes. This way, Linus can generate his own patches from CVS and examine them at will, or on the urging of leutenants.
Another orthogonal approach to the ckernel into separately maintainable parts. There is no excuse for bundling every device driver under the sun when one has loadable modules. Device driver patches would then go to the device driver maintainer (in the same way, and for the same reason ,that Linus does not get Apache patches), offering some relief. Some core device drivers might remain Linus' responsibility, but he'd be more free to concentrate on architectural rather than implementation issues.
Yes, this would fork the kernel, in so much as there would be different bundled device driver sets. I see this no worse than the variety of existing distros, though I would like some central mirrored repository of what drivers were available.
You could've hired me.
Linus and team are architects of technology, the distributions are architects of product. This same scenario goes on in the large Unix system development houses too.
Often there is conflict between the two in terms of goals/objectives, key values, processes, tools, practices, timelines, etc...
Understanding the key values of each team is critical in developing the correct practices, tools, and processes for the teams.
The challenge, I think, for the Open Source Community is to develop tools that will respond to different team values and practices, which results in providing support for the individuals to achieve their objective.
Regards,
Kramer
I'm fairly positive that if he did want to he could find a Linux-related company to back him on this, despite the fact that many are in the crapper (RedHat would take him, IBM would probably pay him a full salary just to work on Linux, etc.)
Maybe he has interests outside of Linux? Ever even consider that?
I think there is a possible solution, if we look
closely at what Linus was saying. He talks about
evolution. He talks about him not being
indispensable. He likes module maintainers. This
is all a part of the solution. The only problem is
that people percieve multiple competing kernels as
a big problem.
The perfect solution would be to have each module
maintainer release their own kernels, maybe not
naming it like -ac but -scsi or -net. These kernels
should only have only those module patches and must
be synced with Linus's kernel. Anybody developing
a patch needs to look into which kernel suits his
work most and sends it to the maintainer. The
maintainer adds it if it fits his kernel, other
wise the guy needs to try a different maintainer.
Once a patch is accepted into one of kernels its
the maintainers headache to sync with Linus. Linus
will be happy and everybody knows that their
changes have been included.
It's very simple, you would gain quality and developmental stability where CVS demands a mature development environment.
Kernel development would actually be forced to break down into real -development -stable trees with definitive release engineering schedules. The VM mess in stable would have never happened.
Linus would loose control of all the cool preX numbers and the even/odd numbering system. OMG!
Here's the Linux version:
-- @rjamestaylor on Ello
Its sloppy, yes, if you keep up with the bleeding edge. But, the bleeding edge is always sloppy. Linux, because it has the number of developers and a well known mailing list exposes that slop to the public.
I wonder, how sloppy would freeBSD development be if you synced your kernel hourly with the dev kernels? How buggy would it be? What does the interdeveloper mail look like in the freebad world?
I run kernels over a year old. There's no slop. I'm (mostly) happy. In about 6 months, I'll switch to 2.4.
If you want less slop, stick with a distribution kernel. They've chosen and tested it out to be stable and reliable.
IMHO (very humble in this case as far al kernel hacking goes, but I have been writing software for a decade, using various source control systems for half of that) the fact that Linus doesn't use any kind of source control/version tracking system is immature, stupid, counterproductive and just plain wrong on his part. It's shortsighted that a project of that size and importance depends on the buddy-system (ie Alan Cox has the source as well, etc..) for backups and revision tracking. When I write a few 100's of lines of utility code that i intend to keep, you bet it's in CVS!
No, CVS or other source control is not a magic bullet, you have to use it right. But if you don't have one, you don't even have a gun.
I'm not saying he should have a CVS tree with every joe haxor and their dog allowed to write to it. But no version history and change tracking capability at all! Gibber!
My Karma: ran over your Dogma
StrawberryFrog
First of, Linus is Right. There, I said it. Why?
Because 10 is the practical limit to the number of people any project manager can be expected to directly deal with. And five is 'almost perfect'. Now, I will agree that, from a technical standpoint, Linus might need a second, not just for the "Bus Factor", but also for sanity-checking sake, to help smooth out issues where others might complain that Linus is playing napoleon. "Well, Alan agrees with me, so suck it"
I also think that we need to start thinking of development along honest PM guidelines. And Linus is right, here again. The maintainers for the various modules should be accepting and testing packages. OR, we need a set of people who will be responsible for various functional units of kernel design, a Portability Penguin, a Process Penguin, a Memory Subsystem Penguin, a Filesystem Penguin, a Networking Penguin, &c, who collect patches from the various maintainers. An org. chart where any single node has more than ten connections is useless, just as Linus suggests. These various, and God Help me, Project Managers, would be responsible for accepting patches and making sure they worked with the more core systems, and that things should be checked from whatever the most central modules are to the rest. And it all needs to be documented. There needs to be a Documentation Penguin, too. Dear god, people. Maybe we should all go read the Mythical Man-Month or something.
But then, I'm a freak. You can't build a car in a Bazaar.
Simply undoing the flawed submission means that you did a poor job of making sure that only quality code goes in. It does not fix anything.
Now, this is the only thing that I think is missing from Linus' vision. An SCC system can store this information efficiently. Linus' system (changelog) can only store this information at the granularity of a pre-patch.I'm a leaf on the wind. Watch how I soar.
Call me mean, but I don't respect anyone for wanting to "keep control" over what is supposed to be free software (well, except for modules), espescially when they decided to make it free. Wha? "It's free unless it becomes a hit and then I want control?" I don't think so -- I think Linus has plenty of leveragable egoboo to not need that kind of power trip.
If Linus wants to play architect, and concentrate on APIs, modularity, and scalability, great. If he wants to bundle components meeting his standard for stability, also great. But, if he can't do it all himself, he shouldn't try to prevent others from stepping in.
If he is worried about dilution of the stability associated with "his" Linux, he should excercize his trademark rights: you can't call a kernel, bundled with loadable modules, "Linux®" unless it has his blessing. Of course, meeting the lesser standard of API complience should garner a "Linux®-compatible" moniker, or "[insert favorite distro]-certified Linux® compatible", with Linus authorizing major distros to do their own testing, and applying that moniker.
Does this mean a forking of monolithic kernel bundles? Yes, but I don't think that seperate module sets are that traumatic of a fork. This is an area that the various distro-makers can move into, and an espescially fertile one for embedded kernel providers.
You could've hired me.
From what I know, the only way Theo eats, pays rent, or goes to the movies, is if he breaks his ass working on OpenBSD. I.e. it is his full time job and he takes it seriously. And the quality shows the the history of OpenBSD's security and reliability. Linus on the other hand has stated many times that he has a life outside of Linux and many other things to do. Fine. Linus, you are a better man than most, and we all owe you a great deal of gratitude. But now that people are using Linux in banks, hospitals, billion dollar companies, etc, maybe Linus should pass his hobby onto some people that will treat it like a profession.
Sorry if I'm out of line.
-... ---