Slashdot Mirror
The Evolution of Linux
Taiko writes: "Kerneltrap.org has posted some of the more interesting messages from a recent kernel mailing list discussion. It started with a post on proper indentation, but turned into something a bit more. There are some posts by Linus and Alan Cox about the nature of design, computer science, Linux development, evolution, and more. Quite interesting and funny."
Will use Outlook only -- "Prayer will defend us from viruses", says school principal.
This will not do good for the acceptance of Linux in the Bible Belt -- Linux evolved through natural selection, while Windows was created by God.
I enjoy reading Linus' thoughts so much.
All around him, people try to make him or Linux more than it really is - and invariably, Linus brings it down a notch and puts it in perspective
It's amazing that this guy gets constantly hero-worshiped, his baby created billion dollars of wealth (at one point, at least), and yet just keeps his feet firmly planted
Compare that to the clowns that get high and mighty because they rUleZ at Quake, or on some IRC channel ... The geek community could learn a LOT from trying to emulate Linus' behaviour.
In all seriousness, would this article have been given a second glance if Linus wasn't involved? If I were to post a message saying "Hey, my friends and I were discussing the meaning of life after arguing about pencils, check out the log," I doubt a single editor on slashdot would have given it a 2nd glance. What kind of sick twist on celebrity worship is this?
Can anyone really say that computing as a field or science was designed? What we have today is the result of a form of evolution and a result of a market economy. Nobody knew where we were going, we just started going someplace.
The company I work for has spent the past 4 years slowly evolving a fairly complex graphics and haptic (see: Intelligent Scalpels Through Touch Technology for more about haptics) API. At the start we had only a vague idea of what it should be like. We knew from our experiences in graphics that it should be scene-graph based -- so we borrowed the VRML design. We knew that we wanted to be able to do a few things with it. This gave us the basic framework to start with, much like Linus had with Linux.
Then we basically evolved the product. Every time we worked on a project that used the API, we learnt more about what it was good at and what it lacked. We modified it, fixed things, extended it with new features. After 4 years we have something far better than we could ever have dreamed of designing.
The most important reason for using this approach was not because we believed in an evolutionary approach to software engineering (I don't think that Linus' advice should be taken too literally). It was because we were dealing with making an API out of cutting-edge research - much of which hadn't been done when we started. We simply couldn't have designed it.
Following that thread, can I now propose Linus' Law:
Any software system with a large enough user base can rely on the accumulated experience of its users to add features, and also picking ideas from smaller systems now and then (at a very low incremental effort).
Corollary. The onus is on the smaller players to come up with new features to distinguish themselves from the masses -- but ultimately it's no-win for them because their *really useful* ideas will be subsumed into more popular systems anyway
I need sleep and I'm quite possibly not thinking straight, but am I right in thinking this would create enormous pressures for specialized players like Sun and Apple (and Be, as they found out) in the long term?
If that is the case, where does that leave the "small is beautiful" rule? Does it mutate to "small is beautiful, provided you are part of a *big* idea that has incredible amounts of 'traction'"?
don't be silly.
...
WTF do (ex-)linux companies have to do with the quote you posted.
I think this quote has a point.
If we go into comparing, let's say, building bridges and os programming, I think we _can_ see the differences in methodologies one needs.
With bridges, we have a well known and accepted theory of their statics, a relativly narrow expectation what we expect a bridge to do, and we can, by using tolerances of a wide margin, account for the fact that something unexpected happens.
In an os, there is not really a broadly accepted theory (micro- vs macro-kernel, VM, filesystems, implemetation language) - at least when we look how different realisations we see in practice.
What do we expect am OS to do, or more precisly, what do we expect an OS to do well?
latency vs throughput, single vs massivly multi cpu, graphics in kernel vs graphics in userspace
Seems we have no real consensus here.
At last, and this is perhaps the most important factor - we can't make an OS more failsafe (or performing better) by introducing margins anywhere. Due to the binary nature of CS it doesn't make sense to use redundancy for many aspects of an OS.
It either works or fails.
1- Get latest kernel source 2- Open a random file, go to a random location in that file 3- Roll a d100, use this table: 0-10 insert a random C keyword 11-20 delete nearest statement 21-50 define a new variable 51-80 delete a variable declaration 81-85 change your keyboard layout to some language, switch off monitor and start typing headlines of slashdot. Stop when you feel like it 86-90 delete rand(20) characters 91 delete file 92 merge file with other random 93 copy file with a new name 94 move file to another location 95+ merge file with a random C source from net 4- try building source. If all goes well,submit a patch. Otherwise roll d6, on 1-5 return to step 2, on a 6 return to step 1.
Gentlemen, you can't fight in here, this is the War Room!
Science provides the tools that engineers use to build stuff.
Science provides a lot of dandy tools. Engineers like tools.
Engineers would be useless without science to provide new raw materials.
Baloney. We (engineers) were building all sorts of impressive stuff long before the invention of science. Check out the Great Pyramid and Yu the Great.
According to Rik:
Biological selection does nothing except removing the weak ones, it cannot automatically create systems which work well.
In short, I believe the biological selection is just that, selection. The creation of stuff will need some direction.
And I have to nod vigorously to that. Even taking the model of accelerated evolution through human breeding of species: you direct two animals together to breed. You don't just let the Ps, the F1s, the F2s, etc. just all wander around in a pen, have a sniper sitting on a post shooting the ones you don't want, and hoping the rest go at it...
Reading it, does anybody else get a strong sense of deja vu? It sounds like the two sides are arguing Evolution vs. Creationism -- well, they *are* -- but in this case they're arguing it over Linux instead of over human beings. Only in this case, we *know* there was a creator, and he says "I didn't create it, it evolved". Which makes me wonder if we ever did find the "creator" of human beings, and what would happen if he/she/it/they said the same thing about us :-) Picture it (and pardon my Eurocentricity):
Us: "God! At last we have found you! Now tell us, please... WHY ARE WE THE WAY WE ARE? WHY ARE WE HERE?"
God: "I dunno. I created you to eat the lions, and you just kinda got out of hand"
bollocks, no way this is insightful!!!
you don't understand the concepts of evolutiuon, and neither does mr. van riel.
biological selection (actually, the terminology is "natural selection") does not work by weeding out the weak ones. natural selection favours the multiplication of successfull ones (ie 'survival of the fittest').
the argument you (and rik van riel) are using, is essentially the same as most creationists use: mutation can only break down and not build up.
this is wrong. read some darwin before you comment on this stuff please.
regards,
meneer de koekepeer
> The strength and robustness of the linux system
This linux system that depending on which "stable" version you download, locks up under high load, corrupts your filesystem when umounting it, invisibly reverts your filesystem to one that can be hosed from a power fail, or kills off processes at random -- like init -- when it starts running out of memory... And that's just what I recall off the top of my head from the last few months.
I don't think Linux is exactly a pile of shit either, but let's not kid ourselves, it's got the same problems that commercial OS's deal with, and the development model hasn't exactly been a panacea in that respect.
I've finally had it: until slashdot gets article moderation, I am not coming back.
Sounds like a couple of harsh extremes to me.
Of course software is designed. But this does not mean that the design is complete, correct, or optimal. And that's where evolution comes in.
All these people who scoff at formal design do have a point: so many times so called formal designs end up being one way paths to the wrong thing.
The formal design advocates repond by saying, "well, you didn't have a correct design." A fat lot of good that does. I've been part of development teams where there is this mantra of design it, check it, double check it, lets not do anything until the design is complete, because failure is uncorrectable. And you end up progressing e v e r s o s l o w l y. This is design by perfection -- the idea is to be so careful about the design that it can't be flawed.
Of course, this never works. Nobody can make anything non-trivial right the first time around. It requires some kind of step-wise refinement. Now, this does not mean the design should be abandoned, but one should design in anticipation of making mistakes. Then, the design permits the local correction of errors, without them becoming a global fiasco.
Design for flexibility then: separate APIs from implementations. Version your APIs so when they're lacking you can produce a new back-compatible version. Don't know all the details about every possible kind of device? Gee, throw in an open-ended IOCTL into the device control API. Refine IOCTLs for similar devices later, when we figure out what they need besides the basics.
The point is that it is possible to design adaptable and refinable systems in order to accomodate the inevitable "opps" with a fix that is local and not global in nature. Now, you can't be flexible in everything and sometimes correcting things hurts: witness the Linux VM. It wasn't really planned to abstract it's API away to allow for interchangable plug-ins, was it. And the VM wars were somewhat painful precisely because one had to chose and couldn't punt.
Nevertheless, experienced software designers try to provide an "out" whenever they can, and think that a particular course might require modification in the future.
You could've hired me.
I didn't read the whole thread rant with Linus et. al. - but from my own experience and observation EVERY successful project mixes both initial design and evolution in design AND implementation. If you fix the design absolutely up front at both the macro level AND of every sub-system in a large project, you will invariably run into huge roadblocks at some point. Something will not work as planned. As I see the Linux Bazaar process, it reaps benefits when this happens - some person or organization stumbles into a roadblock with poor networking code, poor SCSI subsystem behavior at high loads, or an unreliable VM. These emergent behaviors may only affect some small portion of the user base - but the subsystems then enter an evolutionary phase where people varyingly fix what's there or design something new, and some design ends up surviving based on what the most people seem to like and want and in the end, if all else fails, what Linus dictates.
So no, this isn't strict "evolution" after the style of Darwin. If we let purely random decisions drive software and forked every few minutes, the analogy would be pretty complete. It would also take as long to write good software as it does to evolve a well adapted creature. An eternity.
I see where the idea of selective breeding comes in - Linus sees himself and the kernel leading guns as picking and choosing the best patches and suggestions. Up to a point, this means they are exercising design and discretion, but they generally don't "assign" work from their central database of TODO tasks to IBM, Red Hat, and other individuals or organizations participating in kernel development - those organizations and individuals scratch their own itches and their work usually finds its way back into the kernel. Other posters accurately said that a more random evolution could be effected by letting people check in free-for-all into CVS. This is true, but I don't think that would necessarily improve the results and timeline of kernel development.
You have to realize that the comparison here is, as others pointed out, to a monolithic software development process - in the Cathedral, a centralized decision is made - "we are going to make Windows NT better able to support large enterprise database deployments" and a team is assigned to break it down and work through all the implications, then implement. In Linux-land, the interested parties don't call to schmooze with MS biz dev people who pass info down to technical guiding councils, they pony up and write their own patches to the subsystems they see that need improvement. If there are enough interest parties, presumably enough patches will get submitted that the best from all get incorporated into the set of relevant subsystems that effect large enterprise database deployment, and we end up with a Linux kernel that supports exactly that. Of course the primary difference is that at the same time, somebody else may have made complementary and/or conflicting changes to make Linux a better desktop OS. Chaos ensues and flames erupt on kernel-dev and wonderously, eventually, something better for everyone results after compromises are made.
I agree with Linus.. projects that I've spent several years on came out at the end with features and design elements I could never have predicted going in. I've spent 6 months doing design work on pen and paper at the start of a project, and during the years of implementation thereafter, far more 'design' was done by reacting to the state of the code in any given moment and the problems it was having both internally and with regard to the userbase. My biggest project has evolved tremendously, even though I was essentially the only coder working on it for most of its existence. I can't imagine, then, how much less 'designed' by any individual the linux kernel must be, with the hundreds or thousands of developers contributing to it.
On the topic of Sun's doom, I understand why he says that. Sun's software is co-evolved with their hardware, but neither change very quickly. Linux has to cope with a much more wild, much more genetically diverse hardware base, and as a result it tends to move faster to support new types of devices. Solaris on Intel is a joke compare with Linux on Intel in terms of its hardware support.
Of course, there is nothing magical about a process that allows more evolutionary freedom.. if the hackers working on it don't have the good sense to be effective natural selectors and mutators, then the process won't have a terrific outcome. Linux is thriving because it has so darn many hackers working on it, and because it has so very, very many users using it, and because Linus has a deep and proper understanding of both good taste and evolution.
- jon
Ganymede, a GPL'ed metadirectory for UNIX
I must say, the following quote from Linus from the article is one that strikes me with fear and awe:
When someone with the prestige that Linus has says something as powerful as this, I cannot help but feel that this topic is something that he is absolutely passionate about, much in the same way Stallman is passionate about Free Software. Linus doesn't seem like the type of person to use this sort of phrase on a whim; like he says, "I'm deadly serious".
When I was in college, I read the book Software Tools in Pascal by Kernighan and Plauger. The most valuable thing I learned in college was the system of design set forth in that book, which the authors called "left-corner design".
The idea is simple: when creating a program, start with the most important thing the program needs to do. Once you have that working, add more features. Ideally, as you go, you should be releasing working versions to whoever will be using your program.
This is so right in so many ways. For one thing, if you run out of time during a project, at least you have something you can release, and it may very well do much of what the users need. (There is a line in the book to the effect of "80% of the problem solved now is better than 100% solved later.") Also, early feedback from the users can show you what's wrong with your design, before you write a whole bunch of code that you would later have had to rip out. (I seem to recall an example in the book where a large system spec turned out to be totally wrong; the users didn't know what they wanted until they had something to play with.)
I never before noticed that the standard open-source development techniques match up with the left-corner methodology. Open-source projects such as Linux are all about "release early and often".
When I read Linus's comments, I was nodding my head all over the place. You create some code that solves some problem, possibly not very well. You release it. Feedback and patches start to arrive, and the code grows, possibly in directions you never foresaw. The more popular the code gets, the more robust it gets, as people patch it to work in a wide variety of situations and on a wide variety of hardware. This is why Linux has come so far, so fast.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
You sound pretty paranoid.
Honestly you shouldn't be too worried. The shit _hasn't_ hit the fan, and 2.4.16 is ROCK solid. _Yes_, 2.4 took a long time to stabilize. It's there now, after the Van Riel vm was tossed aside. So lets cut the crap and call a spade a spade: 1) Linus is not a stable release maintainer. If linus puts out a kernel it needs to be tested. Linus does not put out release candidates. Only a fool would use a product that has been released without prior testing. 2) 2.4 took so long to stabilize because of the mistaken beleif that a BSD style VM was best for linux. 2.4 doesn't have the infrastructure to handle it (reverse memory mapping, etc). 3) 2.4.16 is a fucking great kernel. Except for a few possible bugs (the source tree is 149MB uncompressed!) I know of no problems whatsoever. 4) 2.5.x is already starting off with a bang. the new block/io layer should kick major ass, along with all the other enhancements planned. 5) Quit your whining. The sky isn't falling, alan cox isn't retiring to the hills to become a hermit, and linus torvalds knows what the fuck he's doing.