Slashdot Mirror
The Evolution of Linux
Taiko writes: "Kerneltrap.org has posted some of the more interesting messages from a recent kernel mailing list discussion. It started with a post on proper indentation, but turned into something a bit more. There are some posts by Linus and Alan Cox about the nature of design, computer science, Linux development, evolution, and more. Quite interesting and funny."
Can anyone really say that computing as a field or science was designed? What we have today is the result of a form of evolution and a result of a market economy. Nobody knew where we were going, we just started going someplace.
The company I work for has spent the past 4 years slowly evolving a fairly complex graphics and haptic (see: Intelligent Scalpels Through Touch Technology for more about haptics) API. At the start we had only a vague idea of what it should be like. We knew from our experiences in graphics that it should be scene-graph based -- so we borrowed the VRML design. We knew that we wanted to be able to do a few things with it. This gave us the basic framework to start with, much like Linus had with Linux.
Then we basically evolved the product. Every time we worked on a project that used the API, we learnt more about what it was good at and what it lacked. We modified it, fixed things, extended it with new features. After 4 years we have something far better than we could ever have dreamed of designing.
The most important reason for using this approach was not because we believed in an evolutionary approach to software engineering (I don't think that Linus' advice should be taken too literally). It was because we were dealing with making an API out of cutting-edge research - much of which hadn't been done when we started. We simply couldn't have designed it.
don't be silly.
...
WTF do (ex-)linux companies have to do with the quote you posted.
I think this quote has a point.
If we go into comparing, let's say, building bridges and os programming, I think we _can_ see the differences in methodologies one needs.
With bridges, we have a well known and accepted theory of their statics, a relativly narrow expectation what we expect a bridge to do, and we can, by using tolerances of a wide margin, account for the fact that something unexpected happens.
In an os, there is not really a broadly accepted theory (micro- vs macro-kernel, VM, filesystems, implemetation language) - at least when we look how different realisations we see in practice.
What do we expect am OS to do, or more precisly, what do we expect an OS to do well?
latency vs throughput, single vs massivly multi cpu, graphics in kernel vs graphics in userspace
Seems we have no real consensus here.
At last, and this is perhaps the most important factor - we can't make an OS more failsafe (or performing better) by introducing margins anywhere. Due to the binary nature of CS it doesn't make sense to use redundancy for many aspects of an OS.
It either works or fails.
True, but I have always maintained that Linus is not, in fact, a geek. His interest in computer technology does form a large part of his life but otherwise he seems to be a fairly normal human being.
Let's go through Linus' claims:
Cat, the other, tastier white meat.
My turn to bite :-)
Linus is saying that Linux is evolving through countless small decisions. There is no One Big Plan. There are just ideas that are thought of. There's lots of code that is written. Some of it get into the kernel, some doesn't but gives new ideas for better design.
If you'd read through the whole discussion, you'd notice how computer science was compared to alchemy. It's a young science that has years to go before genetices can be applied. And we still have years and years of research that has to be done before we understand genetics. There's lots of trial and error being done there too.
Actually Windows has evolved a lot. Just look how much it has changed since Windows 3.1. It sure succeeds because of marketing and aggressive business tactics but that's only helping. MS wouldn't be able to compete against Linux with only Windows 3.1 no matter how aggressive they's be. So don't underestimate the effort behind developing Windows.
Actually Windows evolved towards what people wanted in the nineties. But since Windows 98 it has also had Word Domination-plan which is not good for Windows in the long run. But as Microsoft has loads of cash, they can afford trial and error as long as they learn from their mistakes. And that's one thing they are good at.
OK, linux is evolving (=changing incrementally) and not controlling tightly how it evolves is a nice idea, but this is how far the analogy goes. Linus is taking the analogy too far and use biological evolution out of its context. People do design pieces of code they submit, linus do control which ones are released in the main tree. Both of these facts, especially the latter one, make evolution of linux fundementally different than natural evolution. If you agree with linus please carefully state what do you agree with. Do you agree that any complicated engineering project can not be designed in advance? Or the fact that linux is not particularly directed to a defined goal is a good thing? Or natural evolution is a proof that not designing linux is a good idea? I agree with first two, but third one is plain wrong.
Gentlemen, you can't fight in here, this is the War Room!
> The strength and robustness of the linux system
This linux system that depending on which "stable" version you download, locks up under high load, corrupts your filesystem when umounting it, invisibly reverts your filesystem to one that can be hosed from a power fail, or kills off processes at random -- like init -- when it starts running out of memory... And that's just what I recall off the top of my head from the last few months.
I don't think Linux is exactly a pile of shit either, but let's not kid ourselves, it's got the same problems that commercial OS's deal with, and the development model hasn't exactly been a panacea in that respect.
I've finally had it: until slashdot gets article moderation, I am not coming back.
Sounds like a couple of harsh extremes to me.
Of course software is designed. But this does not mean that the design is complete, correct, or optimal. And that's where evolution comes in.
All these people who scoff at formal design do have a point: so many times so called formal designs end up being one way paths to the wrong thing.
The formal design advocates repond by saying, "well, you didn't have a correct design." A fat lot of good that does. I've been part of development teams where there is this mantra of design it, check it, double check it, lets not do anything until the design is complete, because failure is uncorrectable. And you end up progressing e v e r s o s l o w l y. This is design by perfection -- the idea is to be so careful about the design that it can't be flawed.
Of course, this never works. Nobody can make anything non-trivial right the first time around. It requires some kind of step-wise refinement. Now, this does not mean the design should be abandoned, but one should design in anticipation of making mistakes. Then, the design permits the local correction of errors, without them becoming a global fiasco.
Design for flexibility then: separate APIs from implementations. Version your APIs so when they're lacking you can produce a new back-compatible version. Don't know all the details about every possible kind of device? Gee, throw in an open-ended IOCTL into the device control API. Refine IOCTLs for similar devices later, when we figure out what they need besides the basics.
The point is that it is possible to design adaptable and refinable systems in order to accomodate the inevitable "opps" with a fix that is local and not global in nature. Now, you can't be flexible in everything and sometimes correcting things hurts: witness the Linux VM. It wasn't really planned to abstract it's API away to allow for interchangable plug-ins, was it. And the VM wars were somewhat painful precisely because one had to chose and couldn't punt.
Nevertheless, experienced software designers try to provide an "out" whenever they can, and think that a particular course might require modification in the future.
You could've hired me.
what does survival mean in software???
--exactly what it means in biology. things that survive from a biological point of view are necessarily good or better. sometimes they are. sometimes they're not. humans survived because they were able to overcome certain hardships created by the world. But i dont exactly admire humans; if you read Ishmael by good old mr. Quinn, he clearly (as do I) dislike human nature, despite the fact that we can't avoid it. humans do the exact same thing that microsoft does: they kill everything around them, and take more than they need.
There was one thing that Bill Gates did not foresee: the advent of a FREE os...something that he could not counter. the human race (analogous to M$) has killed everything, and eventually there will come a species that can not be killed off (in my opinion this will be the sentiet AI that I, err...i mean people will create). However, until there comes along something analogous to linux, humans will continue to dominate.
QED
BSD is for people who love UNIX. Linux is for those who hate Microsoft.
If you want to see a system that was more thoroughly _designed_, you
should probably point not to Dennis and Ken, but to systems like L4 and
Plan-9, and people like Jochen Liedtk and Rob Pike.
And notice how they aren't all that popular or well known? "Design" is
like a religion - too much of it makes you inflexibly and unpopular.
I hardly think that plan9's unpopularity is down to that fact that's it's been well designed!
working in it is a joy. It suffers from lack of a good web browser (not exactly a small undertaking) and 23 char filenames (wave bye bye to those ream soon now [tm])
but I guess not everyone likes design. I'm sure more ppl reading this are in an untidy hell hole of a room. If you've not got some dirty crockery in reaching distance of you then I doff my hat to you.
but good design brings pleasure, and working with plan9 brings more joy than frustration.
linux is winning not because it's a great piece of software but rather one of those historical flukes of the right place at the right time and captured people's imagination. Feeding my pc with my first slackware floppy disk set was liberating and discovering the joy of hitting co-operate rather than default has justly brought it's reward.
but hey, come on, keep your mind open. there's always a spare pc lying around, spend an evening with somethign else for a change.
http://plan9.bell-labs.com/plan9
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
I find it disappointing that he so smugly suggests there is no God.
Where did he do this? In the quote you posted, all that I see is that he has confidence in the theory of evolution. If you are suggesting that this process was guided by "the hand of God," then you are talking about something that is different, namely a hybrid of creation "science" and evolution. Just because he cannot see the hand of God at work, he reasons it is not there. How arrogant is that?
I don't know what Linus' stance is on theistic beliefs, but it seems to be you are misrepresenting his statements. How arrogant is that?
-
I understand Linus to be saying that he didn't forsee things like the iPaq or the OS/390 port when he started his terminal program in '91. That's strategic. That doesn't preclude him from designing an API or a data structure. That's tactical.
how to invest, a novice's guide
I didn't read the whole thread rant with Linus et. al. - but from my own experience and observation EVERY successful project mixes both initial design and evolution in design AND implementation. If you fix the design absolutely up front at both the macro level AND of every sub-system in a large project, you will invariably run into huge roadblocks at some point. Something will not work as planned. As I see the Linux Bazaar process, it reaps benefits when this happens - some person or organization stumbles into a roadblock with poor networking code, poor SCSI subsystem behavior at high loads, or an unreliable VM. These emergent behaviors may only affect some small portion of the user base - but the subsystems then enter an evolutionary phase where people varyingly fix what's there or design something new, and some design ends up surviving based on what the most people seem to like and want and in the end, if all else fails, what Linus dictates.
So no, this isn't strict "evolution" after the style of Darwin. If we let purely random decisions drive software and forked every few minutes, the analogy would be pretty complete. It would also take as long to write good software as it does to evolve a well adapted creature. An eternity.
I see where the idea of selective breeding comes in - Linus sees himself and the kernel leading guns as picking and choosing the best patches and suggestions. Up to a point, this means they are exercising design and discretion, but they generally don't "assign" work from their central database of TODO tasks to IBM, Red Hat, and other individuals or organizations participating in kernel development - those organizations and individuals scratch their own itches and their work usually finds its way back into the kernel. Other posters accurately said that a more random evolution could be effected by letting people check in free-for-all into CVS. This is true, but I don't think that would necessarily improve the results and timeline of kernel development.
You have to realize that the comparison here is, as others pointed out, to a monolithic software development process - in the Cathedral, a centralized decision is made - "we are going to make Windows NT better able to support large enterprise database deployments" and a team is assigned to break it down and work through all the implications, then implement. In Linux-land, the interested parties don't call to schmooze with MS biz dev people who pass info down to technical guiding councils, they pony up and write their own patches to the subsystems they see that need improvement. If there are enough interest parties, presumably enough patches will get submitted that the best from all get incorporated into the set of relevant subsystems that effect large enterprise database deployment, and we end up with a Linux kernel that supports exactly that. Of course the primary difference is that at the same time, somebody else may have made complementary and/or conflicting changes to make Linux a better desktop OS. Chaos ensues and flames erupt on kernel-dev and wonderously, eventually, something better for everyone results after compromises are made.
I agree with Linus.. projects that I've spent several years on came out at the end with features and design elements I could never have predicted going in. I've spent 6 months doing design work on pen and paper at the start of a project, and during the years of implementation thereafter, far more 'design' was done by reacting to the state of the code in any given moment and the problems it was having both internally and with regard to the userbase. My biggest project has evolved tremendously, even though I was essentially the only coder working on it for most of its existence. I can't imagine, then, how much less 'designed' by any individual the linux kernel must be, with the hundreds or thousands of developers contributing to it.
On the topic of Sun's doom, I understand why he says that. Sun's software is co-evolved with their hardware, but neither change very quickly. Linux has to cope with a much more wild, much more genetically diverse hardware base, and as a result it tends to move faster to support new types of devices. Solaris on Intel is a joke compare with Linux on Intel in terms of its hardware support.
Of course, there is nothing magical about a process that allows more evolutionary freedom.. if the hackers working on it don't have the good sense to be effective natural selectors and mutators, then the process won't have a terrific outcome. Linux is thriving because it has so darn many hackers working on it, and because it has so very, very many users using it, and because Linus has a deep and proper understanding of both good taste and evolution.
- jon
Ganymede, a GPL'ed metadirectory for UNIX
I don't think I agree with the full analogy. Obviously Code is not randomly generated or selected for mutation. We use intelligence to know what's wrong, then we use knowledge to improve. What really evolves in good software is the design, not the code. Sometimes you have to start from scratch again to implement design changes.
We could probably design a new, better human, but sheer evolution will _NEVER_ result in perfection. Design can perfect many small peices of code. Combining these smaller pieces, one can achieve near perfection in a lot less time than sheer luck. Evolution produces local minima's, where design can find the absolute minimum error, and move toward it much quicker. (Think if multiple layer perceptron networks.)
Karma Clown
The challenge is to come to terms with the fact that the bulk of humanity will never see the deep truth in what he is saying.
The mythology of design is pervasive but just plain wrong. Design only ever happens in marginal increments. Quotes about standing on the shoulders of giants come to mind.
A deeper challenge is that most people are incapable of understanding evolution, not because of any lack of inherent intelligence but because they haven't ever gotten out of the comfort zone.
An interesting but neglected mid-80s paper by Marcia Salner, then at the Saybrook Institute in San Francisco, but now at the University of Illinois at Springfield, as I see following a Google search I now need to spend some time following up on, pointed out that it helps greatly to have got through some genuine crises, firstly to break our naive and seductive faith in the universality of right and wrong answers and secondly to force us to look beyond the naive relativism which first replaces the right-wrong dichotomy.
Evolution, be it biological, social, technical or whatever, is about what works in practice, and even more so about the uses made of its products, because evolution does not happen in a vacuum. (Yes I am using "vacuum" metaphorically. The real vacuum of 3D space is also highly evolved.)
Now I find myself caught up with the even deeper challenge that if too many people actually believed what Linus is saying that the whole system would collapse. It seems only possible to build viable social institutions on rhetoric that does not stand scrutiny.
-- Our systemic servants do not good masters make.
When I was in college, I read the book Software Tools in Pascal by Kernighan and Plauger. The most valuable thing I learned in college was the system of design set forth in that book, which the authors called "left-corner design".
The idea is simple: when creating a program, start with the most important thing the program needs to do. Once you have that working, add more features. Ideally, as you go, you should be releasing working versions to whoever will be using your program.
This is so right in so many ways. For one thing, if you run out of time during a project, at least you have something you can release, and it may very well do much of what the users need. (There is a line in the book to the effect of "80% of the problem solved now is better than 100% solved later.") Also, early feedback from the users can show you what's wrong with your design, before you write a whole bunch of code that you would later have had to rip out. (I seem to recall an example in the book where a large system spec turned out to be totally wrong; the users didn't know what they wanted until they had something to play with.)
I never before noticed that the standard open-source development techniques match up with the left-corner methodology. Open-source projects such as Linux are all about "release early and often".
When I read Linus's comments, I was nodding my head all over the place. You create some code that solves some problem, possibly not very well. You release it. Feedback and patches start to arrive, and the code grows, possibly in directions you never foresaw. The more popular the code gets, the more robust it gets, as people patch it to work in a wide variety of situations and on a wide variety of hardware. This is why Linux has come so far, so fast.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely