Byte Benchmarks Various Linux Trees

urbanjunkie writes: "Moshe Bar has an interesting article, essentially benchmarking the standard kernel (with aa VM) against the -ac kernel (with Rik's VM)." He also raises some very interesting points about how patches (and entire development trees) interact.

Comments

(with a VM)

[d-ude]

of course they wouldn't actually benchmark each kernel and distro separately....too much work i guess

Re:(with a VM)

[Anonymous Coward]

d-ude, that's aa vm, as in that weird foreign guy's vm

Re:(with a VM)

[Anonymous Coward]

What's even stranger:

aa VM is in another Strange Foreigners' kernel.

Mysterys yet unsolved.

Re:(with a VM)

[jo42]

Would have of been nice to see FreeBSD in that comparison...

Interesting conclusion...

[NOT-2-QUICK]

The author came to the quite informed conclusion of:

I prefer the 2.4.17 or 2.4.18pre2aa for my heavy-duty servers. The reverse-mapping patch by Rik, however, has great promise once it has stabilized. Finally, the Red Hat 2.4.9 is a very good kernel, fast and reliable.

While I personally may not have agreed with this synopsis prior to reading the article (and am still not completely sold...), there are certainly some interesting facts and figures to ponder the next time you reload your system or update your kernel...

Re:Interesting conclusion...

[peripatetic_bum]

What a nice article and it seems to come at a time just then everyone is talking about "Fork this and Fork that" that in fact this is exactly what is needed in this healthy debate.

I think perhaps that we should start having "kernel" races if you will, and we could have various categories (ie, I/O races, Stress test races) in which the various trees would compete and by-and-by the best kernel trees would become known.

Please, I would like to hear your comments!

Re:Interesting conclusion...

[utdpenguin]

>>I think perhaps that we should start having "kernel" races

Correct me if IM wrong, but dont races in the kernel cuase horrible security problems?? And who would decide the race conditions?

Re:Interesting conclusion...

[Anonymous Coward]

Unfortunately, the fastest code is rarely the most maintainable or extensible.

Re:Interesting conclusion...

[peripatetic_bum]

Mmmmm, not a bad point, I see what you mean is to say that if we really put kernel tress head to head in preformance, then the developers will ingore security issues? not bad, could be. I need more data from you about that

As for who would decide the race conditions, Im sure that woulnd tbe too hard to come up, of course the obvious downpart would be if the racers themselves choose the race conditions but i think that could be solved no?

Re:Interesting conclusion...

[Rik van Riel]

What a nice article and it seems to come at a time just then everyone is talking about "Fork this and Fork that" that in fact this is exactly what is needed in this healthy debate.

Indeed, forks are (IMHO) the best way of doing development. Doing your development in the main kernel will just lead to contradictory code being integrated and the code never working quite right because it's missing fixes (guess why RH's 2.4.9 runs faster ... it does have the fixes).

One minor nitpick though ... I never released an -rmap VM against 2.4.18-pre3, the latest is still against 2.4.17. I suspect that the crashes Moshe saw are due to some change in 2.4.18-pre3 conflicting with the -rmap VM patch, especially since rmap-11c has survived the kernel torture lab at RH. ;)

Re:Interesting conclusion...

[NOT-2-QUICK]

What you are proposing is a very fascinating concept...essentially, a detailed comparison/benching of various metrics of a kernel's performance performed by an objective third party on a semi-frequent basis (at least, I believe that is where you were headed...).

Immediately, I see a great deal of benefit that could be derived from such a venture for the common Linux user. This information could not only render which kernel is the "fastest", but could also provide information on which kernel design is best optimized for a given task. Effectively, we would have "Open Information on Open Source" (TM).

Alternatively, however, I would never want to see this go to the extreme of a kernel tree being abandoned nor neglected as a result of these "Kernel Olympics" - variety is the spice of life, and IMHO the strength and diversity that drives Open Source!!!

Re:Interesting conclusion...

[koekepeer]

Phew.... a good thing that the thing performs as well as you claim it does...

[quote]
while Rik restricted himself to question whether the standard Linux kernel would ever even finish a stress test
[/quote]

Of course I'm far from serious with this remark, I'm glad to see a Dutchman "die graag zijn kop boven het maaiveld uitsteekt" (let's see how google translates that ;-).

PS. Actually, this is a test to verify whether our average /. moderator has the sense of humor not to moderate me into oblivion. :-P

Re:Interesting conclusion...

[Anonymous Coward]

Google doesn't translate Dutch, however InterTran gives us :

"who willingly one's pate upstairs the mowing field uitsteekt"

I'm guessing it kinda looses something in the translation...

Re:Interesting conclusion...

[Anonymous Coward]

Joke: ->

Your head: ->0

Get the picture?

Re:Interesting conclusion...

[whovian]

After playing with the alternative translation option, I would translate this for us Americans as one who is willing to stick his neck out (note: uit = /out/).

Re:Interesting conclusion...

[dash2]

What a nice article and it seems to come at a time just then everyone is talking about "Fork this and Fork that"

You mean like

Fork this and fork that
Fork it all and fork the f***ing brat

?

(I don't want a kernel that swaps like that
She don't want a kernel that swaps like that...)

Re:Interesting conclusion...

[Anonymous Coward]

Good one!

Re:Interesting conclusion...

[Anonymous Coward]

everyone is talking about "Fork this and Fork that"

Fork you motherforker!

Re:Interesting conclusion...

[koekepeer]

that would be an appropriate translation

(-1: Offtopic) :-P

Pournelle ?

[nick-less]

[from the article...]
> As Jerry Pournelle pointed

I always though he's a sience-fiction author?

---

Re:Pournelle ?

[netringer]

I always thought he's a science-fiction author?

He is.

He also is(was?) the author of the "User's Column" in BYTE magazine.

I haven't seen nor heard of him in the last decade, although I did talk to him on the phone once. (I did tech support and called him back.)

Re:Pournelle ?

[JabberWokky]

I haven't seen nor heard of him in the last decade

He's still writing - good classic Science Fiction, and still writing Chaos Manor, his collumn for Byte. He's also got a site that is similar to Slashdot in "umm... vaguely geeky stuff, lots of visitor feedback and opinionated as hell". It is located (in all its hellishly difficult to navigate, 'spend a few hours finding the good stuff buried deeply' glory) here.

--
Evan "Many many years spent with Byte" E.

Re:Pournelle ?

[markmoss]

Pournelle is a science fiction writer and also a computer columnist for Byte magazine (byte.com now). I think his original degree was in political science (!) but somewhere he learned quite a lot about engineering, and he did the original (DOS/BASIC) programming for his wife's learning to read program. I assume his computer skills are self-taught -- he's got to be at least 60 and there were no programming classes when he was young. He wrote political columns for Intellectual Capital a few years ago, when that website was worth looking at. In the 60's he was something in NASA management -- and also he appeared at least once on national TV in a debate about the Vietnam War.

Re:Pournelle ?

[mkelley]

Not a lot of info, but in one of his BYTE columns, he talked about getting free time on an old Honeywell or IBM mainframe and that is where he learned to program.

Re:Pournelle ?

[(outer-limits)]

I once tried to read one of Jerry Pournelles books, and found it to be totally inane, guilleles, pontificating and absurd, and that was just the first three pages. It's basic premise was that earth was in a new ice age and everyone was freezing to death, but the greenies were still banning the use of hydrocarbons to warm the earth up using the greenhouse effect. Talk about trying to have your cake and eat it too! One the one hand he is saying that the greenhouse effect is real because it would solve the problem of the ice age, on the other he is saying the greenies are to blame. Now, at present, we have the real possibility, despite all the debate, of a greenhouse effect induced heating of the earth, and no indication of an ice age. He takes this concept and turns it on its head.

No wonder his computers never work. I read his column every month just to have a good laugh at how his latest freebie has shat itself because he doesn't know how to set it up, and then gets some poor support guy from the company that gave it to him, (in between plugging his wifes reading program and his sons latest business venture), to get out to his place over the weekend to fix it up.

At the same time, he never stops praising the free enterprise shambles that can't give him broadband in the middle of Hollywood.

Then he signs off on his latest friends book about how the white people are better than non-whites, and how America should just nuke the rest of the world, after it has finished up using all the oil that is there.

Re:Pournelle ?

[Anonymous Coward]

You left out the fact that's he's a Unix-hater and Microsoft Shill. As I recall, Pournelle was one of Microsoft's prime flacks with the "NT will kill Unix" BS that made Byte and other PC mags so unreadable during the mid-late 90's.

Re:Pournelle ?

[Anonymous Coward]

HO HO Ho, What a maroon! Your so stupid. The book was a joke, dummy. Of course it also pissed off all the anti-tech types. You should stop reading SF, maybe romances are more your speed. The purpose of SF is to ask "What if?", not to predict the future. Oh, BTW there is every indication that an Ice Age is possible and that it could appear with amazing rapidity. Meteoroligists are speculating that several different possitive feedback mechinisms are involved in creating Ice Ages. All in all, Niven and Pournelle came up with a very funny premise.

Re:Pournelle ?

[(outer-limits)]

How can I be a moron when you can't even spell the word. The book was not a joke. A joke SF book is something like Hitch Hikers Guide. It was written in an incredibly artless and clumsy style, (Probably why it was on the remainders table). I love SF, which is why I find Pournelle to be so hopeless, he writes as if it is the future, etc, and everyone thinks just like him. His characters sound like Jerry having a conversation with himself.

Maybe an Ice Age is possible, but you won't find the Greenies opposing dealing with a disaster, they just don't feel like needlessly creating one. And that'll be the day greenies actually get in a position of power like he outlines. The middle class love buying bigger, faster, more powerful cars to sit in traffic jams too much to ever give up their dreams of aquiring happiness through believing ads on TV.

I have read plenty of better SF,

Re:Pournelle ?

[markmoss]

You left out the fact that's he's a Unix-hater and Microsoft Shill. Not hardly. Your prejudices are showing.

He did try out and write mainly about stuff that he expected most of his readers would be using -- which is to say, DOS & Windows up until two or three years ago -- but unlike some at Byte, he was always well aware that there were OS's out there other than MS's and Apple's. He has added some Linux systems to Chaos Manor now.

It might do you *nix zealots good if you can set your prejudices aside and actually read Pournelle's comments about why he is still using Windows & MS Word for most work. But too many of you have utterly closed your minds to any discussion of usability by people who just want to get some work done on the computer, rather than spending their time learning weirdly-named text commands... Do you remember that recent slashdot discussion of how Linux gurus will tell people to do basic tasks with difficult text commands, even though they are talking about a specific distro with a GUI that does the job much easier.

Re:Pournelle ?

[Anonymous Coward]

Oh boy, you are a dummy. The "maroon" was a joke
too, a Buggs Bunny quote. Your just angry that Pournelle paint your average tree huggers as the idiots they are, you included. Ever comment of yours diggs you in deeper. Doug Adams wrote comedic farse not jokes. Gee not only do you not understand SF, but you dont know what a joke is. Larry Niven and Jerry Pournelle wrote it as an joke for hard core SF types ( my roommate, at the time, interviewed Niven shortly befor the book was published). This is a fact. Most readers and reviewers understood that it was a joke. As for the envirowackos actions in the book, there is volumnous evidence. Can you say Cassini?

Re:Pournelle ?

[Anonymous Coward]

I remember him fondly because he was
using OS/2 around the same time I
was. However, he always had this
silly axe to grind about SMP
"no one wants to share a processor".
Lucifer's Hammer is a pretty good SF
book he did on asteroid impact in
the 1970s. I don't really consider
him to be very visionary anymore but
he seems to be a nice person.

Re:Pournelle ?

[(outer-limits)]

It is just hard to pick out what are your jokes from all your other spelling mistakes. I was reading SF probably long before you were born, and I get it much better than you. Jerry Pournelle writes turgid prose, is a rascist, inward looking moron. The early byte columns were interesting only because of his friend, who died pretty early on. He takes freebies on the basis that he flogs them to his readers, shamelessly advertises his families' business ventures, thinks the only country in the world that matters is the US, nearly gets himself killed driving alone in a desert without water or supplies, thinks that all the US has to do is blast a few enemies to achieve peace, endlessly praises technology that causes him so much trouble that it will probably send him to an early grave, and is a friend of the loony right. One of the funniest pieces I ever read of his was when he had that lunatic 'newt' gingrich around to a hotel room, and spent about four hours trying to connect to the internet over a modem to demonstrate it to the guy. You could just read between the lines that newt was not a happy guy, and thought jerry was an idiot. Now that was a funny piece. The fact is that greenies are not am amorphous mass of indistinguishable fools. Some are, but many aren't. The fact is, the world is being subject to massive, rapid change, the consequences of which are poorly understood, and for which many feel no responsibility.

[ot] So what's the best kernel to get right now?

[wrinkledshirt]

Sorry for the dumb, offtopic questions.

I'm sitting at home with my fresh install of RH 7.1 and I'm wondering what kernel to upgrade it to. Any suggestions? Is there a stable one in there somewhere that I should go with? Should I stick with the default kernel that's on their now?

If I'm regularly compiling new programs using gcc or g++, is it safe to go from one tree to another, as long as they're all 2.4.x, or what? Do I need to recompile with a new kernel? Or is that a red herring?

The folly of Open Source

[Anonymous Coward]

Open Source has shown itself to be an effective strategy for the implementation of small to medium sized projects. Many eyes, blah blah blah, and the project leader makes the final decision on what goes in. It's very cool.

However, as was discussed on /. before, when projects pass the "medium sized" mark, the effectiveness of having such an open system becomes self-defeating. It's simply a case of too many people having too many different ideas about the direction the project should take.

With closed source, a single person can make all the final decisions about what will make it into the final release, but Open Source has no such system. A person who feels that his ideas are not being taken seriously can fork the project and create his own possibly incompatible tree. This seems to be what is happening now with Linux.

At the early stages of tree forking, there is usually little problem. Most of the forks coexist with each other peacefully with only minor manual tweaking to fix any conflicts. Over time, though, further development along one of the forks will inherently diverge further from the root tree.

The argument that good patches will make it back into the root tree, thus preventing fairly deep splitting, is fallacious. Many of the reasons for forking are precisely because the root project refuses to take a fix from a contributor. What happens afterwards is that users will get 'locked into' a particular strain of the project and will be unable to incorporate other good features that are present in other strains which, for whatever reason, can't be merged into their chosen strain.

Kernel hacking just isn't as fun as it used to be. :-(

Re:The folly of Open Source

[Junta]

I don't think this is a bad thing, really. People have conflicted ideas so two forks form. Each fork is evaluated in the eyes of the hardcore users and in the eyes of distributors such as RedHat. Sure, we look now and think that things will get ugly as all these trees evolve, but as the past has shown, the trees that work better, or are closer to the pre-exsiting "standard" given a tie, dominate and often extinguishes less succeful trees whose ideas didn't pan out, usually folding in the improvements the other tree made succcessful.

The process of having these trees sprout up and played with by businesses and advanced users provide a sort of natural selection for improved evolution of the system. While all this complicated mess plays itself out among the bleeding edge people, the common users have the stock redhat or what have you kernel to play it safe with.

Re:The folly of Open Source

[Hagmonk]

Most of the 'forks' seem to actually be marshalling grounds for maintainers to prove the reliability of their work to the kernel community in general. Once things gain some faith in the community and in Linus' head, they get incorporated in the main tree. The process needs work, but it has gotten the community this far ...

Can you provide some high profile forking examples that have occured in the recent history of Open Source, out of interest? Or evidence of similar forks in Linux. Not, to coin a phrase(?), "soft forks" as I have mentioned here, but "hard forks". Fundemental changes in ideology or personality clashes that have seriously caused a split in the community?

Re:The folly of Open Source

[Anonymous Coward]

Can you provide some high profile forking examples that have occured in the recent history of Open Source, out of interest?

Take a look at how many distros for Linux exist, as a straightforward example of forking. I'm talking about RedHat vs. SuSE vs. Mandrake vs. Debian ad infinitum.

Re:The folly of Open Source

[Hagmonk]

That's not forking, that's diversity. There is a massive amount of compatibility between those Linux distros. If one of the distros dumped gcc as a compiler and tweaked all of the Linux system calls - that's a hard fork.

Re:The folly of Open Source

[Anonymous Coward]

You can redefine fork however you want, I'm not going to go out of my way to find the one fork that fits all your criteria of what a 'hard' fork should be.

There is a massive amount of compatibility between those Linux distros.

The OP says exactly that early forks will be largely compatible with each other.

Re:The folly of Open Source

[Hagmonk]

I will eat my hat if Linux distros like Mandrake, RedHat and SuSE hard fork.

The difference between the fork types is easy to comprehend. A soft fork remains loyal to the original code and strives for compatibility. The hard fork throws compatibility out the window, and serves its own goals completely.

All distros that I know of would fall into the soft fork category. Nobody will strike out on their own, unless they have an army of hackers behind them to maintain the momentum required to keep the project running.

That's also why the Linux kernel will not hard fork in the forseeable future. There would have to be grassroots dissent to encourage enough hackers to jump on the new fork and maintain it.

Re:The folly of Open Source

[Anonymous Coward]

There would have to be grassroots dissent to encourage enough hackers to jump on the new fork and maintain it.

So how exactly did Rob Landey's L-K thread end up?

A quick google for "patch penguin" finds articles from LWN.com, OSNews.com, ZDNet[UK], ZDNet[India], and CNET, all within the first 10 matches.

Re:The folly of Open Source

[Hagmonk]

The media loves to make a big deal out of things like that. In actual fact, the discussion was fairly amicable. And Linus' stance seems to be supported by the maintainers of the various subsystems.

All Rob was suggesting was a formalisation of a de facto practise. Hardly a fundamental paradigm shift in kernel development.

It did unearth a number of gripes people had; eg. Linus dropping maintainer patches. The thrust seems to be that Linus wants to trust ten or so people, and only accept patches from them. So if Linus is dropping your patches, push them on to a 'trusted' maintainer instead.

IMHO Linus needs to step forward and deal with this a little more assertively. But everyone loves a good bit of controversy, and the Linux kernel is the tall poppy of the open-source community that everyone wants to chop down ...

Re:The folly of Open Source

[Anonymous Coward]

OpenBSD. Need I say more? Or is that to far back in history?

John-Mark Gurney

Re:The folly of Open Source

[darkonc]

Conflicting ideas are not the only cause of a fork. Conflicting needs can make a strong argument for forking. The various realtime forks are a prime example here.

Keeping the realtime code in the main code tree would be counter productive -- confusing the development of the larger Linux tree, but not providing much of anything to the 98% of us who have little need for hard realtime in our systems.

Re:The folly of Open Source

[jo42]

Flamebait? Moderator is royal wanker.

Read this and tell me that the Linux kernal isn't a rancid ball of mud.

Re:what's the point?

[ekrout]

I won't disagree, but which is stronger as a "desktop OS"?

That's where Linux shines compared to the *BSDs. Sure, use it on your headless servers, but for a b0x at home, most simply prefer Linux.

Not a red herring

[Dead Penis Bird]

More like a Red Hat.

hmm

[Zanek]

I'm waiting for a standardized procedure to allow downloads of the latest updates and patches,ie: just like MS.

Re:hmm

[DeadeyeFlint]

If you have RedHat use up2date, with Debian use apt-get.

Re:hmm

[Anonymous Coward]

Ok... I'll feed this troll..

go to redhat, and look there's this funny little program that almost nobody knows about. It's called up2date. and it does what MS does but better, faster, and easier. Really they should actually tell someone about this but they keep it secret.... and almost noone knows about it...

sad eh?

This sarcasim brought to you by the number 3.14579

Red Hat 2.4.9 is a very good kernel, fast...WHY?

[srw]

Can anyone explain this? Was the stock 2.4.9 faster and more reliable than our current stable kernel? If there are stability and speed patches in the RH kernel, why haven't they been adopted in the standard kernel? How close is RHs 2.4.9 to Alan Cox's kernel? I'm assuming he has a strong influence on RHs kernel.

Riels rmap is nice......

[CDWert]

The rmap patch is nice.....

It makes a big difference on loading the hell out of my woefully under memoried workstation at work. My home machine, used mostly for surfing has seen a dramatic imporvment in free memory and is no longer swappin , it has 512 meg, so it really shouldnt swap too much, but was constantly with the stock redhat kernel, as well as 2.4.17 plain vanilla, trhe 2.4.18 -rpma12a has been ROCK solid.

On my server however with 256 megs , the stock redhat roll has done nicley with the minimal load its under.

I am a little leary about using the rmap in prouction as of yet, it seems to be killing things each nigh, (no shit) that dont drop with 2.4.17 or 2.4.9

I would like to see an option at configure to select a VM, I think the preemptive added would be fun too, I know its a pain because of the way it all intergrates to the other code, but thats my desire, it seems to be alot of other peoples desire as well, its funny how I bitched and moaned about the Riel VM , that was in the kernel prior to AA's , but since then and all the patching that was done I think Riels would give it a run for its money .....

Re:Riels rmap is nice......

[Rik van Riel]

I am a little leary about using the rmap in prouction as of yet, it seems to be killing things each nigh, (no shit) that dont drop with 2.4.17 or 2.4.9

Interesting. I've not managed to run into bugs like that on my computers here, so you must be running a very different workload to trigger such a bug.

Would you have the time to help me debug this problem and is it still happening with the latest rmap VM ?

Re:Riels rmap is nice......

[Anonymous Coward]

Oh god you have only 512 meg of ram? what are you stupid or a freak? everyone knows you need 4.7 terabytes of ram to run, and windows XP uses a pentabyte.

sheesh....

Listen if you cant run at 128 meg then you either are doing hefty science research or have a horribly botched install.. NOBODY needs more than 512 meg of ram except for high level researchers. Hell you can run a linux redhat install WITHOUt a swap space fine in 256 meg of ram.
yes, you heard me... NO SWAP SPACE... it works fine :-)

Re:Riels rmap is nice......

[Anonymous Coward]

> My home machine, used mostly for surfing has seen a dramatic imporvment in free memory and is no longer swappin , it has 512 meg, so it really shouldnt swap too much, but was constantly with the stock redhat kernel, as well as 2.4.17 plain vanilla, trhe 2.4.18 -rpma12a has been ROCK solid.

?? I'm running 2.4.17 at home (Debian-based system), 128Mb, and it *never* swaps. What do you do (above/beyond the surfing) that 512Mb isn't enough?..

Re:Riels rmap is nice......

[Anonymous Coward]

What about the inclusion of the rmap patches against the Red Hat rawhide kernel? Have you been working with the red hat folks directly or have they just been putting it together following your patches?
They also have :

the low-latency patches,
ingo's O(1) patch
and 2.4.18pre7

among others.

Also has anyone else tested rawhide 2.4.17-0.12 or 2.4.17-0.13 (13 was just recently released)?

Re:Riels rmap is nice......

[fsck!]

I love the smell of cooperation in the morning. It smells like... victory.

Re:Red Hat 2.4.9 is a very good kernel, fast...WHY

[keeg]

2.4.9 was the last official kernel from linus which used Rik van Riel's VM which was introduced in 2.3.x. (The switch to Andrea Arcangelis VM occured in 2.4.9->2.4.10) Alan Cox and Red Hat used this in their kernels, and the Red Hat kernel was heavily patched with the patches from Rik van Riel which Linus "reportedly" dropped (among other things). The Red Hat kernel is also _very_ well tested, as all their kernels are. You might not like their distro, but their kernels are usually among the more stable.

Re:Red Hat 2.4.9 is a very good kernel, fast...WHY

[Rik van Riel]

Thanks to Alan Cox, Red Hat (and most Linux distributions) do have the patches for my VM that Linus didn't have the time for.

Stable kernel?

[sitturat]

It is hard to believe that all this is going on with what is meant to be a stable kernel version, ie 2.4.x

So far the VM has been replaced twice, and now the rmap patch is apparently going to be added despite the fact that "something is seriously messed up in the reverse-map implementation".

Have they saved any experimental code for the 2.5.x kernels, or will that now be stable?

Re:Stable kernel?

[Rik van Riel]

It is hard to believe that all this is going on with what is meant to be a stable kernel version, ie 2.4.x

So far the VM has been replaced twice, and now the rmap patch is apparently going to be added despite the fact that "something is seriously messed up in the reverse-map implementation".

Ummmm, -rmap is still under development. If there are any plans to put it into 2.4.x, people sure haven't told me about them. ;)))

(and personally, I'd prefer to keep -rmap separate for quite a while more ... development is much more efficient in a fork)

Re:Stable kernel?

[Tack]

The only reason I am replying to this is that I accidentally modded you down (Overrated) when I meant to mod you up (underrated). So because there is no "undo moderate" feature I am forsaking moderating this discussion in order to fix my mistake. :)

Sheepishly,
Jason.

Re:Stable kernel?

[SilentChris]

Yes, let us all bow down and forgive our sins.

Sad.

Re:Stable kernel?

[liquidsin]

That "van Riel" guy is a total karma whore. Everything he's posted is modded "5 - informative". He must have like 90 points just from this one discussion! What's up with that? ;)

Re:Stable kernel?

[Anonymous Coward]

Huh?

Sad.

Re:Stable kernel?

[puetzk]

he wrote the rmap VM, so he's definitely the right guy to answer a lot of questions about it

doesn't answer why he's getting modded to high heaven tho :-)

Re:Stable kernel?

[WNight]

"Stable" doesn't necessarily mean what you think. It refers to the APIs. All releases in 2.4 are supposed to be compatible, but 2.5 is allowed to do things that break code between versions.

Now, by keeping the interfaces stable, they usually make the kernels more stable (in the uptime sense) as well, but that's not exactly the goal.

If you blindly upgrade to the latest kernel, you will be bitten. It'd be like grabbing Mozilla nightlies, or if MS released IE nightly builds, or whatever. There will be a few stinkers.

Frankly, if you're the type to complain when a kernel is buggy instead of sending in bug reports, you probably should be at least two full releases behind the latest. (When 2.4.17 comes out, upgrade to 2.4.15) The exception being when there's been a big hole found in a line of kernels. And if you need the stability, consider downgrading to the kernel before the problem one instead of upgrading to a less-tested one.

Re:Red Hat 2.4.9 is a very good kernel, fast...WHY

[LunaticLeo]

There was no stock 2.4.9 in the test; only RedHat's highly modified 2.4.9.

RH 2.4.9 is a lot like the ac kernels. Mainly because, Alan runs that part of the RH distro. But, there are many concerns that RedHat addresses. They work with corperate customers and partners (like Oracle) to make sure the kernel they ship is as stable and fast as they can get it. So the RH kernel does diverge from the "plain" AC kernel.

Of course they submit all their patches back to Linus. But Linus just hasn't been keeping up with them. Linux accepts patches based on three factors: his previous experience with the developer submitting the patch, the "correctness" of the patch, and the phase of the moon. And the phase of the moon is the dominant factor, because even Alan Cox complains that Linus won't accept his patches.

So the RH kernel is excellent because Alan Cox, RedHat, and RedHat's corperate partners make sure the kernel is fully fleshed out. This is the kind of vetting that Linus doesn't do.

"If it compiles it is Good, if it boots it is Perfect!" -Linus Torvals.

Why use an unstable patch?

[Hagmonk]

Most of those 'forks' are going to be maintained by kernel hackers to marshall patches for eventual inclusion in Linus' tree. I wouldn't put them anywhere near a production server.

There are various patches like the Robert Love's preempt patch which might be considered production quality. And perhaps some collections of production quality patches exist out there. But I wouldn't say -ac or -dj are in that category.

Or any of the patches marked 'preXYZ'. They're 'pre' for a reason you know. I'd be thrashing them on test servers, then giving feedback to the maintainer of that series. Let the maintainer declare them stable first.

You'll find in environments ambivalent to Linux that you really need to prove its stability to management first. Trying a new whiz bang kernel can have unforeseen side effects, in meetings that you'll never be invited to; and whose outcome you will only learn when it's too late to change it. "We let Bill convert server X to Linux and then it corrupted the filesystem. Clearly Linux carries more business risk than expected."

Why is this funny?

[megaduck]

C'mon guys. Show a little open-mindedness. One of the things I really missed from the Windows world when I switched to Linux was the "Windows Update" feature. Want to install the latest security or feature patches? Click a check box and hit "install". No dependencies, no patch conflicts, no esoteric config options, it just worked. Admittedly Ximian's Red Carpet comes close, but it's still a little quirky sometimes.

I know there will always be those people who want to manually tweak their kernel (god bless 'em!). There's a lot of us, though, that don't want to deal with it. I'd rather have one-click shopping for all of my patch needs so that I can spend more time writing code or playing Quake. MS understands this. Apple understands this. Why doesn't the Linux community understand?

Re:Why is this funny?

[reaper20]

Some of us do understand. I've got apt-get update and upgrade in my crontab, which emails the results to me every night at 12:00am.

That means 'Aunt Tilly' never even has to even update her box, ever.

Granted the typical 'desktop user' wouldn't know how to do this ... but it is possible to do - maybe an install time option for this would work.

Re:Why is this funny?

[kinnunen]

> Want to install the latest security or feature patches? Click a check box and hit "install".

That's bullshit. Latest security patches find their way to windows update several days, if not weeks, after their initial release. Windows Update is great service but unfortunately Microsoft uses it mainly for distributing the latest version of IE/MP/Messenger. I'm sure that one of the factors causing long delays is that I don't use an english language version of Windows, but my friedns who do say it isn't much better. I hope to see more security patches in Windows Update now that Microsoft has promised to take the matter seriously, we'll see how it goes.

And before anyone says this is just FUD from a linux fanboy let me say that the only OS in my PC is Windows 2000 and I have no plans to change that.

Re:Why is this funny?

[kaladorn]

I'd rather have one-click shopping for all of my patch needs so that I can spend more time writing code or playing Quake. MS understands this.

Oh good! 1 click to the latest MS Insecurity Patch!

I'm not inately against one click patching, but to suggest that Linux can't do it and Windows gets it right (seems to be what you're implying) while ignoring the side effects of some of MS more hysterical patches is somewhat like putting the cart before the horse. When MS can deliver security (of at least a reasonable degree) along with its patching capabilites and other neat ideas, then it'll be something impressive. Until then, it'll just be another Security Chasm waiting to be installed.

Re:Why is this funny?

[DrSkwid]

debian has it

freebsd has it

in fact both of those also upgrade third party software that's not part of the OS

windows update will never upgrade mozilla for you or KDE whereas apt-update & cvsup-portsupgrade upgrade EVERYTHING you've installed (if installed via apt-get or /usr/ports)

[my understanding of the debian process is through the grapevine so if I'm off the mark, be cool; not a fool]

Re:Why is this funny?

[megaduck]

I'm not saying that the *nix community doesn't have the technology to match Windows Update. My complaint is that to a newbie the freebsd/linux methods are non-intuitive and difficult in comparison. Both OS X and Windows have very easy update tools that apply the latest patches in three or four mouse clicks. While the ports or apt-get systems may be techically better, the ease of use still isn't there.

Re:Why is this funny?

[Anonymous Coward]

Latest security patches find their way to windows update several days, if not weeks, after their initial release.

Which is probably as it should be, right? Having done all your internal testing, first test your security patch among those external sites who care most about it (who read the mailing lists or check the security areas on the website manually), then, once a week (or 2?) goes by, assuming you've gotten no stability complaints or other f*ckups from the security patch, roll it out to the entire world via Windows Update. Seems like a sensible, moderately conservative release strategy to incrementally larger customer sets over time, not incompetence.

Now if it took over 2 weeks, I'd agree with you. Incompetence.

Re:Why is this funny?

[smcv]

When did your Windows box last upgrade its kernel, then?

So Windows Update consists of DLL upgrades (or shared library upgrades, to use the Linux terminology) - a lot like, say, an APT update on Debian, or whatever the equivalent's called on Red Hat.

If you want to be messing with custom-compiled kernels, that's your choice, and it's not as trivial as updating system libraries and programs. But then it's not a choice Windows even gives you, so I can't say I'm complaining :-) From what I've gathered about Red Hat, Windows has much the same approach as a stock install of RH, in fact - you get a standard, intensively-tested kernel which isn't exactly on the cutting edge, but is stable.

One thing Microsoft's strategy does have in its favour is that kernel-specific device drivers aren't a problem when all the copies of, say, Win2k have the same kernel. However, don't some Linux distros distribute packages of kernel modules compiled for their standard kernel (like the modified 2.4.9 kernel Red Hat use)?

Re:Why is this funny?

[Pastis]

Let's compare Windows update with another typical Linux update.

1. Windows 2000
2. 
   I installed windows 2000 on my new Inspiron 8100 yesterday. And I used the so cool 'Windows update' function.
   Note that it requires to use IE (No IE, no windows update? At least the help doesn't describe any command line method.)
   
   • First I installed the OS (one reboot - pretty normal)
   • Then I installed Service Pack 2 (another one)
   • Then a group of security fixes (reboot)
   • Then another (missing ???) security fix (reboot)
   • Then IE 6 (reboot)
   • Then a new security fix, probably for IE6 (reboot)
   • Then DirectX 8.1 (reboot)
   • Then new drivers (reboot)
   • I also modified the fonts to large fonts: reboot.
   
   Until I had the drivers installed, everything was done with IE 5.0, in 640x480 mode, and (I think) 8 bits; so ugly!
   And if you do that from home on a modem, you have to restart your connection at every reboot. That's optimum.
   And when this is finished, you can now start installing non-windows applications!
   
   I forgot to mention that the first time I installed windows 2000 on a logical partition, is wasn't able to boot it after I installed Linux. Impossible to repair as my disk was not detected correctly.
   [Perhaps because w2k doesn't support DMA 100.
   When I face this kind of problems I am pretty happy to be able to recompile a new kernel.]
   
   I completely reinstalled it on a new Primary partition and, after I installed the drivers, I had a crash (blue screen) during reboot. [Note I was updating the win modem driver while using it to download the driver from the web]. Anyway, it was impossible to restart (crashing even in FailSafe mode or with 'last known good configuration'), and impossible to repair even with an up-to-date ERD. Had to restart the install from scratch for the third time! Damn!
   
   
3. Debian 3.0
4. 
   
   On the other hand, I have installed Debian 3.0 (aka testing) from floppies on the same machine. I rebooted once during install, and then another once again after I installed and tweaked the new kernel.
   Of course, you are not obliged to compile the kernel. I bet that if you use Mandrake or Red Hat, you will have a pretty good hardware detection process. I just like Debian and like to tweak the kernel.
   I did everything on the command line. Didn't have to have a full graphical installation (I don't like too much laptop built-in mouses).
   

To summarize:
Windows: requires graphical environment and a mouse to install correctly plus requires having at least 9 non-necessary reboots (with modem disconnections /reconnections)to have an up-to-date system. Don't mention the un-repairable crash and the non-recognized partitions...

Debian GNU/Linux: two reboots (one optional). All in command line, with a 15 seconds boot. No problem. There are perhaps GUI front-end to apt, but I do not need them. Can something be simpler than 'apt-get upgrade'?

I let you decide who is the easier and who is my winner.

So to come back to the subject of the thread: download kernel and patches.

• Note that windows update only updates your Windows stuff, none of your other applications. (and without applications and OS is nothing, right?)
• Note that possible dependencies problem may arise after you upgrade non-Windows applications.
• Note that you can automate the install of the updates on Linux (cron the apt-get command). That's even better than on windows, where you can only be warned that you need to update your system.
• 
  
  • 
    
    I could go on...

Re:Why is this funny?

[DrSkwid]

now you're changing the subject :

I'd rather have one-click shopping for all of my patch needs so that I can spend more time writing code or playing Quake. MS understands this. Apple understands this. Why doesn't the Linux community understand?

is what you said, the Linux/FreeBSD community has addressed this problem.

And you don't even have to press anything ever again once you've started the process, no going to windows update, no clicking anything. I get an email once a week telling me what's been updated!

See the new 0.9.8 mozilla update today? My mozilla will self upgrade to that on the first Friday after it hits the ports. (Friday so I have sat & sun to sort it out if it goes wrong!).

How is windows update less painful than that?
Unless it's on MS' critical update list I won't even know there's an upgrade available without looking manually!

How often do you go your software vendor's homepage to see if there's a new version? That could be upwards of 20 web sites a week and most of the time it will be pointless.

I use software that isn't handily announced in News For Nerds (such as Xchat) and I can't be arsed to wade through freshmeat every week!

How much easier will it need to be to get your "ease of use" approval?

Interesting ?

[koekepeer]

If only there were options like (-1:Clueless), or perhaps (+1:Pretending). That would be an interesting intelligence test for your average /. reader ;-)

This posts' moderations:

(+1, Funny)(-1:Overrated)(-1:Overrated)(-1:Overrated)(- 1:Overrated)(-1:Overrated)

Re:Interesting ?

[Anonymous Coward]

In your .sig, it's SPEECH, not SPEACH.

Just a friendly reminder.

Re:Interesting ?

[Anonymous Coward]

grammar nazi going anonymous now?

I stand corrected

[sitturat]

oops!

Re:I stand corrected

[Anonymous Coward]

I stand corrected

Ahem.. most likely, You sit corrected.

Re:[ot] So what's the best kernel to get right now

[JediTrainer]

If you're using a fresh install, stop now.

You should use RH 7.2 instead. It comes with kernel 2.4.7 or something, but can (and should) be upgraded to RedHat's kernel 2.4.9 via up2date. The RedHat kernel is quite stable and fast.

Re:[ot] So what's the best kernel to get right now

[reaper20]

I'll probably get flamed for this - but I installed 2.4.17 from Red Hat's Rawhide distro (find it on rmpfind.net) and it worked like a champ.

rpm -Uvh kernel-2.4.17-athlon.rpm or whatever it was and rebooted. I'm sure everyone will chime in on how 'evil' it is to install a kernel by rpm, but for my typical desktop box, it's no big deal.

I think you'd need to get the new modutils package, but i don't recall.

Ummm no.

[Anonymous Coward]

The FreeBSD operating system
is way more mature than Linux 2.4 or Linux 2.5.

Thats amazing considering none of the 4.4BSD derivitives have existed as long as Linux. Nice try though.

Re:Ummm no.

[cbv]

Thats amazing considering none of the 4.4BSD derivitives have existed as long as Linux. Nice try though.

Following your way of thinking, Windows would have to be more mature than both, Linux and BSD....

I don't see how 'maturity' and 'time of existence' are possibly related.

Re:Ummm no.

[bovinewasteproduct]

Ok, lets see here...

Linux got its start somewhere around 1991.

The *BSDs are all based on 4.4BSD-Lite2 which is based on 4.4BSD. 4.4BSD has a history going back to 32V for the VAX (which you can now download as ancient UNIX from Caldera who released it under a BSD style license, see the slashdot article).

Even if you take Minix (I've still got the book) as an ancestor to Linux, I think the *BSDs are just a little bit older.

To tell you had bad it was, FreeBSD had to start all over again back in 1994 because a little bit of AT&T code (which had to exist prior to 1979) had crept into 4.4BSD-Lite1 which is why FreeBSD 2.0 was an almost total re-write from 1.0 (I was there).

Here is a question for you, what is the oldest copyright in the Linux tree? Can it beat the 1979 Copyright by The Regents of the University of California in FreeBSD? 1979 is about the time that they added such things a Virtual Memory and Job Control to Unix. Which were BSD firsts BTW.

It's about the uptime, stupid!

[Mullen]

A major problem with alot of linux admins is shown in the article. It's not about how fast your kernel is, especially when it comes to a 2 second difference in 50 seconds of computing time, but how long your machine will stay up.

If a user compiles 35 gigs of code on a 6 processor box and it takes 5 minutes longer, he is not going to complain. If he compiles 35 gigs of code on a 6 processor box and it crashes half-way through the compile, your going to here it from your boss.

Benchmarking kernels is plain pointless. Take a machine for each kernel, put it under real load and tell me how many times it crashes in 100 days, and I will you which kernel I want to use.

Re:It's about the uptime, stupid!

[Anonymous Coward]

someone moderate this up plz

Re:It's about the uptime, stupid!

[Anonymous Coward]

100 days? That's ridiculous. There aren't going to be any crashes in just 100 days.

The problem with modern OSes like Linux, is that by the time you have run the system long enough to know whether or not it's stable, your system is obsolete.

Re:It's about the uptime, stupid!

[Anonymous Coward]

your going to here it from your boss

Yes, you will certainly hear it from him, and he might even say, "You're fired!" Especially if you spell like this guy.

Re:It's about the uptime, stupid!

[AaronStJ]

A major problem with alot of linux admins is shown in the article. It's not about how fast your kernel is, especially when it comes to a 2 second difference in 50 seconds of computing time, but how long your machine will stay up.


It is about how fast your kernel is, if you want to use Linux on your Desktop. I, personally, don't , for a couple of reasons. One is, I don't have to fiddle with obscure text files every time I want to change some little thing. But the major reason is that when running it in a desktop enviroment (Ximian Gnome, the whole shebang), it's noticiably sluggish. Now from what I've gathered, this is partly due to the windowing system, etc., buty also largely due to VM issues and such like. If you want the average guy to run Linux on the desktop, you're going to have to design a system that doesn't feel sluggish. I hyave a fairly nimble computer. My windows should appear where I want them when I want them, not a tenth of a second later.

Oh, and testing how often a kernel crashes in 100 days? I sure hope your kernel doesn't crash at all is that long. Even my Win2k box can veat 100 days uptime easily.

Re:It's about the uptime, stupid!

[jandrese]

Interestingly, this is why I hate Win2k. Win2k seems to always want to swap background apps out, even if you have plent of ram available. This is beyond irritating when I minimize netscape to work in a text editor, then bring it back to the front, only to have to wait 2 minutes for the thing to swap back in (I'm on a laptop--slow HD).

A friend of mine actually forced Win2k down to the minimum amount of swap allowed (2MB IIRC) to avoid this problem.

In contrast, my FreeBSD box (my home desktop machine) feels much snappier (although I use Windowmaker and no desktop) and the cacheing works wonderfully. I've noticed the cache helping me along on several occasions (generally when I'm organizing directories or programming and lots of things start happening instantaniously as the files are still in cache).

Re:It's about the uptime, stupid!

[AaronStJ]

Interestingly, this is why I hate Win2k. Win2k seems to always want to swap background apps out, even if you have plent of ram available. This is beyond irritating when I minimize netscape to work in a text editor, then bring it back to the front, only to have to wait 2 minutes for the thing to swap back in (I'm on a laptop--slow HD).

Really? I never have this problem at all. Win2k has always been snappy as it comes for me.

This is wild speculation, but there's an option (in conrol panel->system, I believe) to switch between optimized for applications and and optimized for background services. You may want to check and make sure that's set correctly.

Re:It's about the uptime, stupid!

[jandrese]

I never noticed much difference between the two settings actually.

You must not run Win2k with very large (40-200MB resident) applications that get sent to the background much do you? Even Mozilla triggers this behavior on a consistant basis.

And don't get me started on the sounds in Win2k. Every time that stupid beep plays the kernel chugs for a moment to read in the sound effect, slaughtering any hope you have of maintaining say a bitstream to a CD-Burner.

The VM is one area where MS really pooched Win2k IMHO.

Re:It's about the uptime, stupid!

[AaronStJ]

I run photoshop occasionally, and that soaks up a a good 100MB of memory, depending on what I'm working on. Even then, it Window's is pretty snappy on my machine, better than I remember X being on the same machine. On the other hand, I have very recently done a fresh format/reinstall, and I've got toadmit, Windows does tend to need that far too often.

Alltogether, though, I think we can both agree that VMs and kernel speed *is* important, especially in a desktop enviroment. We've both had bad experiences with GUIs due to kernel problems. Which is really the point I was trying to originally make.

What's in a number?

[gTsiros]

Numbers numbers numbers... what does all of this mean? I can't see the point benchmarking something generic like that. The kernel first of all wasn't meant to be used like that so it might very well not perform well on the benchmark.

So, pushing the limits a little, i could say that this isn't much different than 'bogomips'

Note that the physical counterpart thought (where you test run, for example, a car for days on end to measure it's endurance) does not apply here.

Then again, this might actually mean something regarding the low level procedures responsible for allocating memory. Never read source, tho, probably never will, out of my league :/

If you support forks so much...

[Free Bird]

then why is your -rmap patch a diff against the -aa VM? I thought your own old VM should be superior, have you stopped believing in it? And correct me if I'm wrong, but isn't RH porting it to a more recent 2.4 kernel?

Or is the article incorrect?

Re:If you support forks so much...

[Rik van Riel]

Nice troll ... ;)

My -rmap VM is a patch against marcelo's standard 2.4 kernel, because that is the thing people have. It just doesn't make sense to release patches against kernels nobody has.

Also note that -rmap replaces pretty much all parts from the -aa VM I don't agree with, while at the same time integrating some parts from the -aa VM that I do like.

Rik van Riel soaking up the karma

[joshtimmons]

I think Rik's discovered a good new way to karma-whore.

1. Write a VM.
2. Get a fine article written about your work
3. Have somebody post the article on /.
4. Post. Post. Post.

Rik's put in at least 3 comments in this tree and they're all being mod'ed up.

I'd better start on my own VM!

(Or, I'll write a reverse disk-sector mapper)

Do you think the BSD projects reflect this?

[twilight30]

I take your points about forking, but as a counter-example I'd think about the BSDs instead. They all operate under the same license, all forks from roughly the same code base.

The advantage here is that with three BSDs you have three separately-tuned operating systems that attack different problems very well, yet maintain a certain level of commonality and compatibility.

Call me a starry-eyed optimist, but my exposure to this open source fad started with the Wired article in the autumn of 1997. In it the writer painted a picture of a 'computing epic', one collectively started and maintained. I still think that metaphor is accurate and useful.

Re:[ot] So what's the best kernel to get right now

[teg]

I'm sitting at home with my fresh install of RH 7.1 and I'm wondering what kernel to upgrade it to. Any suggestions? Is there a stable one in there somewhere that I should go with? Should I stick with the default kernel that's on their now?

The current kernel supported by Red Hat for Red Hat Linux 7.1 is 2.4.9-21, which you can see does a good job in this test.

Number of crashes in 100 days?

[alexhmit01]

It's good to see that Slashdot users still don't use computers for anything... I'm not looking for a system with better uptime than Win95, and that seems to be all you guys want.

I can't have multiple crashes in 100 days. If you are doing real load, spend real money, get real systems.

Don't build machines with your screw driver, get QA'd servers. Don't roll your own kernel, let Redhat test it.

These types of tests are useful as commentary and recommendations for what people should do in the development process.

Re:Number of crashes in 100 days?

[Anonymous Coward]

So they got you hooked with "professional" servers, "certified" admins, "blah blah blah"... Some of the people here do this work and know the end results. For the guys in suits it's great until they hit some snags and realize this ain't worth shit. Do you sell such solutions? Oh you do? How about that.

Re:Number of crashes in 100 days?

[Free Bird]

You obviously lack the technical knowledge to build your own server, so you buy a lame-ass and badly designed A-brand.

That is good for you, but please don't troll these boards with your stupidity. It's a shame you got modded up like that.

Hopefully Someone Has an Answer...

[jschmerge]

Since I've seen posts from at least one kernel developer in response to the attached story, I figured that this might be a good place to ask the following question:

A little while ago, I wrote an application that uses an incredible amount of memory... A very space inefficient implementation of Eratosthenes' Sieve. In essence, what the algorithm does is cycle through the entire contents of memory sequentially many, many times (not a completely correct description). What I found with the following three kernel versions:

• 2.4.4
• 2.4.8
• 2.4.17

is that any time the program's footprint exceeded the physical ammount of available memory, performance degraded exponentially. I found this to be very suprising, considering that I was only exceeding the physical 1024 Megs of memory by less than 10 Megs. About the only difference between the three kernels I listed above is that the 2.4.17 kernel would kill of memory intensive processes a lot quicker than the other 2 versions.

My question for the Kernel gods out there is as follows: are there any stable 2.4.x kernel releases out there that would handle this type of stress without the performance degradation that I've experienced with these kernels?

Re:Hopefully Someone Has an Answer...

[rmull]

While I'm not a kernel hacker, I think I know why this happens.

As soon as you fill up physical memory, you're using swap space. The kernel will either swap other programs to disk to make room for yours, or it will swap parts of yours out that aren't being used. Either way, disk i/o is very expensive - hence the performance decrease. No kernel can help that, really. Once you're out of memory, you're out.

Re:Hopefully Someone Has an Answer...

[jschmerge]

I'm sorry that I wasn't clearer in the original post... With all three kernel versions, what I experienced was not only performance degradation, but a change in runtimes from about 15 minutes to over 8 hours. Unfortunately, I never managed to really quantify times or run a thorough set of benchmarks, but it seems that such a drastic difference in runtimes would imply that the VM's in all of these kernels are not handling a worst-case scenario correctly.

Re:Hopefully Someone Has an Answer...

[Anonymous Coward]

I think that's easy to understand. Since all paging algorithms basically use some LRU strategy (least recently used, the pages which aren't used the longest time get swapped out), what happens is that you will never find a page already in memory. The memory you're going to access will always be just swapped out.
Probably no vm is going to change this, since it's impossible to know what pages you're going to access in the future. It's up to you to change your algorithm so it doesn't operate sequentially on all the memory (called "blocking algorithms" IIRC).

Re:Hopefully Someone Has an Answer...

[Anonymous Coward]

When you're accessing memory sequentially, the optimal strategy for page removal is no longer LRU (Least Recently Used) but rather becomes MRU (Most Recently Used).

In theory, the VM could detect sequential scans and switch its page swap selection policy for that process. However, detecting sequential scans with nothing more than a single referenced bit from the hardware is nontrivial.

Even if you implemented this, there would still be cases where your access patterns tricked the VM into switching, even though LRU (or perhaps a completely different selection strategy) would have been more optimal. This brings us to an important truth in VM design: there is no perfect VM, all VMs will have cases where they fall over. The goal is to simultaneously maximize performce for common loads and minimize cases where everything goes to hell.

Re:Hopefully Someone Has an Answer...

[Spy Hunter]

If you're using a program that cycles through all available memory + 10Mb, of COURSE it will be slow! The VM can't squeeze 1044 Mb of data in 1024 Mb of RAM, no matter what ugly hacks it uses, and as soon as it needs more memory than it has, it has to hit the disk. As soon as you start hitting the disk, you get horrible performance decreases because your program is stalled until the data can be read from disk (which is orders of magnitude slower than RAM).

The reason VMs usually work well is because most programs don't actively use all the memory they allocate at once, so the VM swaps out the memory that isn't being used. If your program uses all the memory it allocates, the VM has no choice but to use the slow disk to store some of it. No magic VM will solve your troubles.

Re:Hopefully Someone Has an Answer...

[chriton]

I wonder if there has ever been a debate about declaring a an access pattern type when allocating memmory? This would give a developer the power to tell the VM something about how s/he is going to be using the memmory.

A few basic choices would allow certain optimizations for sequential access, fully random access, interleaved access, an a few more.

More sophisticated choices could be possible for heaps & trees that would allow you to tell a VM which chunks of data are likely to be accessed next (based on what leaves are below a particular node).

I also wonder if operating systems will ever have deep introspection similar to speculative execution in modern day processors. Probably not... but you never know.

Re:Hopefully Someone Has an Answer...

[dangermouse]

You generally don't optimize for the worst case... I'd wager that a single process requiring all physical RAM and then some is a rare occurrence.

Besides, think about the access time differences between RAM and your hard drive... you're going from a few nanoseconds to a few milliseconds; that's a thousand times slower. Your test went from 15 minutes to 480 minutes, which is only an increase of a factor of 32. That means it was talking to swap what, 30% of the time?

I'm not entirely sure how the math breaks down here, exactly, but it really doesn't seem too bad considering the OS itself (and anything else you might have had running) was taking up some of that physical RAM, too.

Re:Hopefully Someone Has an Answer...

[tal197]

If you're using a program that cycles through all available memory + 10Mb, of COURSE it will be slow!

Nonsense. That assumes that you always throw out the least-recently-used pages from memory (so, in this case, you throw out each page just before you access it).

A good VM could spot this access pattern and throw out the page just used (which won't been needed again for ages) instead, keeping the pages that are about to be used in memory.

There are other places where this helps, eg playing a video or audio file from disk, where recently accessed data is never needed again and should be quickly thrown away (rather than paging out other processes, etc).

Re:Hopefully Someone Has an Answer...

[markmoss]

A good VM could spot this access pattern and throw out the page just used (which won't been needed again for ages) instead, keeping the pages that are about to be used in memory.

Please, please, give us the algorithm that will detect this usage pattern without either using great gobs of memory to record past usage, or using a lot of CPU time. Until someone does this, the LRU algorithm is simple and works fairly well for the most programs.

On a case-by-case basis, it's possible to revise most of the exceptional algorithms so as not to fight with LRU memory management:

Multimedia playing, where the most recently used (played) page is never needed again: free() it! De-allocated pages aren't saved, I hope. Only an idiot would keep all the objects generated and used just once - but there must be a lot of idiots writing programs.

Sieve of Eratosthenes:
(1) Compress the array as much as possible so as to keep as much in memory as possible. You don't need to store the numbers, just whether they are crossed out yet or not. Use an array of bits, initialized to zero, where b(i) is set to 1 when number i is "crossed out." Maybe there are even more clever storage schemes?
(2) Change the bottom to top sweep to a back and forth sweep. That is, the sieve takes the next prime number p and "crosses out" numbers p+p, p+p+p, and so on to the end. Then it comes back to the beginning, finds the next prime number (not crossed out) p1, and crosses out all the multiples of p1. So let's do this sweep in reverse. Integer divide t = n/p1, where n is the last number in your array. Cross off t (which is in the 2nd to most recently used page), t-p1, t-p1-p1, ... down to 2*p1. Next sweep is "forward" (starting at P2+p2), then reverse, etc. This uses pages in reverse order of how they are swapped out, so it minimizes swapping.
(3) Rearrange the algorithm to take a whole block of primes and apply all of them to one block (page) of the array at a time. Or, if you want a _really_ big array, write it to apply several thousand primes to one track of the hard drive at a time...

#1 is obvious and easy, if you are using a language that supports 1 bit objects. #2 doubles the code, but the program is so short that it doesn't really matter. #3 takes some thought and clever coding to run efficiently and accurately.

The problem is, there aren't enough really good programmers out there to figure out how to optimize every program so as to fool every VM into allocating the memory efficiently. So it certainly would help if there was a way to _tell_ the VM to switch to a different allocation scheme for particular objects. Problem: objects (to the programmer) and pages (to the VM) are not in any particular correspondence. The Sieve program, for instance, really needs one page for scalar variables (p, n, i, etc.), which is always being referenced, and so must stay in memory under the LRU method, while the array takes many pages which are best swapped as MRU. What you actually get from a C compiler is something like the first page containing both scalars and the beginning of the array. So you cannot tell the VM to swap all pages containing the array by MRU...

Re:Hopefully Someone Has an Answer...

[hoggy]

Walking repeatedly and sequentially over All Memory + K tends to result in worse case performance. The reason is that when you hit the +K at the end of the loop you page out the oldest memory, which is the K at the beginning of the sequence. When you restart at the beginning you need to get this back in, so it pages out the next oldest K, which is of course the next part of the sequence. Thus you end up paging in and out the whole 1GB on each iteration, resulting in a massive slowdown.

(The real answer is less simplistic, but this is close enough to the truth.)

Re:Hopefully Someone Has an Answer...

[haro]

Please, please, give us the algorithm that will detect this usage pattern without either using great gobs of memory to record past usage, or using a lot of CPU time. Until someone does this, the LRU algorithm is simple and works fairly well for the most programs.

My favourite alternative to LRU is a simple throw out a random page. In most "normal" cases LRU will perform better, as obviously it will never throw out a heavily used page, but the random method does not meet the wall once a program like this comes along. You are in effect scarifying the peak performance for better protection against cases like this.

Re:Hopefully Someone Has an Answer...

[markmoss]

My favourite alternative to LRU is a simple throw out a random page. Interesting suggestion.

How about a hybrid strategy: exempt the 25% of pages that were most recently used from swapping, but then pick the one(s) to swap out randomly from the other 75%. For the Sieve, this would mean that the few pages used for the program and the continually accessed scalar variables would stay in memory, along with OS pages, and the rest of the top 25% would be waste space. But the other 75% would be managed by random swapping, which is a lot better than what happens with the normal algorithms. And for more normal programs -- with 256M RAM becoming standard, anything that is in frequent use should stay in the most recently used 64M and not be affected by the randomizing.

Or is it practical for an OS to recognize when VM is thrashing, and shift strategies from the normal (LRU or some approximation) to random?

Re:Hopefully Someone Has an Answer...

[WNight]

Well, it's an open problem, which means I don't have a solution, but I have a few ideas...

The first is to have algorithms for five or six different VM algorithms, a LRU, an MRU, and some more complex ones optimized for various tasks. Then, start everything off with the one that proves best in a random sampling of programs. But, while actually using the LRU (for instance) to swap, let the other routines simulate their own swapping. When one starts to pull ahead of the others, use it, but keep simulating.

Now, as long as there is plenty of swap space, regardless of free physical memory, it may be a win to keep large statistical studies of memory usage in different processes. Of course, this sort of thing would get turfed when the system actually started to run out of memory, but in the meantime the slight slowdown due to having less available RAM might be offset by swapping less, or with proper speculative swapping, not having to halt a process.

Also of use would be memory pools, to avoid having to kill userspace processes just to let the kernel alocate enough memory to keep running.

Of course, there will always be pathological conditions which render even the best VM incapable of dealing. (Imagine an FTP server that didn't write out uploaded files to disk, keeping it all in memory, eventually it'd run out or physical and virtual memory. Or the case of a process running some study on the data which necessitated accessing pages in an effectively random manner.)

I think we should work on putting an upper limit on swap time instead of making it a little snappier for every day usage. (I'd sacrifice 3fps in Quake3 to keep Photoshop from effectively locking my computer when I manipulate 200+ MB images.) We should also consider killing processes to be the very last resort before forbidding the kernel any more memory, and this should come about only from a bug in the kernel. (All parts should be designed as not to grow unbounded.)

But then, I'm an armchair VM designer. It's an educated guess, but still only a guess.

Re:Hopefully Someone Has an Answer...

[nivedita]

If your description is approximately correct, the way to solve this is not with a clever VM, but to simply rewrite your program to explicitly maintain roughly 10Mb of your array on disk. You could do this by `wiring' down most of your array so it isn't swapped out (can be done only if you are root). Of course, the _real_ solution is to rewrite your program to use a smarter algorithm, or at least implement the sieve slightly more intelligently.

Any mainstream VM algorithm will essentially wind up reading your entire array from disk every time you make a pass through it. MRU/random/hinting will not help (since the CPU processing is so much faster than I/O, significant overlap cannot be achieved). The trick in this case is essentially not to pageout at all for most of the array, and let the last 10Mb be accessed purely from disk.

Re:Hopefully Someone Has an Answer...

[WNight]

This has been hotly discussed. The problem is that it would require a complete refit of all programs to be of any real use, or it would be a performance patch on one system that made the programs incompatible with another system.

But yes, it would easily solve the program of accessing 1.1GB of data on a 1GB system. You simply tell the VM that .1GB at the start and .1GB, .55GB into the data, are to be swapped out before the rest. As you've then got .9GB marked to be kept in RAM (while possible, at least) the VM simply throws out one of those chunks to make room for the other, reducing the swapping from 1.1GB/round to .2GB/round. Furthermore intelligent precaching could get the second .1GB of swappable data ready as soon as you've gone through the first .1GB, so as to have it ready by the time you get there, and vice versa.

This could be implemented as an out-of-band request to the VM, or could be handled by writing a new malloc() that's backwards compatible.

You'd call newmalloc() in the program, specifying how much RAM, what swap pattern is likely to work, and if any of it should be considered higher priority than the rest. Ideally this would be processed by system libraries and the improved VM would read it, but on a system without those libraries the _newmalloc() function in your program is used, and it simply sums up what you've asked for and places a simple call to malloc(), giving you the ram, but without the performance options.

I'm sure a few high-end systems have this sort of thing, I can't imagine writing weather simulations to run on a Cray for instance, and having to use your own manual hacks for memory management, so I'd assume they have something... I doubt they bothered with the fallback though, most things written for a Cray wouldn't be worth running elsewhere.

Don't understand Moshe's conclusion

[DaveWood]

Under the section "Allocation and Swapping Results," I assume larger numbers are higher times and therefore worse. By the numbers, 2.4.18pre2aa (the Arcangeli kernel) seems to be the fastest overall, due to the 5th run (I would consider it the common case) results. Yet Moshe says:

"From the above figures it seems that the old van Riel VM is somewhat faster (considerably faster in the case of 2.4.9) than the new Arcangeli VM..."

Is my math wrong? The RVR VM in 2.4.9 is ever so slightly faster on the 2nd run and slower on the 5th, and the slowest of all is the newer one in 2.4.18pre3rmap. What's my mistake?

Moshe's politely indicating that van Riel was an ass when asked for comments; we can conclude either that Moshe didn't have a proper recent RMAP kernel to test with (as a result), or that the recent RMAP kernels are hit and miss.

From looking at van Riel's comments here, he vehemently believes his kernel is perfect and Moshe just got it wrong... The problem is that lots of people seem to "get it wrong" with that VM, including Linus... Overall in Rik I get the sense of an aggressive person who may have trouble admitting mistakes or accepting failure; not good traits in a programmer, since it's humility and communication skills which can often be the critical factors in a team programming effort... and lack of them can cause exactly the kinds of problems we've observed.

Re:Don't understand Moshe's conclusion

[Anonymous Coward]

From looking at van Riel's comments here, he vehemently believes his kernel is perfect and Moshe just got it wrong... The problem is that lots of people seem to "get it wrong" with that VM, including Linus... Overall in Rik I get the sense of an aggressive person who may have trouble admitting mistakes or accepting failure; not good traits in a programmer, since it's humility and communication skills which can often be the critical factors in a team programming effort... and lack of them can cause exactly the kinds of problems we've observed.

ehh... when i think "top rank" open-source programmer, I think: Theo de Raadt (his early work on netbsd/sun alone is brilliance), RMS (GCC, emacs, etc), Jordan Hubbard (of FreeBSD fame), Linus Torvalds, Alan Cox, etc.

Of that group, Only Jordan and Alan strike me as having "humility and communications skills." Linus is often curt, sometimes cordial, and sometimes very rude. Theo is almost always rude. One look at RMS' TCL outburst and 'communications skills' obviously don't belong in the same sentance with that guy. I'm not trying to troll or flame here, I'm just saying that expecting open source (or Free Software, whatever floats your boat) programmers who run insanely popular projects and who get treated like gods whenever they show up in Slashdot (ooh.. it's Alan, +5 - not that he doesn't usually deserve it), at a Con, or on IRC or something... expecting these guys to have normal-sized human egos is asking a lot. It's clearly asking too much of most of them.

also please note that i'm not slotting rick in next to AC, Linus and John... yet. Just making a comparison.

Re:Don't understand Moshe's conclusion

[ahde]

I believe you are right concerning the benchmark. I, too, saw the same thing, and had to jump to the bottom to see his conclusion. I think either on the table or the summary he switched the description. Since he twice defended the rmap's speed in the summary, it may be the table -- which would align with other benchmarks of the AA vm suffering under extended load.

Ironically, Rik's rmap patch wound up being unable to complete the test, and his complaint about using 2.4.18pre3 seem disengenius since he was asked politely for info on testing against it, and refused.

Re:Don't understand Moshe's conclusion

[mark_lybarger]

vehemently??

come on... the guy takes pride in his work, enjoys working creatively in the linux kernel code. he not believe his kernel is perfect (why would he conclude the rmap is still under development?). how about a little love? you know a show of hands for AA, RVR, Alan, hell even Linus. they spend their time writing, developing, arguing over a linux kernel so i can have a choice in the OS i put on my machine. so i can spend my time developing wannabe c++ applications for KDE. thanks guys for the work and dedication.

Re:what's the point?

[Arandir]

I'm using FreeBSD as a desktop OS at home and as a workstation at work. It's simply great!

There are only two advantages I see to Linux that makes it a better "desktop" OS. First, they put DRI in the kernel. They're working on this now for FreeBSD, but there's a lot of resistance to keep userland stuff out of kernelland. If all you care about is running games, then FreeBSD is not it. Go get a PS2 or an XBox.

Second, the popularity of Linux gives it a greater pool of developers to draw from. So Linux gets new device drivers faster. But you still will *not* get Linux shoehorned into this week's premium super buy at CompUSA. With Linux you have to wait around three months to get a driver for brand new hardware. With FreeBSD you have to wait about six months. If you buy computers more often than once every six months, stick with Linux. As for myself, I had no problems with FreeBSD on a stock Dell Optiplex GX240.

In terms of desktop software, don't worry a bit about it. They're the same on both platforms. Identical. Staroffice, GNOME, KDE, Xmms, Gimp, Mozilla, etc.

Re:what's the point?

[Moderator]

Second, the popularity of Linux gives it a greater pool of developers to draw from. So Linux gets new device drivers faster.

You would think that most companies would embrace the BSD license over the GPL, as it allows the companies to write drivers without having to release the source.

Value in diversity

[ondelette]

Then, if they are all compatible with each other, where is the harm???

You have *diversity*. Unlike the M$ world where you don't.

Sure, it is hairy at first... but once you get used to the fact that Linux is all about diversity... having choices... many, many choices... you come to like it. In fact, the monolithic Windows really gets on your nerves once you fully appreciate Linux.

Small amd medium sized

[Anonymous Coward]

And this differs from the MS development model how?

You can even view all the different versions of apps they've put out as various "forks". Heck, some of them are even compatible to some degree (but not too much - else we peon's wouldn't be forced to upgrade so soon).

Re:Small amd medium sized

[Anonymous Coward]

And this differs from the MS development model how?

Microsoft is the sole arbiter of what goes into a Windows release. There are no other companies selling "Windows -ac".

If an MS developer writes something that really enhances some particular feature, it is his responsibility to make sure that it folds back into the main tree without causing conflicts. With the Linux patches, it is the user's responsibility to make sure patches are compatible.

Re:Small amd medium sized

[Anonymous Coward]

this should take me more than 20 seconds but just in case it doesnt hmmm. whaddaya say.

Re:VM -ac kernel?

[Anonymous Coward]

Security through obscurity works like this: it takes someone who knows how to patch a kernel to use all those security exploits in Linux.

Try XP

[mangu]

No, I do not mean eXtreme Programming -- for the ultimate centralized source repository you should try Windows XP Shared Source.

Free vs. Open vs. Net BSD

[mangu]

And they all forked from THE Original Berkeley Software Distribution, didn't they?

Re:what's the point?

[modecx]

Well, except that you can release drivers (that need to interact with the kernel) as kernel modules [in binary form]. Nvidia, and lots of other companies have drivers in the form of binary kernel modules.

There is just no such problem, imho.

A suggestion

[nusuth]

I can't answer the kernel related part of your question, however a possible workaround is forcing future reads from active pages by preaccessing them. E.g, before starting on processing [N,N+K) cluster, read value(s) of N+K, N+2K, N+3K up to N+JK and write them back. For a suitable K,J pair, you should experience linear performance drop. A K=8000 and J=1 sounds reasonable to me.

&ltspeculation> AFAIK kernel uses a hairy buddy system (but I did not check that myself), if that is the case, not whole memory is accessible in an arbitrary allocation sequence. So your analysis of exceeding 1024Megs of memory by less than 10 Megs is incorrect. You have to allocate progressively smaller page sizes that follow fibinocci(sp) series (or something like that, see your local kernel hacker for more info) if you want whole your data in memory. &lt/Speculation>

question...

[keepper]

This is not a flame, am really just curious...

I compiled and ran the "memory hogger" application
and it did not eat more than 424K on FreeBSD..

I know you have a lot more to do thatn to answer silly simple questions BUT.. why is that :-D..?

Thx in advance..

Re:question...

[Anonymous Coward]

Perhaps FreeBSD notes that if you're writing a zero to a page that hasn't been demand mapped into your address space yet, you don't actually need to do anything at all?

If this interpretation is correct (I'm at work, so I don't have time to check the code), this is a trivial micro-optimization that only helps benchmarks.

Re:question...

[jquirke]

Perhaps calloc() on FreeBSD marks the page as "to be zeroed" on first touch?

Whereas Linux's calloc() actually zeroes the area manually?

--jquirke

Seti@home

[Txurlo]

I don't know you guys, but my all-time favorite benchmark tool is Seti@Home. =) The distro which spits out more work units, that's what I'll take. And my linux box does 5 times as fast as my w32, mind you. But you already knew that, did you? =)

Re:Seti@home

[Anonymous Coward]

How will Seti@Home show the difference between kernels? Yea, there may be some small difference (I doubt it though), but they would not be great enough to declare one kernel better than the other.

Re:Seti@home

[Anonymous Coward]

> I don't know you guys, but my all-time favorite benchmark tool is Seti@Home. =)
The distro which spits out more work units, that's what I'll take.
> And my linux box does 5 times as fast as my w32, mind you. But you already knew that, did you? =)

Poor test; very little I/O, very little networking, and the working set's small enough that once it gets going, no paging or other VM stresses.

A large-scale software download and upgrade cycle, a la' a Debian apt-get update involving 50-100 files, would be a better all-around test.

Re:Seti@home

[Explo]

I don't know you guys, but my all-time favorite benchmark tool is Seti@Home. =) The distro which spits out more work units, that's what I'll take. And my linux box does 5 times as fast as my w32, mind you. But you already knew that, did you? =)

You don't happen to run commandline client on Linux and the 'gui' client on Windows, do you? ;) The difference between gui and commandline client is huge on both Windows and Linux.

It's Troll Tuesday

[Anonymous Coward]

At least, in Europe and the East it is already Tuesday.

Re:what's the point?

[GigsVT]

There's a big problem in your argument. You assume that when someone "buys a new system" they are getting all of the very newest hardware.

I venture to say I am not alone, in that when I upgrade, I always upgraded to hardware that was already out at least 6 months anyway (even when I ran windows), that is where the sweet spot is in price/performance ratio usually.

So, one could buy a new system every 6 months, with no driver hassles at all, if they just trail the market, like I'm sure millions do. (at least everyone who isn't rich and wants the most for their money).

You can't count crashes in 100 days

[Anonymous Coward]

Unles you use a lot of decimal digits, the answer is zero, no matter which Linux kernel tree you have. The less reliable hardware modules in my system have 100000+ hours MTBF, and I make sure they will not fail due to software.

Re:You can't count crashes in 100 days

[Anonymous Coward]

> Unles you use a lot of decimal digits, the answer is zero, no matter which Linux kernel tree you have. The less reliable hardware modules in my system have 100000+ hours MTBF, and I make sure they will not fail due to software.

Parent post was talking about crashes related to kernel bugs, like the ones described in the article.

RTFArticle, then post.

Re:[ot] So what's the best kernel to get right now

[MindStalker]

Nothing evil about it at all. I compile my kernal because I have to patch in some drivers, and like to specify certain modules to be included in the kernel and certain mods to not be. But if a straight rpm kernel works properly why not. (you do get a slight preformance increase with a compiled kernel btw, as its optimized to your hardware exactly when compiled, but I digress)

why didn't author measure page faults?

[brer_rabbit]

I've asked this before when people do benchmarks and I'll post it again: why don't these authors measure page faults? Measuring the total time the process runs is good, but if you can measure how many page faults occured you'd be providing a lot more infomation.

So how do you measure page faults? Be sure your kernel is configured with "BSD process accounting". Then use a shell like tcsh. The man page of tcsh describes the "time" variable, you can set it to report the number of major/minor page faults that occured during the lifetime of the process.

I did my own unscientific test back in November. I ran 32 simultaneous instances of mpg123 on a just-booted machine. Among other things I measured the number of page faults. The results for the then-current kernels I had were:

kernel: 2.2.20 2.4.8 2.4.12
mean elapsed time: 88.6 86.5 88.4
mean page faults: 7833 7285 8990

those number are the means of the 32 values from each process. Anyway, you get the idea.

Re:why didn't author measure page faults?

[Anonymous Coward]

You don't actually need BSD process accounting in your kernel, and you don't need to use tcsh (although tcsh will work).

You can just use /usr/bin/time (write out the whole path, otherwise bash's builtin time takes over) on any old kernel.

It's about the frames per second, stupid!

[Blue Lozenge]

Why would I want to play Quake at a measily 60 fps for 2 years straight, when I can play it for at most 1 hour at a blazing 61 fps? :)

This is nearly impossible to measure

[roystgnr]

If you see a Linux kernel crash, it is either hardware failure or a kernel failure. Since hardware failures (especially RAM and power supply) can be imposssible to repeat, the only way to prove that it is a kernel failure is to find the kernel bug. If you find the kernel bug, then the bug gets fixed, and suddenly your crash data is for an obsolete kernel.

You could try to take a statistical run at it, of course, but I suspect the number of machines required to give meaningful results would be on the order of Google's farm.

Re:what's the point?

[motox]

As far as gaming goes you could also keep you PC and install Windows. You don't need a PS2. FreeBSD is nice but Linux has got better marketing ad a name. Everybody heard "Linux" at least once, much less people know FreeBSD.

I thought about it some more...

[nusuth]

You should do something like starting with a large J (K*J comparable to but not exceeding total memory size) and read K*J pages without any preaccessing. E.g read values at offsets N,N+K,N+2K,...,N+KJ, then continue normal processing until you reach N+KJ. Repeat procedure with new N=N+KJ. This way you hint the VM that these pages are active, and should not be swapped out. K should be selected as large as possible, not exceeding a page boundary. Too large K will make fail to mark some pages as active, too small K will consume unnecessary processing time while hinting.

If everything goes according well, your kernel should swap out and swap in exactly M amount of memory for each preaccesing pass, where M is the amount of data that does not fit in the main memory. So if you have total memory T, total data T+M and K*J size prefetch, total swapped data per whole process would be M*(T+M)/(K*J).

Re:I thought about it some more...

[jschmerge]

Just to let you know, I tried your strategy and several others, none of them have seemed to work for me... I should probably just post the problem with some carefully constructed benchmarking code to the lkml and see what they have to say. Anyway, this thread of discussion is really going nowhere... I was kind of hoping to get some insight from the kernel developers which hasn't happened. Thanks for the suggestions, but I've already tried every way I know of hacking around the problem without any success.

Re:I thought about it some more...

[WNight]

This precaching idea will only help if your program spends more time in a block of data than it took to fetch, otherwise you'll catch up and be blocked.

If your algorithm is doing something bloody quick (optimized CRC, etc) you'll probably work faster than the disk I/O. If however you're calculating weather patterns or Conway's Life, or anything involving a bit of CPU and reprocessing the block of data in a non-linear fashion, you'll likely be okay.

If the routine involves looping through 1GB of data 100 times and you don't have the RAM, it will end up reading 100GB from disk, eventually. Your process may be dead simple and go from 15M for a 990MB dataset to 8Hr for 1GB, or it might be complex and go from 7:50 for 990MB to 8HR for 1GB, but if it takes 7:45 to read the 100GB, you've established a lower-bound on how long it'll take to process.

What were you doing to this data as you looped through it?

You might want to consider doing your own VM. If you're just 5% or so over available RAM you could manage this by writing to a file and deallocating two 6% chunks of memory back and forth. Make then each as far apart (in the dataset) as possible so that after loading and processing one, you have as much time as possible to let the VM load and precache the next set without blocking.

Because of the patches... but WHICH patches?

[Chazmati]

Is there any telling what patches are used in the Red Hat kernel releases? Or is that a trade secret? I know the source is there, but when you start talking about multiple patches, it seems like it would be a tough one to figure out.

It seems like this may be the key source of competitive advantage for a Linux distribution vendor: the know-how to optimize the kernel and other software to make a fast, stable system.

Re:Because of the patches... but WHICH patches?

[puetzk]

sure, I'd expect each is in a patch file packed in the SRPM

picking the right ones is, as you said, the trick. And RH's stresstest labs are pretty godo at it

Re:Because of the patches... but WHICH patches?

[Anonymous Coward]

I use the redhat patches on my debian box. Its pretty easy, just download the SRPM, install using rpm, manually apply the patches to the source and make-kpkg away.

Say what you will about the userspace components of Redhat's distribution, but their kernel QA is unbeatable.

Re:Superbowl Commercials were great!

[Anonymous Coward]

Your ideas intrigue me and I wish to subscribe to your newsletter.

Re:Pournelle ?-fiction.

[Anonymous Coward]

"Then he signs off on his latest friends book about how the white people are better than non-whites, and how America should just nuke the rest of the world, after it has finished up using all the oil that is there."

Now your just making this up.

thx

[keepper]

Now, the bigger question is.. why would linux's vm allocate empty pages?

Re:thx

[Anonymous Coward]

Huh? It only allocates pages on the first access, just like FreeBSD's VM or any other demand paged VM.

The test allocated the memory with calloc to write 0s to all the memory, so that it would actually be demand paged and stress the VM.

This is all well and good

[The Bungi]

The raw performance of the VM is certainly important and all, but what I'd like to see are some *application* benchmarks among the various kernel trees. Star Office, the window managers, KDE, GNOME, etc. Graphics. Storage. Networking. Unles we're talking heavy metal servers running the usual suspect daemons the average user doesn't really give a hoot if the VM is well-designed or not - only if The Gimp runs quickly enough.

Re:Superbowl Commercials were great!

[Anonymous Coward]

send me money

Kernel Test Suites Article

[goingware]

Some of the test suites in Using Test Suites to Validate the Linux Kernel might be good for benchmarking.

Yes, I post the link to this here all the time, I think it's useful to people.

Re:Kernel Test Suites Article

[Tower]

+1, thanks for helping find better memory death tests :) Nice basic h/w test links all in one place...

Re:Kernel Test Suites Article

[Anonymous Coward]

I wish linux kernel had unit tests for testing the kernel itself. I suppose even kernel development would be a bit faster as it would be harder to
introduce subtle bugs...

Your standards are too low

[Anonymous Coward]

Why can't I have both stability and speed?

Byte magazine is coming back!

[Anonymous Coward]

Look at the end of the article - March 2002 Byte Magazine is coming back.

Re:[ot] So what's the best kernel to get right now

[bruceg]

Any idea when RH's 2.4.17 will be the "supported" kernel? I am using vanilla 2.4.17 now, since the ieee1324 drivers lockup my RH 2.4.9 kernel.

Re:what's the point?

[Arandir]

Oh of course! But I see so many posts to the FreeBSD lists from people with one week old technology that I had to at least point it out. The problem is that people don't use the latest releases. They go to Fry's or CompUSA and pick up a shrinkwrapped box that will often be several releases behind.

Re:what's the point?

[Arandir]

Well, except that you can release drivers ... as kernel modules

Ditto for FreeBSD. Unless of course that is what you meant.

Linus' good manners and motivating approach?

[Anonymous Coward]

From the article:

While Linus seems to always show good manners and a motivating approach...

Are we talking about Linus Torvalds or another Linus that I am not familiar with?

Is he nicer on the Linux Kernel list than he is on the GCC mailing list? GCC is actually more complex than any one of Linux's kernels! Furthermore, it would not be enjoying the widespread portability if not for the GCC compiler he claims to loathe so much.

Rik van Riel Chat

[sfrenchie]

Wow! Reading this story at +5 is like seeing Rik van Riel have a conversation with himself!

Re:Rik van Riel Chat

[garcia]

talk about whoring...

He's the fucking pimp.

50 karma in a single story.

Re:Rik van Riel Chat

[sg_oneill]

Grow up you fking teenaged dipshits.

You've got one of the leading mojo's in Linux Kernel development prettymuch being interviewed by "the people" here and answering Q's etc, and you decide to lambast him as a "whore" because people are actually interested in what he has to say. Note that Linus, RMS , ERS , A-Cox or any of them pretty much have never(as far as I remember) have shown up in here, so that's a pretty nifty thing IMHO

And for christ sake, promoting interesting responses what Karma is for you idiots. (and Karma cap stops it going nuts). Get a life.

Re:Rik van Riel Chat

[Anonymous Coward]

umm, moron...

I wasn't upset about it. Jesus you're gay.

Re:Pournelle ?-fiction.

[(outer-limits)]

Try this, or his support of the bell curve .

How can you take a guy seriously when he writes a book in which science fiction writers are guys who save earth from an alien invasion.

His knowledge of history is woeful as he does not appear to have heard of the Marshall plan. Without that, we would have had WWIII already, and with one after WWI, maybe no WWII.

Comparision of evolution..

[kesuki]

In the context of evolution Torvalds represents natural selection, and kernel developers changing the code represent mutation, albeit a poor representation because the developers have some sense and purpose in what they do, while mutation "in nature" has absolutely none.

I have to disagree with the notion that this isn't Exactly how evolution occurs. Evolution isn't Random, Virus are the vector for almost all radical natural genetic manipulation. The purpouse of a virus is to use the resources of a host organism to replicate it's DNA sequences because Virus are too small to replicate themselves. In doing so It injects it's DNA inside a host cell and basically exploits the RNA strands inside to replicate it's own DNA. While doing this The virus can in fanct find new DNA within the host and borrow it for it's own protection. In many cases this is to make it resistant to antibodies produced by the body (HIV.) Now, not all Virus destroy the host cell especially when antibodies destroy it before it can complete it's task. In some instances the Virus may act as a Vecor Borrowing the DNA from one species, and inserting that code in another. Many virus can cross infect species. For example humans and pigs can catch influenza from each other. Geese and pigs can also catch influenza from each other, while humans and geese cannot infect each other with influenza.
Virus are acting for thier own self promotion and preservation. When a DNA stand from one species makes that species less able to destroy them they would try to splice that DNA into as many species as they can. Comparing that to kernel developers
is pretty straight forward. They try to 'infect' the kernel tree with the 'code' they've produced for any number of reasons. Being known for coding on linux, to get promotions at a linux friendly workplace, for the challenge and fun of contributing to the linux kernel, or just to fix something so that they can do something with linux that they were trying to do but couldn't.
This introduces variation along the same analog as virus changed DNA. As for 'uncorrected errors' in the DNA strand the only thing we can prove comes from that is cancer. Thus that type of 'mutation' is analog to 'bugs' in the code of the linux kernel. Unfortunately humans aren't anywhere near as good as the roughly 99.7% error correction rate of replicating double helix DNA strands, so code tends to get a lot more malignant tumors (root exploits) to cut out.

Um, lemme think about it Mullen...

[/Idiot\]

Let's see, if you are testing a kernel for a production system with real tests (you know, you have probabbly read about them!) then if it goes down once in 100 days then it's off my list.

Aim a little higher than - as another respondant said - Windows 95 (which wasn't designed for serious use anyway!).

400meg/sec swap out

[Anonymous Coward]

Hmmm, I sure wish I had a system that could swap out over 400meg/sec. I want his 18gig HD's! 200meg/sec to each of them!

Just checked the source, vmstat doesn't normalize based upon how long it really took. Without time stamp information on each line of the vmstat, the swapping rate is not reliable. Almost all OS's have this problem. You aren't suppose to do benchmarking with vmstat.

John-Mark Gurney

Another issue...

[Chicks_Hate_Me]

I'm using kernel 2.2.20 with the Riel VM (I'm guessing) on a Debian box. I've noticed that everyday my memory slowly creeps away. First I had 100MB free RAM and now it is down to 20MB after 14 days and still is going down. Could this be related to the van Riel VM? or perhaps the ext3 patch I'm using. Sorry if this is seems offtopic.

Re:Another issue...

[mallie_mcg]

I'm using kernel 2.2.20 with the Riel VM (I'm guessing) on a Debian box.

No idea, sorry.

I've noticed that everyday my memory slowly creeps away. First I had 100MB free RAM and now it is down to 20MB after 14 days and still is going down. Could this be related to the van Riel VM? or perhaps the ext3 patch I'm using. Sorry if this is seems offtopic.

This sounds like it could be a memory leak. This happens when ever a program allocates memory and does not free that memory.

Re:Another issue...

[Anonymous Coward]

2.2.x has never had any Riel VM.

2.2.x where X is large ( > 19 or so, I think), has AA's VM which is similar to, but less developed than his 2.4.x (where x > 10) VM.

server this, server that

[paulbd]

it seems that the vast majority of commentary here all assumes that a linux machine is run as a server, or at best, some kind of generic desktop machine. while linux may be very good at running servers, and may be capable of acting as a good generic desktop machine, some of us are interested in it for much more specific tasks such as realtime audio processing, editing, synthesis and recording. we care about those extra 2 seconds spent in the VM code during a benchmark. we care about all the extra paging that goes on with some designs. we care about the internal operation of the buffer cache and how it affects attainable peak i/o performance because we stream, and when i say stream, i'm not talking about measly HTTP numbers, i'm talking about 64-128 streams of 24 or 32 bit 96khZ audio data. stop assuming that a top-end linux box is a server, please.

Re:server this, server that

[Anonymous Coward]

Paul, while I agree fully with you that latency is a huge issue, your DAWs shouldn't even have swap enabled, because once you hit swap, all bets are off.

BTW, I love ardour.

Re:server this, server that

[paulbd]

true, swapping (actually, its technically called paging - swapping generally refers to moving an entire address space between RAM and a storage device) is the end for such systems. however, the design of the VM code affects things long before we get to swapping. in particular, it interacts in important ways with the buffer cache, which can affect streaming performance on most filesystems. ps. you'll love ardour even more when its finished!

Re:server this, server that

[Anonymous Coward]

"it interacts in important ways with the buffer cache, which can affect streaming performance on most filesystems."

Eh? In 2.4.x the buffer cache is primarily used for filesystem metadata, of the sort that should only be accesed before the 'firm' realtime part of a application runs. Filesystem access, via read (2) or mmap (2) should go through the page cache entirely. Longer term, the buffer cache should go away completely.

I anxiously await a finished ardour. What ever happened to Quasimodo?

Re:Red Hat 2.4.9 is a very good kernel, fast...WHY

[Anonymous Coward]

Are we so starved of intelligent opinions...

Why, yes.

Supply and demand

[Anonymous Coward]

For those who think the free/open source development process is somewhat socialistic, I disagree. What we see happening is with a large supply of high quality patches coming out for the kernel, the value of the patches are going down. So, for the authors to get what they think it is worth, we now see 'branding' of patches, by the likes of Rik. Advertising and talking up a patch increases it's value, creates a loyal client base.

We probably will see a 'recession' in patch production due to the overheated supply and low demand. Then the quantity of patches will decrease, raising their value and prominence, which will then start the cycle over again. And I think the attempts at 'regulating' the supply and demand of patches will have the same effect as it does in the real economy. Supply will stabilize, quality will drop, and eventually the whole thing will stagnate.

Derek

Please MOD parent up and ANSWER

[Civil_Disobedient]

I agree completely. While the tests performed are very useful for those people running server processes, the entire article ignores most end-user applications. He didn't even bother installing X on his test systems (though naturally one doesn't need KDE to keep a company running). I would love to see a benchmark comparison for real end-user user applications. And since this is Linux, have a separate section devoted strictly to the stability of these apps.

Sadly, I don't think this question will be answered today, for so many zealots still think you can "sell" Linux without convincing Joe User that open source can answer all their needs just as well as other, more Redmond-based solutions. While I have no gripes with the article, I simply wish Linux gurus would step back from their BASHes and see that there's a whole world of regular people just trying to use computers as tools for their jobs and varied interests. I want to know how GIMP will run. I want to know how good Konquerer is compared to IE. I want to know how the speed of KDE/GNOME's UI is compared to Windows 2000. Unfortunately, I think this will probably be too "beneath" most kernel hackers to address.

Listen, people. The reason that the Evil Empire is doing so well is that thousands of people are buying their software. Thousands of people that have better things to do than trying to keep their computers operating. You like programming? That's fine. You probably make a living in the engineering field. Personally, I make a living doing design and advertising. That means I will take the solution that is reasonably fast and doesn't crash while I'm editing a 600 meg graphic file or a 4 gig video. I love open source software (and wish sourceforge would offer more Win32 app development, but whatever), and will support it in a more knee-jerk way than any MS solution, provided people actually have the guts to compare the two.

2.4.17 of course!

[Codifex Maximus]

The linux-2.4.17 kernel rocks the world. I have had not a single problem since compiling and installing it AND I've also had more things begin working.

Before 2.4.17, I couldn't get sound to work with Return to Castle Wolfenstein... trying to run the game with sound would just put the machine into a probable race state... I'd try to run a program and it would just hang... switch to another virt term login run top and hang...

I'd have to restart the machine because I couldn't even kill the processes. But with 2.4.17, NO MORE PROBLEMS.

Me is a happy camper.

Re:2.4.17 of course!

[Forrestina]

2.4.17 works fine on all my laptops. but.... my 3 eepro100 ethernet cards in my primary server... are fucking up left and right. they had less problems in 2.4.16, but not by much.

i never had a single problem with these cards under 2.2 kernels. and i really liked them. so, what the heck is the matter? i know they are one of the most common ethernet cards in servers out there. one would think they would be well supported.

Re:what's the point?

[Anonymous Coward]

You would think that most companies would embrace the BSD license over the GPL, as it allows the companies to write drivers without having to release the source.

Then they're not embracing the BSD license then are they? They're merely using what is free.

I don't trust him

[Anonymous Coward]

Moshe Bar... Ever since this guy compared FreeBSD 4.4 and Linux 2.4 and "raised" the FreeBSD MAXUSERS setting to 20, I don't trust a single word he says. And look at him, just look at him! Do you trust that shaven face?

Re:Pournelle ?-fiction.

[Anonymous Coward]

The Marshall plan had next to nothing to do with preventing WWIII. The development of atomic weapons was the true culprit.

Of course, whiney librals have to make up reasons we haven't had another world war to avoid facing the stark truth.

You go girl!!

[Anonymous Coward]

Dont take no crap from the haters

picking nits

[darkonc]

As a reader from Australia noted in an e-mail to me, "evolution is by definition an undirected process with natural selection." You might, however, argue that there is undoubtedly some direction in the Linux kernel development. ....

I'm going to disagree with this notion of evolution. Evolution is not undirected. The current environment gives a good deal of direction to the sorts of evolution that occurs. For example: evolution appropriate to tropical beaches is unlikely to occur in the arctic tundra.

Similarly, Evolution in the Linux world is also mostly in reaction to environmental needs. Where the difference in randomness comes is that the mutations that lead to biological evolution are generally random in nature -- but environment and statistics choose which mutations lead to enhanced viability.

For Linux, patches are generally in direct response to specific needs. The nature of these changes are directed by nature but generally random in form -- ranging from the icky to the elegant. Fork maintainers like Torvalds and Cox are more like the social interactions which can shift the survivability of an otherwise brilliant mutation/patch. Although this social rejectin will deeply affect survivability, an especially bright change can still give a survivabillity edge that makes up for the rejection

This is really the pleasant aspect of the Linux community; exactly those people who are busy and sought after by many journalists and hackers are also those who take time to answer questions with enthusiasm and a very positive attitude. Thank you Andrea, thank you Alan.

This is something of a chicken and egg proposition. Those people "who take time to answer questions with enthusiasm and a very positive attitude" are precisely those who will likely be sought out by Journalists and others. I mean -- come on! Are you going to take your question to someone who regularly beats into submission anybody who comes to them with (what they consider) a non-profound queston?

Journalists, especially often need their question answered today , and can't be bothered to wait for someone who berates them for half an hour for asking a straightforward question -- especially if they then forget what the origina question was (no known anectdote comes to my mind).

Re:picking nits

[markmoss]

Linux "evolution" is thus a combination of Darwinian and Lamarckian. (Lamarck thought that acquired traits would be inherited -- that is, a giraffe's neck was long because its ancestors had stretched to reach high-up leaves. Darwin postulated natural selection of random variations -- that is, the tallest proto-giraffes got the most leaves and were more likely to survive, so the next generation was taller and the tallest of them survived better, etc. In biology, natural selection is so well established as to be basically a _fact_. Lamarckism is discredited; it was a nice idea, but now that we know about DNA there is no way the length of the neck could change the DNA in the gonads, and a long time ago some lab experiments seeming to show Lamarkian inheritance were proven fraudulent.)

The author's tests don't look very stressful...

[tym]

I fail to see how the code used stresses a VM system excessively or realistically. Simply writing to memory once and then throwing it away after a time almost never happens in real programs.

A more realistic test would force the VM to get the best working set in limited memory. For example, two processes allocating (and writing random data to) 100% of real memory (each) and regularly accessing 40-55% of that would stress things nicely. Adjust the accessed percentage to test the system in various situations...

Re:The author's tests don't look very stressful...

[Anonymous Coward]

A truly realistic test would be to run a process that didn't blow out the available memory.

With memory prices being what they are today, paging is more or less useless.

If our DB server or webservers ever hit swap nontrivially, we'd fire our sysadmin for being an idiot.

Re:Pournelle ?-fiction.

[Anonymous Coward]

What are your reasons for categorizing people in black-and-white as "liberals" and presumably "conservatives"? Moron.

I hate you

[Anonymous Coward]

Perhaps the reason slashdot has so many trolls is because people like you are just plane rotten dirty scoundrels.

I hate you (_O_) (_O_)
Yes, I hate you /
I really hate you /_
I hate you alot _______
I hate you quite a bit / \

I hate you.

Dual P4 boards

[sirsnork]

Since when was it even possible to have a Dual P4 board? Dual Xeon: Yes, Dual P4: No!

just do your own "magic" vm :)

[warrior]

Perhaps a "magic" VM could solve these problems. The magic here would involve the CPU checking it's instruction stream against it's hardware assisted TLB. It could speculatively pre-fetch pages before they're need. Hrm, though, we are still talking about ms access times to disk, oh well, some day... so try this: if his code is quite regular (not too many branches), he should be able to do this. Okay, so this could be quite difficult, but given the description of his program, just make a "false" access to some of the data you're going to need in the future, causing the os to load that page, and hoping that it doesn't swap out your current page, you should already have your data in memory by the time your code rolls around to the fetched page. Keep in mind that your program is going to have to be multi-threaded so that it doesn't choke on the "false" access. This technique would require intimate knowledge of your compiled program, but if you're working on a program that uses data of this size and minimizing waiting on page swaps would save you minutes/hours/days of compute time it'd be well worth it.

Michael

Re:Pournelle ?-fiction.

[(outer-limits)]

Where do you think WWII came from. Without the crushing poverty of the reparations from WWI, WWII might never have happened. Which logically leads to the Marshall plan being a vital part of the West beating the Stalinist Russia. Russia wasn't beaten with weapons, it was beatan because the Russians themselves did not want Communism. Without a Marshall plan, a beaten and destabilised West Germany would have given the Communists a chance of outdoing the West with the East, with a weak West much more open to invasion from the East.

Bad taste.

[Anonymous Coward]

Bad taste that man, really.

Re:Red Hat 2.4.9 is a very good kernel, fast...WHY

[Anonymous Coward]

Rik van Riel's VM which was introduced in 2.3.x

Actually, in 2.4.0-testX (test9, IARC).

Lemme Guess...

[Threed]

"Too much work at interrupt"?

Switch to netgear cards / ng_tulip driver. I've never tried to gauge them for performance, but at least the driver is solid.

Need help building 2.4.18pre2aa

[Anonymous Coward]

Anyone know what I can do to build this kernel

using SuSE 7.2 with vanilla 2.4.9
gcc version 2.95.3 20010315 (SuSE)

It fails with

make[3]: Leaving directory `/usr/src/2.4.18pre2aa2/drivers/video'
make[2]: Leaving directory `/usr/src/2.4.18pre2aa2/drivers/video'
make all_targets
make[2]: Entering directory `/usr/src/2.4.18pre2aa2/drivers'
make[2]: Nothing to be done for `all_targets'.
make[2]: Leaving directory `/usr/src/2.4.18pre2aa2/drivers'
make[1]: Leaving directory `/usr/src/2.4.18pre2aa2/drivers'
make CFLAGS="-D__KERNEL__ -I/usr/src/2.4.18pre2aa2/include -Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fomit-frame-pointer -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=i686 " -C mm
make[1]: Entering directory `/usr/src/2.4.18pre2aa2/mm'
make all_targets
make[2]: Entering directory `/usr/src/2.4.18pre2aa2/mm'
gcc -D__KERNEL__ -I/usr/src/2.4.18pre2aa2/include -Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fomit-frame-pointer -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=i686 -c -o memory.o memory.c
memory.c: In function `copy_page_range':
memory.c:227: warning: implicit declaration of function `kmap_pagetable2'
memory.c:263: warning: implicit declaration of function `kunmap_vaddr'
memory.c: In function `pte_alloc':
memory.c:1435: warning: implicit declaration of function `unlikely'
memory.c:1443: `new' undeclared (first use in this function)
memory.c:1443: (Each undeclared identifier is reported only once
memory.c:1443: for each function it appears in.)
memory.c:1450: warning: implicit declaration of function `kmap_pagetable'
memory.c:1424: warning: unused variable `pte'
make[2]: *** [memory.o] Error 1
make[2]: Leaving directory `/usr/src/2.4.18pre2aa2/mm'
make[1]: *** [first_rule] Error 2
make[1]: Leaving directory `/usr/src/2.4.18pre2aa2/mm'
make: *** [_dir_mm] Error 2

specialized boxes

[timothy]

Yes!

Specialized boxes are what really show off what nice things commodity hardware and malleable source code are capable of.

I'd like to see a lot more companies sell products analogous to the video-editing stations offered by Linux Media Arts; offer a known-to-work combination of hardware and software so that people at least have the opportunity to flock to it.

Audio workstations, please? I'd pay a thousand bucks for a $500-in-parts workstation (which these days can be quite a powerhouse in absolute terms), if I could hook a few pre-amps or a portable Mackie mixer up to it and start multitracking.

(I'm aware of a lot of all-in-one recording consoles, and even own one, but a) screen real-estate is nice b) there are a lot of things which a "real computer" could do as an audio workstation which would be fun to experiment with and c) ever tried to enter in text labels with the insane rotating-knob method of an Akai DPS1200?)

OTOH, the people who most care about this sort of benchmark often *are* the people who want them for servers, or who are themselves developers trying to benchmark raw performance to help them make programming / design choices, so it's not that surprising. For a lot of specialized uses (TiVo, say), adequate performance to do their job doesn't seem to be a problem at the moment ...

timothy

More nits

[meehawl]

I'm going to disagree with this notion of evolution. Evolution is not undirected. The current environment gives a good deal of direction to the sorts of evolution that occurs. For example: evolution appropriate to tropical beaches is unlikely to occur in the arctic tundra.

Actually, that is wrong. Evolution happens in an undirected sense. In Artic environments, mutations both beneficial and non-beneficial for that environment occur. There's no doubt about Evolution.

The Theory of Natural Selection, however, adds a directed development, in the sense that beneficial mutations will produce more offspring, and in the long run should predominate. There's the doubt about Natural Selection... in the long run we are all dead.

Re:More nits

[darkonc]

Natural selection is an aspect of evolution -- and the aspect which makes it directed.
The mutations are random, but the resulting evolution is directed by the needs of the environment.

Re:More nits

[meehawl]

Environments don't have needs. People see a direction to evolution because of an anthropomorphic error, a illogical conflation of local experience with hubris.

The universe we observe has precisely the properties we should expect if there is, at bottom, no design, no purpose, no evil and no good, nothing but blind pitiless indifference.
A quote by one of everyone's favourite uber-athiest, Dawkins.

Re:More nits

[darkonc]

I guess I shouldn't be surprised given the subject of the thread....

Evolution is directed by the needs generated by (or arising from) the immediate environment.

and -- yes, I agree. The earth and nature have no needs, per se. As an environmentalist, I would say that we have needs, and if we sate our needs and desires in a way inconsistent with what Mother nature 'needs' (i.e. what would be required for it to stay stable and supportive of thriving human life), we're going to pay bigtime, in the future.