Slashdot Mirror
Debate on Linux Virtual Memory Handling
xturnip sent us a good piece running over at Byte about Linux's VM. Somewhat more technical then the stuff we usually see online, this one talks about different VM systems, and the egos in the kernel. Its worth a read.
Oh please. Have you ever worked on a commercial software project? I've seen just as much if not more ego in moronic engineering team meetings at my enterprise software company. Without a single strong technical leader OR a group of smart people who all equally respect each other's opinions, the SAME THING happens on a commercial project. I've watched a Director of Engineering call meetings almost every day for 3 weeks in a row because he didn't know how to solve exactly this sort of problem. In the end he just decided to go with what the person with the most years of experience said and to get the CEO to give him blanket license to make that technical decision, though none of the other engineers agreed with it - they were all too conflict averse to speak up and too afraid about losing there jobs just as the economy was tanking (he made a bad decision indeed and the project suffered greatly for it, getting delayed by 3-4 months and even then never delivering a large portion of the promised features because this architectural decision made them impossible). That company (mine, unfortunately) is most likely going out of business soon. So don't give me this crap that ego only adversely affects Open Source projects.
Moshe Bar argues two points I vehemently disagree with:
(1) Alan made a mistake in not switching to Andrea's VM. Alan is trying to maintain a stable kernel. Switching out large chunks of the VM is the last thing to do to achieve those goals. Alan will switch in due time.
(2) The preemptible kernel is unfit for certain scenarios. Everyone I know loves the preemptible kernel. It gets good reports on lkml and the kernel news sites - Hell, it even got good comments here!
I realize this is an editorial, and I understand everyone has an opinion, but if it isn't true it isn't true. An opinion can't contradict fact.
Tim
Look at the success of EGCS and GCC. That was a successful split and merge. It led to a better GCC in the end while supporting both stable and advance versions of gcc in the interim.
Joe
Joe Batt Solid Design
i think it is mainly to do with stability and the proven ability of the old vm code. basically, the new vm code was a complete rewrite from scratch and was incorporated into the main kernel straight away. the problem therein is the fact that the code will not have been as throughly tested and proven as the old vm. it may well be that the new vm is rock solid but it hasnt been in use for as long as the old vm to prove it. what alan is doing is sticking with the old vm as it is pretty much proven to work well and not fall over.
its not exactly trivial to rewrite an entire vm so there are bound to be problems with it. these problems come out through testing. i would have thought such a major rewrite would have been put in a development kernel first rather than into a "stable" kernel tree. that way, developers can test it first and iron out any problems rather than everyone upgrading to the new vm _then_ a major problem found.
the new vm may be brilliant and fast, but alan has a point in sticking with the old code. major rewrites should belong in development trees until fully ready for a stable release.
Not really; it's hard to quantify the benefits of a text editor; and although one may make a statement such as, "Editor foo enables me to edit files 50% faster," it doesn't mean much; because the reference is subjective to personal preferences.
With VM in the kernel, it's pretty obvious when things aren't working as well as they should -- mainly because it is possible to write test programs (scaffolding) that check to see if a designed system is performing to specifications -- read _The_Mythical_Man-Month_ for an excellent explanation of the merits of specification-based "scaffold" testing.
--
I Hit the Karma Cap, and All I Got Was This Lousy
Good, this would be an interesting benchmark.
Then, I changed the memory down to 192-MB RAM so as to stress the VM system more.
ok, this is fair, but you should also run with the same memory configuration you originally ran.
I also upgraded to the latest stable versions of Sendmail (8.12.1) and MySQL (version 3.23.42). Finally, I compiled everything with the latest version of gcc, 3.0.2, and tuned the two instances to the best of my knowledge (softupdates and increased maxusers for FreeBSD, and untouched default values for Linux).
NO!!!! why would you do this? Don't you want to know how the earlier linux/FreeBSD kernel compares to a later ones? Now instead of modifying one variable you've modified 3,846 variables. It's going to see if any improvements in FreeBSD/Linux are due to an updated kernel, compiler, mysql, etc etc. Go back to your original setup and only change the kernel, since I believe that's what you want to benchmark.
In addition to this it seems that he has implemented VM with reverse mapping also. Therefore it should be clear that he previously thought this was the best method. I've understood that the issue between Rik's and AA's VMs is that Rik's is optimized for normal swapping and AA's for OOM case. Because VM performance really matters only when OOM happens I think AA's should be superior. The real difference depends on benchmark, of course.
Both systems seem to be somewhat equal. AA's needs less swap but Rik's is claimed to be better performer. If AA's system is simpler then that's what should be used. Select maintainability over questionable performance increase. This is like quicksort - there's a point when you usually get better performance bubble sorting the little pieces quicksort generates during the whole sort. The smart version isn't always the best. Nowadays CPUs can easily do a bunch of dumb operations faster than one smart operation.
_________________________
Spelling and grammar mistakes left as an exercise for the reader.
Isn't that basically Microsofts argument as to why its ok for them to be a monoply? That competition is not efficient in the software industry?
While I normally take issue with the way Linus bullies a kernel issue based on what he perceives is technical merit, I have to agree with him on this one. First, if a better way of doing things has been found, even though it's in the middle of a stable series, it should be changed in order not to propagate wrong coding. I've been coding for a long time now and I still believe that if an error, bug or better coding scheme is found, it be implemented as soon as possible.
The problem with leaving the change till the 2.5 series is that the 2.5 series is nowhere in sight and development kernels usually take more than a year to cycle through (no matter what the kernel hackers say). The fact that 2.5 hasn't even begun may be an indicator of how long 2.5 will take to finish.
I really hate to say this, but I'm wondering if jumping ship to freebsd (etc) makes sense. I've been a major linux supporter for quite a long time, but I know that the *bsd guys have had their act together (good smp, good networking under load, etc) for a long time.
would it be all that crazy to adopt the VM system from the 'establishment' (bsd)? frequently the linux codebase DOES borrow from bsd. why is the VM system all that different?
--
"It is now safe to switch off your computer."
Um, not meaning to tell you how to do your job, but if your admining Linux machines with new kernels, and _not_ following the lists, or Kernel Traffic at least, then I don't think you can really complain about not hearing about it. It's been well known scince 2.4.[012] that there was a VM issue at heavy load.
Actually, I'd say they're more like 'sporks' than 'forks'. Nobody who makes them intends for them to take over from the main Linus kernel tree.
The price of having virtual memory is terrible performance once paging between active processes starts.
When that happens, you are running a lot more processes that can fit into memory. Without VM you would not be able to do that at all.
A basic problem with shared libraries is that you load in the whole library, needed or not, when you need any function from it.
False. Any decent VM does demand paging. Only the pages that are needed are loaded from the executable. The parts of the program that are never executed are never loaded from disk, notwithstanding read-ahead optimization. A shared library is just an extention of the executable so the same rules apply. Further, a shared library can be used by multiple processes and only *one* copy of it is loaded into memory.
I'd argue that it's time to go back to a swapping model - all of an app has to be in before it runs.
That would be absolutely stupid. It would slow down the system tremedously. Se above about demand paging.
Without VM, you would need to increase the memory requirements by a factor of N, where N is the number of processes running concurrently. Further, the startup time of each process would always be slower since all of the code would have to be read in memory. With VM part of it is already there (shared libraries), and the code is loaded on demand.
In short, this is the biggest pile of uninformed garbage. You *really* need to take an OS course before you can talk about OS design.
___
If you think big enough, you'll never have to do it.
No, sorry but I dissagree.
A clever VM system is more important than ever, when you combine it with an effective disk cache.
Yes, Ram is cheaper and faster than ever, but stopping your OS from using a swap file/partition is gonna stop your OS from efficiently using your ram.
Your machine should be allocating as much memory as possibe to a disk cache, even if this reduces the available memory to the extent that active processes start paging, because the swap file paging is optimised by the disk cache too.
It is usually better to swap out pages and keep your cache large than to keep rarely used pages in memory at the expense of cache, because even if you need those pages they will quite likely be in the cache.
Even if you have a shedload of memory (especially so!) you will get better use of your memory if you use some of it as a disk cache than if you don't page rarely-used pages to disk.
Hey, Linus f***ed up with accepting the original RVM for 2.4. And now he was between a rock and a hard place. "Marketing" considerations meant that 2.4 needs to be "non-experimental"; that the client base could go to the 2.4 version, and use it with little concern that their server would crash. That would not have been the case with RVM versions up the last month.
So what was Linus to do? Keep dragging out 2.4 until RVM could fulfill minimum "marketing" requirements of stability? How long is that going to take? Do you want to wait and let M$ marketers talk about how amateurish Linux was; that professionals did not use Linux's current "stable" release, but a version that hasn't improved in 3 years?
So Linus decides to commit A REALLY BAD PRACTICE, and changes to a less tested VM over the initial 2.4 VM. Its another f**kup if AVM is buggier than RVM. (But Linus had reason to believe it wouldn't be so, with relatively limited testing.)
Is it still an f***kup even if AVM turns out to be more stable than RVM? If so, are you saying its preferable for new Linux development to be shutdown for another 6 months to a year? And Zdnet to opine on how the "stable" 2.4 kernel is DEMONSTRABLY unreliable? I'll take a "manager" that makes mistakes and makes decisions based on product survival over a manager that religiously follows an engineering practices manual.
There is no America. There is no democracy. There is only IBM and AT&T and DuPont, Dow, General Electric, and Exxon
No, there have been quite enough delays associated with 2.3/2.4 already. More than enough. And there will continue to be delays until the processes get ironed out.
What would have been preferable, IMO, would have been if more resources had been devoted to fixing and tuning the VM we already had (RVM, for good or ill). Linus could have put his foot down. He could have said "There will be no 2.4 VM except for RVM. The price for admission to the next round of VM redesign is that you help us fix RVM." People - notably Andrea - would have listened, and contributed more constructively. They know that Linus's good will is like currency. But Linus didn't say that. Alan Cox pretty much has, and kudos to him for having the courage to do so. What Linus did was take a bad situation and act in a way that nine times out of ten would make it worse. Maybe he'll get away with it this time because AVM in its current state is more robust than RVM in its current state, but that would actually be a bad thing because it will only reinforce the bad decision-making and we'll get burned next time instead of this time.
First off, are good reviews from places like ZDnet the goal for Linux development? Second, do you think it's better for the stable 2.4 kernel to be subtly, unpredictably unreliable? Better the devil you know, and all that.
Most importantly, what if Linus's gamble - and that's what it was - hadn't succeeded? What would the ZDnet reviews be like then? What kind of ammo would that provide for everyone who wanted to claim that open-source development processes weren't all they're cracked up to be? Yeah, it looks (so far, knock wood) like we've been lucky this time, but I don't think relying on luck is a good thing.
The two aren't as diametrically opposed as you make them out to be. Good engineering practices are good because they help increase either the speed or the reliability with which product can be delivered. Slavish adherence to any dogma is a bad thing, but so is the belief that everything you're doing is OK just because you managed to win one game of chicken. My point is that this scenario is going to be repeated. I'd rather encourage responsible driving than watch what happens when Linus plays one game of chicken too many and brings everyone else along for the ride.
Slashdot - News for Herds. Stuff that Splatters.