Rik van Riel on Kernels, VMs, and Linux

Andrea Scrimieri writes " An Interesting interview with Rik van Riel, the kernel developer, in which he talks about the Linux's VM, particurarly about his own implementation (which was recently adopted in Alan Cox's tree). With some controversy towards Linus Torvalds. "

Comments

Wondering...

[prisoner-of-enigma]

I'm wondering why both VM's can't be included in a distro and allowing the end user to select the one he/she wishes to compile into the kernel? Are the two implementations THAT mutually exclusive?

BTW, this kind of bashing between the high priests of Linux is not good. You can bet your bottom dollar that MS is going to use this conflict to fuel their propaganda machine, saying Linux is a fractious OS run by a bunch of young upstarts who can't agree on anything.

Re:Wondering...

[reaper20]

I think this kind of infighting is great, as long as it doesn't get out of control.

We get to see arguments and competing subsections of the kernel - this is SO one of the most underrated benefits of open source. Users of some other OS's don't have this benefit. I am not a programmer, so to me, I don't really understand/care the benefits of different VM systems, but I know that some other, smarter people do, and they're all trying to figure out the best way to do it, and that's good enough for me.

I say let them go at it, let the best code win - it can only help us. And who cares how MS construes this, I'd like to see them open up their development model and see what kind of conflicts they are having.

OT - but kerneltrap.org has a good interview with Alan Cox today....

Re:Wondering...

[kaisyain]

this kind of bashing between the high priests of Linux is not good

Sure it is. How else are we going to find out where our disagreements are and work through them? Or, at the very least, learn not to make the same mistakes in future projects. The problem of the Linus bottleneck has been known for a long time. This "bashing" is not new, it's just current.

Having Linus Torvalds around helps insure that, for the average user, there is no splintering of development effort -- just use the Linus kernel. But it also severely hinders improvement because you are limited to what Linus likes or dislikes.

And despite what may be the common conception on /. Linus is not an all knowing genius. He makes mistakes. Perhaps this is one of those mistakes. The real question is whether the benefits of the stewardship he provides compensates for the hindrances his authoritarianism creates.

Re:Minor nit...

[Rik van Riel]

Both Alan's and Michael's kernels are including my -rmap VM now.

This is quite interesting since I haven't begun tuning -rmap for speed yet ;)

Good decision to remove Rik's VM from mainline.

[PastaAnta]

I think it was an excellent decision of Linus to remove Rik's VM from the mainline kernel. If not for technical reasons then for political reasons.

Rik's VM obviously needed to be fixed and/or tuned, but apparently lacked the necessary attention from Rik. If Linus had not removed the VM, it would probably have been the situation for a while. Instead we now have TWO VM's which are rather stable and Rik working full speed to make his VM the best.

Competition is good! Which VM will be the best for the future will be determined by Survival Of The Fittest(tm)

It can be argued though, that it was not the right time during 2.4, but Andreas VM seemed to stabilise rather quick with the high level of attention to the problem. Sometimes it takes drastic measures to get results...

Interesting

[4of12]

I have a lot of respect for Rik van Riel, but I think that Linus made a good decision to "cut bait" on his VM implementation for 2.4.

It was not that Rik's ideas were bad, it was just that their complexity and implementation were going to take too long - they should have been hashed out in 2.3 instead of 2.4.10.

I'm looking forward to having Rik prove his reverse mapping technology implementation in 2.5.

May the best ideas ultimately win, and may the giants of the kernel not take offense at each other. It would be a real shame if something stupid like Linus' lossy source code control system put off Rik so much the Linux community at large lost his wonderful contributions.

Here's to hoping that Linus gets more sensitive in some cases, and that Rik gets less sensitive in some cases.

Re:Multi-proc 'big iron'..

[Rik van Riel]

Indeed, it is important to optimise the VM to work right on such large machines. I guess what I wanted to say is that the VM isn't just optimised for high-end machines, but also for machines on the low end.

To be honest though, optimising for machines of different sizes really is a no-brainer compared to having to make the VM work with really diverse workloads ;)

On developer spats and high drama

[ajs]

Open Source's biggest PR dilema is this sort of argument.

Make no mistake, every company has developers that do this. There's two differences in the Open Source world: 1) you can't just fire an Open Source developer who won't "play ball" with management's edict 2) it's usually public.

These are actually both really good things. The fact that you can't silence someone leads to repeated analysis of a problem. OSS' biggest benefit is that it brings massive peer review to bare not just on the code, but on the process.

The fact that it's public feeds into that, and is equally good.

The problem is PR. The Linux kernel is starting to look like anarchy to non-developers. I suggest that the process works, so we should all take a deep breath and leave it be. However, we all need to take the front lines on PR. Spin is all-important. This is not a "spat" or a "fight", this is "parallel development" and "peer review". The joy of this kind of spin is that, unlike most spin, it's TRUE! This guy is pissed at Linus. Linus has dumped his code. Yet, the two of them keep working hard to meet their customers' demands and producing what they feel is the best possible product.

Please, don't foster the idea that we're a bunch of anarchists producing code that's any less functional than the rest of industry, because quite the opposite is true.

Re:Patch bot is the answer?

[Rik van Riel]

The problem is simple: maintainers of any parts of the kernel get flooded by email, maintainers of the whole kernel (Linus, Alan, Marcelo) get flooded even worse.

You really cannot expect these people to read all their email all the time, so patches and bugfixes get lost and may need to be resent various times before they get noticed.

Add to that the fact that many of the people writing these patches are also extremely busy and may not get around to resending the patch all the time (I know I don't).

The solution here would be to have the patch re-sent automatically as long as it still works ok with the latest kernel version ... this can all be checked automatically.

Fear Factor

[ChaoticCoyote]

An honest environment -- such as fostered by "free" software -- is both good and bad. On one hand, I (as a programmer) am comforted to read the kernel mailing list and other resources that let me know exactly what is happening with my tools. I don't need to wonder what's happening with "free" software -- and this is more comforting to an engineer like myself than is the closed-door, silence-is-golden, hide-the-bugs policy of a Microsoft.

On the other hand: Show this interview to an MIS manager who need 24/7/365 reliability, and she is going to be very nervous about deploying a Linux-based solution. You can talk until you're blue in the face about reliable distros and the open road to sofwtare quality -- what the MIS/corporate person sees is chaos and feels a lack of COMFORT .

"Out of sight, out of mind" is a philosophy many people adhere to, especially when dealing with complex issues they can not or do not want to grasp. From waste storage in Nevada to the the war in Afghanistan, most people lack the time and initiative to understand what is really happening; they go on appearances and marketing, and ignore complex and disturbing facts.

Technology is no different. The MIS manager doesn't want to hear about VM conflicts or file system bugs or different kernels -- such issues are beyond their capability and desire to understand. Buying Microsoft is (or was, until recently) comforting, because no one ever saw the internal debates and code battles and what-not that any development team expresses. Even recent security disclosures about WinXP are unlikely to shake the faithful -- but those same people will run in fear from the blunt honesty of Linux.

Ignorance may be bliss, but it can also get you killed. I know people whose lives depend on cars, but they have no knowledge of how to check the oil. Most MIS managers simply want to drive software; if it looks good (like a Jeep Liberty), they don't pay attention to whether it is safe (the Liberty performs poorly on crash tests).

I doubt, however, we're going to change human nature -- and I'd rather have spirited debate and even some nasty contention if it means that people are striving to make Linux the "best" it can be.

OOM Killer must die

[Salamander]

Rik is an extremely bright (and likeable) guy, but his adherence to the OOM killer concept is disappointing. I've seen a lot of dumb ideas gain currency in the computing community or some part of it; OOM killer is the dumbest. If your process was allowed to exist in the first place, it should not be killed by the VM system. The worst that should happen is that it gets suspended with all of its pages taken away. If that doesn't free up any memory then neither would killing it (modulo some metadata - read on). If there are other processes waiting for the one that's suspended, then eventually they'll go to sleep, their pages will be released, and the suspended process will wake up - which won't happen if you killed it. There are only two differences between the two approaches:

• Suspension does not take irrevocable action; the suspended process can still be restarted.
• Suspension bears the cost of retaining the metadata for the suspended process so it can be restarted.

The usual whine from OOM-killer advocates is that you can still get into a situation where all of that retained metadata clogs up the system and essential system functions can't allocate pages. However, that's preventable too. All you need to do is preallocate a special pool of memory that's only available for use by those essential system processes - either individually or collectively. The size of that pool and the exact details of how it gets allocated (e.g. which processes are considered essential) could be treated as site-specific tuning parameters. The same idea can then be further generalized to allow definition of multiple private pools, creating a semi-hard barrier between different sets of tasks running on the system (if you want one; the default pool is still there otherwise). This actually fits in very nicely with other things like processor affinity and NUMA-friendly VM, which I know because I once worked on a kernel that had all of these features.

In short, there's no need for the OOM killer. Plenty of systems, many of which handle extreme VM load much better than Linux, have been implemented without such a crock. Rik contends that a lot of people make suggestions without actually understanding the problem, and he's right, but I also submit that sometimes he also rejects suggestions from people who do know what they're talking about. This row has been hoed before, and Rik's smart enough that he should know to avoid the NIH syndrome that afflicts so many of the other Linux kernel heavyweights.

Re:OOM Killer must die

[Salamander]

If you have enough swap space (whether you call it "tertiary" or not) for all writable memory, you're not overcommitting. By definition.

You might (or might not) be overcommitting. Up to you. However, even if you are, you're not waiting until the last second and then going postal instead of taking concrete steps sooner to avoid total memory exhaustion. For example you could say that, once you start dipping into the overcommit pool, fork() will start failing but existing processes can continue. You could say that only certain processes that are being allowed to run to completion will be able to allocate new swap space; anyone else will just get suspended the first time they try. Once you have set a high watermark somewhere short of total exhaustion, you can do any number of things, even if you're overcommitting. Some of those measures are pretty drastic, but still better than the OOM killer.

To a certain extent, perhaps, these "softer" approaches just slow down what might be an inevitable march to OOM. In theory, you could still reach the total-exhaustion deadlock that OOM-killer is supposed to deal with, although it really doesn't because it doesn't in any way guarantee that your system will really be any more useable than if the deadlock had occurred. In practice, though, you'd be hard pressed to find a system that (a) allows overcommit, which is only necessary with VM systems that are broken (wrt how much swap they allow) to start with, (b)takes such drastic measures before going OOM, (c) does in fact hit OOM anyway, and (d) would benefit from an OOM killer if it had one. Without such an existence proof, claims that an OOM killer is necessary are pretty bogus.

As I've said, these aren't new ideas just off the top of my head. These are approaches that are proven to work. Ask yourself: how is it that so many systems get by just fine without an OOM killer? There are answers out there.

It's funny that you accuse Rik of NIH

Actually I didn't. I accused other Linux kernel hackers of NIH, and tried to warn Rik about becoming more like them. I know Rik's smarter than that, but sometimes even smart people submit to "common nonsense".