Torvalds Has Harsh Words For FreeBSD Devs

An anonymous reader writes "In a relatively technical discussion about the merits of Copy On Write (COW) versus a very new Linux kernel system call named vmsplice(), Linux creator Linus Torvalds had some harsh words for Mach and FreeBSD developers that utilize COW: 'I claim that Mach people (and apparently FreeBSD) are incompetent idiots. Playing games with VM is bad. memory copies are _also_ bad, but quite frankly, memory copies often have _less_ downside than VM games, and bigger caches will only continue to drive that point home.' The discussion goes on to explain how the new vmsplice() avoids this extra overhead."

Comments

Wrong Side of Bed?

[AKAImBatman]

Ok, let me see if I've got this straight:

• Copy on Write saves you real memory, cache memory, and CPU time by pretending that each forked process has a true copy of a memory segment when it in fact is looking at the original. That is, right up until a fork tries to write to that memory location, in which case an exception is handled by making an actual copy to a new location and allowing the write.
• Linus believes that the exception will occur enough in real world usage that it will be slower than just doing the copy in the first place.
• Linus wants to push the manual use of zero-copy memory sharing through the vmsplice() routine. He believes that the programmer will always know better than the system when to share memory.
• Linus doesn't like "VM Games" despite the fact that Virtual Memory, Memory Mapped Files, Disk I/O, Write Caching, etc, etc, etc, are all already "Memory Games" and "VM Games"

Do I have that right?

If so, I'm not really seeing his issue. Or at least not as hard-line as he sees it. The issue of memory copy performance is a tricky one, especially since CPU cycles are not the be-all to end-all of performance. Does the exception generated really cost that much more than he believes, or is it often eclipsed by the cost of the extra memory read/writes and CPU waits that are normally generated by a copy? Is it really feasible to expect program developers to do manual memory management in a day in age when programs easily weigh in at hundreds of megs?

I'm just not sure that Torvalds is really looking at all sides of this. He may be right, but I'd like to hear more discussion between the *BSD guys and Torvalds before we put this matter to rest. And preferrably without the insults this time. :-/

Links:

Copy on Write as explained by Wikipedia
FreeBSD page on Zero Copy Patches
Duke Uni Research

Re:Wrong Side of Bed?

[qwijibo]

I don't consider myself an expert in kernel programming, but I definitely think someone is off base if they're expecting programmers as a whole to do the right thing. Many programs seem to work by coincidence rather than design. People didn't do their memory management right in the days when it was necessary. Now that a lot of people are moving towards languages that handle the memory management for them, I expect even fewer to worry about it. That does mean that the programmers of the programming languages are the ones who are responsible, but I'd personally rather have the kernel take a more active role in memory management.

Re:Wrong Side of Bed?

[mrsbrisby]

One thing that concerns me about making all of these copies is that it seems like a quick and easy way to blow out your L2 cache. That could in the long run have a worse performance penalty than having to play the VM tricks with CoW.

No it won't. The only way to avoid the copies is to avoid the pagefaults. Since userland doesn't get explicit notification in FreeBSD of when the pages are safe to use, the process should wait as long as possible (e.g. until malloc() starts failing)

The idea Linus pushes here is explicit notification- via select() or poll() and returnable via recvmsg(). That way the userland knows exactly which pages can be reused.

The result is that it's faster and easier to develop userland programs to take advantage of it. It's also easier to degrade gracefully into read()/write() until the FreeBSD people see the light and add support for this too.

It's really a clever idea.

Re:Wrong Side of Bed?

[Nato_Uno]

He doesn't care what the FreeBSD developers are doing... ... until someone advocates copying their ideas into the Linux kernel. Then he cares very much.

He's not saying "The FreeBSD people should rewrite that part of their OS," he's saying "don't put that crap into the Linux kernel."

Re:Wrong Side of Bed?

[hackstraw]

One thing that concerns me about making all of these copies is that it seems like a quick and easy way to blow out your L2 cache. That could in the long run have a worse performance penalty than having to play the VM tricks with CoW.

Right. Especially with multithreaded apps as Linus pointed out. Also the TLB misses could get expensive as well, and again the TLB misses will be more of an overhead with multithreaded apps.

I don't believe that COW is completely evil. It exists, obviously for a reason, but I would agree with Linus on a much less harsh tone (depending on mood).

Oh, and isn't "VM" a trick to begin with?

Re:Wrong Side of Bed?

[outZider]

Here's my -1, Troll.

Funny that we just had an article about how many Linux users and enthusiasts exclude other people by being complete dicks, and here you are, acting like a dick. Of course, I don't know you from Joe Blow, so maybe I just misunderstood your obviously angsty response.

"That's obvious."
"Correction: when _you_ start using up a lot of memory Linux totally sucks. When I start using up a lot of memory, Linux acts exactly as I expect, and better than FreeBSD."

Linux acts exactly how I'd expect, too. It completely sucks when it comes to memory and process management. Linux may have a better threading kernel, but that's the only thing that seems to save it in the real world. After only six years of administering servers professionally and for my own use, it has come down to Linux on the desktop, and FreeBSD for Real Work(tm). Many large companies that depend on their data agree with me, and those who use Linux or Windows just throw more machines at the problem.

At least Linux is free compared to Windows, right?

Re:Wrong Side of Bed?

[Just Some Guy]

No BSDer has been willing to reproduce the tests, as it will only confirm what the marketplace has already decided ... Linux is the superior OS.

...but still inferior to Windows, right? I mean, we're only looking at number of installations, after all. Furthermore, McDonald's clearly has the best hamburger and Velveeta beats Hoffman's Super Sharp.

I like Linux. I'm typing this on a Gentoo box. However, I'd never pretend that it's better in every single aspect than any other OS in existence. The BSD guys have a few tricks up their sleeve, and even Redmond manages to get things right on rare occasion.

Re:Hmm, where have I heard this recently?

[dow]

Are the *BSD people are nicer? Or at least more tactful?

No. Thats why there is more than one BSD. Issues come up, and booom crash goes the fork. Pity.

Linus sometimes calls people idiots

[cpu_fusion]

And in other news...
Grass is green;
Oil is overpriced;
Absolute power corrupts absolutely.

He's just a developer

[krewemaynard]

Here we go again, imposing "role model" status. Linus is just a guy. Sometimes he gets his buttons pushed, sometimes he's doing the pushing. BFD. Maybe you'd be a little pissy too if Slashdot posted a story every time you did or said something. Linus Prefers Gas-X, Says Bean-o Is For Douchebags. Who cares? (BTW, Linus didn't really say that, I made it up. Don't wanna get the Bean-o people on his case too.)

As far as this whole VM thing goes, time and testing will tell the true story. Meanwhile, maybe we could try NOT deifying Linus (any more)?

Re:Linus is turning into a dictator

[Bogtha]

Dictator? Are the FreeBSD developers somehow unable to keep their implementation now that Linus deems it stupid?

You might feel he's being a bit of an arsehole, but that doesn't mean he's a dictator. He's not stopping anybody from doing anything, he's merely sharing his opinion of a development technique on a mailing list dedicated to discussing the development of his kernel.

Just say what you mean

[lymond01]

"I claim that Mach people (and apparently FreeBSD) are incompetent idiots."

Linus, who's becoming more outspoken as he ages, needs to find that line between anonymous forum geek and software spokesperson...and then not cross it. Calling anyone an incompetent idiot is both non-constructive if you're hoping to improve a situation, and just plain unfriendly in an area where cooperation amongst developers is so crucial (open source).

Re:Linus is turning into a dictator

[Lumpy]

No he is simply getting less tolerant of "sloppy" programming. He is one of the very very few that believes in doing it the way that gives you the best speed. Something that takes 4+ operations compared to a way of doing it with only 2 operations and you get less problems = performance gains that add up. Just because your typical machine has 4 dual core 8Ghz processors and 22 terabytes of ram does not mean you can slack off and write the whole thing without paying attention to performance.

the BSD guys have their reasoning and if you read more info about this it is not a shot in the dark that Linus is taking but he is frustrated that after many discussions nobody cares as much as he does on the performance issues.

Go back and read what Linus did back in the early days, it's no different today than what it was in 1990, he will call a duck a duck.

Lesson on Tact..

[corellon13]

Linus, you may be right and you may be very smart, but you should try a little tact. Here's a good definition for it that I learned from a drill sergeant: "Tact is the ability to tell someone to go to hell and look forward to the trip."

Being nice and respectful doesn't mean you can't tell it like it is.

Re:Linus is turning into a dictator

[rtaylor]

No he is simply getting less tolerant of "sloppy" programming.
You'll forgive me for taking that with a grain of salt so long as memory over-commit remains the default mode of operation within Linux.

He has the stones to back it up

[Gothmolly]

Linus has frequently called people idiots, and ignored patches, and done stuff his own way for a very long time now. He's quite successful at it. Perhaps what most people need to realize is that he is that good, that he can. The average read-Slashdot-during-work-while-coding Slashdotter is not in his league, so decrying his adhominem attacks, or "I would do X instead" arguments just dont hold much water.

Re:Wrong side of compiler

[arivanov]

More likely he had some really bad acid the previous night.

After all he did more than 6 revisions of the Linux VM using CopyOnWrite before this latest fad.

Possibly more.

Off the top of my head that is at least 1 in the 1.2 tree, 1 in the 2.0 tree, 1 in the 2.2 tree, 2 in the 2.4 tree and more than 2 in the 2.6 tree, all of which being CopyOnWrite and at least some of which has been hailed as the next best thing after hot bread.

As far as the technical point he is possibly correct for x86 where COW goes through the fault mechanism and causes some TLB and cache abuse which is really bad on modern CPUs. I am not sure as far as other architectures are concerned, because IIRC (I may be wrong) the memory mapper hardware on the old Sparc was designed for COW in first place.

Anyway, before calling somebody else an idiot for something you have happily done for 10+ years till yesterday it may be nice if you look at yourself in the mirror. Because I never remember any branch of FreeBSD reaching the point where you can do a find /usr -exec cat {} > /dev/null \; to hang the system. That is 2.6.16 at your service (from rc4 onward) on at least two x86 subarchitectures where I had the time to test it. That is besides the unkillable processes in [S] state on an nfs flock in 2.6.14 (yep, that is a gem which no other unix has managed so far), besides the OOM idiocies in 2.6.10, besides deliberately making it absolutely impossible to backtrack any more interesting patch to a previous kernel without employing a team of kernel developers because the VM and locking is not compatible across any kernel version since 2.6.9 and even when it is something else is changed like the tty layer, besides.... Aarghh.....

What harsh words?

[Inoshiro]

Andy went out and said that he thought the Linux approach was wrong, and archaic, and that people should go and wait for GNU.

Linus said that he felt this was wrong, and that being a prof is no excuse for Minix being the mess it was (and Minix was a mess in the late 1980s/early 1990s). He also apologized if he came off as too harsh for his writing about how people should be able to throw away an old design in favour of a new one anyway, etc.

It was very polite compared to some of the non-Andy/Linux replies.

Re:RTFA, please. Or at least my summary here.

[joe_bruin]

Thank you. I've read more than 30 high-modded posts in this article, and yours is the best explanation of the issue by far.