Slashdot Mirror
Linux 2.4 VM Documentation
popoutman writes "Mel Gorman has announced the availability of a guide to the 2.4 kernel VM
including a walkthrough of the VM code. Anyone interested in obtaining a solid understanding of the Linux 2.4 VM will certainly want to take a look at this documentation. Mel says that the effort is at least several weeks from being finished, but that he's releasing it now with the hopes of getting feedback to be sure he's on the right track.
He also notes that the 2.5 VM is still too much of a moving target for him to document it just yet." See also a Kerneltrap story.
We design aned document things first, work out the bugs in the high level design and then code.
Well maybe not all the time and with serveral itterations, but I only manage your credit raiting, not you kernel VM.
thank God the internet isn't a human right.
...that virtual memory works because of small, magical faeries and gnomes.
/syle
heh, it can stand for a few things in the realm of computers:
1. virtual memory (using secondary storage to allow the program to address more memory locations than there is in physical RAM)
2. virtual machine (a machine implemented in software on a real machine, which *could* have different opcodes, or the same ones as the real machine)
3. VM - the operating system by IBM which allows one of their mainframes to act like many (stands for virtual machine)
( Also, there's the VM in VMS, the DEC Vax and Alpha operating system, which stands for VIrtual Memory as in #1)
I've always wondered why, in today's world of gigabytes of memory in personal computers, why such a big deal is made about virtual memory..
I'm not trying to troll, and maybe someone can explain this to me, but as far as I know, VM is used when a computer runs out of RAM. You take some of what's in the RAM and toss it on to the hard drive. When you need it again, you reload it back into RAM and use it.
I've got 1.5GB of RAM in the box I'm typing this on, and the only reason I don't have more is that I have no use for it.
At my work, we've got some systems that can eat up 2GB of RAM, but we just put 4GB in the box and don't worry about it. We've found that if we let anything touch VM it slows things down way too much.
Anyways, I was hoping someone could enlighten me as to why VM has been such a big deal recently.
Define your function "Olog", please. Surely Mr. "Wagner LLC Consulting Co. - Getting it right the first time" couldn't have meant O(log n)... :)
People who don't know what it stands for would not want to read the document anyway, don't you think?
This is a good question -- there are several features the VM system allows for in addition to overcommitting physical memory. If your system has enough memory to run with no swap/paging space set up, the VM system still provides these features without ever paging or swapping out memory.
The first is demand paging. That is, only those pages of an executable that are needed are brought into memory, and on-demand. Yes, this saves memory, but more importantly, it makes program startup much faster. Without demand paging, the whole program would need to be brought into memory at startup. Nowadays, when disks are much slower relative to CPUs, than they used to be, this makes a big difference, especially in the Unix shell-pipeline style of programming, where you run many different short-lived programs.
Secondly, the VM system allows for shared memory segments between processes. This allows for shared memory, threads and shared libraries.
Finally, the VM system implements caching of the file system, which we all know and love as a good use for all the memory we stuff into our machines now.
Documenting the VM is a good idea. I hope it is accepted into the /Documentation directory in 2.4.x kernel tarballs but I'm not sure whether you'd get a reply from the maintainer, Marcelo Tosatti. I sent him a carefully written email in August 2002 documenting an invalid config in 2.4.19 which causes the build process to fail. It was disappointing he never responded and apparently did not fix the bug which is still present in 2.4.20 and more recent patches.
Why oil price increase equals economic trouble (Score: Interesti
Good point -- this is such a given on Unix systems that I didn't word it very well. What I should have said is that it enforces memory separation and protection between most memory segments and most processes, and allows for sharing of segments when explicitly setup. This is perhaps the most important thing the VM system does.
yeah, right. let's also integrate to the kernel Perl, Python, Tcl, Lua, Emacs (for Elisp), Guile, Hugs, OCAML, Bash, Apache (for PHP) and Gecko (I want my Mozilla to work faster too!). I wonder, why is X server still not there? And don't forget about at least two CORBA brokers: Gnorba (everyone would love faster Gnome) and OmniOrb (just for a case). Hey, let's put everything into the kernel! Ooops... It's not kernel anymore and it doesn't want to run either. What was the mistake?
Less is more !
VM does NOT mean just paging/swapping. that is a small part of it. mac os and windows users constantly misuse the term. the vm does demand paging, copy-on-write, file/page cache, buffer cache, shared mem, mmapped files, and more. the most important of which is protection.
I suspect it would be a waste of your time. Lets look at a bit of history.
There used to be a kernel-space HTTP server. It was integrated into the kernel for a specific reason: zero-copy access to the network interface memory. It was fast and relatively feature-poor. If it crashed (fortunately, a rare occurance), you got a kernel panic.
Along came a user-space, zero-copy HTTP server. It was faster and had a few more features to boot. Being a user-space program, if it crashed, you got a core-dump. It could also be run in a chroot jail, a gigantic step more secure than running in ring-0.
Two lessons can be read from this:
1. Don't integrate something with the kernel unless there is a specific advantage you hope to gain from it. Will making a JVM part of the kernel really speed it up? Are you sure?
2. Don't under-estimate the speed of a properly designed user-space Linux program. The kernel developers have done a magnificant job tuning the kernel and providing APIs for performance-critical apps.
OMG! You've reverse-engineered windows!
Why not fork?
I did a lot of research on VMs about a year ago, and while the different VM systems out there really are interesting they're all kind of limited by the whole "page-level protection" model we've stuck with since the '70s. There are a lot of clever algorithms in the FreeBSD and 2.4 VMs to deal with high load and paging, but the core VM stuff (mapping memory, protection, sharing etc) is largely the same, and is limited by the MMU capabilities of current CPUs. All CPUs basically seperate processes into unique memory contexts, and map (using some sort of page table or reverse page table) physical memory pages of fixed size into the contexts. This solution is sub-optimal for object oriented systems with large scale sharing (the page tables get unweildy, and certain secondary structures, like reverse maps*, grow out of control). What I'd like to see is something new, just for the hell of it being new.
Here's what I'm thinking. Ditch pages and memory contexts entirely. Instead, divy up a 64-bit virtual address space among individual processes, say 48 bits apiece. If a process wants to access memory outside it's 48-bits, it would need to access it through special pointers (which, thanks to a tagged-RAM architecture) could written to by the OS (allowing the OS to define its own protection and sharing mechanisms). Does anybody know of any existing systems that work even vaguely like this? Or of a different MMU architecture at all? I was hoping that AMD might at least include software TLB management, because there is some nifty stuff you can do with that, but it looks like Hammer will use the same VM mechanism that came out with the 386!
A deep unwavering belief is a sure sign you're missing something...
Because it's important. :-)
Some people think that virtual memory means paging to secondary memory, such as disk. It doesn't. Virtual memory systems can support this, but many OSes (e.g. QNX) support virtual memory with no disk paging. (OK, QNX does support disk paging, but only as an afterthought, so that QNX can be self-compiled. GCC takes a lot of memory.)
Virtual memory provides a virtual address space for each process. The benefits include:
This is just off the top of my head.
sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
5 root 9 0 0 0 0 Z 0.0 0.0 0:18 kswapd
...going back at least through 2.2 kernels and it still happens with 2.4.18 (not the latest, but no real reason for me to build a new kernel unless it solves this). I've never read a thread on this that arrives at a solution or even a description of what is going on. I first encountered it when I went from Mandrake 8 (2.4.8 kernel) to 8.1 (2.4.18).
My kswapd went zombie after ~18days uptime; it had gone zombie after about 70 days prior to that reboot. I've found references to this occurance
Fortunately, I have plenty ram and this doesn't seem to otherwise affect anything.