Tuning Linux VM swapping

Lank writes "Kernel developers started discussing the pros and cons of swapping to disk on the Linux Kernel mailing list. KernelTrap has coverage of the story on their homepage. Andrew Morton comments, 'My point is that decreasing the tendency of the kernel to swap stuff out is wrong. You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine. Get it out on the disk, use the memory for something useful.' Personally, I just try to keep my memory usage below the physical memory in my machine, but I guess that's not always possible..."

Comments

God no...

[0123456]

"You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine. Get it out on the disk, use the memory for something useful."

I absolutely despise the way that XP swaps out applications in order to make the disk cache larger. I have 1GB of RAM on my machine precisely so I don't have to wait two minutes for it to swap my web browser back in after it's swapped out... yet if I copy a 2GB file from one drive to another, the stupid operating system will swap out all the applications it can just to make the cache larger.

Please, please, don't take Linux down the same braindead route as Microsoft has done for XP. It's utterly insane to swap out my browser so that a 2GB file can be copied two seconds faster when I then have to wait two minutes for the browser to swap back in. Or at least provide some kind of '#define STOP_VM_SWAPPING_STUPIDITY' so that I can disable it.

Re:God no...

[Shakrai]

Please, please, don't take Linux down the same braindead route as Microsoft has done for XP. It's utterly insane to swap out my browser so that a 2GB file can be copied two seconds faster when I then have to wait two minutes for the browser to swap back in

Does it really make it faster anyway? Unless parts of that 2GB file were already in the cache then how is the cache going to make it transfer any faster?

As a side note I haven't noticed Linux swapping much out in favor of the cache. My home grown samba/sql/dhcp/nat/intranet server has 768 megs of memory. As of today (43 day uptime -- Linux 2.4.25) there is only 2,528k in SWAP. 8,444k of free memory, 191,952k used for buffers, 296,004 used for cache and the rest for applications.

I wouldn't mind seeing Linux swap out programs that aren't touched in several days/weeks (like the 12 agetty processes on my monitor less machine -- yes I know I could disable them if I wanted) but I definitely don't want to see it swapping out that browser I used 5 minutes ago in favor of increasing the disk cache size. Now if I launch Quake that's a different story.

As far as the other posts about rule of thumb for swap size go -- I stopped using the 1:1 or 2:1 ratio a long time ago. I have a 256meg swap partition on my 768meg Linux box. That's pretty much as big as I go with swap spaces. Are you seriously going to setup a 768 (or worse x2) swap space? A) You'll never use it, B) If you do use it your machine will barely be useable.

As far as XP's stupidity goes look under My Computer -> Properties -> Advanced -> Performance Settings -> Advanced and make sure both options (processor scheduling and memory usage) are set to "Programs" and not "background services" or "System cache". That may (or may not -- it is Windows after all) help you a little. On the flipside of the coin I discovered that I needed to reverse the memory option on my Windows 2000 Terminal Server to prevent stupid HP print drivers from sucking up 100% of the CPU and 90% of the physical memory.

Re:God no...

[ckaminski]

Mozilla has the problem specifically because it's memory footprint gets so large with all those tabs. If you don't use process separated IExplore processes, you get the same problem with IE when it's footprint gets up around 70+ MB.

The only way to stop this madness on XP is to turn off the swapfile. I'd REALLY hate to see Linux go down this route. Big bloaty applications need to stay IN MEMORY unless there is memory pressure being exerted on the system. That is the only time swapping should occur.

This reminds me of an old convo I had ...

[DikSeaCup]

I had this conversation with a fellow sysadmin, about the time that RAM was fairly cheap and we had a budget.

She had just procured a new Sun machine with 2 GB of RAM. Mind you, disk space hadn't grown all that significantly and you could still get machines with 9 GB drives.

The original practice was to make swap 2xRAM. So when the student she had putting the machine came to her and said, "What do I make swap?" she responded "Twice the RAM."

He said, "Are you sure? That's like almost half the boot drive."

She thought about it for a second and said, "Oh, yeah. I guess just make it the same as the RAM."

So this begs the questions: What do you make your swap now? When does your rule of thumb change? And remember when you could run a "fast" linux box on a P100 with 64MB of RAM and 128MB of swap?

Re:This reminds me of an old convo I had ...

[drinkypoo]

The reason the original practice in Sun shops was to have swap be twice the RAM is that SunOS4 swaps real ram to swap space on a 1:1 basis such that the first n bytes (where n is the number of bytes of physical memory) correspond to the first n bytes of the swap file.

When SunOS5 rolled around, this was no longer necessary, and your swap is additive, so you only need as much swap as, well, you actually need.

On my linux firewall system with 256MB real RAM, I have 512MB swap space. On my Windows system with 1GB real RAM, I have 768MB of swap space. This number is actually a hold-over from when I only had 512MB of RAM, I could probably decrease it to just about nothing now.

Amusingly enough my system has ~480MB of real RAM free, and is using 701MB of my paging file. Go windows! Like I need 480MB free all the time. Still, it is nice not to have to swap something out if I start a big application - but Windows is awful about returning from swap.

Some other more or less useless data points: My Indy (running gentoo) with 128MB has 256MB swap, which has been enough. I probably could have gotten away with 128MB but believe it or not my primary concern is whether I'll be able to compile some of the biggest C++ programs without the larger amount of swap. Certainly 128MB will not do it, even when you are booted from the gentoo installer CD and there's nothing much running.

Re:This reminds me of an old convo I had ...

[13Echo]

I don't normally make my swaps more than 512 MB on my Linux machines. In fact, when I had 1204 MB of RAM on my last machine, it only ever touched the swap once (when I was compiling Mozilla). The machine was so responsive with 1024 MB of RAM, it virtually never needed to use the swap with that much RAM.

Now that I have a newer machine, and RAM prices have increased (had to replace SDRAM with DDR), I only have 512 MB in my home machine. It seems to be nearly as responsive, practically never needing to touch the swap. I've only ever seen it use a few MB of the swapfile. When partitioning my Linux drives, I almost always have more than one drive in the machine. HDA1 normally gets the root partition. HDB1 is normally my swap, at the front 512 MB of the drive, followed by home on HDB2. This system makes everything snappy.

Even on my work machine, which is only a p3 450 with 256 MB of RAM, things operate quite well under Gnome 2. I have two drives in that machine as well, and the swap is on a seperate drive from the root partition. Programs can load from one drive while simultaneously swapping (if necessary) to a second drive. Even with Gnome 2 running, in addition to my browser and several other apps, only a few KB of space is being used on the swap.

I can't see most desktop Linux users needing more than 512 MB of swapfile space, assuming that they have at least 256 MB of RAM. The general rule of thumb, though, is to put the swap partition at the front of the drive for the best performance, in the event that it does need to get used.

I've really been impressed with Linux's memory management, even in the 2.2/2.4 series kernels. I've heard that 2.6 even makes some improvements as well. When I used Windows 2000, on the other hand, it INSISTED on using the swap even with a gig of RAM, even after I tweaked it for the best performance. I even used a RAID0 array, and Linux is still faster and more efficient at managing memory WITHOUT the RAID array. I was surprised that the array wasn't even really needed on Linux for fantastic disk access speeds with my 3 year old 7200 RPM drives.

Of course, the rules will be different for server application. More swap is probably a necessary thing. It's possible, however, that users of Linux (on the desktop) may not even need a swapfile with more than 512 MB of RAM.

Re:This reminds me of an old convo I had ...

[TheLink]

Well, here's my thoughts on swap.

First you should worry about how your O/S does "memory overcommit".

Many O/Ses overcommit mem. How they handle the case when it turns out there really isn't any mem left (including swap) is what you'd want to know. Some O/Ses (and versions of O/S) effectively kill -9 random processes till there's enough RAM to run. Some applications intentionally allocate large amounts of mem and usually don't every use them. So they usually won't work if you have overcommit turned off (and not enough RAM+swap).

If you having tons of swap just to avoid your O/S poor handling of mem overcommit, you may end up in a death spiral of swapping. Running processes page by page off your HDD isn't fun to watch (it's so 50s or was that 60s :) ).

My HDD transfers at max 40-50MB/sec, random seek transfer maybe about 11MB/sec.

At worst case how long does it take to swap out and swap in the largest process you'd ever have, given the speed of the HDD? Can you wait that long? Can the app wait that long? Will the machine be dead for practical purposes?

So if you can wait 20 secs, maybe 512MB is ok, assuming the pig process only uses half or so of your swap (plus whatever physical RAM you have).

But with a small swap, you may run out of mem and hit the memory overcommit scenario.

I'd still keep swap - just so that when my machine runs out of mem starts slowing down, rather than slamming full speed into a hard wall.

Problem

[FreeLinux]

Personally, I just try to keep my memory usage below the physical memory in my machine, but I guess that's not always possible..."

No it isn't possible. With today's RAM prices I almost always have more physical RAM than the system requires. But, due to aggressive VM swapping there are still hundreds of megs swapped out to disk when there is no need at all. This means that those applications, when their time does finally come, are slow because they must be retrieved from disk first. It's really annoying sometimes. Yet, even with excess RAM turning off swap is disasterous.

Re:Problem

[Trifthen]

No, turning off swap is not disastrous. We've turned it off on our production web server cluster that routinely serves 60Mb sustained traffic. We've turned it off because we have 2GB of ram in these machines, and Linux insisted on preferring buffers and cache over our running applications. Fuck that, we said. With over 1GB Of buffers and cache, we had RAM to spare; bye-bye swap.

The big issue

[MrIrwin]

The main cuase of memory usage on Linux is the use of many different shared libraries and not bloated apps.

I think developers could do more at a library level. For example.....dare I suggest using common sub libraries within libraries, that is people like KDE and GTK get thier heads together and say "are thier functions we include in our libraies that could just as well be linked to an underlying library?"

Other reasons

[Halo1]

Another reason to gradually and pro-actively swap things out, is that when another program later needs a lot of memory, your system doesn't come to a grinding halt because suddenly a lot of stuff has to be swapped out at once (followed by zeroing all that memory, since you don't want to have one program leaking data to another).

At least, that's the rationale I've read behind OS X's strategy of swapping things out long before all physical memory is used (and of keeping a pool of zeroed memory pages ready to fulfill most requests). Note that this does not require superfluous swap-ins if your reuse strategy is balanced properly, as the fact that something is swapped out doesn't mean that the memory which contained that data will be cleared/reused immediately (i.e., if it's needed again shortly afterwards, that page can be reactivated without having to go to disk).

Under most desktop OS'es, programs can even give some hints to the system regarding their usage of a memory region using e.g. the madvise() system call.

Dumb Swapping is Computer Abuse

[stuffduff]

Programmers have put a lot of time and effort into the VM swapping algorithm; mostly with the intention of being prepared to have a lot of memory ready and waiting for the next thing it will be asked to do. Unfortunately that's not so much of an issue with cheap ram and disk storage and faster and faster front side buses. What we really need is more intelligent swapping, which can only come about when the VM gets a set of API hooks (would make for a great 'shared object') that would enable the system administrator (and maybe someday the end user) to assist an intelligent VM manager to establish priorities and consistently respect those priorities.

Unfortunately the current crop of best guess VM managers end up denying the end user the experience of their computer's peak performance. Coupled with the horrible state of application bloat, modern 'state of the art' hardware and software combine to give us less and less in terms of overall performance. Software developers throw more code at the cpu to add functionality with little or no concern for performance. And hardware manufacturers add more and more 'special instructions' and 'pipelining' which the majority of software is completely unable to access. If anything it's more like a bunch of dysfunctional co-dependents than an industry that is cogent as to what really needs to be going on. If the folks dealing with processors and the application software could take a page from the gamers (look at the high levels of integration between game engines and video cards) for example, and more effort put into consolidating functionality in dlls and shared libraries; we would be amazed at how truly fast these machines could perform.

Not amused

[MrLaminar]

Actually, I haven't been very impressed by the whole swapping thing under Linux lately. I'm running 2.4.22 with a 400MB swapfile.

Some apps _can_ make the system unresponsive enough to ignore keystrokes, which is *very* annoying. At other times, xmms will stop playing while the disk goes crazy... Switching from emacs to Firefox after 10 minutes usually takes an extra 5 seconds to redraw the window and load all the stuff again.

Running GNOME2 on this laptop is also quite noisy on the disk. It swaps all the time...

The answer is . . . "It depends"

[kmankmankman2001]

The universal IT answer of "It depends" applies here as well. Yes, having Mr. Bloaty App glob onto scads of memory that are then not referenced for long periods of time can have a negative impact on other apps if the system becomes memory constrained. And, Yes, if the memory manager swaps a bunch of unreferenced memory out to disk and Mr. User has to wait a long time for Mr. Bloaty App to become responsive because it was his memory that got swapped out, that's a problem, too. The ideal is to be able to address this (haha, bad pun) at the application level and not simply at a global level. This has been the standard on the mainframe (MVS, OS/390, z/OS) operating systems for a long time, where there is a very sophisticated virtual memory manager. If there are, say, a 100 apps and 2 of them are very sensitive to response time, most of them aren't, and 10 are just dead dogs you couldn't care less about how nice is it to be able to actually tell the system that? The 2 "loved ones" then receive preferential storage treatment at the expense of the other, "less loved ones" and the dead dogs are always first on the pecking order of who to steal storage from. The memory manager then is acting to maintain the responsiveness of the applications (the reasons we run OS's in the first place) to meet the needs and expectations of the user(s) (the reasons we run the Apps). Without that ability, arguing over "more swappy" vs. "less swappy" when it's only applied at a global, default, level is not especially productive except within the context of attempting to establish, perhaps, where the best general-use default happy setting is - for the general-use default system we all use (is that you? I know it's not me).