Slashdot Mirror
How Much Virtual Memory is Enough?
whitroth asks: "Ten years ago, Received Wisdom said that virtual memory should be, on the average, two to two-and-a-half times real memory. In these days, where 2G RAM is not unusual, and many times is not that uncommon, is this unreasonable? What's the sense of the community as to what is a reasonable size for swap these days?"
lots
Under Windows it seems it'll swap out whether the free RAM is needed or not, no matter what (there's a registry setting to change this though). Under Linux, you won't swap much anyway unless you need it.
I run a Core Duo laptop with 1GB of RAM and have never swapped out in Linux, no matter what I was doing.
"You can either have software quality or you can have pointer arithmetic, but you cannot have both at the same time."
My rule of thumb these days is 1.5x RAM, unless you're at 2GB, in which case I go with 2GB swap as well. This is for *gasp* Windows, though.
Back when I had 512MB of memory, I had a 512MB swap partition, but I noticed that I never came close to using all of it.
When I got my new machine with 1G, I never bothered to make one at all, and I've never had a problem with it. If I do ever find myself in a situation where I need some swap space, I could always just create a swap file. It's a lot more convinient because it wouldn't have to be a fixed size, doesn't take up space when I don't need it, and I have one less partition
Especially if you have 2G or more, I don't see a real reason to use swap
640k should be enough for anybody.
2X physical memory for under 2G RAM
2G swap for up to 8G RAM
+1G swap for every 4G RAM beyond that
I disable virtual memory on computers with more than 1GB of ram unless the user is going to be manipulating large images. Never had a problem yet.
If you really want to know, I use 1-2 GB swap with 1GB ram and the same for 512MB ram.
However, you might just do what I do and try out different values to figure out what works. If you're talking about a linux system a real-time memory/swap usage graph can be added to most window managers so that you can see what's happening. You could also try to estimate usages based on what the machine is expected to do.
-Tim Louden
I use 4x750 GB hard drives (RAID), purely for virtual memory. It increases the speed on the RAM preprocessing directive, but demodulates the core processing utility monitor. I find it to be a good setup, especially for running Naibed Linux.
Stupidity is like nuclear power, it can be used for good or evil. And you don't want to get any on you.
According to MS, it's 1.5 times the total RAM. I assume you're asking because you're trying to avoid a fragmented page file. While the benefits of an unfragmented page file are dubious at best (since it will be randomly accessing different parts of the page file), it's better to err on the side of caution: If you have 2GB of memory, you likely have an equally compensating-for-something hard drive, so you probably won't miss 3GB of space, or even 4. It's better to waste a little space than have Windows run out of Virtual Memory. Otherwise, just let it do its dynamic page file adjustment thing.
If you're asking about creating a swap partition for Linux then 1.5X is also recommended. Just be generous, unless -- for some reason -- you've got 2GB of RAM and a 50 meg hard drive. Too much is always better than not enough.
https://www.eff.org/https-everywhere
One of the real advantages of using swap isn't to avoid memory exhaustion at all; by moving infrequently accessed pages from memory you make more room for the disk cache, thereby possibly improving overall system performance by reducing hard drive reads.
just let windows set it for you.
Linux has futzed with this a lot (and lets the user tweak VM behavior a lot, but /proc/sys/vm/swappiness goes a long way...), both linux and Windows will swap well ahead of not having free memory (for good reason). Just wanted to go into detail because I keep seeing people complain that they see swap used in linux or windows when they still have free memory, not realizing this isn't a bad thing generally.
There are generally two strategies:
-The common-sense one where you swap when you run out of memory. This makes a lot of practical sense on systems with limited write cycles (flash based swap, though you really never ever should do that anyway), and systems that want to spin down drives to conserve power for battery conservation. Performance wise (this may surprise people who haven't spent time thinking about it), this can often be bad. Avoiding swapping is generally only good on systems where resource utilization is carefully managed and you know it won't swap ever (the IO operations of unneeded can interfere with the productive activity of a constantly busy system). This is actually a vast minority of systems in the world (no matter how l33t one may think themselves, they most certainly don't have a usage pattern that would be impacted by the extraneous IO operations of occasional write to swap.
-Pre-emptive swapping. When the IO subsystem is idle and the system can afford to copy memory to swap area, it does so (depending on criteria). Generally speaking it will select memory not accessed much and write it to disk, but leave the memory copy in place if the physical memory is not immediately needed. A fair amount of swap used in an apparently underutilized system is duplicated in physical memory and swap space. The benefit here is that if the process reads back that memory, it doesn't incur any penalty in reading it back despite it being also in swap (the system may make certain decisions on what is the best swap candidate and write to disk different data). The benefit of writing this stuff to swap even when not needed is clear when an application comes along that allocs more memory than the system has free in physical space. In the first strategy, this means the malloc blocks while data is written to disk, and the new application starting or needing a lot of data is severely impacted. In the pre-emptive swap case, system notices the condition, knows what memory it has a backup in swap of that hasn't been used lately, and can free that memory and satisfy the malloc pretty much instantly.
To those who have 1GB of RAM or so it becomes less likely that the system will have to flush memory from physical RAM, but there is a balance to be struck between memory used directly invoked by applications, what the application memory access pattern is, and what ram you can use to buffer filesystem access. If your total application memory allocation is 75%, it still may make sense performance wise to only keep 50% of your physical memory dedicated to the applications, (the other bit relegated to swap), and 50% of the memory to buffer disk I/O.
XML is like violence. If it doesn't solve the problem, use more.
I think it was one of the Live Journal guys at OScon that said, "If your server starts to swap, you've lost the battle".
With all of our 64bit 4GB of ram minimum hosts floating around, there is no longer a point to having swap -- if you server really is swapping, it's under a huge load and the io is making the problem worse. Let the OS kill a few processes to get it back under control
-- DrZaius - Minister of Sciences and Protector of the Faith
My strategy generally is to use a file for swap rather than a partition, even in linux. I figure that if memory has to be swapped in from disk, it's already crappy going to disk so the extra overhead doesn't matter much, and I have freedom to adjust it up or down depending on my needs. (This is a desktop/laptop circumnstance). I generally start at 512MB or so, increasing maybe if IO is faster on the drive. I view swap like a rumble strip on a road before a stop sign. With no swap, you don't realize a process leaked memory until it's too late, with swap, while it eats through your swap the performance will degrade and you'll see the end coming ahead of time, and may be able to head it off with a kill. It may be well an good your 4GB of ram is technically capable of handling the same load your 1GB RAM+1GB swap handled in the past, but having some noticable impact when things start going wroing is nice. I realize theoretically there are better approaches, but nothing gets in your face like poor performance and tons of disk accesses.
On a production server or a problematic system where I want support and the OS likes to dump a core to swap, I'll ensure a generous swap partition is available (generally observed active swapx1.5+physical memory size). In this case a file-backed swap may depend on layers of the kernel that are in an invalid state, and a swap partition is more likely to be reliably writable. The only system I would even theoretically hibernate on is my laptop, and I only ever suspend to ram or shutdown completely, so I don't consider my laptop as needing a swap partition of any significant size.
XML is like violence. If it doesn't solve the problem, use more.
This is completely wrong.
/tmp.
In Windows, your RAM is saved to a file called "hiberfil.sys" which is the exact size of your physical RAM. Your swap file stays exactly the way it is, otherwise you'd lose the data that was swapped to it.
In Linux, it depends on what program you are using to suspend, but typically, it's a file in
I have 4GB of physical ram (ddr2-6400) and 4gb of swap. There are actually a few reasons for this, YMMV (obviously I think the answer to this question depends on what you do).
I have a lot of things running which, usually, are doing nothing. For instance, apache2, mysql, postfix, and courier-imapd-ssl are always running, but they're rarely actually *doing* anything. (If I get a hit or an email, it's relatively rare as I hardly have very little hosted off of my home box - nevertheless, I do want these running). So I'm happy to let these get swapped out. When I start up matlab, and start dealing with huge datasets, I know it's going to swap most of these out. That's good. It will also swap out some of my matlab data that's loaded but not currently being used (and yes, it's quite possible to have >4gb in your workspace). For me, I have the swap because I need it. Figure out what you need, and you will have the answer to your question.
all this swapping talk is giving me nightmares.
a few years ago, we had a customer with multiple colocated servers complaining that sometimes they crashed for no apparent reasons.
after much debugging, we figured out one of their script was leaking memory. eventually consuming all ram (2.5G) + all swap (1-2g).
now the real problem is this. those were LIVE processes so the system was constantly paging back and forth, using 90%-95% cpu just to swap the damned things in and out and starving the actual processes.
linux 2.4, linux 2.6 (early 2.6). same deal. amazingly, the distro made a difference, redhat was pure hell, debian slightly better (though still not acceptable)
freebsd was much smarter, it just killed the offending processes.it sure wasn't ideal, but at least the server was still serving its clients
to this day, i never put more than 256m as swap even on servers with 4G of ram. that's where we had the least problems.
Sure when you had 128MB of ram, and you had a 256MB swap.
But dude, my next box will have two GIGABYTES of RAM!
Every one of your usage options assumes you'll run out of physical ram. Maybe if the OS is wasting it on pointless disk caching, but don't you think the programs in memory should have priority over blind disk caching?
Lest a foolish reader believe your two options (swap immediately, or swap as lazily/late as you can) are the only two possibilities, how about swapping when, say, only 20% of physical RAM is left? That way my Firefox and Eclipse don't swap to disk and take twenty seconds to swap in when I have 500MB of GODDAMN FREE RAM!
Hey, I'm just your average shit and piss factory.
On Mac OS X, swap is stored (by default) in files in the /var/vm directory on the boot hard drive, instead of on a separate partition. So there's no limit to how much is used, nor a predefined minimum amount of space used, the swap space expanding and contracting as needed. That seems reasonable.
Signature.
Disk is always far cheaper and more plentiful than memory. If you have four gigs of memory, what's wrong with carving eight gigs of swap out of your terrabyte RAID? If you have that much memory in the first place, then you're probably running large apps. Do you and them a favor and give them a little breathing room.
Dewey, what part of this looks like authorities should be involved?
You have a GUI to run: 600 MB for firefox, 1800 MB for OpenOffice.org, 100 MB for X, 100 MB for desktop odds and ends, 300 MB for Evolution or Thunderbird, and 10 MB for old-style stuff running in the background.
Total: 2910 MB
Yep, you need a gigabyte of swap. OpenOffice.org was made 64-bit clean for a reason. If you plan ahead, not wanting to reallocate disk space in the next few years, you'll allow for this:
2 GB for firefox, 5 GB for OpenOffice.org, 1/2 GB for X, 1/2 GB for desktop odds and ends, 1 GB for Evolution or Thunderbird, and 10 MB for old-style stuff running in the background
That's 9.01 GB. You're exactly 7.01 GB short, so you'll be needing that swap space before you know it.
But... but... the rule of thumb says to have twice as much swap as RAM!
It's a pet peeve of mine that so many system administrators appeal to "rules of thumb" about decisions such as this, instead of actually thinking it through. Sys admins pass around these nuggets of wisdom with unquestioning reverence, like they were handed down from some bearded UNIX guru sitting on a mountaintop. These rules either 1) happen to reflect reality, 2) do not reflect reality, or 3) reflected reality 20 years ago but nobody got around to issuing some sort of "revocation rule of thumb". :)
My experience is that very little swap is needed these days, and the rule of thumb falls into category #3. Long gone are the days that the OS demanded swap space for all process memory.
If I have a machine with 1GB of RAM, I'll usually give it 512MB of swap or so. As discussed elsewhere in this thread, a little bit of swap is good for pre-emptive swapping and for emergencies (to avoid the dreaded Linux "oom killer".) Also, if you're going to use hibernate, you'll want at least as much swap as real memory.
One thing to consider is whether or not you're using tmpfs for /tmp. For performance, I recommend using tmpfs for /tmp, and basically treat the swap partition as your /tmp partition. It may seem counterintuitive, "why would it be faster than a filesystem when it's backed to disk anyway, and my filesystems caches just file if need be?" The answer is that tmpfs never needs to worry about consistency. On the kernel.org machines, we have seen /tmp-intensive tasks run 10-100 times faster with tmpfs than with a plain filesystem. The downside, of course, is that on a reboot, tmpfs is gone.
Man, it's utterly depressing to see the same useless "rules-of-thumb" still in effect when the original question is asking if the rule-of-thumb is a good idea.
1) Page space is not swap space. There's a small distinction that's generally lost (and generally ignored). Page space is used to move memory pages to and from disk. Swap space is technically to move entire processes out to disk. The difference is mainly based on when your OS was created (i.e., technological underpinnings) and no need to get into it now... but the difference is meaningful.
2) Page space is not *free*. There's a misconception that if you have 500G of disk space then "how does it hurt" to put 8G of swap on 4G RAM. Depending on your OS, the size of the page table can grow remarkably depending on how much memory (RAM + VM) is allocated. This means that adding 2G of page space may not cost anything, but adding 2.5G may suddenly take up another chunk of real, non-pageable memory because the page table cannot itself be paged. This means that if your app is thrashing, then adding page space may make it worse.
3) Even with lots of RAM, it's still often a good idea (depending on your usage) to have some page space. Modern OSes will still page out unused pages to use RAM for better stuff. I.e., if you have a huge file open in a graphics application, but are not actively using that application for a length of time (an hour, say) then the OS will page it to disk. This makes better use of your physical RAM. On some OSes the OS will use page space even if free RAM is available. It can then toggle a page out by flipping a bit in the page table and not have to do an expensive write.
4) In some systems you can overcommit memory. Applications tend to request a lot more memory from the OS than they'll actually use. This is useful in many instances but it again depends on your usage. If you're running a single application that doesn't dynamically allocate memory then you can run pageless. If a new app requests memory that's not available then it will get a failure on malloc request. This can be desirable in some circumstances.
5) There are benefits to running page space on a separate disk, but for the vast majority of home users, the difference is negligible. This applies to Windows and Linux. Once you start stressing the VM subsystem then a separate disk is highly desirable.
6) You can create page files on Unix/Linux. It's not desirable generally because of the extra filesystem overhead and possibility of fragmentation. But hey, in a pinch it works.
7) Why this 2x RAM rule? A lot of it comes from old VM subsystems that needed a "picture" of the entire memory space. This made the page-out algorithms easier to code. Newer algorithms don't require the 2X RAM.
KL
There's no real hard and fast rule anymore. And setting it against a static value (like physical memory) is incredibly wasteful.
It's a much better idea to set it interactively. Use the system without adjusting the Virtual Memory for a while. Then take a look at your usage and set your virtual memory against that usage.
For instance.
If you're in a Windows machine, let it run normally for a few days.
Run everything the way you normally use it.
Multiple apps, multiple instances, games out the ass, everything.
Then open up the Task Manager and look at the Performance tab.
Take a look at the Peak value under "Commit Charge".
Set your virtual memory, min and max, at about 10% above that value to leave yourself a little headroom.
Normally this will be enough to deal with your maximum swap requests.
If, somehow, you begin bumping against virtual memory limits again AFTER that, bump it another 10%.
If you still have problems, keep bumping it in 10% increments, and start looking for apps that are memory leaking.
Chas - The one, the only.
THANK GOD!!!
Don't you hate meta-sigs?
My strategy generally is to use a file for swap rather than a partition, even in linux.
What I find curious is that you have a strategy. On what relevant experience do you base this strategy? 1 GB of disk space costs less than $0.50. Set up 3 GB of VM if it makes you feel good. The latte you drink while you set it up costs more than the extra disk space!
So go for it!!! Who cares what you do? Heck, give yourself 10x the RAM and see if it actually makes any difference!!! (it won't)
This is sort of like asking: "Which goes faster: the yellow Pacer or the red Pacer?"!
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Before talking about swapping, pagging, and virtual memory please learn and understand this equation: Virtual Memory = Physical Memory + Swap(or Page) file. I let the OS (windows) manage my page file. The current generation of windows OS's (2k, xp, & 03) mange the swap file much more efficently than windows 9x did. All this mumbo-jumbo about tweaking the swapfile came about because these old versions of windows needed to be tweaked, they had memory problems and tweaking the swapfile would improve performance and stability. But the current generation of windows has no need for tweaking the swapfile, save for some special server situations. Its a waste of time to personally manage the swapfile for a desktop/development computer.
I never user more than 512Mb of swap. If you have a runaway process, you can let it live but you avoid a lot of trashing. If more than one process start consuming RAM like crazy you, actually want them to die from an out-of-mem error otherwise your whole system will grind to a halt while it spends most of its time unswaping one and the other. At 512Mb you can do a little excess of memory usage but won't go beyond what you can unswap in a time quantum (mostly).
Smarter per-process ressource quotas would probably be better and it would be nice to have a trashiness function according to the disk speed but so far 512Mb sounds like the limit between using the resset button or just taking a coffee break when you see the HD led blinking like a strobe.
It is just easier to try the approach where you consume a lot of RAM first and to re-code if it doesn't work. I work in bioinformatics and we often have huge datasets, I alway try to load the whole thing and to make the computation in RAM. Only when I get and out-of-mem error do I segment the dataset and try a smarter approach. That might explain my choice for 512Mb and the right threshold for other people might be bigger or lower but I'm pretty sure that its bellow 8Gb.
That raises the question: is swapping obsolete? Or to put it more explicitly, has the speed difference between modern CPUs and hard drives become so large, and RAM so cheap, that it's better to consider running out of RAM to be indicative of a software failure? That way you end up with a system where one or more processes may fail (or be terminated) but at least the machine remains usable and doesn't swap itself into non-responsiveness.
In my experience, the answer is yes: with 2GB of RAM in my machine, I never need to swap, and in the few instances where swapping did occur, it was because of buggy software (memory leaks) and manually terminating the offending processes is what I needed to do resolve the memory shortage. So why not just have the OS do that automatically?
Or to put it a third way, is there any situation where swapping is helpful, anymore?
I don't care if it's 90,000 hectares. That lake was not my doing.
"Memory is like an orgasm. It's a lot better if you don't have to fake it."
-- Seymour Cray, on virtual memory.
You are not entitled to your opinion. You are entitled to your informed opinion. -- Harlan Ellison
Or to put it a third way, is there any situation where swapping is helpful, anymore?
:)
Sure. Consider Andrew Morton's logic:
http://kerneltrap.org/node/3000
In your average program, most code never gets executed, and most data is never used. For a long-lived process, swapping out the unnecessary bits frees the memory for disk cache.
While you may improve overall performance, by minimizing the average completion time for operations, the downside is responsiveness. As a user, I don't care if Firefox reads cached images a few milliseconds faster (by reading from cache instead of disk) if I have to wait 3 seconds for Thunderbird to respond to my clicks (because it has to swap in) after I've been browsing for a while. Average speed be damned!
Having said that, I just set my swappiness to 100.
What the original article didn't mention, and none of the replies seemed to go into, is the fact that with current CPUs, effectively all RAM is 'virtual':
Only on-chip memory, i.e. cache, is "real" these days, and all accesses to DRAM will be handled in paging units of 64/128 bytes or so. If this sounds familiar, it should! CPUs with 1 to 4 MB of real memory and lots of virtual memory is what the mainframes and minicomputers had about 20-30 years ago.
What this means is that now, just like then, all performance-critical code needs to be written to keep the working set within the amount of "real" memory you have available. When you passed this limit, you needed to make sure that you handled paging in suitably large blocks, to overcome the initial seek time overhead.
Today this corresponds to the difference between random access to DRAM and burst-mode (block transfer) which can be nearly an order of magnitude faster.
In the old days, when you passed the limits of your drum/disk swap device, you had to go to tape, which was a purely sequential device. Today, when you pass the limits of DRAM, you have to go to disk, which also needs to be treated as a bulk transfer/sequential device.
I.e. all the programming algorithms that was developed to handle resource limitations on old mainframes should now be ressurected!
"those who forget their history, are condemned to repeat it"
Terje
"almost all programming can be viewed as an exercise in caching"
The VM Size listed in Activity Monitor is not the size of the swapfiles. I'm running on a machine with 1 GB RAM, my VM size is 6.53 GB, but I only have 128 MB of swapfiles. You might find it interesting to intall MenuMeters http://www.ragingmenace.com/software/menumeters/ to keep track of what's going on.
And is 1000 times faster ?
So, to answer the original question :
Optimal amount of Swap ? 0 !
Even my old PIII 1Ghz takes 2 gigs of ram. Newest system we have at the office takes 64 Gigs.
=> right now I consider the Linux systems @ work as having a problem if they use swap...
Adding just a wee bit of RAM to your system and seeing swap disappear means your perfomance just exploded on this particular task...
Best Regards,
D.
It takes 40+ muscles to frown, but only four to extend your arm and bitchslap the motherfucker
We've found that 512 Megs of swap is more than enough for our 2 and 4 Gig machines. Why even have swap? Here is an example:
1) On a system with zero swap, when apache gets slammed (say you get to the top of digg or slashdot), apache starts consuming lots of memory to handle new inbound requests. When it runs out, the machine grinds to a halt because it can't allocate more and requires a power cycle. (Setting a low max children really only helps if you are happy denying traffic to the people who are trying to see your site...it's best to plan for capacity and put quite a few servers load balanced).
2) On a system with any appreciable swap (IMHO, more than 128 Megs, up to 512 Megs), if you're monitoring the system (watch -n 1 df -h, for example) and all of a sudden it starts using swap, the machine is on the edge of dying. This gives you an early warning that maximum machine performance/throughput is occurring. You can restart apache or shut it down or similar, you can do something to temporarily lower or remove load from that machine. This doesn't give you *much* time, but it gives you some.
In our real world experience, at digg and slashdot loads you have about 10-15 seconds to stop apache once it starts swapping. After that, the performance degrades so bad that the machine becomes catatonic, the same as #1, requiring a power reset (obviously because virtual memory on HD is magnitudes slower than RAM, as numerous others have suggested). The key here is that you must realize that some swap is good for allowing unused programs to be swapped out, such as login terminals that just sit there. It's great for detecting problems, but if your heavy app is the one utilizing swap, your machine is about to crash anyway.