Understanding Memory Usage On Linux

Percy_Blakeney writes "Have you ever wondered why a simple text editor on Linux can use dozens of megabytes of memory? A recent blog posting explains how the output of the ps tool is misleading and how you can get a better idea of how much memory a process really uses."

Comments

Not only shared libraries

[pontus]

Nice article.
It could also have mentioned mappings on /dev. For example, the X server, on a system with a 256MB graphics adapter, will map all that memory into its address space, making X look huge, even though it's not using all that much system RAM. This will show up as a device-backed mapping in the maps file.
On a related note, X also looks big because it's holding pixmaps belonging to various applications (Firefox comes to mind).

Re:Not only shared libraries

[ratboy666]

The "problem" is the concept of a COW page (copy on write). Coupled with the semantics of mmap().

In a nutshell: I can use mmap() to map /dev/zero into memory, for (pretty much) as big as I want. 200MB? Its now mine.

I can have a pointer to this memory.

The problem? The memory doesn't exist. What I have is a pointer, and a guarantee that enough backing store exists to satisfy it.

If I read through that pointer, I will see zeros. It *is* /dev/zero after all. However, I can write into the memory. If I write something, the page that is changed is copied and replaced; taking memory AT THAT TIME. Sparsely.

The mmap() call can map a file (backing store) and allow data to be shared. Memory does not need to be used until the data is read (or written). And this time, the backing store doesn't even need swap (because the file is the backing store).

All of which means non-changeable may be altered. Changeable may be non-existent or shared. Try to teach that to your DG tools.

A page of code that is shared - may becomes a page of code that is private. A page of data that is unwritten doesn't have to exist. Even if it is read! A page of data that is written may STILL be shared.

"ps" and the other tools could walk through typical process maps, counting up pages, and figuring out what each was for, but that may be a bit too intensive. The pages aren't "cross referenced" for that purpose. Besides, the page could be COWd, and then swapped. Should THAT count against the memory of the application? Maybe, maybe not.

So "ps" by default gives you an idea of the "big picture" for each process.

Ratboy

Re:Linux file & memory management shines

[Anonymous Coward]

Many people don't know, but executable files are not 'loaded' in the Windows sense - they're just mapped into memory.

Apparently, some people don't know that modern NT-based Windows versions also behave in exactly the same manner.

top

[Kupek]

Run top. Check out the column that says SHR. Subtract it from VIRT if you want to know the virtual memory usage of a process excluding shared libraries, or subtract it from RES if you want to know the physical memory usage of a process excluding shared libraries. Problem solved.

I don't like how he phrases that what ps reports is "wrong." It's not wrong, or even "wrong." It reports exactly what Linux tells it (through the proc filesystem). It's just might not be what you expect it to be, which means you don't understand the tools and the system. When ps reports that a process' virtual memory usage is xKb, that is correct. In the address space for the process, xKb have been allocated. Shared or not, they're still in the address space.

More tips

[typical]

The thing is, when you fork it maps the memmory and marks everything as copy on write, when something needs to write to part of the memmory, then it will make the copy for each process.

A couple other tips:

* Each thread in a process shows up as consuming the same amount of memory (either this only happens under Linuxthreads or I don't have any threaded applications running on my system).

* Device mappings show up as consumed memory (which generates plenty of XFree86/xorg complaints). If you want to find out how much memory xorg/X11 is actually using (bytes in cached pixmaps on behalf of each process and sans device mappings), try this program (contains a tiny program that lists how much memory X is using for other programs by caching pixmaps and a perl script that lists how much memory X is using sans device mappings).

* The article mentions the fact that shared libraries show up in every application's memory usage. So, for example, glibc alone adds 1.5MB to the memory usage of every process. But Win folks may not realize how significant this is. Most Windows applications ship with their own copies of almost all shared libraries used, which means that there is a huge amount of wasted memory under Windows that *actually affects you*. Under Linux, instead of shipping shared libraries with applications, folks have built tools to automatically download the latest shared libraries and use those across multiple applications. Result -- only one copy of the library need be in memory at a time. This means that it's actually reasonable to run a box with 128MB of memory and three remote users using the thing. You simply can't pull that under Windows and expect usability.

* This may not sound significant, but Linux's VM is (anecdotal evidence, of course) really solid. When I run out of memory under Windows, performance rapidly degrades -- bring an application to the foreground, and the system just starts churning. Under Linux, you can push a ways into VM and things generally keep functioning pretty well (this is one of the causes of people talking about "applications loading faster under WINE than Windows" when they're trying to prove that WINE is 'faster' than Windows -- good disk I/O and VM code).