Slashdot Mirror
Understanding Memory Usage On Linux
Percy_Blakeney writes "Have you ever wondered why a simple text editor on Linux can use dozens of megabytes of memory? A recent blog posting explains how the output of the ps tool is misleading and how you can get a better idea of how much memory a process really uses."
Nice article. /dev. For example, the X server, on a system with a 256MB graphics adapter, will map all that memory into its address space, making X look huge, even though it's not using all that much system RAM. This will show up as a device-backed mapping in the maps file.
It could also have mentioned mappings on
On a related note, X also looks big because it's holding pixmaps belonging to various applications (Firefox comes to mind).
'top' gives you stats on what percent use programs make of memory and cpu.
Even so, it isn't perfect. Something will bog down my machine, I will run 'top' and discover that no process is using more than 10% of the available resources. OK, so why is my machine bogging? I guess there's no perfect tool.
That's the point. Nobody cares about how much actual memory a C/C++ app touches.
Making Java "part of the system" won't help much either because the libraries aren't the same. You could argue that at the bottom of the pyramid its still libc that's being used, but we still have and need all the wrappers on top of the library to make it compatible with Java code.
So until people find it normal to run more than one or two java applications at once, Java will be deemed a memory hog. It's sort of a rut that Java is in right now, because nobody would really run more than two Java applications at once. My computer—granted a 5-year old 1.7ghz P4 with 386mb of RAM—can barely handle Eclipse at any reasonable speed. God forbid I also run something else)
Linux (and to be fair, Unix-like systems in general) shine at file & memory management. Many people don't know, but executable files are not 'loaded' in the Windows sense - they're just mapped into memory. This design improves performance and gives the system better performance under swapping (not thrashing, mind you). Things like mem mapped files are integral in the way the system is designed and implemented. That's one of the very reasons why a Linux machine usually runs faster and more reliable than a equivalent Windows machine... even if has less memory. The Apache tuning example is great, and it shows how much performance you can squeeze out of a good design.
Devin's blog also has an excellent posting on Apache performance. "Tuning Apache, part 1" (and the comments) is the sort of succinct empirical advice it is always nice to find.
I'm kinda curious who you heard that from. Embedding Mozilla when you've got an already existing binding (such as for Gtk) is trivial, but writing the binding from scratch is no easy task. Gecko is a beast and the need to integrate its own drawing layer with yours makes it hard to integrate as an embedded browser. In its defense, it was never designed or intended for such a purpose. KHTML is only easier if you're using Qt (and you *did* obey the license, right?), otherwise you need to provide mappings from all the Qt primitives used by KHTML to your own. Easier than embedding Gecko, but still not trivial.
http://en.wikipedia.org/wiki/Windows_NT
Update your knowledge.
Java has concurrent GCs now that do not freeze the entire VM while being run. And I've never seen the GC go "out of whack" and hang permanently (though I've seen many apps do this due to poor thread/resource management).
Throw the bums out!
Because there is not just one moderator. Everybody can moderate. So there are always a few people who think that's funny. But by not being an Anonymous Coward, but logging in instead, you can set a threshold to all posts, which will exclude most of them...
I run a P2, 266 at home, with 256 MB of RAM. KDE 3.4 runs pretty slow. I've turned off a lot of the eye candy, but still the response time is quite slow. Windows 2000 on the other hand is quite speedy, I can't speak for windows XP, because I don't run it. The problem is, is that this isn't really a fair comparison, as the Windows 2000 UI, it more comparable to something like sawfish. Well, the look is similar, but Even straight X Windows has a better feature set. So, I could use Sawfish, but If I start up a KDE Program, then it takes forever just to start it up.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
On a related note, if anyone is curious how memory management library calls such as "malloc" work, you might check out my article on the subject.
Engineering and the Ultimate
Run top. Check out the column that says SHR. Subtract it from VIRT if you want to know the virtual memory usage of a process excluding shared libraries, or subtract it from RES if you want to know the physical memory usage of a process excluding shared libraries. Problem solved.
I don't like how he phrases that what ps reports is "wrong." It's not wrong, or even "wrong." It reports exactly what Linux tells it (through the proc filesystem). It's just might not be what you expect it to be, which means you don't understand the tools and the system. When ps reports that a process' virtual memory usage is xKb, that is correct. In the address space for the process, xKb have been allocated. Shared or not, they're still in the address space.
The thing is, when you fork it maps the memmory and marks everything as copy on write, when something needs to write to part of the memmory, then it will make the copy for each process.
A couple other tips:
* Each thread in a process shows up as consuming the same amount of memory (either this only happens under Linuxthreads or I don't have any threaded applications running on my system).
* Device mappings show up as consumed memory (which generates plenty of XFree86/xorg complaints). If you want to find out how much memory xorg/X11 is actually using (bytes in cached pixmaps on behalf of each process and sans device mappings), try this program (contains a tiny program that lists how much memory X is using for other programs by caching pixmaps and a perl script that lists how much memory X is using sans device mappings).
* The article mentions the fact that shared libraries show up in every application's memory usage. So, for example, glibc alone adds 1.5MB to the memory usage of every process. But Win folks may not realize how significant this is. Most Windows applications ship with their own copies of almost all shared libraries used, which means that there is a huge amount of wasted memory under Windows that *actually affects you*. Under Linux, instead of shipping shared libraries with applications, folks have built tools to automatically download the latest shared libraries and use those across multiple applications. Result -- only one copy of the library need be in memory at a time. This means that it's actually reasonable to run a box with 128MB of memory and three remote users using the thing. You simply can't pull that under Windows and expect usability.
* This may not sound significant, but Linux's VM is (anecdotal evidence, of course) really solid. When I run out of memory under Windows, performance rapidly degrades -- bring an application to the foreground, and the system just starts churning. Under Linux, you can push a ways into VM and things generally keep functioning pretty well (this is one of the causes of people talking about "applications loading faster under WINE than Windows" when they're trying to prove that WINE is 'faster' than Windows -- good disk I/O and VM code).
Any program relying on (nontrivial) preemptive multithreading will be buggy.
What gets me is how some distro builders see a security warning about setuid/setgid binaries using lazy so loading, and decide that using -Wl,-z,now is a good thing to add. Excuse me, but that will pull in EVERY library at link time, whether used or not, often leading to some MAJOR bloat.
Yes, it "fixes" the "problem", but so would using rpath to DSOs not writable by users or ensuring that LD_LIBRARY_PATH doesn't point to user writable directories. Without the load time bloat.
Regards,
--
*Art
--
Given enough personal experience, all stereotypes are shallow.
Because what ps reports is the truth, from a certain point of view.
Shared and static libraries both have advantages and disadvantages. The main advantages of shared libraries are that they save hugely on memory and disk space. It also means that you often upgrade a library and all the applications will reap the benefits.
You will find that most libraries install both shared and static versions to give applications the choice over which to link to. Shared libraries normally outweigh the benefits of static ones, so the linker will usually use shared ones. There is normally a build option to override this.
BTW, when you 'javax.swing.*;' you aren't actually pulling in more code, just making more symbols in the namespace available to you.
That is how it should be read I think. To start 200 instances of your Java proggie you pretty much did the same thing as starting 200 threads in single virtual machine. These threads show in ps output as operating system processes and they map entire address space of virtual machine which is why their sizes are identical.
Memory usage of Java actually scales very nicely with silly number of threads. A couple of months ago I created a small server which opened lots of listener sockets in their own threads.
With one thread the size of the virtual machine about 40 megs which pretty much for a simple application but when I created more server threads the amount of added memory was very small. With 100 listener threads it was like 60 megs, with 400 it was 80 megs and finally with 3000 server threads the amount of used memory was only 290megs!
It is true that these threads were not actually doing anyting except listening on their sockets but I thing it is very impressive nevertheless.
That's just not true, as someone else has swillden points out in this comment to the current story. Nobody should follow your suggestion.
Based on your over-simplified claim (which I'll call "wrong") the 43 java threads on my Tomcat box are using 3.0GB of RAM total, minus 426MB shared, which is impossible on a box with 256MB of RAM and 512MB swap.
More generally, the problem with ps (and top) is that they fail to highlight the most important piece of information: the amount of unshared memory each process is using, or, as TFA calls it, the "marginal cost" of each process.
Instead, they give you the total memory available to each process. That number is irrelevant to a user of that process. It won't tell you, for example, how much memory you'd save if you killed off any given process. It won't even tell you how much total memory (shared+unshared) that process is using... as others have pointed out, ps's number includes unused copy-on-write device-mapped memory.
ps is at best deceptive, if not actually wrong.
When I moderate, I only use "-1, Overrated". That way, I never get meta-moderated!