Slashdot Mirror
Understanding Memory Usage On Linux
Percy_Blakeney writes "Have you ever wondered why a simple text editor on Linux can use dozens of megabytes of memory? A recent blog posting explains how the output of the ps tool is misleading and how you can get a better idea of how much memory a process really uses."
Nice article. /dev. For example, the X server, on a system with a 256MB graphics adapter, will map all that memory into its address space, making X look huge, even though it's not using all that much system RAM. This will show up as a device-backed mapping in the maps file.
It could also have mentioned mappings on
On a related note, X also looks big because it's holding pixmaps belonging to various applications (Firefox comes to mind).
How can they diminish EMACS like that ?
Try statically linking a program that uses just a few glibc calls and it's pushing 800k. Now add in libc++, Qt/gtk, Xlib, kde, boost, xml, etc and you're talking a lot of memory. This is what gets me about people who say "well Java performs okay now, but it uses so much memory".
A typical C/C++ based app uses just as much memory, it's just shared between processes. And for that matter, startup time of the first thing using kde/gnome isn't all that great either. Isn't it about time some effort was put into making Java or Mono part of the system, so it can be shared like C apps do?
...no process is using more than 10% of the available resources...
But hey, 10 processes are using 10%...
Hey! That's my sig you're smoking there!
How about going one step further than just blogging about it and actually submitting a documentation update to the ps man page. That way future confusion of the ps output could be avoided. Of course I guess people have to actually read the man page (In honor of slashdot, I didn't read it before posting this comment ;-)
If you wanna see a real OS with memory hogs, run Emacs...
His next story will cover that.
Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
Linux (and to be fair, Unix-like systems in general) shine at file & memory management. Many people don't know, but executable files are not 'loaded' in the Windows sense - they're just mapped into memory. This design improves performance and gives the system better performance under swapping (not thrashing, mind you). Things like mem mapped files are integral in the way the system is designed and implemented. That's one of the very reasons why a Linux machine usually runs faster and more reliable than a equivalent Windows machine... even if has less memory. The Apache tuning example is great, and it shows how much performance you can squeeze out of a good design.
Firefox does indeed suffer from some very serious memory management-related issues. For anyone who doesn't have an ideological connection to the project, it's obvious why that is.
About 8 months back I attempted to embed Gecko within an existing graphical user interface toolkit. Having heard so much from the open source community about how easy it was to do, I thought it would go rather quickly. Of course, it did not. The lack of up-to-date documentation (if such documentation there at all) and solid examples were some of the big problems.
But the overall architecture struck me as the worst part of Mozilla. Like it or not, it's overly complicated and convoluted in many areas. I admit that it's not easy to build well-designed software, but they so completely missed the boat it's unbelievable. However, it does make it obvious as to why many people complain about Firefox and Seamonkey running so slowly, in addition to suffering from huge memory consumption.
As for the embedding of Gecko, I said to hell with it. I took a page from Apple, and used KHTML instead. The loss in portability by not going with Gecko was well worth the far quicker development time, the lower memory consumption, the increased responsiveness, and the higher degree of stability of KHTML.
Cyric Zndovzny at your service.
Devin's blog also has an excellent posting on Apache performance. "Tuning Apache, part 1" (and the comments) is the sort of succinct empirical advice it is always nice to find.
Typical Slashdot response, blame the users for the browser's bloat. 99% of the users of Firefox are not programmers and wouldn't have the slightest clue what is going on. They just want to look at porn without popups or getting infected with spyware via IE's ActiveX vulnerabilities. Asking them to download some script, set environment variables, and then file bug reports is unrealistic since most of them can't even tell the difference between a web browser and a web site. That's what beta testers are supposed to be doing but we all know that 90% of the beta testers never bother to file any bug reports, even when the browser crashes.
All those shared libraries are also part of the reason that KDE and GNOME can take so long to start up, and why more memory and a higher-RPM hard disk can speed things up. It does make me laugh sometimes that Emacs is now one of Linux's fastest-starting desktop apps.
Indeed. You're completely correct. If the Mozilla crew want to take on Internet Explorer, then they can't have the general public debugging their software for them. That just won't fly.
Part of the problem is that it's far too easy for bugs to creep into Mozilla. The code is a small step above horrible, and the architecture isn't much better. A lack of up-to-date documentation leads to programmers not knowing which XPCOM interfaces are deprecated, and which aren't.
You can look at browsers like Konqueror and Opera, which offer a very comparable feature set to Firefox, yet do no suffer from the drawbacks. Not only that, but Konqueror and Opera are often described as feeling far more responsive, while being extremely stable. It's things like that which really impress the average Jill and Joe. Excessive memory usage will just perplex them, and likely result in them going back to Internet Explorer.
Cyric Zndovzny at your service.
Top will show you the same as ps does, ps calls /proc//statm and asks whats going on. The problem on linux is the copy on write principle wich saves heaps of memmory, but makes it virtually impossible to figure out what belongs to what. The thing is, when you fork it maps the memmory and marks everything as copy on write, when something needs to write to part of the memmory, then it will make the copy for each process.
However asking the process how much memory it has allocated will show all memory including stuff that is marked copy on write - that is, I could have 100 processes showing they each use 1.4MB of memory, because they all share the same libray, but in fact, its the same copy they are all using so I'm only using 1.4 MB instead of 140MB (+PCB et. al)
The whole discussion should be grounded in the reality of alternatives. A typical M$ system will grind it's way into swap space on start up, before the user loads anything! The very latest and greatest Linux distros run well on Pentium IIs and the like, which XP refuses to install on.
Friends don't help friends install M$ junk.
I'm kinda curious who you heard that from. Embedding Mozilla when you've got an already existing binding (such as for Gtk) is trivial, but writing the binding from scratch is no easy task. Gecko is a beast and the need to integrate its own drawing layer with yours makes it hard to integrate as an embedded browser. In its defense, it was never designed or intended for such a purpose. KHTML is only easier if you're using Qt (and you *did* obey the license, right?), otherwise you need to provide mappings from all the Qt primitives used by KHTML to your own. Easier than embedding Gecko, but still not trivial.
What I really wanna understand is the memory usage in Windows.
w00t
A nice article, been looking for more information on this. So often you read items in program FAQs or such giving a disclaimer on how ps memory usage is misleading, but they offer no better way. Okay, so ps memory usage information is pratically useless; now what am I supposed to use?
I was hoping for a bit more, though; like, say, a small program that lets you see both the aggregate virtual memory total as well as the memory used specifically by the program. Add a few options for how to handle the only-one-app-using-a-library situation. Doesn't seem like it'd be that hard, and very useful.
Is not that there's not a perfect tool, the problem is that it's a problem which is impossible to solve properly as I see it
Take a shared library. For whatever reason, process 1 uses only the first half of the library. Thanks to demand-loading, only that half is loaded in mem, and that's what accounts as RSS for that process, say 10 MB.
Now a process 2 is launched and it uses the other half of the library. Now, all the library is loading in memory, and even if the first process is not using and has not requested to use the second half, its RSS will grown because somebody else use other parts of the library.
I don't think it's something you can or want to "solve": That's a consequence of the design ideas behind shared libraries. Deal with it.
Something will bog down my machine, I will run 'top' and discover that no process is using more than 10% of the available resources. OK, so why is my machine bogging?
The "feature" that I find annoying about top, though it's really rather necessary for a CLI program, is that only the most CPU-intensive programs at a given instant get to the top. This isn't a problem with truly CPU-intensive programs that are constantly running. But all too often there's a program that's spiking to 30% or more CPU intermittently, and so the program might flash at the top every now and then, but for the most part it's low on the list where you can't see it. I'm not saying that top is bad, it's a very nice command line tool that works well; I'm just saying that the CLI has its limitations, and thus top does too. I find that KSysGuard works pretty well for this, since the processes all stay in the same place, and you can see when a process flashes %40 or whatever in the CPU column, and then kill it. You can use ps for this as well to an extent, but it's much harder (hit ps over and over and scroll up (or worse, use 'less' or 'more') to see how much CPU is being used by each process).
I have discovered a truly remarkable proof of this theorem that this sig is too small to contain.
So, people don't know how to interpret the output of ps? And that's a Slashdot frontpage story?
Slashdot isn't only about breaking tech news; it's about keeping geeks generally informed. Many Linux geeks (including myself) probably learned something from the article that they didn't know. It's a well-written, informative article, and I'm glad Slashdot posted it because otherwise I probably would have never seen it. Not every Slashdotter already knows everything there is to know about Linux like you apparently do, and I imagine this isn't quite "common knowledge," so it's helpful for some of us.
What have I done wrong in my settings to deserve such trivial items?
No one forced you to click on "Read More." Sorry that you wasted a couple seconds reading the summary and realizing you already knew all about ps, but you didn't need to waste even more of your time trolling.
I have discovered a truly remarkable proof of this theorem that this sig is too small to contain.
Because there is not just one moderator. Everybody can moderate. So there are always a few people who think that's funny. But by not being an Anonymous Coward, but logging in instead, you can set a threshold to all posts, which will exclude most of them...
The JVM serving this page currently has an uptime of 32 days. But in the past it's had uptimes of over 200 days. Neither it, nor any of the other Tomcat servers I run, has ever gone out of whack. Java (Tomcat, Weblogic and others) powers the web servers of many of the world's biggest websites, serving millions of pages of dynamic content every day. If it was unreliable, that wouldn't be happening.
I'm old enough to remember when discussions on Slashdot were well informed.
Of course. EMACS - eight megabytes and constantly swapping.
Because there is nothing quite like seeing you've got 20 Oracle instances at 1gb each on a 4gb box. :)
On a related note, if anyone is curious how memory management library calls such as "malloc" work, you might check out my article on the subject.
Engineering and the Ultimate
The architecture of Java doesn't allow it to share library memory space like this. The effect of this is Java programs, appear to use about the same amount of memory as compiled programs when, in fact, they are using quite a bit more. This is why running a Java program that takes up 25 megs of memory can seem to suck the life out of a computer while a compiled executable using 25 megs doesn't. Java is probably really using about 10x more memory.
It's also why systems running a Java framework with multiple programs executing in the same Java process do so much better than ones where everything is in its own process. This is Java's sweet spot, where these JVM architecture disatvantages have the least impact.
This is my understanding of how Java's libraries work. Someone let me know if I'm missing something here.
set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
Run top. Check out the column that says SHR. Subtract it from VIRT if you want to know the virtual memory usage of a process excluding shared libraries, or subtract it from RES if you want to know the physical memory usage of a process excluding shared libraries. Problem solved.
I don't like how he phrases that what ps reports is "wrong." It's not wrong, or even "wrong." It reports exactly what Linux tells it (through the proc filesystem). It's just might not be what you expect it to be, which means you don't understand the tools and the system. When ps reports that a process' virtual memory usage is xKb, that is correct. In the address space for the process, xKb have been allocated. Shared or not, they're still in the address space.
The first person to use a system might load 128Mb worth of libraries and applications. The second and all subsequent users may only use 15-30Mb worth of RAM for each additional user. e.g. A 1Gb RAM system could handle 30 concurrent users rather than 8.
Deleted
Load and CPU usage are different things. Load is a very tricky topic. The gist of it is that it is the average number or processes that were waiting to do some amount of processing. It is then scaled based on a logarithmic algorithm to give you a rough picture of what is happening. So lets say you have an SMTP server with a dozen processes all trying to disk access and the disk is also busy updating its locate database. Your disk is hammered. Your processor is not. But you have so many processes competing for IO that it bogs down the process scheduling eventually, which can make everything sluggish. Your CPU usage might not be heavy, but that doesn't mean the system isn't bogged down trying to do other things. CPU usage is an important part of system load, but not the only thing going into it.
Jeremy
>> Have you ever wondered why a simple text editor on Linux can use dozens of megabytes of memory?
Correct me if I'm wrong but... doesn't the fact that KEdit uses a lot of libraries that consume resources and impact system performance -- whether shared or not -- still means that it is a hog? I mean, if a seemingly simple application is consuming "dozens of megabytes of memory", saying "oh, it's OK, because most of it is being shared and already commited", does not really excuse it. What if those libraries are not currently being used by any other process?
In order for the shared memory to lessen the impact on the system, the user must be running some other processes that share the same libraries. This to me is a *BIG*, and unwarranted, assumption by the developer, as evidenced by his example of someone running the Gnome environment but running a single KDE application.
-dZ.
Carol vs. Ghost
The thing is, when you fork it maps the memmory and marks everything as copy on write, when something needs to write to part of the memmory, then it will make the copy for each process.
A couple other tips:
* Each thread in a process shows up as consuming the same amount of memory (either this only happens under Linuxthreads or I don't have any threaded applications running on my system).
* Device mappings show up as consumed memory (which generates plenty of XFree86/xorg complaints). If you want to find out how much memory xorg/X11 is actually using (bytes in cached pixmaps on behalf of each process and sans device mappings), try this program (contains a tiny program that lists how much memory X is using for other programs by caching pixmaps and a perl script that lists how much memory X is using sans device mappings).
* The article mentions the fact that shared libraries show up in every application's memory usage. So, for example, glibc alone adds 1.5MB to the memory usage of every process. But Win folks may not realize how significant this is. Most Windows applications ship with their own copies of almost all shared libraries used, which means that there is a huge amount of wasted memory under Windows that *actually affects you*. Under Linux, instead of shipping shared libraries with applications, folks have built tools to automatically download the latest shared libraries and use those across multiple applications. Result -- only one copy of the library need be in memory at a time. This means that it's actually reasonable to run a box with 128MB of memory and three remote users using the thing. You simply can't pull that under Windows and expect usability.
* This may not sound significant, but Linux's VM is (anecdotal evidence, of course) really solid. When I run out of memory under Windows, performance rapidly degrades -- bring an application to the foreground, and the system just starts churning. Under Linux, you can push a ways into VM and things generally keep functioning pretty well (this is one of the causes of people talking about "applications loading faster under WINE than Windows" when they're trying to prove that WINE is 'faster' than Windows -- good disk I/O and VM code).
Any program relying on (nontrivial) preemptive multithreading will be buggy.
What gets me is how some distro builders see a security warning about setuid/setgid binaries using lazy so loading, and decide that using -Wl,-z,now is a good thing to add. Excuse me, but that will pull in EVERY library at link time, whether used or not, often leading to some MAJOR bloat.
Yes, it "fixes" the "problem", but so would using rpath to DSOs not writable by users or ensuring that LD_LIBRARY_PATH doesn't point to user writable directories. Without the load time bloat.
Regards,
--
*Art
When you can remember running 60 users on a mainframe with about 1MB RAM and a processor no faster than a 386.
The closest I've come to dealing with it was writing exmap.
This is a (moderatly ugly) gtk+ tool which uses a loadable kernel module to work out which pages are used by more than one process. If a page is used by N processes, each process is credited with PAGE_SIZE/N bytes.
I believe it "solves" the problem you describe above. The biggest problem is that it provides a little too much information, so perhaps I should simplify it a bit.
(Known problems with current 0.8 version: some of the tests fail intermittently and some systems with pre-linked elf binaries can cause errors. Should fix up both with the next release).
The "feature" that I find annoying about top, though it's really rather necessary for a CLI program, is that only the most CPU-intensive programs at a given instant get to the top. [...] I find that KSysGuard works pretty well for this, since the processes all stay in the same place
This has nothing to do with CLI vs GUI programs, and everything to do with what you're choosing to sort by. You can change the sort order in top.
If you sort by PID or process name or something else less volatile than CPU percentages, the processes all stay in the same place in top, too. However, if you're looking for programs that are using a lot of CPU over time, it's probably worth sorting by cumulative CPU time instead.
Read the man page or the interactive help (hit "?").
I just wanted to add my confirmation that the Apache article is an excellent tip.
I had been experiencing issues reaching the max clients on a busy apache server serving around 6mbit/sec of images at peak times, and had been forced to increase the maximum child process setting to a very large number to cope with the peak daily periods.
Having just made the changes recommended in that article, ie. changing the keep alive timeout to around 2 seconds rather than the default of 15 - we've gone from an average of 100+ child processes to a constant of 20-30.
I'd advise anyone experiencing problems hitting their max client setting (the example he gives is a slashdotting, in my case it's serving loads of individual images) to try this setting out.
That is how it should be read I think. To start 200 instances of your Java proggie you pretty much did the same thing as starting 200 threads in single virtual machine. These threads show in ps output as operating system processes and they map entire address space of virtual machine which is why their sizes are identical.
Memory usage of Java actually scales very nicely with silly number of threads. A couple of months ago I created a small server which opened lots of listener sockets in their own threads.
With one thread the size of the virtual machine about 40 megs which pretty much for a simple application but when I created more server threads the amount of added memory was very small. With 100 listener threads it was like 60 megs, with 400 it was 80 megs and finally with 3000 server threads the amount of used memory was only 290megs!
It is true that these threads were not actually doing anyting except listening on their sockets but I thing it is very impressive nevertheless.
That's just not true, as someone else has swillden points out in this comment to the current story. Nobody should follow your suggestion.
Based on your over-simplified claim (which I'll call "wrong") the 43 java threads on my Tomcat box are using 3.0GB of RAM total, minus 426MB shared, which is impossible on a box with 256MB of RAM and 512MB swap.
More generally, the problem with ps (and top) is that they fail to highlight the most important piece of information: the amount of unshared memory each process is using, or, as TFA calls it, the "marginal cost" of each process.
Instead, they give you the total memory available to each process. That number is irrelevant to a user of that process. It won't tell you, for example, how much memory you'd save if you killed off any given process. It won't even tell you how much total memory (shared+unshared) that process is using... as others have pointed out, ps's number includes unused copy-on-write device-mapped memory.
ps is at best deceptive, if not actually wrong.
When I moderate, I only use "-1, Overrated". That way, I never get meta-moderated!