Slashdot Mirror


Understanding Memory Usage On Linux

Percy_Blakeney writes "Have you ever wondered why a simple text editor on Linux can use dozens of megabytes of memory? A recent blog posting explains how the output of the ps tool is misleading and how you can get a better idea of how much memory a process really uses."

248 comments

  1. Not only shared libraries by pontus · · Score: 5, Informative

    Nice article.
    It could also have mentioned mappings on /dev. For example, the X server, on a system with a 256MB graphics adapter, will map all that memory into its address space, making X look huge, even though it's not using all that much system RAM. This will show up as a device-backed mapping in the maps file.
    On a related note, X also looks big because it's holding pixmaps belonging to various applications (Firefox comes to mind).

    1. Re:Not only shared libraries by Anonymous Coward · · Score: 1, Interesting

      is this the same for FreeBSD --and other Unix-like-- by any chance?

      especially if DRM/DRI is used I would guess.

    2. Re:Not only shared libraries by OhHellWithIt · · Score: 2, Interesting

      Nice article, indeed. It didn't tell me what I'd hoped it would, though, that there is a tool to give me memory stats the way Data General's AOS/VS II tools did. In the DG world, one component of process memory organization mapped a process' space into four areas: shared code, unshared code, shared data, and unshared data. The DG equivalent of "ps" reported shared memory and unshared memory, and it made short work of determining which processes were pigs and which weren't. I haven't found that in either Unix or Windows. It would be really useful in troubleshooting my wife's accursed W2K system.

      --
      "Who controls the past controls the future. Who controls the present controls the past." -- George Orwell
    3. Re:Not only shared libraries by Anonymous Coward · · Score: 0

      Why a simple text editor can use dozens of megabytes of memory? Well, if as the article suggests, you discount stuff like shared memory... the last thing remaining is that it's probably written in Java. Oh, hang on... that wouldn't be "dozens of megabytes" it would be "hundreds"... so perhaps we can rule Java out too.

    4. Re:Not only shared libraries by jnik · · Score: 1

      Hmm. It told me enough to identify which processes are pigs...use pmap -d and look at the bottom for writeable/private. Not exactly the same as the DG tool, but you get the bottom line.

    5. Re:Not only shared libraries by ratboy666 · · Score: 5, Informative

      The "problem" is the concept of a COW page (copy on write). Coupled with the semantics of mmap().

      In a nutshell: I can use mmap() to map /dev/zero into memory, for (pretty much) as big as I want. 200MB? Its now mine.

      I can have a pointer to this memory.

      The problem? The memory doesn't exist. What I have is a pointer, and a guarantee that enough backing store exists to satisfy it.

      If I read through that pointer, I will see zeros. It *is* /dev/zero after all. However, I can write into the memory. If I write something, the page that is changed is copied and replaced; taking memory AT THAT TIME. Sparsely.

      The mmap() call can map a file (backing store) and allow data to be shared. Memory does not need to be used until the data is read (or written). And this time, the backing store doesn't even need swap (because the file is the backing store).

      All of which means non-changeable may be altered. Changeable may be non-existent or shared. Try to teach that to your DG tools.

      A page of code that is shared - may becomes a page of code that is private. A page of data that is unwritten doesn't have to exist. Even if it is read! A page of data that is written may STILL be shared.

      "ps" and the other tools could walk through typical process maps, counting up pages, and figuring out what each was for, but that may be a bit too intensive. The pages aren't "cross referenced" for that purpose. Besides, the page could be COWd, and then swapped. Should THAT count against the memory of the application? Maybe, maybe not.

      So "ps" by default gives you an idea of the "big picture" for each process.

      Ratboy

      --
      Just another "Cubible(sic) Joe" 2 17 3061
    6. Re:Not only shared libraries by TekPolitik · · Score: 1
      Try this:

      #!/bin/sh
      for i
      do
      [ -d /proc/$i ] || { echo "$i: No such process"; continue; }
      echo $i:
      awk '
      {
      split($1, vals, "-");
      val = strtonum("0x" vals[2]) - strtonum("0x" vals[1]);
      types[$2] += val
      }
      END {
      for (i in types)
      printf "\t%s %9d\n", i, types[i]
      }
      ' < /proc/$i/maps
      done
    7. Re:Not only shared libraries by BokLM · · Score: 1

      Yes, it's the same. FreeBSD has shared libraries too.

    8. Re:Not only shared libraries by OhHellWithIt · · Score: 1

      Thanks! The awks on my system don't have the strtonum function, but I see what you've done, and you've prompted me to read the fine man page on proc.

      --
      "Who controls the past controls the future. Who controls the present controls the past." -- George Orwell
  2. Before you start bitch about Firefox memory leaks by tqft · · Score: 0, Flamebait

    Have you been here? http://dbaron.org/log/2006-01#e20060114a
    and helped do something about it?

    Also see here
    http://www.squarefree.com/2006/02/04/memory-leak-p rogress/#comments
    about progress made and being made.

    --
    The Singularity is closer than you think
    Quant
  3. A 'simple' editor ? by Anonymous Coward · · Score: 4, Funny

    How can they diminish EMACS like that ?

    1. Re:A 'simple' editor ? by aurb · · Score: 5, Funny

      They said 'text editor', not 'operating system'.

    2. Re:A 'simple' editor ? by Mjlner · · Score: 1

      If you would have cared enough to RTFA and had the OP chosen to quote completely, you would notice TFA said "simple KDE text editor". More specifically, they were talking about KEdit. KEdit is not Emacs.

      --
      Lemon curry???
    3. Re:A 'simple' editor ? by Anonymous Coward · · Score: 0

      That should be obvious without even needing to RTFA. Everyone knows that EMACS stands for "Eight Megs and Constantly Swapping", not the dozens of megs mentioned in the summary. They must be talking about another editor.

    4. Re:A 'simple' editor ? by darthgnu · · Score: 1

      KEdit in not Emacs ? Surely, you must mean KEINE is not Emacs.

      /me ducks

      Seriously, I can't believe KEdit needs to map that much librairies. Surely, this must go along with the "KDE is flexible" principle that Linus stated recently, not that it is a bad thing...

      --
      Freedom is strength, Ignorance is peace, War is slavery.
    5. Re:A 'simple' editor ? by the_greywolf · · Score: 2, Funny

      speaking of which... has anyone ported a text editor to it yet? it would seem EMACS is a good OS, but with no VI, i just can't figure out how it's useful...

      --
      grey wolf
      LET FORTRAN DIE!
    6. Re:A 'simple' editor ? by Surye · · Score: 1
  4. My own favorite is 'top'. by Anonymous Coward · · Score: 1, Informative

    'top' gives you stats on what percent use programs make of memory and cpu.

    Even so, it isn't perfect. Something will bog down my machine, I will run 'top' and discover that no process is using more than 10% of the available resources. OK, so why is my machine bogging? I guess there's no perfect tool.

    1. Re:My own favorite is 'top'. by bumby · · Score: 5, Insightful

      ...no process is using more than 10% of the available resources...

      But hey, 10 processes are using 10%...

      --
      Hey! That's my sig you're smoking there!
    2. Re:My own favorite is 'top'. by Splab · · Score: 5, Insightful

      Top will show you the same as ps does, ps calls /proc//statm and asks whats going on. The problem on linux is the copy on write principle wich saves heaps of memmory, but makes it virtually impossible to figure out what belongs to what. The thing is, when you fork it maps the memmory and marks everything as copy on write, when something needs to write to part of the memmory, then it will make the copy for each process.
      However asking the process how much memory it has allocated will show all memory including stuff that is marked copy on write - that is, I could have 100 processes showing they each use 1.4MB of memory, because they all share the same libray, but in fact, its the same copy they are all using so I'm only using 1.4 MB instead of 140MB (+PCB et. al)

    3. Re:My own favorite is 'top'. by diegocgteleline.es · · Score: 3, Insightful

      Is not that there's not a perfect tool, the problem is that it's a problem which is impossible to solve properly as I see it

      Take a shared library. For whatever reason, process 1 uses only the first half of the library. Thanks to demand-loading, only that half is loaded in mem, and that's what accounts as RSS for that process, say 10 MB.

      Now a process 2 is launched and it uses the other half of the library. Now, all the library is loading in memory, and even if the first process is not using and has not requested to use the second half, its RSS will grown because somebody else use other parts of the library.

      I don't think it's something you can or want to "solve": That's a consequence of the design ideas behind shared libraries. Deal with it.

    4. Re:My own favorite is 'top'. by lasindi · · Score: 2, Interesting

      Something will bog down my machine, I will run 'top' and discover that no process is using more than 10% of the available resources. OK, so why is my machine bogging?

      The "feature" that I find annoying about top, though it's really rather necessary for a CLI program, is that only the most CPU-intensive programs at a given instant get to the top. This isn't a problem with truly CPU-intensive programs that are constantly running. But all too often there's a program that's spiking to 30% or more CPU intermittently, and so the program might flash at the top every now and then, but for the most part it's low on the list where you can't see it. I'm not saying that top is bad, it's a very nice command line tool that works well; I'm just saying that the CLI has its limitations, and thus top does too. I find that KSysGuard works pretty well for this, since the processes all stay in the same place, and you can see when a process flashes %40 or whatever in the CPU column, and then kill it. You can use ps for this as well to an extent, but it's much harder (hit ps over and over and scroll up (or worse, use 'less' or 'more') to see how much CPU is being used by each process).

      --
      I have discovered a truly remarkable proof of this theorem that this sig is too small to contain.
    5. Re:My own favorite is 'top'. by Anonymous Coward · · Score: 0

      so if you are using a real operating system (like Solaris) then you DO have real tools .... prstat (which won't put it self at the top like top does) .... and microstat accounting you wont have processes hiding behind the clock ticks ....

    6. Re:My own favorite is 'top'. by jallen02 · · Score: 4, Informative

      Load and CPU usage are different things. Load is a very tricky topic. The gist of it is that it is the average number or processes that were waiting to do some amount of processing. It is then scaled based on a logarithmic algorithm to give you a rough picture of what is happening. So lets say you have an SMTP server with a dozen processes all trying to disk access and the disk is also busy updating its locate database. Your disk is hammered. Your processor is not. But you have so many processes competing for IO that it bogs down the process scheduling eventually, which can make everything sluggish. Your CPU usage might not be heavy, but that doesn't mean the system isn't bogged down trying to do other things. CPU usage is an important part of system load, but not the only thing going into it.

      Jeremy

    7. Re:My own favorite is 'top'. by jbert · · Score: 3, Interesting

      The closest I've come to dealing with it was writing exmap.

      This is a (moderatly ugly) gtk+ tool which uses a loadable kernel module to work out which pages are used by more than one process. If a page is used by N processes, each process is credited with PAGE_SIZE/N bytes.

      I believe it "solves" the problem you describe above. The biggest problem is that it provides a little too much information, so perhaps I should simplify it a bit.

      (Known problems with current 0.8 version: some of the tests fail intermittently and some systems with pre-linked elf binaries can cause errors. Should fix up both with the next release).

    8. Re:My own favorite is 'top'. by woolio · · Score: 1

      I particularly love it when "top" itself is consuming 99% of the CPU...

      I've seen this while running it as root

      [/me ducks for cover at the flame war over whether top should be run as root or not]

    9. Re:My own favorite is 'top'. by Corgha · · Score: 2, Informative

      The "feature" that I find annoying about top, though it's really rather necessary for a CLI program, is that only the most CPU-intensive programs at a given instant get to the top. [...] I find that KSysGuard works pretty well for this, since the processes all stay in the same place

      This has nothing to do with CLI vs GUI programs, and everything to do with what you're choosing to sort by. You can change the sort order in top.

      If you sort by PID or process name or something else less volatile than CPU percentages, the processes all stay in the same place in top, too. However, if you're looking for programs that are using a lot of CPU over time, it's probably worth sorting by cumulative CPU time instead.

      Read the man page or the interactive help (hit "?").

    10. Re:My own favorite is 'top'. by waferhead · · Score: 1

      I personally find that I/O has far more effect on the "feel" and responsiveness than 100% CPU usage.

      ferinstance---I can be running several background processes, which keep the CPU usage pegged at 100%.

      The GUI works fine, smooth, minimal lag. (K7Barton2800+/1G ram/Nvidia card+driver) as long as these processes are NOT doing heavy file I/O.

      If mythcommflag runs (niced), or anything else with even moderate HD access (whereisdb), the system CRAWLS at ~20% CPU or less usage.

      Is there some method of setting up some sort of qos-like bandwith limiter to the HD for niced processes?
      (or anything else?)

    11. Re:My own favorite is 'top'. by Schraegstrichpunkt · · Score: 1
      [/me ducks for cover at the flame war over whether top should be run as root or not]

      Only if you trust the kernel to not give you malicious data in /proc.

    12. Re:My own favorite is 'top'. by lasindi · · Score: 1

      If you sort by PID or process name or something else less volatile than CPU percentages, the processes all stay in the same place in top, too. However, if you're looking for programs that are using a lot of CPU over time, it's probably worth sorting by cumulative CPU time instead.

      I stand corrected. :-) Thanks for the tip!

      --
      I have discovered a truly remarkable proof of this theorem that this sig is too small to contain.
    13. Re:My own favorite is 'top'. by timeOday · · Score: 1
      all too often there's a program that's spiking to 30% or more CPU intermittently, and so the program might flash at the top every now and then, but for the most part it's low on the list where you can't see it.
      At each moment on a uniprocessor system, every program is either using 100% or 0% of the CPU. There's no in between. The idea of a program that never grabs 100% of the CPU on occasion is wrong.

      Maybe you would be interested in running top and hitting 'T', which sorts programs by the total CPU time they've used since invocation. If you launch top with S (cumulative mode), those times will include the times of all child processes as well.

    14. Re:My own favorite is 'top'. by timeOday · · Score: 1

      It's bizarre, isn't it, that we have such elaborate schemes for prioritizing processes' CPU access, yet nothing for IO access. IMHO Process priorities should apply to disk access. They should also apply to network send queues.

    15. Re:My own favorite is 'top'. by the_greywolf · · Score: 1
      Is there some method of setting up some sort of qos-like bandwith limiter to the HD for niced processes? (or anything else?)

      have you looked at Con Kolivas' patchset? it sets the CFQ I/O scheduler as default and implements a process scheduler tooled specifically toward responsiveness and interactivity.

      i've been running it for some months and have found that even under severe I/O traffic, the system remains usable overall. (the only catch is that the CFQ scheduler is still feature-incomplete and doesn't schedule I/O traffic by niceness properly yet.) i've had a Java app (Eclipse and Azureus both) push my load average up past 30.0 because of I/O and the system still remained as usable and responsive as ever (even though anything that did I/O would halt painfully, including starting new apps).

      it's worth a try in any case. -ck kernels just "feel" better, IMO, and there are a lot of other people who seem to agree.

      (also, you can set your scheduler to CFQ or whatever whenever you please: echo cfq > /sys/block/hda/queue/elevator)

      --
      grey wolf
      LET FORTRAN DIE!
    16. Re:My own favorite is 'top'. by Anonymous Coward · · Score: 0

      your favorite is top? My favorite is bottom! We should definitely get together and have some hot anal sex!!!!

    17. Re:My own favorite is 'top'. by WuphonsReach · · Score: 1

      Or try 'atop'... although the big reason I run a screen session with atop running in addition to regular "top" is because atop shows me which physical devices are busy. Which is more useful when tuning database systems where you're balancing loads between multiple arrays.

      "dstat" also works well as a histogram, although I'm not sure how much support it has for tracking individual processes.

      --
      Wolde you bothe eate your cake, and have your cake?
  5. Extra, extra, read all about it by tgv · · Score: 0, Flamebait

    So, people don't know how to interpret the output of ps? And that's a Slashdot frontpage story? What have I done wrong in my settings to deserve such trivial items?

    1. Re:Extra, extra, read all about it by mrjb · · Score: 4, Funny

      His next story will cover that.

      --
      Visit http://ringbreak.dnd.utwente.nl/~mrjb/growingbettersoftware to download your free copy of the book
    2. Re:Extra, extra, read all about it by Anonymous Coward · · Score: 0

      What have I done wrong in my settings to deserve such trivial items?

      Yeah, this story sucks. I so very much want to read another Vista or Xbox story from Zonk. You hear that Slashdot editors? Please post more Microsoft lovefest stories!

    3. Re:Extra, extra, read all about it by tgv · · Score: 0, Flamebait

      This is Slashdot. It's not your www.i-am-a-grandmother-who-has-been-saddled-up-by- her-grandson-with-linux-and-has-just-discovered-th e-command-line.com
      People here know how to deal with linux and are interested in more complex things and in news. This is neither. Although I have to agree with the person that wrote "give me another xbox story instead"...

    4. Re:Extra, extra, read all about it by lasindi · · Score: 5, Insightful

      So, people don't know how to interpret the output of ps? And that's a Slashdot frontpage story?

      Slashdot isn't only about breaking tech news; it's about keeping geeks generally informed. Many Linux geeks (including myself) probably learned something from the article that they didn't know. It's a well-written, informative article, and I'm glad Slashdot posted it because otherwise I probably would have never seen it. Not every Slashdotter already knows everything there is to know about Linux like you apparently do, and I imagine this isn't quite "common knowledge," so it's helpful for some of us.

      What have I done wrong in my settings to deserve such trivial items?

      No one forced you to click on "Read More." Sorry that you wasted a couple seconds reading the summary and realizing you already knew all about ps, but you didn't need to waste even more of your time trolling.

      --
      I have discovered a truly remarkable proof of this theorem that this sig is too small to contain.
    5. Re:Extra, extra, read all about it by tgv · · Score: 1

      Yes, you're right. Perhaps this really is more interesting. Sad, isn't it?

    6. Re:Extra, extra, read all about it by tgv · · Score: 2, Informative

      Because there is not just one moderator. Everybody can moderate. So there are always a few people who think that's funny. But by not being an Anonymous Coward, but logging in instead, you can set a threshold to all posts, which will exclude most of them...

    7. Re:Extra, extra, read all about it by tgv · · Score: 0, Flamebait
      Slashdot isn't only about breaking tech news; it's about keeping geeks generally informed.


      So, what's next? ls -h? Regular expressions? tee?


      Sorry that you wasted a couple seconds reading the summary


      Slashdot only publishes 20 to 30 stories per day, and it's annoying enough to find Star Wars trivia between them. There is a special "Linux" section that is devoted to stuff like this, but even there the level is higher. So what's the point of publishing this on the front page?


      Hindsight is always perfect, but I really should have added an irony warning symbol.

    8. Re:Extra, extra, read all about it by suitepotato · · Score: 1

      Summation: buy more RAM, consider learning to program and be a memory management master while you enjoy the benefits of the increased RAM while you play TuxRacer.

      --
      If my grammar and spelling are off, I am [distracted/tired/careless] (take your pick)
  6. The only thing running by 0xABADC0DA · · Score: 4, Insightful

    Try statically linking a program that uses just a few glibc calls and it's pushing 800k. Now add in libc++, Qt/gtk, Xlib, kde, boost, xml, etc and you're talking a lot of memory. This is what gets me about people who say "well Java performs okay now, but it uses so much memory".

    A typical C/C++ based app uses just as much memory, it's just shared between processes. And for that matter, startup time of the first thing using kde/gnome isn't all that great either. Isn't it about time some effort was put into making Java or Mono part of the system, so it can be shared like C apps do?

    1. Re:The only thing running by chris+macura · · Score: 2, Informative
      A typical C/C++ based app uses just as much memory, it's just shared between processes...

      That's the point. Nobody cares about how much actual memory a C/C++ app touches.

      Making Java "part of the system" won't help much either because the libraries aren't the same. You could argue that at the bottom of the pyramid its still libc that's being used, but we still have and need all the wrappers on top of the library to make it compatible with Java code.

      So until people find it normal to run more than one or two java applications at once, Java will be deemed a memory hog. It's sort of a rut that Java is in right now, because nobody would really run more than two Java applications at once. My computer—granted a 5-year old 1.7ghz P4 with 386mb of RAM—can barely handle Eclipse at any reasonable speed. God forbid I also run something else)

    2. Re:The only thing running by CyricZ · · Score: 3, Insightful

      What do you mean, making it "part of the system"? Are you suggesting that they be embedded within the Linux or BSD kernels, for instance? I would hope not, because for serious use that is a recipe for disaster.

      Part of the problem with Java is that each VM has traditionally had its own copy of the Java class library. When you consider how huge the standard library is these days for Java, it's no wonder that even a small Java program consumes so much memory. And running several programs, each duplicating data from the others, is wasteful.

      Apple has had for years a JVM that shares classes between numerous virtual machine instances. It thus reduces unnecessary memory consumption.

      --
      Cyric Zndovzny at your service.
    3. Re:The only thing running by Max+Threshold · · Score: 4, Interesting

      The problem is that the bulk of Java's libraries aren't shared. At least, that's how it looks to pmap.

      Sun JVM running a simple "Hello World" program that sleeps 1000ms between messages in an endless loop:

      mapped: 260888K writeable/private: 199604K shared: 54652K

      It wouldn't be hard to create a launcher that would run them all on the same virtual machine. Such a launcher would a candidate for the system integration you suggest. After all, if you needed to run Windows apps on your Linux box, you wouldn't run multiple instances of VMWare, would you?

    4. Re:The only thing running by Anonymous Coward · · Score: 0

      Just to compare though, my Athlon 1.4 with an effective 384mb memory runs Eclipse wonderfully. I even prefer it to my PowerBook (1.5ghz/1gb) for developing. Granted, that's an effective 384 - VMWare runs on the machine and is set to use 385mb of a total 768mb.

    5. Re:The only thing running by mi · · Score: 0, Troll
      It wouldn't be hard to create a launcher that would run them all on the same virtual machine. Such a launcher would a candidate for the system integration you suggest. After all, if you needed to run Windows apps on your Linux box, you wouldn't run multiple instances of VMWare, would you?
      Actually, a JVM is even less stable than Windows. It was not designed as a real OS. The garbage-collection, for example, will freeze the entire VM for as long as it needs to run -- and sometimes it goes out of whack and hangs permanently...
      --
      In Soviet Washington the swamp drains you.
    6. Re:The only thing running by mi · · Score: 1
      A typical C/C++ based app uses just as much memory, it's just shared between processes [emphasis mine -mi].
      Well, is not this the whole point, eh?
      --
      In Soviet Washington the swamp drains you.
    7. Re:The only thing running by archeopterix · · Score: 1
      Isn't it about time some effort was put into making Java or Mono part of the system, so it can be shared like C apps do?
      Huge effort required, with some hard design decisions looming ahead. Jar files store java bytecode, which is (usually) jit-compiled at runtime to native code by the VM. My guess is that sharing the bytecode alone isn't enough and probably happens already (A disk page read by several different processes is loaded into 1 physical memory page which is then mmap'ped by the OS to the processes' address spaces). So you want to share the generated native code, which at present can even differ between JVM instances, due to different jit options or even different execution patterns.

      This is harder than sharing C libs which are native code already - you have to either precompile the shared jars or make the shared library manager understand the intricacies of the java jit compiler.

    8. Re:The only thing running by Ulrich+Hobelmann · · Score: 1

      Sharing Java Class Code between processes? You mean like what Apple has developed years ago, and even given back to Sun, so they could include that same code into the stock JVM (IIRC since version 1.5)?

      The downside is that "modern" Java, but also C++ programs only consist of comparably little code, but TONS of objects allocated on the heap. No code sharing will help, there.

    9. Re:The only thing running by Max+Threshold · · Score: 1

      Sounds like all the more reason not to run more than one of them. :o)

    10. Re:The only thing running by jsight · · Score: 3, Informative

      Update your knowledge.

      Java has concurrent GCs now that do not freeze the entire VM while being run. And I've never seen the GC go "out of whack" and hang permanently (though I've seen many apps do this due to poor thread/resource management).

    11. Re:The only thing running by diegocgteleline.es · · Score: 1

      Well, libraries are shared, sure. But libraries can be bloated.

      I mean, it's nice that people explains that all that memory that processes use is shared between processes, but the linux desktop platform couplement (kernel libc Xorg qt libkde app) is far from being perfect. Just take a look at the upcoming gnome 2.6.14: Too many performance improvements in the sahred libraries, right?

    12. Re:The only thing running by 0xABADC0DA · · Score: 1

      I agree completely, although an alternative solution is to put the JVM into the kernel, where it could manage different user-level processes that all shared the same optimized code (essentially a process would hold the native code a Java app uses, if any). Of course, this would mean at least some optimizations would be global across all Java-using programs. That isn't necessarily bad though. Apple does this somewhat in userland with their version, but it still has a lot of unshared code that could be shared.

    13. Re:The only thing running by vrai · · Score: 1

      We run the latest version of Sun's JVM and have had to disable concurrent garbage collection. Why? Because it was causing the JVM to shutdown without reporting an error; one second it would be running, the next it was gone. While concurrent garbage collection has obvious advantages, it is clear that Sun's implementation has a little way to go before it can be considered stable.

    14. Re:The only thing running by m50d · · Score: 1
      This is what gets me about people who say "well Java performs okay now, but it uses so much memory".

      But python, for example, is in the situation. But, it doesn't use so much memory, because it does the right thing and wraps the system libs for such things, instead of reimplementing everything.

      A typical C/C++ based app uses just as much memory, it's just shared between processes.

      But processes from many languages use these same libraries.

      Isn't it about time some effort was put into making Java or Mono part of the system, so it can be shared like C apps do?

      It's Java/Mono that need to make such efforts - I believe Mono does so, by using thin wrappers around gtk+, libxml etc. rather than writing their own window toolkit, xml parser and so forth. The only reason Java doesn't do it is idiocy on the part of the Java people, it's not the system's fault.

      --
      I am trolling
    15. Re:The only thing running by cerberusss · · Score: 1
      I agree completely, although an alternative solution is to put the JVM into the kernel

      This has actually been done some years ago, see the kissme project.

      --
      8 of 13 people found this answer helpful. Did you?
    16. Re:The only thing running by swillden · · Score: 5, Interesting

      The problem is that the bulk of Java's libraries aren't shared. At least, that's how it looks to pmap.

      There's clearly something pmap is missing. I just tried exactly what you said, and on my system, pmap reports:

      mapped: 209820K writeable/private: 169696K shared: 33660K

      So then I ran 200 copies. pmap reports the same stats for each of them, but that's clearly absurd. 170MB writable/private multiplied by 200 instances is 34GB of RAM. But my laptop has 2GB of RAM, only 136KB of swap is being used, and 800MB of my RAM is being used for cache. So those 200 java instances, plus everything else I have running at the moment (which is quite a bit), are consuming about 1.2GB of RAM. That's quite a bit, but nothing like what pmap would lead me to believe.

      Thinking about it, I think I can see what's going on. pmap is showing the mapped size of each process. Each JVM individually mmaps a huge amount of memory, because it maps in all of the Java libraries. However, mmapped pages don't consume any virtual memory unless they're actually used. This trivial program only uses a tiny portion of the libraries, so only a very small part of the mapped pages is actually read. Each JVM also mmaps a big block of anonymous memory for use as its heap but, again, the mapped address space doesn't consume RAM/swap if it's never touched, and this trivial app doesn't use much heap.

      My conclusion: pmap *also* overestimates memory usage, because some portion of the mapped address space isn't actually in use. RSS, on the other hand, only measures memory that is actually in use, but doesn't distinguish between memory that is shared and memory that is not. VSZ is the most pessimistic measure, since it includes all mapped memory, shared and unshared.

      Looking again at my 200 Java processes confirms this. Each has an RSS of 6.7MB, which is too much to be "correct" (in the sense that 200*6.7MB is more RAM than my whole system is using), but not much too much, which tells me that a lot of that 6.7MB RSS is unshared.

      Looking at the pmap output in more detail, I see that most of the memory is mapped in three big anon blocks -- probably the heaps used by the generational garbage collector. The libraries are smaller and they're (duh) read-only which I'm pretty sure means the libs *are* shared across multiple instances of the JVM, because I believe that multiple processes that mmap the same file in read-only mode only get a single shared copy.

      That means the bulk of the actual memory usage is writable, not libraries, and it's all unused heap space. Assuming the generational GC does the obvious thing and unmaps the whole "dead" generation block, the bulk of the heap space will usually be unused... and the JVM actually will "give back" heap that it no longer needs, at least in part. RSS should show that. Hmm... how to construct a test case to verify it...

      Bottom line, I think: Java apps do use a little more actual memory than C/C++ apps, and trivial Java apps do use a lot more actual memory than trivial C/C++ apps, but it's not nearly as bad as pmap shows because the GC will always have a lot of extra memory mapped that has never been touched (assuming it does unmap and remap the dead generation, and it would be stupid not to).

      Enough of my rambling, semi-informed speculation. Anyone who knows more about how this stuff works, please weigh in and correct me.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    17. Re:The only thing running by mi · · Score: 2, Insightful
      Update your knowledge. Java has concurrent GCs now that do not freeze the entire VM while being run.
      The latest 1.4.2 still freezes the entire VM. I support a Java application for a living -- keep your evangelism to newbies...

      May be, 1.5 will bring some wonderful improvements in this area, but so far it, apparently, has not -- see another response to your posting.

      And I've never seen the GC go "out of whack" and hang permanently (though I've seen many apps do this due to poor thread/resource management).
      Of course, anything GC does is triggered by the application, and some apps are better than others with resource management.

      But the "you need not worry about memory management" was one of the top items in Java evangelism, and now the 'net is full of advice on how to manage memory in Java to avoid GC-related problems. Oops.

      Some developers, though, continue to believe that "don't worry" hype. Hard to blame them, because without it, there are even fewer advantages to Java.

      --
      In Soviet Washington the swamp drains you.
    18. Re:The only thing running by Simon+Brooke · · Score: 1
      A typical C/C++ based app uses just as much memory, it's just shared between processes. And for that matter, startup time of the first thing using kde/gnome isn't all that great either. Isn't it about time some effort was put into making Java or Mono part of the system, so it can be shared like C apps do?

      A virtual machine is a virtual machine. There's nothing whatever to stop you running multiple Java threads running different applications in the same virtual machine - this is, after all, exactly what application servers like Tomcat do. Each Web-app in Tomcat does not by default run in a separate virtual machine, instead they all share the same one, and consequently share the same copy of the Java class library. This is like many C programs running on the same real machine.

      --
      I'm old enough to remember when discussions on Slashdot were well informed.
    19. Re:The only thing running by swillden · · Score: 1

      But the "you need not worry about memory management" was one of the top items in Java evangelism, and now the 'net is full of advice on how to manage memory in Java to avoid GC-related problems. Oops.

      Perhaps that was one of the top items pushed by evangelists, but it should have been obvious to anyone who thought about it that while GC makes memory management *easier* (and it does!), it doesn't mean you can ignore memory issues entirely.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    20. Re:The only thing running by typical · · Score: 1

      Apple has had for years a JVM that shares classes between numerous virtual machine instances. It thus reduces unnecessary memory consumption.

      So it's kind of like a slow version of native compiled code from gcj?

      --
      Any program relying on (nontrivial) preemptive multithreading will be buggy.
    21. Re:The only thing running by Anonymous Coward · · Score: 0

      The 'native compiled' gcj code is 'slow.' The JVM JITC beats it in throughput, it just has a longer startup time.

    22. Re:The only thing running by Anonymous Coward · · Score: 0

      "Isn't it about time some effort was put into making Java or Mono part of the system, so it can be shared like C apps do?"

      No. Java memory usage sucks. Deal with it.

    23. Re:The only thing running by Anonymous Coward · · Score: 0

      Give Netbeans a fixed heap space of 64MB and then work with a marginally-complex project. Be sure to turn on GC statistics reporting so you can watch in awe as the splendor of NetBeans sucks up the entire heap and enters a cycle of full collections that never ceases, rendering NetBeans completely unresponsive.

    24. Re:The only thing running by Walles · · Score: 1
      If SUN's VM doesn't work for you, you *could* try another one. Here's the 5.0 release of the same thing.

      Since it's an independent implementation it contains different bugs from SUN's VM, and you may have better luck with it.

      Or it could be that your VM process uses up too much memory and gets killed by the kernel's OOM handler. In that case JRockit will disappear even faster for you.

      Disclaimer: I work for BEA.

      --
      Installed the Bubblemon yet?
    25. Re:The only thing running by Anonymous Coward · · Score: 0

      > jar files store java bytecode, which is (usually) jit-compiled at runtime to native code by the VM.

      And as soon as it's JIT'd, it loses any possibility of shared mapping unless it's actually in the same classloader.

      Not to mention that the JIT has to do its profiling and JIT thing every time you run the app. .NET assemblies, pointedly, cache JIT'd code, and therefore can demand-load it as needed instead of blowing up your runtime with an entire freaking optimizing compiler.

    26. Re:The only thing running by Anonymous Coward · · Score: 0

      The JVM does use shared lib's. Try running ldd on it.

    27. Re:The only thing running by Anonymous Coward · · Score: 0

      I agree completely, although an alternative solution is to put the JVM into the kernel,

      [Gibber]

    28. Re:The only thing running by ultranova · · Score: 1

      We run the latest version of Sun's JVM and have had to disable concurrent garbage collection. Why? Because it was causing the JVM to shutdown without reporting an error; one second it would be running, the next it was gone.

      Were these graphic applications ? Because if they are, you might want to try running them from command line (with the oncurrent garbage collector enabled) and see if the thing throws a NullPointerException just prior to shutdown. FreeCol did just this: it started throwing NPE's with incremental garbage collector enabled - I know, it isn't the same as concurrent, but it's still worth checking - because it contained a threading bug (it started threads from object constructors, which is a no-no in Java) which didn't show up under the standard collector.

      Multithreading is truly an endless source of bugs...

      --

      Forget magic. Any technology distinguishable from divine power is insufficiently advanced.

    29. Re:The only thing running by ultranova · · Score: 1

      The latest 1.4.2 still freezes the entire VM. I support a Java application for a living -- keep your evangelism to newbies...

      Um, the latest Java out is 1.6 beta, and the latest stable is 1.5.

      --

      Forget magic. Any technology distinguishable from divine power is insufficiently advanced.

    30. Re:The only thing running by Anonymous Coward · · Score: 0

      Java 1.5 broke a LOT of shit, anyone doing anything serious with Java is likely still using Java 1.4.2 to keep things sane.

      Along with the world's crappiest implementation of generics ("hmm, I need to create an array of the generic type and - oh, it's literally impossible to do so, cool"), they totally fucked up the XML libraries and screwed up various Swing features. Java 1.6 is supposed to totally fuck Swing up, I can't wait for that!

    31. Re:The only thing running by Max+Threshold · · Score: 1

      Fascinating, Captain.

    32. Re:The only thing running by Anonymous Coward · · Score: 0

      There's actually quite a bit of work at Sun that is doing exactly what you suggest.

      This describes a system for running multiple Java apps in one virtual machine.
      http://research.sun.com/projects/barcelona/papers/ oopsla01.pdf

      However, there are safety issues with such an approach. If one application manages to crash the JVM, all the applications running on the shared JVM will stop. So other approaches were also investigated such as sharing memory between virtual machines

      http://research.sun.com/projects/barcelona/papers/ ecoop02.pdf

      and building Java classes into shared libraries that allow non-changing parts of the class representation in memory to be shared across JVMs.

      http://research.sun.com/projects/barcelona/papers/ ipdps03.pdf

    33. Re:The only thing running by ArsonSmith · · Score: 1

      Nahw, gcj code is a slow version of native compiled code all on it's own.

      --
      Paying taxes to buy civilization is like paying a hooker to buy love.
    34. Re:The only thing running by Anonymous Coward · · Score: 0

      Yeah - glibc, xlib, and various other system libraries to integrate with the system.

      It doesn't share any of the (checks) 46MB class library. (And that's compressed. Who knows how big it would be uncompressed.)

    35. Re:The only thing running by Anonymous Coward · · Score: 0

      Your right.

    36. Re:The only thing running by espressojim · · Score: 1

      List l = new ArrayList();

      That was hard, how?

    37. Re:The only thing running by jsight · · Score: 1

      The latest 1.4.2 still freezes the entire VM. I support a Java application for a living -- keep your evangelism to newbies...

      I support several. Keep your falsehoods to Ruby fans. ;)


      May be, 1.5 will bring some wonderful improvements in this area, but so far it, apparently, has not -- see another response to your posting.


      "Will"? You do realize that 1.5 was released well over a year ago, right? Anyway, the best VM on the market for Java right now is BEA JRockit, which is very stable with the concurrent garbage collector turned on.


      Of course, anything GC does is triggered by the application, and some apps are better than others with resource management.


      True, but if these apps are hanging completely it is VERY unlikely to be the garbage collector (unless they are simply being given too much RAM and are trashing during GC very badly).


      But the "you need not worry about memory management" was one of the top items in Java evangelism, and now the 'net is full of advice on how to manage memory in Java to avoid GC-related problems. Oops.


      Anyone who thinks that they don't need to think about resource management at all in any language is incredibly naive.
  7. man page update by suso · · Score: 4, Insightful

    How about going one step further than just blogging about it and actually submitting a documentation update to the ps man page. That way future confusion of the ps output could be avoided. Of course I guess people have to actually read the man page (In honor of slashdot, I didn't read it before posting this comment ;-)

    1. Re:man page update by Stumbles · · Score: 1

      Yeah that'd be a good idea. In this case I don't think RTFM would have made any difference.

      --
      My karma is not a Chameleon.
    2. Re:man page update by TallMatthew · · Score: 3, Informative
      How about going one step further than just blogging about it and actually submitting a documentation update to the ps man page. That way future confusion of the ps output could be avoided.

      Because what ps reports is the truth, from a certain point of view.

    3. Re:man page update by Buck2 · · Score: 1

      A man page is traditionally supposed to be concise. The kinds of misinterpretation problems that he explained in his article don't lend themselves to concise explanations.

      AFAIK, this is also why the 'info' page system was attempted. Feeling constricted by the consensus that man pages should be as pithy as possible, 'info' was created so that you could have pages and pages of explanation. Unfortunately, it seemed that 'info' was/is hard to work with because it assumes EMACS familiarity.

      Um, in any case, a whole writeup like what this guy did doesn't really belong in a man page.

      --

      As my father lik@(munch munch)... ....
  8. Emacs by Touisteur · · Score: 2, Funny

    If you wanna see a real OS with memory hogs, run Emacs...

    1. Re:Emacs by typical · · Score: 1

      Emacs gets flack for memory usage based on using a whole couple megs of memory back in the day -- but do you want to compare its usage to Visual Studio's or Visual Slickedit's?

      BBedit might use less memory -- haven't been following BBedit recently...

      --
      Any program relying on (nontrivial) preemptive multithreading will be buggy.
    2. Re:Emacs by Anonymous Coward · · Score: 0

      Real developers use Notepad / GEdit. :)

    3. Re:Emacs by Anonymous Coward · · Score: 0

      those bloated beasts? check out ed.

    4. Re:Emacs by Anonymous Coward · · Score: 0

      ed is scary. i'm happy with vi.

  9. Linux file & memory management shines by carribeiro · · Score: 2, Informative

    Linux (and to be fair, Unix-like systems in general) shine at file & memory management. Many people don't know, but executable files are not 'loaded' in the Windows sense - they're just mapped into memory. This design improves performance and gives the system better performance under swapping (not thrashing, mind you). Things like mem mapped files are integral in the way the system is designed and implemented. That's one of the very reasons why a Linux machine usually runs faster and more reliable than a equivalent Windows machine... even if has less memory. The Apache tuning example is great, and it shows how much performance you can squeeze out of a good design.

    1. Re:Linux file & memory management shines by Anonymous Coward · · Score: 5, Informative
      Many people don't know, but executable files are not 'loaded' in the Windows sense - they're just mapped into memory.

      Apparently, some people don't know that modern NT-based Windows versions also behave in exactly the same manner.

    2. Re:Linux file & memory management shines by lisaparratt · · Score: 1

      Surely most of that efficiency disappears the second your executable comes under the scrutiny of the runtime linker? Doubly so if any relocation has to occur.

    3. Re:Linux file & memory management shines by Rezonant · · Score: 1, Informative

      Sorry, you're wrong. Windows NT has done this (mapped executables into memory instead of loading them) since 1992 or so. It's nothing specific to linux.

    4. Re:Linux file & memory management shines by Anonymous Coward · · Score: 0

      No, because the linker only needs to read a handful of pages to do it's job, and those pages can usually be discarded once it's done.

      Although I guess you could end up with some horribly perverse exectuable format that scatters the symbol table across the executable and forces the linker to load the entire binary into memory to perform the link. I can't see anyone being daft enough to do that in the real world.

    5. Re:Linux file & memory management shines by JesseMcDonald · · Score: 4, Informative

      Actually, modern runtime linkers use a table of offsets rather than embedding the relocated symbol addresses directly into the executable code, and the relocations themselves are handled by mapping the file contents into virtual memory at the necessary addresses. With those two techniques combined, it is almost never necessary for the in-memory version of the executable to differ from its on-disk representation where the code and constant-data sections are concerned. When a typical application begins execution, nearly all of its virtual memory will be mapped directly onto the executable file and its shared libraries, and loaded on demand. The initialized-data section must be copied into virtual memory, and the uninitialized-data section and the stack are typically allocated as they are accessed on a page-by-page basis. Aside from a handful of housekeeping data for the linker and the C libraries, the rest of the virtual memory consists of read-only memory-mapped files.

      --
      "The state is that great fiction by which everyone tries to live at the expense of everyone else." - Bastiat
    6. Re:Linux file & memory management shines by Anonymous Coward · · Score: 3, Informative

      Please do yourself a favor and educate yourself before making any future bogus claims.

      The following two articles respectively deal with executable and libary loading in Windows:
      http://msdn.microsoft.com/msdnmag/issues/02/02/PE/
      http://msdn.microsoft.com/msdnmag/issues/02/03/Loa der/

    7. Re:Linux file & memory management shines by ooze · · Score: 1

      Well, but with Vista this advantage will be gone again, and .Net bytecode will be translated, although all it will ever run on is x86 systems. And on top of that you need the whole .Net framework and everything else. I always ask myself how an operating system with zero apps can take up several GB of space. Maybe it's because a Windows Graphics driver package of NVidia or ATI is as big as the whole kernel source code, where all all hardware support is implemented in.

      --
      Just because I can imagine doing a hippopotamus, doesn't mean I'd like to do it.
    8. Re:Linux file & memory management shines by cnettel · · Score: 1
      Not completely true. They (MS) have actually detailed the scenarios that enable (marketspeak on) shared pages, and those that do not. The bottom line is that common libraries in the global assembly cache will almost always stay shared, and you can make sure that your own assemblies do, too, if you do not mess up the settings.

      This is somewhat similar to the fact that you can't share a relocated module between processes in the "native" world. If you customize the loading of an assembly too much, the actual pages won't stay the same and then it's obviously impossible to share them.

    9. Re:Linux file & memory management shines by typical · · Score: 1

      Under Windows, for various reasons (including the lack of a sane update and library dependency management system), applications all have their own damn copies of all the shared libraries. Which means that memory doesn't get "shared", as it does under Linux. It just gets wasted.

      --
      Any program relying on (nontrivial) preemptive multithreading will be buggy.
    10. Re:Linux file & memory management shines by ooze · · Score: 1

      Oh, I wasn't referring to the sharing part. I was referring to the "just mem-mapping into memory" part. That certainly won't work with bytecode and is the very reason Java and .Net applications always take so long to load.

      The sharing is is for sure. Java shares all the classes in one VM instance, and old dlls are also shared. The problem with Windows is just that they have even less discipline (and means) in organizing their files, libraries and executebales, in short:ressources than the Unix world. So it's not really uncommon for several Windows applications to use their very own private copy of the same dll. Or to install dlls in system wide library locations that never get used by any other application (and are a pain to get rid of).
      This even more blatant lack of discipline and means to enforce it (like a central packaging/installation/deinstallation system, or even something as trivial as a file hierarchy standard, or labrary naming conventions) is what is th4e main source of the Windows cancerous growth, and the many many places you have to attack the system.

      --
      Just because I can imagine doing a hippopotamus, doesn't mean I'd like to do it.
    11. Re:Linux file & memory management shines by gregorio · · Score: 1
      Under Windows, for various reasons (including the lack of a sane update and library dependency management system), applications all have their own damn copies of all the shared libraries.
      No, they won't. You're just bullshitting, I'm sorry.
    12. Re:Linux file & memory management shines by Anonymous Coward · · Score: 1, Informative
      Apparently, some people don't know that modern NT-based Windows versions also behave in exactly the same manner.
      No, they don't. The absense of fork() on Windows is proof that they don't.

      On Unix, to create a new process, you fork(). That creates a copy-on-write image of your process, e.g. when one of the processes changes something, it gets its own copy of the data, but stuff that is not changed stays shared between them.

      On Windows, they only have CreateProcess(), which creates a new process image based on an EXE, i.e. it loads the EXE again (AFAIK with a brand new data segment, not shared with the original). They do not have a model like fork() (and threads are not the same; you need a seperate address space)
    13. Re:Linux file & memory management shines by ultranova · · Score: 1

      The initialized-data section must be copied into virtual memory,

      Why ? Can't it just be mapped with MAP_PRIVATE ?

      --

      Forget magic. Any technology distinguishable from divine power is insufficiently advanced.

    14. Re:Linux file & memory management shines by Arandir · · Score: 1

      On Unix, to create a new process, you fork(). That creates a copy-on-write image of your process, e.g. when one of the processes changes something, it gets its own copy of the data, but stuff that is not changed stays shared between them.

      But afterwards you almost always do an exec(), which conceptually "loads" the executable.

      --
      A Government Is a Body of People, Usually Notably Ungoverned
    15. Re:Linux file & memory management shines by Anonymous Coward · · Score: 0

      That's not the point.

      The point is, Windows doesn't have anything like fork(), because given the Windows process model, fork() doesn't make sense.

    16. Re:Linux file & memory management shines by Anonymous Coward · · Score: 0

      Almost always, perhaps (though even that is debatable). But certainly not always. Ergo, there is more memory sharing in Unix.

      Look at some simple Unix code which forks a different process without doing exec(). It's very, very common. Often a similar effect can be done in Windows with threads, but, my point was that the Windows behavior is not "exactly" the same as has been claimed.

    17. Re:Linux file & memory management shines by Anonymous Coward · · Score: 0
      It just gets wasted.
      That's funny. I'd have to be wasted to use Windows, too.
    18. Re:Linux file & memory management shines by Quantam · · Score: 1

      I think you're using a more rigid definition of "exactly" than the GP. On Windows, executables (I'm using this term interchangeably with 'modules', so as to include DLLs) are loaded as memory mapped files (pretty sure this is what the GP meant by 'exactly'). Any regions of the executable that require updating (such as import tables, code that requires pointer relocation, etc.) are set as copy on write, so that if they do get written to, it'll be saved. For multiple copies of the same process (here I am using the term to mean separate copies of the program, not multiple processes of the same program that have been forked), this is no less efficient than on Unix/Linux (read-only data segments on Windows are always memory mapped). For cases of a process forking to spawn a new thread of its own, Unix/Linux would be more efficient; but remember that Unix was designed to do this (as it lacked threads), whereas Windows was designed to use threads for this purpose, which are even more efficient than Unix forks.

      You're right in that it's not exactly the same, as Windows typically does not use the forking process (I should note that there are indeed internal Windows APIs for forking). However, the only time forking provides a benefit over Windows process creation is when spawning a new thread of the same program, which I already discussed.

      --
      You have tried to support your argument with faulty reasoning! Go directly to jail; do not pass Go, do not collect $200!
    19. Re:Linux file & memory management shines by Quantam · · Score: 1

      You're pretty close, but a couple of points to mention. First, while relocations are indeed minimized (by things like import and export tables), at least on the x86 (well, there is a way, but I know Windows doesn't use it, and I have no information on Linux, but I can make an educated guess that it doesn't) you cannot use relative pointers to data. This generally means that all global variable accesses are absolute, and must be relocated (but only if the module can't load at its preferred address). This can potentially drastically increase the amount of stuff that gets copied on write in the executable.

      Also, the initialized data section is copy on write, at least on Windows. There's also a utility on Windows called Bind that can store the actual function addresses to linked modules in the executable, so that no import table fixups are necessary if the linked module loads at its preferred address (and is the same version you bound the executable to).

      --
      You have tried to support your argument with faulty reasoning! Go directly to jail; do not pass Go, do not collect $200!
    20. Re:Linux file & memory management shines by typical · · Score: 1

      Install a Fedora RPM and look how many shared libraries are installed private to that RPM. You will find damn few packages like this.

      Now install just about any commercial software package for Windows, and count the number of DLLs that get dumped in application-private directories.

      --
      Any program relying on (nontrivial) preemptive multithreading will be buggy.
    21. Re:Linux file & memory management shines by JesseMcDonald · · Score: 1

      I was only considering functions, not global data structures, but this page appears to indicate that PIC-enabled ELF programs (the default/prevalent configuration on Linux) use a Global Offset Table (GOT) for static data, which is itself referenced through a relative address from the code. Thus, relative addresses are indeed used for both functions and data in shared libraries. According to the page, Windows(tm) actually does something similar for data symbols marked with dllimport, at least in recent versions of Microsoft(tm)'s C compiler, although programmers did formerly have to take care of data relocation manually.

      On the other hand, Linux does use copy-on-write for initialized data; I only meant to say that at some point in the application's execution the data would have to be copied (when it is written to), and thus the initialized data section cannot be permanently shared like the executable and constant data sections. However, that wasn't precisely what I said, so thanks for pointing out the discrepancy.

      --
      "The state is that great fiction by which everyone tries to live at the expense of everyone else." - Bastiat
    22. Re:Linux file & memory management shines by gregorio · · Score: 1
      Install a Fedora RPM and look how many shared libraries are installed private to that RPM. You will find damn few packages like this.

      Now install just about any commercial software package for Windows, and count the number of DLLs that get dumped in application-private directories.
      Yet Windows will not load a different module if the private and public dlls are the same ones. If a programmer decides to use his own copy of windows dlls, that's his problem.
  10. Re:Before you start bitch about Firefox memory lea by CyricZ · · Score: 5, Interesting

    Firefox does indeed suffer from some very serious memory management-related issues. For anyone who doesn't have an ideological connection to the project, it's obvious why that is.

    About 8 months back I attempted to embed Gecko within an existing graphical user interface toolkit. Having heard so much from the open source community about how easy it was to do, I thought it would go rather quickly. Of course, it did not. The lack of up-to-date documentation (if such documentation there at all) and solid examples were some of the big problems.

    But the overall architecture struck me as the worst part of Mozilla. Like it or not, it's overly complicated and convoluted in many areas. I admit that it's not easy to build well-designed software, but they so completely missed the boat it's unbelievable. However, it does make it obvious as to why many people complain about Firefox and Seamonkey running so slowly, in addition to suffering from huge memory consumption.

    As for the embedding of Gecko, I said to hell with it. I took a page from Apple, and used KHTML instead. The loss in portability by not going with Gecko was well worth the far quicker development time, the lower memory consumption, the increased responsiveness, and the higher degree of stability of KHTML.

    --
    Cyric Zndovzny at your service.
  11. "Tuning Apache" is also excellent by volts · · Score: 4, Informative

    Devin's blog also has an excellent posting on Apache performance. "Tuning Apache, part 1" (and the comments) is the sort of succinct empirical advice it is always nice to find.

  12. Re:Before you start bitch about Firefox memory lea by Anonymous Coward · · Score: 5, Insightful

    Typical Slashdot response, blame the users for the browser's bloat. 99% of the users of Firefox are not programmers and wouldn't have the slightest clue what is going on. They just want to look at porn without popups or getting infected with spyware via IE's ActiveX vulnerabilities. Asking them to download some script, set environment variables, and then file bug reports is unrealistic since most of them can't even tell the difference between a web browser and a web site. That's what beta testers are supposed to be doing but we all know that 90% of the beta testers never bother to file any bug reports, even when the browser crashes.

  13. Hah... by Anonymous Coward · · Score: 0

    Have you ever wondered why a simple text editor on Linux can use dozens of megabytes of memory?

    No. I use ps and... oh.. OH... who passed this 08/15-user-FUD-article?

  14. Forgot to mention startup times... by soboroff · · Score: 2, Insightful

    All those shared libraries are also part of the reason that KDE and GNOME can take so long to start up, and why more memory and a higher-RPM hard disk can speed things up. It does make me laugh sometimes that Emacs is now one of Linux's fastest-starting desktop apps.

    1. Re:Forgot to mention startup times... by Anonymous Coward · · Score: 0

      It does make me laugh sometimes that Emacs is now one of Linux's fastest-starting desktop apps.


      Exactly. Whether or not the memory usage is shared between apps, it is ridiculous to include megabytes of shared libraries into a text editor.
      prelink or not, it has to be loaded sometime.

      Motif didn't need it (remember mr hungry/lousy memory lib?), gtk1 neither.
  15. GtkMozEmbed is useless for what I wanted. by CyricZ · · Score: 1

    First of all, I wasn't embedding Gecko into a GTK+ application. Had you bothered to read my post, you would have seen that I was embedding it into another toolkit. So using GtkMozEmbed outright was for the most part out of the question. I didn't want to involve any more unnecessary code than I had to.

    Second of all, it doesn't serve as a good example. Look at the code yourself. It's horribly convoluted. While it does indeed provide a fairly usable interface to the programmer who is using it, what's under the hood is an utter mess. That's directly caused by the poor design of Gecko itself.

    --
    Cyric Zndovzny at your service.
  16. Re:Before you start bitch about Firefox memory lea by CyricZ · · Score: 3, Interesting

    Indeed. You're completely correct. If the Mozilla crew want to take on Internet Explorer, then they can't have the general public debugging their software for them. That just won't fly.

    Part of the problem is that it's far too easy for bugs to creep into Mozilla. The code is a small step above horrible, and the architecture isn't much better. A lack of up-to-date documentation leads to programmers not knowing which XPCOM interfaces are deprecated, and which aren't.

    You can look at browsers like Konqueror and Opera, which offer a very comparable feature set to Firefox, yet do no suffer from the drawbacks. Not only that, but Konqueror and Opera are often described as feeling far more responsive, while being extremely stable. It's things like that which really impress the average Jill and Joe. Excessive memory usage will just perplex them, and likely result in them going back to Internet Explorer.

    --
    Cyric Zndovzny at your service.
  17. A practical measure and perspective. by twitter · · Score: 2, Insightful
    Using the pmap -d trick gives some insight but the amount of swap space used is what actually slows down system response. The author notes that a user who mostly runs either KDE or Gnome will pay a greater marginal cost for running the one Gnome or KDE application that's different. That's true, but it's also hard to avoid and it often does not matter, even on a modest system with 256 MB RAM. You would think that running konqueror, kontact, gimp and gnumeric on Enlightenment or Window Maker would suck down resources. It does, but it might not be enough to get you into swap space. Just run top and see. A low resource window manager can use fewer resources than a full Gnome or KDE Window Manager, despite the magic of shared libraries. DSL and Feather GNU Linux distributions run on P1s because they come with very low resource programs, which may or may not share many libraries. A little swap use does not hurt, but things get slow when too much gets in there.

    The whole discussion should be grounded in the reality of alternatives. A typical M$ system will grind it's way into swap space on start up, before the user loads anything! The very latest and greatest Linux distros run well on Pentium IIs and the like, which XP refuses to install on.

    --

    Friends don't help friends install M$ junk.

    1. Re:A practical measure and perspective. by Rezonant · · Score: 0

      Huh? XP installs just fine on a PII 400 and I've even seen it run decently on a 233MHz machine (with 256MB RAM)...

    2. Re:A practical measure and perspective. by Lussarn · · Score: 2, Insightful

      Sure, I've been running both Linux and Windows for quite some time. And I've NEVER encountered a 1 minut swap session after closing an application in Linux. On windows it happens everyday if I run a big program like a new game (As thats what I have windows for).

      I can only imagine those swap sessions on a 233Mhz machine. Linux does handle memory way better.

    3. Re:A practical measure and perspective. by geniusj · · Score: 1

      You're likely talking about differences between OS's various VM systems. I'm not sure which way is best, but here are some observations I've made . .

      Solaris will go to swap very early on, likely swapping inactive pages, even if there is plenty of physical memory to be had.

      FreeBSD doesn't seem to go to swap until it damn near runs out of physical memory to use.

      Linux is somewhere in between the two. It doesn't go to swap quite as early as Solaris, and also not as late as FreeBSD.

      Which way is best? Not sure, probably depends on what you're running and its memory usage patterns.

    4. Re:A practical measure and perspective. by Anonymous Coward · · Score: 0

      WinXP installs fine on P2. I have installed it into a P2 233Mhz with 64Mb ram! (It didn't run too fast, but it was better than win98). If you wonder, the swaping is an issue... It does help to have more than 1 physical HD and split the swap among them.

      Also, Linux is not magical either. Trying to install SuSe (8.0 I think) into this machine was impossible. KDE loading time was ridiculous and completly unusable.
      Debian's APT wasn't too good either. It takes a while to do whatever it does after you install some package...

    5. Re:A practical measure and perspective. by CastrTroy · · Score: 2, Informative

      I run a P2, 266 at home, with 256 MB of RAM. KDE 3.4 runs pretty slow. I've turned off a lot of the eye candy, but still the response time is quite slow. Windows 2000 on the other hand is quite speedy, I can't speak for windows XP, because I don't run it. The problem is, is that this isn't really a fair comparison, as the Windows 2000 UI, it more comparable to something like sawfish. Well, the look is similar, but Even straight X Windows has a better feature set. So, I could use Sawfish, but If I start up a KDE Program, then it takes forever just to start it up.

      --

      Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
    6. Re:A practical measure and perspective. by PeterBrett · · Score: 1
      You would think that running konqueror, kontact, gimp and gnumeric on Enlightenment or Window Maker would suck down resources. It does, but it might not be enough to get you into swap space. Just run top and see.
      I find it quite ridiculous that logging into Windows XP -- which has very few programs installed on it, basically AV + some games -- causes my pagefile to start grinding, while I can run many instances of Konqueror, KDevelop, OpenOffice.org and watch a DVD and listen to music all at the same time on my Linux installation without swapping anything at all...
    7. Re:A practical measure and perspective. by QCompson · · Score: 1

      I'm sorry, but the "very latest and greatest Linux distros" do not run well on Pentium IIs and the like. They run slower than an overweight grandmother with no legs. Sure, you can not use Gnome or Kde, but heaven forbid you try to open an application like firefox or openoffice: slooooww.

    8. Re:A practical measure and perspective. by Dr.+Evil · · Score: 1

      Windows just handles memory differently.

      The VMM will agressively swap out "unused" memory pages in order to increase the size of the disk cache.

      So if you're running a game or something, closing the game will cause all kinds of inactive pages long swapped out to come back into active memory.

      The downside to the Linux approach is that your disk performance is not as good as it could be while you're playing the game. This is because a lot of RAM is tied up with memory pages which are only going to be used to shut down the game. Windows would rather use those inactive pages to cache disk activity.

      Which approach is better depends on what your system is for.

    9. Re:A practical measure and perspective. by Trelane · · Score: 3, Informative
      Linux is somewhere in between the two. It doesn't go to swap quite as early as Solaris, and also not as late as FreeBSD.
      It's also quite informative to note that the swappiness of the Linux kernel may be changed dynamically, via /proc/sys/vm/swappiness.
      --

      --
      Given enough personal experience, all stereotypes are shallow.
    10. Re:A practical measure and perspective. by twitter · · Score: 1
      The problem is, is that this isn't really a fair comparison, as the Windows 2000 UI, it more comparable to something like sawfish. Well, the look is similar, but Even straight X Windows has a better feature set. So, I could use Sawfish, but If I start up a KDE Program, then it takes forever just to start it up.

      It would be more fair to compare to packages that came with Woody or Potato, which are closer to year 2000 than the latest and greatest GNU Linux distribution is. KDE 2 and Gnome 1 are both speedy on those machines as are Window Maker, Afterstep and Enlightenment. The bigger window managers have grown with hardware available. Not all window managers have gone that way, and there are new low resource alternatives.

      Window Maker, Afterstep and Enlightenment are still usable on that machine and new versions play well with KDE applications such as the Kicker, which you can stick to a single virtual window. I used Sarge on a 233 MHz P2 laptop with less than 256 RAM for more than a year this way. The only problem program was Open Office. It was slow but it ran for the few times I needed to deal with M$ formats.

      Fluxbox and XFCE also work and there are probably others I have not tried much.

      --

      Friends don't help friends install M$ junk.

    11. Re:A practical measure and perspective. by CastrTroy · · Score: 1

      I've tried all those supposedly fast ones. They are fast, and use very little memory. Until you try to open a KDE app in them. Once you open a KDE App, it loads up a bunch of KDE libs, and takes more memory because it has loaded its own libs, and kdes libs. If you can get by without running any KDE of Gnome apps, then the other ones work great. However, I find that usually I end up loading 1 or 2 KDE apps, meaning I might as well just run KDE.

      --

      Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
    12. Re:A practical measure and perspective. by Gnight · · Score: 1

      I recommend XFCE. I'm using it on a Celeron 466 with 256Mb RAM (debian sarge). Not exactly fast, but very usable as long as you stick with GTK apps.

  18. Re:Before you start bitch about Firefox memory lea by arkanes · · Score: 4, Informative
    About 8 months back I attempted to embed Gecko within an existing graphical user interface toolkit. Having heard so much from the open source community about how easy it was to do, I thought it would go rather quickly.

    I'm kinda curious who you heard that from. Embedding Mozilla when you've got an already existing binding (such as for Gtk) is trivial, but writing the binding from scratch is no easy task. Gecko is a beast and the need to integrate its own drawing layer with yours makes it hard to integrate as an embedded browser. In its defense, it was never designed or intended for such a purpose. KHTML is only easier if you're using Qt (and you *did* obey the license, right?), otherwise you need to provide mappings from all the Qt primitives used by KHTML to your own. Easier than embedding Gecko, but still not trivial.

  19. WinNT 3.1 released July 1993 ;-) by Anonymous Coward · · Score: 1, Informative
    1. Re:WinNT 3.1 released July 1993 ;-) by Anonymous Coward · · Score: 0

      oooooo wikipedia as an authoritative reference!

      classy!

  20. The numbers all go to eleven. by Anonymous Coward · · Score: 0

    But hey, 10 processes are using 10%...

    Look, right across the board, eleven, eleven, eleven and...

    1. Re:The numbers all go to eleven. by grolschie · · Score: 1

      Do you wear black?

    2. Re:The numbers all go to eleven. by Anonymous Coward · · Score: 0
      Do you wear black?
      "It's like, how much more black could this be? and the answer is none. None more black".
  21. Try using large chunks of Sys V shared memory by Anonymous Coward · · Score: 1, Funny

    And then have a lot of processes attach to those chunks of shared memory.

    It's kind fun to see your 4 gig machine have processes using 100+ gigs of RAM on it because there are 50 processes attached to a 2 gig chunk of Sys V shared memory.

    And it's even more fun when a noob admin notices it and gets management all spun up over "the huge memory leak" the overexcitable twit found in your apps. :-P

  22. Linux is no memory hogging Operating System by cciRRus · · Score: 3, Interesting

    What I really wanna understand is the memory usage in Windows.

    --
    w00t
    1. Re:Linux is no memory hogging Operating System by Trelane · · Score: 1
      What I really wanna understand is the memory usage in Windows.
      For Microsoft or non-Microsoft apps?

      *Trelane removes his tongue from his cheek

      --

      --
      Given enough personal experience, all stereotypes are shallow.
    2. Re:Linux is no memory hogging Operating System by Anonymous Coward · · Score: 0

      Bill and company want to understand that too.

    3. Re:Linux is no memory hogging Operating System by Kjella · · Score: 1

      What I really wanna understand is the memory usage in Windows.

      I got 2GB of RAM, and it is completely retarded. For one, it loves to swap to disk for no good reason. That wouldn't be so bad if it just marked them as "already swapped out, ready to be reclaimed" but it seems to have to reload stuff from swap even though I got a low commit charge and plenty system cache. Another wierd thing I see is the disk trash on exit. It seems to want to load lots of crap into memory so it can be freed. Yay. That is particularly neat if the reason you closed it was to improve speed and get your machine working, instead it'll slow to a crawl as it does heavy IO. That said, in the Qt toolkit (which I prefer) I haven't seen any way to hint to the system how memory should be used. For example playing a big movie seems to "blank out" the read cache when there should have been a nice little flag that said that I'll quite likely only read this once, and in order so the OS can give me a nice long pipeline (basicly, if you got free disk time go for it) and throw it out after being read.

      --
      Live today, because you never know what tomorrow brings
    4. Re:Linux is no memory hogging Operating System by Trogre · · Score: 1

      What I really wanna understand is the memory usage in Firefox

      --
      "Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
  23. Overdue by Chris+Pimlott · · Score: 4, Interesting


    A nice article, been looking for more information on this. So often you read items in program FAQs or such giving a disclaimer on how ps memory usage is misleading, but they offer no better way. Okay, so ps memory usage information is pratically useless; now what am I supposed to use?

    I was hoping for a bit more, though; like, say, a small program that lets you see both the aggregate virtual memory total as well as the memory used specifically by the program. Add a few options for how to handle the only-one-app-using-a-library situation. Doesn't seem like it'd be that hard, and very useful.

    1. Re:Overdue by Anonymous Coward · · Score: 0

      heh, kinda obvious, but whatthehell!

      Why don't you learn how to program in C and then write it yourself, instead of sitting around on your lazy ass waiting for someone else to do it?
      why don't YOU get of your ass and write it...

      hehe

  24. Ignorance about Java by Simon+Brooke · · Score: 2, Interesting
    Actually, a JVM is even less stable than Windows. It was not designed as a real OS. The garbage-collection, for example, will freeze the entire VM for as long as it needs to run -- and sometimes it goes out of whack and hangs permanently...

    The JVM serving this page currently has an uptime of 32 days. But in the past it's had uptimes of over 200 days. Neither it, nor any of the other Tomcat servers I run, has ever gone out of whack. Java (Tomcat, Weblogic and others) powers the web servers of many of the world's biggest websites, serving millions of pages of dynamic content every day. If it was unreliable, that wouldn't be happening.

    --
    I'm old enough to remember when discussions on Slashdot were well informed.
  25. Re:Before you start bitch about Firefox memory lea by Anonymous Coward · · Score: 1, Interesting
    It's things like that which really impress the average Jill and Joe. Excessive memory usage will just perplex them, and likely result in them going back to Internet Explorer.

    I beg to differ

    1. Average Joe uses Win32 and doesn't even know how to see how much memory a program is using
    2. Average Joe is already used to crashing apps and just accepts it as a part of life. Stability will not impress Average Joe
  26. Sure by arvindn · · Score: 2, Funny
    "Have you ever wondered why a simple text editor on Linux can use dozens of megabytes of memory? "

    Of course. EMACS - eight megabytes and constantly swapping.

  27. Also applies to shared memory segments by AtariDatacenter · · Score: 2, Insightful

    Because there is nothing quite like seeing you've got 20 Oracle instances at 1gb each on a 4gb box. :)

  28. Re:Before you start bitch about Firefox memory lea by SillyNickName4me · · Score: 1

    You will notice how much faster Konqueror is on a low-end machine. I'm currently playing around with a so called 'thin client', which mostly suffers from having less memory (256mb), a not too fast cpu (800mhz via p3 compatible) and little storgae (128mb). The cpu and memry matter for a browser.

    Now, I can run either Firefox or Konqueror on this machine, in both cases without needing swap. Both will slowly grind to a hold after a couple of hours of intensive use due to running out of memory. No idea if it is Konqueror or some other kde component that it uses, but it seems there is little difference between the 2 when it comes to slowly wanting to eat more and more memory.

    Konqueror renders a lot faster, and stays more responsive, untill the moment a page tries to use some kind of javascript. When pages do, Firefox just goes on, being slightly sluggish just like it already was, while at times Konqueror grinds to a virtual halt for a few secs, and is slow as hell untill another page gets loaded. Doesn't always happen, but often enough to prevent using it.

    And yeah, I got both of them installed, but not on the 128mb internal storgae, both come from an external usb conencted disk.

  29. Memory Management by johnnyb · · Score: 3, Informative

    On a related note, if anyone is curious how memory management library calls such as "malloc" work, you might check out my article on the subject.

  30. This issue with PS hides a huge Java issue by egarland · · Score: 2, Interesting

    The architecture of Java doesn't allow it to share library memory space like this. The effect of this is Java programs, appear to use about the same amount of memory as compiled programs when, in fact, they are using quite a bit more. This is why running a Java program that takes up 25 megs of memory can seem to suck the life out of a computer while a compiled executable using 25 megs doesn't. Java is probably really using about 10x more memory.

    It's also why systems running a Java framework with multiple programs executing in the same Java process do so much better than ones where everything is in its own process. This is Java's sweet spot, where these JVM architecture disatvantages have the least impact.

    This is my understanding of how Java's libraries work. Someone let me know if I'm missing something here.

    --
    set softtabstop=4 shiftwidth=4 expandtab nocp worlddomination
    1. Re:This issue with PS hides a huge Java issue by Anonymous Coward · · Score: 1, Interesting

      That's true for the most part, and also a "feature" not a "bug" of java (seriously). With all the talk of virualization today (Xen, VMWare Server released, Sun pushing containers, IBM with Power virtualization), it's important to note that Java has been virtualizing hardware for years. Many of the apps that folks are placing in virualized OS instances can be better and more efficiently done in Java. It doesn't *replace* hardware virtualization but is a better alternative in many cases.

    2. Re:This issue with PS hides a huge Java issue by Anonymous Coward · · Score: 0

      I believe the upcoming Java 1.6 (or whatever the hell they call it now) aka Mustang can share libraries.

  31. top by Kupek · · Score: 5, Informative

    Run top. Check out the column that says SHR. Subtract it from VIRT if you want to know the virtual memory usage of a process excluding shared libraries, or subtract it from RES if you want to know the physical memory usage of a process excluding shared libraries. Problem solved.

    I don't like how he phrases that what ps reports is "wrong." It's not wrong, or even "wrong." It reports exactly what Linux tells it (through the proc filesystem). It's just might not be what you expect it to be, which means you don't understand the tools and the system. When ps reports that a process' virtual memory usage is xKb, that is correct. In the address space for the process, xKb have been allocated. Shared or not, they're still in the address space.

    1. Re:top by Anonymous Coward · · Score: 0

      This is a silly argument. When people use ps, or free, or any number of tools that show memory usage they get from the kernel, they should display that information to the user in a manner that is clear and consise. To use another computer/car analogy, it's like making a car with a spedometer that reports the angular velocity of the wheels in radians per hour, and then berating the driver for not understanding that, or being able to readily convert that to useful information (mph, or in our case, the actual ammount of memory your 1 process is using).

    2. Re:top by Anonymous Coward · · Score: 0
      or subtract [SHR] from RES if you want to know the physical memory usage of a process excluding shared libraries. Problem solved.

      You are assuming that all of the shared memory is resident, which is almost never the case.

    3. Re:top by Kupek · · Score: 1

      It is clear and concise, assuming you have a basic understanding of how virtual memory works in an operating system. The kernel is simply reporting the statistics it keeps track of for use in memory management policies. Your analogy doesn't work because you're just comparing human-intuitive units to non-intuitive units. However, the concept of "How fast am I going?" is simpler than "What is the memory usage of a process?" A more complex question requires a more involved answer.

  32. It's also why Linux is so good at multi user by Colin+Smith · · Score: 3, Interesting

    The first person to use a system might load 128Mb worth of libraries and applications. The second and all subsequent users may only use 15-30Mb worth of RAM for each additional user. e.g. A 1Gb RAM system could handle 30 concurrent users rather than 8.

    --
    Deleted
    1. Re:It's also why Linux is so good at multi user by Doctor+Memory · · Score: 2, Interesting

      /me flashes back to the day, when I was one of sixty-four CS students editing, compiling and testing on a single VAX running VMS. With one (1) megabyte of memory. That's right, dear friends, roughly a quarter of the cache of your average disk drive today. Makes me wonder how much memory my box would use if I killed X and ran everything from the console...

      --
      Just junk food for thought...
    2. Re:It's also why Linux is so good at multi user by Anonymous Coward · · Score: 0

      I used to do that too. For ADA class. It meant you could take a lunch break while compiling "hello ada world"

  33. Aren't they still resource hogs? by dzfoo · · Score: 2, Insightful

    >> Have you ever wondered why a simple text editor on Linux can use dozens of megabytes of memory?

    Correct me if I'm wrong but... doesn't the fact that KEdit uses a lot of libraries that consume resources and impact system performance -- whether shared or not -- still means that it is a hog? I mean, if a seemingly simple application is consuming "dozens of megabytes of memory", saying "oh, it's OK, because most of it is being shared and already commited", does not really excuse it. What if those libraries are not currently being used by any other process?

    In order for the shared memory to lessen the impact on the system, the user must be running some other processes that share the same libraries. This to me is a *BIG*, and unwarranted, assumption by the developer, as evidenced by his example of someone running the Gnome environment but running a single KDE application.

          -dZ.

    --
    Carol vs. Ghost
    ...Can you save Christmas?
    1. Re:Aren't they still resource hogs? by Anonymous Coward · · Score: 0

      Correct me if I'm wrong but... doesn't the fact that KEdit uses a lot of libraries that consume resources and impact system performance -- whether shared or not -- still means that it is a hog?

      Ah, but we're comparing it to Emacs. And next to that behemoth KEdit looks positively svelte.

    2. Re:Aren't they still resource hogs? by Reemi · · Score: 1

      Depends on how you look at it.

      From the article (don't have a Linux system at hand) one sees the following libraries mentioned:
      kdeinit, ld-2.3.5.so, kedit.so, libkparts.so.2.1.0, libkio.so.4.2.0, libkdeui.so.4.2.0, libkdesu.so.4.2.0, libkwalletclient.so.1.0.0, libkdecore.so.4.2.0

      Most of them are general KDE libraries, so already in use by your normal KDE launch.

      Furthermore, if only a fraction of e.g. libxxxx.so is used, the other part will be swapped out. Ok, you might be running low on swap, but your memory is not used to much.

      Swapping could be reduced by making libraries smaller, but is this really worth the efford to save ourselves swap-space? The additional costs (overhead and management [during development / deployment version mismatches])

      Reemi

    3. Re:Aren't they still resource hogs? by diegocgteleline.es · · Score: 1

      notice that in windows, graphical apps use much less mem than in linux - not because they're any better, but because the graphic subsystem is in kernel space and the memory used in the graphics subsystem is accounted as such.

  34. Re:Before you start bitch about Firefox memory lea by Anonymous Coward · · Score: 0

    When doesn't CyricZ troll /. and other places. look at his history of posting. LOL LOL

  35. If ps is misleading, why not fix it? by Anonymous Coward · · Score: 0

    The story should really be "Why doesn't ps give human readable output by default?"

    If the knowledgeable system admin wants the output the way it is now, he can just modify it - ps -auxgiveittomebaby and get what he wants.

    1. Re:If ps is misleading, why not fix it? by Ossifer · · Score: 1

      There's a problem here--what is the correct metric for memory usage of a single process?

      Is it only private core (non-device) memory, since "those libraries would be loaded anyways..."?

      Or how about private core memory plus the portion of shared libraries being utilized, i.e. P + Li/Ni for all shared libraries 1..i where Li is the size of library i, and Ni is the number of processes currently sharing Li?

  36. Right - and wrong by PhilipPeake · · Score: 1
    The article is factually correct. Ps does include shared libraries in its size, so the incremental amount of memory used by an application may well be much less than the total of its memory footprint -- assuming that the shared libraries are already loaded by some other application.

    However, this still does not excuse a text editor for having a 25MB footprint!

    When that editor runs it WILL require 25MB of memory to run (well, the working set may be a little smaller, but if we don't want any paging we need the full 25MB).

    The "oh well its only shared libraries" excuse is just that - an excuse. Admittedly, the fault may not be with the author of the text editor application but more with the hyper-bloated "desktop" that he links to his application -- however, he should be aware of the implication of using those bloated desktop libraries, and probably do a bit more work to avoid their use.

    The thrust of the article should not have been to excuse the size of those applications because "they are only shared libraries", but to make developers, particularly desktop developers ashamed of the bloated, slow systems that they produce -- seemingly in a never ending attempt to make Windows look fast and lightweight.

    1. Re:Right - and wrong by swilver · · Score: 1
      I'm sorry, but I couldn't care less if my text editor consumes 25 MB of memory (shared or otherwise). 25 MB is peanuts, and I'll gladly use such an editor if it suits my needs, provides a snappy and clean interface and its base foot print doesn't increase when editing larger files.

      Unfortunately, for most GTK Linux apps at the moment (some of which I cannot live without), the "snappy" part isn't true because (so I'm told) GTK doesn't make proper use of graphics card acceleration. I'd glady use applications that use TWICE as much memory if they worked as fast and smooth as under Windows.

      KDE apps seem a lot more responsive, unfortunately due to the licences under which QT is available, a lot of people opt for GTK instead.

      (*) Note: with Sluggish performance I mean menu bars being slow in opening when I move the mouse over them, half-a-second delays in UI actions that are instant on Windows in the same application, that kind of stuff -- just enough to annoy and keeps me using Windows as a front-end while using Linux as just a server. Tested with Debian, Ubuntu & Kubuntu, on a machine with the binary nvidia drivers, 2 GHz, 2 GB RAM

    2. Re:Right - and wrong by Sebastopol · · Score: 1

      Maybe you don't care on your 1-user system.

      But if everyone thought the way you did, and say, you had a limited pool of generic compute servers that hundreds of you had to share, and everyone wanted to load their pet tools and eat up all the RAM rather than using the IT standard-build suite, I bet you'd have a completely different attitude.

      --
      https://www.accountkiller.com/removal-requested
    3. Re:Right - and wrong by PeterBrett · · Score: 1
      KDE apps seem a lot more responsive, unfortunately due to the licences under which QT is available, a lot of people opt for GTK instead.
      I didn't know there was a problem with the GNU General Public License...
    4. Re:Right - and wrong by lasindi · · Score: 1

      The "oh well its only shared libraries" excuse is just that - an excuse. Admittedly, the fault may not be with the author of the text editor application but more with the hyper-bloated "desktop" that he links to his application -- however, he should be aware of the implication of using those bloated desktop libraries, and probably do a bit more work to avoid their use.

      Look at it this way. Let's say you (and everyone else in your community) wanted a way to go across town. You have two choices: you can either build a public transportation medium or even system, or every person can provide their own transportation.

      If you want to provide your own transportation, you basically have to start from scratch. If you build a bicycle, that would probably take many hours and something like $50 of parts to build, or maybe $200 or more if you choose to buy a prebuilt one. You could even walk for miles and miles for free if you wanted to.

      So once you're done, you'll report to your community "I spent $200 on a bike, and I spent about a month clearing brush and building a road for my route to the other side of the city." Sounds pretty reasonable.

      But suppose for a moment the city decided to build a system of roads that everyone can use, and then they even set up a bus or subway for people to commute with. In the end, the project costs, say, $20 million. Ouch.

      But let's look at the big picture. Yes, you alone spent only $200 on your bike, but if everyone has to do that, and you have, say, 100,000 people in your city, that's $20 million in bikes. Plus, you have to clear brush every time you want to go to a different place.

      Now, I'm not trying to take my analogy too far, but if we had a 'ps' for this city, it would ask a given citizen, "How much did it cost for you to get across town?" and he would say "$200" in the first city and "$20 million" if he lived in the second.

      This is why shared libraries aren't just an "excuse." To get a simple GUI text editor running, it takes a *lot* of software. You have to have a basic C (or other language) library, a windowing system, a window manager, and widget toolkit before you can even start on the actual text editor code. Your suggestion is that everyone write their own libraries for their own programs, so that they can say "hey! my text editor takes only 500 KB of memory." I'm sure that if you wrote your own specialized C library, windowing system, and widget toolkit for your own text editor, it would show up smaller on ps than Kedit. But no one will want to use your program or your libraries because your libraries will only work for your program. That means your program can't interoperate with any other software because they're all using different libraries, and when they want to run other software, they will have to run a completely different windowing system and widget toolkit. Plus, they will run out of memory very quickly because they will have thirty different programs doing the same task (windowing) by themselves inside their own memory.

      Code reuse isn't an "excuse for bloatware." It's an effective way to avoid reinventing the wheel and not having thirty different wheels spinning when one will suffice. Shared libraries help make RAM use more efficient, make programmers more efficient, and help make users more efficient through a consistent interface. I'd rather use 25 MB once for 30 different programs than use 30 programs that each use 5 MB, and I definitely would never want to write the latter.

      --
      I have discovered a truly remarkable proof of this theorem that this sig is too small to contain.
    5. Re:Right - and wrong by swilver · · Score: 1

      It's incompatible with other open source licences, like the one used by Mozilla or Eclipse. It means they cannot use QT without also releasing their code into the GPL, or opting for the commercial QT license. GTK works because it is LGPL.

    6. Re:Right - and wrong by coolGuyZak · · Score: 1
      I would like to add to this that if you use desktop A (or a certain set of .so's) and want to use an application from another (that uses another set of .so's), the tradeoff is that you will have to wait longer for those libraries to be loaded into memory. I.e. if you use Enlightenment, don't moan when it takes forever to load KWrite--that is the tradeoff for not using KDE, but wanting some of their functionality.

      The inevitable response will probably bemoan that the KDE/GTK libraries are bloaty... but we (the KDE or GNOME people) like it that way. ;)

  37. More tips by typical · · Score: 5, Informative

    The thing is, when you fork it maps the memmory and marks everything as copy on write, when something needs to write to part of the memmory, then it will make the copy for each process.

    A couple other tips:

    * Each thread in a process shows up as consuming the same amount of memory (either this only happens under Linuxthreads or I don't have any threaded applications running on my system).

    * Device mappings show up as consumed memory (which generates plenty of XFree86/xorg complaints). If you want to find out how much memory xorg/X11 is actually using (bytes in cached pixmaps on behalf of each process and sans device mappings), try this program (contains a tiny program that lists how much memory X is using for other programs by caching pixmaps and a perl script that lists how much memory X is using sans device mappings).

    * The article mentions the fact that shared libraries show up in every application's memory usage. So, for example, glibc alone adds 1.5MB to the memory usage of every process. But Win folks may not realize how significant this is. Most Windows applications ship with their own copies of almost all shared libraries used, which means that there is a huge amount of wasted memory under Windows that *actually affects you*. Under Linux, instead of shipping shared libraries with applications, folks have built tools to automatically download the latest shared libraries and use those across multiple applications. Result -- only one copy of the library need be in memory at a time. This means that it's actually reasonable to run a box with 128MB of memory and three remote users using the thing. You simply can't pull that under Windows and expect usability.

    * This may not sound significant, but Linux's VM is (anecdotal evidence, of course) really solid. When I run out of memory under Windows, performance rapidly degrades -- bring an application to the foreground, and the system just starts churning. Under Linux, you can push a ways into VM and things generally keep functioning pretty well (this is one of the causes of people talking about "applications loading faster under WINE than Windows" when they're trying to prove that WINE is 'faster' than Windows -- good disk I/O and VM code).

    --
    Any program relying on (nontrivial) preemptive multithreading will be buggy.
    1. Re:More tips by Alioth · · Score: 2, Interesting

      The VMM on Windows in particular is a bit nasty when it comes to how it decides to push pages out to swap. The Windows VMM only looks in the CPU's TLB to look for candidate pages to trim from a process's working set, and since this is only 64 page table entries - and recently used ones at that - it's not hard to get Windows into the situation where a process can have a gigantic working set, virtually all unused - but the VMM can't swap it out to let a process that actually needs that memory to get at it (and you get a swapping storm).

    2. Re:More tips by jadavis · · Score: 1

      Linux's VM is (anecdotal evidence, of course) really solid.

      Can someone please explain where the OOM Killer fits in? Once I was running a PostgreSQL database on the same machine as a process with a memory leak. Linux decided to kill PostgreSQL. What gives?

      How do other operating systems deal with that?

      --
      Social scientists are inspired by theories; scientists are humbled by facts.
    3. Re:More tips by runderwo · · Score: 2, Informative

      The TLB is nothing more than a page table cache. On IA-32, a program has no control or ability to view the contents of the TLB besides to flush it via CR3. Saying you can look into it is like claiming you can look into L1 or L2 cache via your program (notwithstanding exceptions such as cache-as-RAM during firmware initialization). The only way you can know the contents of such caches is to know what memory accesses are performed in what order, so if you have that knowledge, then yes you could "look" into the cache using it. But I don't see how that is useful for a mechanism such as you claimed the Windows VMM implements.

    4. Re:More tips by Kupek · · Score: 2, Informative

      Each thread in a process shows up as consuming the same amount of memory (either this only happens under Linuxthreads or I don't have any threaded applications running on my system).

      Under LinuxThreads, each thread had its own PID. Under NPTL (Native POSIX Thread Library) all threads from the same process share the same PID, but each thread has a unique TID (which you can get with the Linux specific call gettid()). Calling getconf GNU_LIBPTHREAD_VERSION from a prompt should tell you what library and version you're running for pthread support.

      Anyway, this is a round-a-bout way of saying you're right. Since LinuxThreads uses a unique PID for each thread, if you queried the kernel for memory info, it would tell you that each process (thread) was invidivually consuming xKb. That's non-intuitive behavior, but I think the blame belonged to LinuxThreads, not the kernel; LinuxThreads was abusing the concept of a PID. Thankfully this has been changed in NPTL.

    5. Re:More tips by timeOday · · Score: 2, Interesting
      I agree, OOM Killer is just slightly better than a spontaneous reboot. (Or maybe worse, since it's less obvious what's going on.)

      The problem is, there's just no good way to handle low memory conditions.

    6. Re:More tips by 3rd_Floo · · Score: 1

      How about a kernel halt with a blue background and a breif nonsensical message? That would get your attention! Ohhh wait...

      Seriously though, the OOM Killer is a more 'friendly' solution than a Kernel OOPS! at least... As long as you dont mind sniping processes vs halting the system. But your very right, there is NO happy way to handle an OOM.
      Maybe just have the kernel pass a redirect to your browser to a certain site?

  38. Loading unneccessary libraries by arth1 · · Score: 2, Informative

    What gets me is how some distro builders see a security warning about setuid/setgid binaries using lazy so loading, and decide that using -Wl,-z,now is a good thing to add. Excuse me, but that will pull in EVERY library at link time, whether used or not, often leading to some MAJOR bloat.
    Yes, it "fixes" the "problem", but so would using rpath to DSOs not writable by users or ensuring that LD_LIBRARY_PATH doesn't point to user writable directories. Without the load time bloat.

    Regards,
    --
    *Art

  39. EMACS - eight megabytes and constantly swapping by newandyh-r · · Score: 1

    In the days when I was first involved in setting-up Unix systems (when SUN 3s were the fancy new toy), 8MB was a huge deal on a system with 4MB RAM. Thus we essentially banned emacs and stuck to vi. Now my thought would be "only 8MB, whats the worry".
    But has emacs also had feature creep since then, with corresponding memory growth?

    Worrying too much about effeciency (memory or processor) can be bad. Failing to keep it at least in the back of your mind while designing and coding leads to some of the awful performance we see in some programs nowadays ... and not only from Microsoft.

    1. Re:EMACS - eight megabytes and constantly swapping by TeknoHog · · Score: 1
      But has emacs also had feature creep since then, with corresponding memory growth?

      The 8MB mark is still pretty accurate... this is for the console version with a couple of text files open:

      $ ps aux|grep emacs
      teknohog 7724 0.0 1.4 8356 7180 pts/4 Ss+ Feb01 0:34 emacs
      --
      Escher was the first MC and Giger invented the HR department.
    2. Re:EMACS - eight megabytes and constantly swapping by Anonymous Coward · · Score: 0

      According to the output of `ps waux', my current xemacs session consumes 49 megabytes (VSZ) of which 38 megabytes are resident (RSS). But note that the session has been up for three weeks, and includes approximately 16 megabytes of text files in 134 open buffers. So, one could say that there is some bloat, but it's certainly snappy enough on a halfway modern computer.

  40. On OS X Java libraries are shared by MarkWatson · · Score: 1

    .. and I thought that Apple passed the code to do this back to Sun over a year ago.

    1. Re:On OS X Java libraries are shared by pjt33 · · Score: 1

      Sun introduced class data sharing between VMs in Java 1.5.

  41. Nostalgia is ... by newandyh-r · · Score: 3, Interesting

    When you can remember running 60 users on a mainframe with about 1MB RAM and a processor no faster than a 386.

    1. Re:Nostalgia is ... by the_greywolf · · Score: 1

      i had 38kB all to myself on a 1.023MHz processor. it felt like the whole world belonged to me.

      i was a 6-year-old king.

      --
      grey wolf
      LET FORTRAN DIE!
  42. Re:Before you start bitch about Firefox memory lea by Anonymous Coward · · Score: 1, Interesting

    For real fun with Firefox, download a RELEASE COPY of the source code. I want to stress this - not a beta, not a nightly, grab an actual release copy. Compile it with debugging enabled, and run it.

    Then start counting assertion errors in the RELEASE COPY of Firefox. It can be a fun drinking game, for every assertion error, drink.

    It makes you wonder how Firefox 1.5 was ever released, since merely STARTING the damned thing causes like three assertion errors before the first window is even displayed.

  43. Why did "vmmap" disappear from OS X? by SgtUnix · · Score: 1, Interesting

    I'm running OS X 10.4.4 and I know for sure that in the past there has been a utility similar to "pmap" that was called vmmap.

    The path was /usr/bin/vmmap

    But it's somehow disappeared from OS X. Anybody know why?

    1. Re:Why did "vmmap" disappear from OS X? by Psykechan · · Score: 1

      Very interesting. I've got two 10.4.4 systems here and one has it and the other does not. The only thing I can think of is that the one that does have it has the latest Apple dev tools installed.

    2. Re:Why did "vmmap" disappear from OS X? by myBotPiko · · Score: 1

      I think so too.. I've got a machine running tiger with the dev tools installed and vmmap exists here.

  44. Re:Before you start bitch about Firefox memory lea by swillden · · Score: 1

    Not only that, but Konqueror and Opera are often described as feeling far more responsive, while being extremely stable.

    Actually, I would really prefer to use Konqueror (I use KDE and I really like the integration), but I use Firefox because Konq is so much slower. I just tested it again and Konq is *much* slower on my system. I picked a site that I hadn't visited for months (to make sure Firefox didn't have a cache advantage), typed the URL into each browser, clicked on Konq and hit enter, then clicked on Firefox and hit enter -- the result was that Firefox had the full page displayed nine *seconds* before Konqueror did, in spite of Konq's half-second headstart.

    Thinking that maybe Firefox had an advantage because it was loaded and had been in use for a while, I (laboriously) browsed around a bit with Konq and then tried the same test again with another rarely-visited site. Same result.

    Konq wasn't always slow, either. Back in the KDE 3.0 days, it was my primary browser, but during some update it just got terribly slow, so I switched to FF.

    --
    Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
  45. Re:Before you start bitch about Firefox memory lea by dustmite · · Score: 1

    About 8 months back I attempted to embed Gecko within an existing graphical user interface toolkit. Having heard so much from the open source community about how easy it was to do, I thought it would go rather quickly. Of course, it did not.

    I'm not sure why you struggled, but I had no trouble embedding Gecko in a wxWidgets application - it took me a couple of hours at most.

  46. Why do libraries have to be shared? by MichailS · · Score: 1

    My c0d1ng sk1llz are not 1337, but I do know why there are such a thing as shared libraries. They seem to bring problems though. They are failure sources as you cannot ascertain that every recipient of a software package has the required library installed. In fact, my failure rate in compiling bleeding-edge Linux software is staggering. I can count a few successes and a metric shitload of failures due to missing libraries. Talk about DLL hell! And as it seems, libraries are filled to the brim with features that are certainly not used at every given time. Thus, what is the point with loading the whole nine yards if all you need is a simple function? Need one JOptionPane? Import the whole freaking javax.swing.*; ! Wouldn't it be reasonable to be able to statically link to [i]only the feature you are using[/i]? To only "check out" that piece of the library as you compile? Perhaps this is already a well-known capability of coding languages, but I haven't heard of it.

    1. Re:Why do libraries have to be shared? by Anonymous Coward · · Score: 1, Informative

      Shared and static libraries both have advantages and disadvantages. The main advantages of shared libraries are that they save hugely on memory and disk space. It also means that you often upgrade a library and all the applications will reap the benefits.

      You will find that most libraries install both shared and static versions to give applications the choice over which to link to. Shared libraries normally outweigh the benefits of static ones, so the linker will usually use shared ones. There is normally a build option to override this.

      BTW, when you 'javax.swing.*;' you aren't actually pulling in more code, just making more symbols in the namespace available to you.

    2. Re:Why do libraries have to be shared? by Anonymous Coward · · Score: 0

      You could use apt? No more dependecy hell (well you can get it if you try - I tried dist-upgrading from debian to ubuntu and back, that did it).

      Apologies if you prefer to roll your own/are using some unsupported processor.

    3. Re:Why do libraries have to be shared? by gnovos · · Score: 1

      USE="static" emerge -e world

      --
      "Your superior intellect is no match for our puny weapons!"
    4. Re:Why do libraries have to be shared? by Anonymous Coward · · Score: 0

      Use your distribution's packages instead of compiling from source, they will handle dependencies for you.

      If your distribution sucks and either (A) doesn't include packages you want or (B) handles dependencies poorly, I recommend switching to Debian or Gentoo.

      As for java... Java does not have "linking" in the same way that more traditional languages do. The "import" keyword just changes the current file's namespace settings.

    5. Re:Why do libraries have to be shared? by fforw · · Score: 1

      your example of including javax.swing.* doesn't mean the VM will load the whole swing package. the imports only exist on source level and are resolved to class and method refs in the class file. so no matter how much you import, the class file will only reference those classes that are really used..

      --
      while (!asleep()) sheep++
  47. mod-ups by pato101 · · Score: 1

    Please mod parent up, and also parent of parent deserves even more positive mods. I finally understood what it is going on and why is it so difficult to show the *real* numbers because the heavy memory optimization that it is going on. Impressive.

  48. Re:Before you start bitch about Firefox memory lea by aCapitalist · · Score: 1

    As for the embedding of Gecko, I said to hell with it. I took a page from Apple, and used KHTML instead. The loss in portability by not going with Gecko was well worth the far quicker development time, the lower memory consumption, the increased responsiveness, and the higher degree of stability of KHTML.

    What exactly is the portability problem with KHTML? I would be interested in a KHTML browser for windows and it's been said that with Qt4 and the work of various developers you'll be able to get an almost complete KDE 4 environment in windows.

  49. Nice (If Math Heavy) Load Average Article by saudadelinux · · Score: 1
    --
    I didn't think the house band in Hell would play this badly.
  50. Agreed - excellent article by a16 · · Score: 2, Interesting

    I just wanted to add my confirmation that the Apache article is an excellent tip.

    I had been experiencing issues reaching the max clients on a busy apache server serving around 6mbit/sec of images at peak times, and had been forced to increase the maximum child process setting to a very large number to cope with the peak daily periods.

    Having just made the changes recommended in that article, ie. changing the keep alive timeout to around 2 seconds rather than the default of 15 - we've gone from an average of 100+ child processes to a constant of 20-30.

    I'd advise anyone experiencing problems hitting their max client setting (the example he gives is a slashdotting, in my case it's serving loads of individual images) to try this setting out.

  51. 200 instances and 170 megs by gini_ · · Score: 2, Informative

    That is how it should be read I think. To start 200 instances of your Java proggie you pretty much did the same thing as starting 200 threads in single virtual machine. These threads show in ps output as operating system processes and they map entire address space of virtual machine which is why their sizes are identical.

    Memory usage of Java actually scales very nicely with silly number of threads. A couple of months ago I created a small server which opened lots of listener sockets in their own threads.

    With one thread the size of the virtual machine about 40 megs which pretty much for a simple application but when I created more server threads the amount of added memory was very small. With 100 listener threads it was like 60 megs, with 400 it was 80 megs and finally with 3000 server threads the amount of used memory was only 290megs!
    It is true that these threads were not actually doing anyting except listening on their sockets but I thing it is very impressive nevertheless.

    1. Re:200 instances and 170 megs by swillden · · Score: 1

      That is how it should be read I think. To start 200 instances of your Java proggie you pretty much did the same thing as starting 200 threads in single virtual machine.

      If I understand what you're saying, no. I think you're saying that the 200 instances consume 170MB, but that's not correct. Even if I start only a single instance, pmap shows 170MB. Each process shows the same values regardless of how many of them there are.

      As near as I can tell (by starting, say, 400 copies of the program looking at free memory, then killing them all and looking again), each instance consumes about 2.4MB of memory, not the 6.7MB that RSS reports, and certainly not the 170MB that pmap reports.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
  52. Article useless also by k8to · · Score: 1

    So, great, you can split out the pages associated with a lib versus allocated directly. But this still doesn't show you how much memory the program is actually using. My bittorent client is actually using 55 megs of ram, but pmap will show 161 megs of writable/private.

    Why? Because some is swapped out, and some is allocated but not used. Sure, swapped out is a cost, but in most apps things may get swapped out and never used again, thus the real cost of that memory usage is quite low.

    Additionally the library cost is not to be dismissed. On my system I have many libraries loaded for just one process.

    --
    -josh
  53. another tool by jbert · · Score: 1

    (shameless plug)

    To help address the underlying problem which the blog points out I've written a tool called Exmap.

    It uses a loadable kernel module to work out which pages are really shared between which processes, to provide some accurate numbers on memory usage in the face of complications such as demand-paging and copy-on-write.

    It isn't intended to analyse the memory usage of a specific process so much as a system of processes using shared libs/mmap'd files. In particular it doesn't do any heap analysis (for that you probably want valgrind/massif or memprof).

    But if you've got many processes running which use shared libs, this can help you work out your "real" memory usage. It'll also help you look at the usage of individual ELF sections and (if you're running non-stripped binaries) symbols.

    It currently has some rough edges (e.g. no 64-bit support, some problems on pre-linked systems), sorry about that.

  54. reverse map by CaptnMArk · · Score: 1

    I've always thought that a much more useful data would be a report of which physical (and swap) page belongs to which process (or file / shared library).

    Does anyone know of a tool that would do this (and do it accurately)?

  55. Re:Before you start bitch about Firefox memory lea by Anonymous Coward · · Score: 0

    Average Joe is already used to crashing apps and just accepts it as a part of life. Stability will not impress Average Joe

    Stability may not, but recoverability sure as hell does. Although it's pretty hard to reliably crash Opera, when the "Joe Users" who've seen Opera crash on me then see it start right back up with everything just as it was previously, they are impressed beyond belief.

    I try explaining to them that they could do the same thing with Firefox, but they have to install and configure an extension; for some reason, they are nowhere near as impressed with that idea.

  56. Re:Before you start bitch about Firefox memory lea by Anonymous Coward · · Score: 0

    http://primates.ximian.com/~federico/news-2005-11. html#moz-imagesFederico Mena-Quintero firefox memory useage analysis

    " When you run Mozilla or Firefox and load a web page with images, it stores the uncompressed images as pixmaps in the X server. In particular, it seems to maintain live pixmaps for all the images in all the tabs that you have open; even if a tab is not visible, the images will be in your X server's memory."

  57. The trouble with shared libraries by Animats · · Score: 1
    Shared libraries aren't necessarily a win on a single-user system. On the plus side, if you're running things that use a shared library, you get a gain. On the minus side, if you use one function in a shared library, you bring in the whole library.

    It's worthwhile, as an exercise, to build an application with all libraries statically linked. That gives you a real size number to compare against. If that number is much smaller than the size with shared libraries, linking with shared libraries may be a lose. Remember that if you run two instances of the same application, you share the code, so shared libraries are a win only when you have different apps sharing the same library.

    This is also a load time issue. Bringing in a huge number of shared libraries is one of the things that runs up program launch times.

  58. Re:Before you start bitch about Firefox memory lea by Anonymous Coward · · Score: 0

    I guess you have used wxMozilla.
    I wonder If you find embedding Gecko a little unstable... for example, browsing to Google's personal homepage (http://www.google.com/ig) just make the application (with Gecko embedded) crash.
    Not to mention that you need to run your application inside a mozilla base installation directory.
    So, yes it's easy to embed Gecko, but is it practical?

    guinux

    P.S.: This discussion has get way far from the original post =)

  59. Re:Before you start bitch about Firefox memory lea by CyricZ · · Score: 1

    The average user probably does not care about how much memory is being consumed, in terms of a numerical value. But when they're told by their Linux-using friend that Firefox is a superior browser to Internet Explorer, and then find their system performance dropping after switching to Firefox, they won't be impressed.

    And stability sure will impress the average user. I recall many regular users who were very surprised by the massive stability increase between Windows 98 and Windows 2000 or XP, for instance. It wasn't such a big deal for those of us who were already aware of NT, but the average user of Windows 98 or ME sure did notice the stability improvements offered by XP.

    --
    Cyric Zndovzny at your service.
  60. Re:Before you start bitch about Firefox memory lea by CyricZ · · Score: 1

    There really isn't any problem besides KHTML depending on the KDE libraries, which as of the KDE 3.x releases aren't easily ported to Windows.

    Indeed, once KDE 4 is released it shouldn't be a problem. But KDE 4 is still under heavy development at this time. I've heard estimates that it will be at least another year before things start to really stabilize with the development of KDE 4.

    --
    Cyric Zndovzny at your service.
  61. Thrashing since Jan 27th by Anonymous Coward · · Score: 0

    Speaking of Linux memory usage...
    My FC4 test box started thrashing Jan 27.
    top managed only one screen of data which showed the load at 61. It took 30 minutes, but the poweroff command was finally accepted. It reported changing the run level to 0. Only output since then has been from the oom-killer reporting mostly crond processes and free swap which has gradually dropped from 660000kB to 460000kB.

  62. Should I use Office instead of OpenOffice on XP? by andywww · · Score: 1

    Does this shared libraries phenomenon apply to XP as well? Am I suffering a major performance penalty by running Firefox and OpenOffice instead of utilized the shared libraries between XP, Word, and IE?

  63. mod parent Overrated by Darkforge · · Score: 2, Informative

    That's just not true, as someone else has swillden points out in this comment to the current story. Nobody should follow your suggestion.

    Based on your over-simplified claim (which I'll call "wrong") the 43 java threads on my Tomcat box are using 3.0GB of RAM total, minus 426MB shared, which is impossible on a box with 256MB of RAM and 512MB swap.

    More generally, the problem with ps (and top) is that they fail to highlight the most important piece of information: the amount of unshared memory each process is using, or, as TFA calls it, the "marginal cost" of each process.

    Instead, they give you the total memory available to each process. That number is irrelevant to a user of that process. It won't tell you, for example, how much memory you'd save if you killed off any given process. It won't even tell you how much total memory (shared+unshared) that process is using... as others have pointed out, ps's number includes unused copy-on-write device-mapped memory.

    ps is at best deceptive, if not actually wrong.

    --

    When I moderate, I only use "-1, Overrated". That way, I never get meta-moderated!

    1. Re:mod parent Overrated by Kupek · · Score: 1

      I don't know how your jvm implements threads, but if it's like LinuxThreads and each thread gets its own PID, then yes, naively looking at top or ps won't tell you the whole story because each thread is reported as a seperate process when it's not. You have to account for the fact that all of those threads are sharing the same address space.

      Keep in mind that when you say "ps is at best deceptive, if not actually wrong" that you're implying that the Linux kernel is at best deceptive, if not wrong, about the memory usage it reports. All ps and top do is query the kernel; they're not coming up with these numbers on their own. The kernel, and by extension ps and top, are reporting the correct numbers for a Linux system. If you attribute different meaning to those numbers, then that's your fault, not the fault of the tools you're using.

    2. Re:mod parent Overrated by Darkforge · · Score: 1

      Remember when you said to just subtract SHR from VIRT, "Problem Solved"? That's not true. LinuxThreads, device memory mapping, and other gotchas mean that there's no easy way to tell just how much memory you'll get back when you kill a given process.

      I'm perfectly happy to blame the kernel if it comes to that, for not giving ps and top enough information to do their job. But it makes no sense to blame the user for believing the deceptive information ps reveals.

      --

      When I moderate, I only use "-1, Overrated". That way, I never get meta-moderated!

    3. Re:mod parent Overrated by Kupek · · Score: 1

      I think the problem here is that people are attributing the wrong meaning to the statistics the kernel is reporting. Nowhere does it (or the man pages for ps or top) claim that the memory statistics it reports for a process is the same amount of memory you'll get back when you kill that process. The information is not deceptive; it's an accurate representation of the sate of the system.

  64. Re:Before you start bitch about Firefox memory lea by dustmite · · Score: 1

    Funnily enough I didn't use wxMozilla, as I had some problems getting that to work or I think to compile - I can't remember exactly why as it was quite long ago, but I vaguely recall it had something to do with (lack of) Unicode support at the time, and I needed to build in Unicode mode. I decided to rather use wxActiveX and wxIE and use the ActiveX Gecko wrapper that emulates the IE API, so that I would have the option of easily choosing between IE and Gecko if one or the other gave too many problems (as the app was a Windows-only app I could do this). Both IE and Gecko were basically 'equally easy' to embed using this strategy - basically just one line of code change and a recompile to switch between the two. The actual process of embedding was also quite easy. But I found both though to be quite buggy or problematic --- just in different ways. The IE component is extremely buggy indeed and I've spent probably two weeks in total now just finding workarounds for the various bugs and "annoying retarded behaviour" aspects of it (and some of the workarounds created new problems). But the Gecko component had problems too --- particularly using in-memory streams was not properly supported and I needed that. (At least I thought I needed that, but ironically I ended up not using them anyway and having to find a crappy workaround because as it turned out IE also has other (different and random) problems with in-memory streams .. *sigh*.) But at the time, on weighing the pros/cons of each I ended up choosing to go with IE. The whole 'having to have that mozilla base directory installed' was definitely a factor, that sucks.

    Embedding browser components was certainly a more horrible experience than I felt it should be. (But seemingly this is not unique to Gecko. Embedding IE is one of the most painful and annoying programming experiences I've ever had.)

    I don't recall having any stability problems with Gecko, or having any crashes when using it or anything like that.

    P.S.: This discussion has get way far from the original post =)

    Yup :)

  65. Imagine ... by Anonymous Coward · · Score: 0
    Imagine a KNotepad or GNotepad using 3.78 GigaBytes of RAM!!!

    No, no, no, they are not memory leaks, they are analyzing for the preventing of memory leaks!!!

    What??!!

    Yes, there are using a page per character.

    Is it bad for you?

  66. what about apportioning it? by idlake · · Score: 1

    Maybe the way to deal with shared memory is to just apportion it to applications. If there are 5 applications sharing 100M then each gets 20M. That way, we can get a better idea of how much impact each individual application has, assuming everything else remains the same.

  67. Re:Dupe by Anonymous Coward · · Score: 0

    So true, so true.

  68. MAP_NORESERVE by Anonymous Coward · · Score: 0

    Now, there's not even a guarantee that your mmap()'d memory has backing store behind it.

  69. Activity Monitor by Macka · · Score: 1


    For a quick result the Activity Monitor has the option under the View --> Columns menu to show not just the usual Real Memory and Virtual Memory, but also Private Memory and Shared.

  70. Re:Before you start bitch about Firefox memory lea by larry+bagina · · Score: 1
    Average doesn't know or care how much memory is being used, but he knows and cares when his machine grinds to a halt and is unresponsive.

    Stability might not impress Average Joe, but when Firefox crashes every 20 minutes (Firefox 1.5, i'm looking at you), he will be even less impressed. I can't recall IE/XP Crashing more than a handful of times. I've never seen Opera crash. Firefox crashes like a demolition derby.

    --
    Do you even lift?

    These aren't the 'roids you're looking for.

  71. Correction by Quantam · · Score: 1

    I'm almost positive that Linux does not use aforementioned method of relative addressing, as the only way to do that is using segmentation.

    --
    You have tried to support your argument with faulty reasoning! Go directly to jail; do not pass Go, do not collect $200!
  72. Re:Before you start bitch about Firefox memory lea by rmdir+-r+* · · Score: 1
    Not only that, but Konqueror and Opera are often described as feeling far more responsive, while being extremely stable.

    Unless, of course, you use them on a regular basis. Opera crashes all the time, and still Konqueror suffers from all sorts of wierd slowdowns and crashes.

  73. ps -auxgiveittomebaby by r00t · · Score: 1
    ERROR: User name does not exist.

    According to the help message, this is because you asked for:

    • ps (the command of course)
    • -a all w/ tty except session leaders
    • -u by effective user ID (supports names)
    • xgiveittomebaby (a user name; see above "-u" option)

    That's, uh, a strange user name you have there. If I drop off the "-" to use BSD syntax, it seems to fail on the "i" option.

  74. Don't talk back, by Anonymous Coward · · Score: 0

    mime is money.

  75. Re:Before you start bitch about Firefox memory lea by fast+penguin · · Score: 1

    As a side note, it should be said that KHTML is also available for GTK.

    --
    My worst enemy gave me a copy of Windows for Christmas.