Slashdot Mirror

← Back to Stories (view on slashdot.org)

Mozilla MemShrink Set To Fix Firefox Memory

Posted by Soulskill on from the old-processes-never-die,-they-need-to-be-killed dept.

darthcamaro writes "If you're like a lot of Firefox 4 users out there, you've probably noticed that Firefox has a serious memory problem — it uses more than it really should. At long last, Mozilla developers are finally set to take this issue seriously with a dedicated team called MemShrink that are focused on the problem. 'It's pretty clear by now that this is a much bigger problem than any one person can likely tackle,' Mozilla Developer Johnny Stenback said."

22 of 375 comments (clear)

  1. bound to fail by demonbug · · Score: 3, Funny

    Any gains they make will be eaten up by the rapidly increasing version number.

    1. Re:bound to fail by MrEricSir · · Score: 5, Funny

      64 bits is only 8 bytes. An unsigned 64-bit number can take Firefox all the way up to version 18,446,744,073,709,551,615.

      So that should take them at least until through next Thursday.

      --
      There's no -1 for "I don't get it."
    2. Re:bound to fail by MobileTatsu-NJG · · Score: 3, Funny

      I consider this a Firefox problem not a version problem. I have always found that Firefox uses allot of memory.

      You're right. And the way they're going to fix that problem is they'll recruit a team of people to find out exactly where this memory usage is coming from, then they'll come to Slashdot and write posts describing how it's the user's fault.

      --

      "I like to lick butts!" by MobileTatsu-NJG (#32700246) (Score:5, Informative)

  2. You can fix the version number inflation too. by Anonymous Coward · · Score: 3, Insightful

    Don't be a Chrome Clone, make the next release Firefox 4.1.

  3. Easy solution by muffen · · Score: 3, Funny

    Bundle in MagnaRAM with Firefox.

  4. Re:All browsers are consuming more memory. by MrEricSir · · Score: 4, Insightful

    A lot of it's got to be the increasing size of web pages in general. Now that most folks have higher bandwidth connections, web designers don't focus on keeping the download size small.

    Multiply that increase by the size of your cache (how many times can you click "back" without hitting the disk?) and you can see the full scope of the problem.

    --
    There's no -1 for "I don't get it."
  5. Re:And probably too big... by maxume · · Score: 5, Informative

    My experience has been that both leaks and overall memory use have gone down between 3.0 and 4.0.

    At the moment, Firefox is at about 375 megabytes, with 16 tabs open. It has been open for 3 weeks. I do have browser.sessionhistory.max_total_viewers set to 3 and the anti-malware databases disabled though.

    --
    Nerd rage is the funniest rage.
  6. Re:All browsers are consuming more memory. by SimonTheSoundMan · · Score: 5, Informative

    A useful new feature in latest Nightly versions from Mozilla is about:memory. It gives you a full tree view of where the memory is being used. Of my 360MB which the browser is currently using, 101MB is JS, 46MB is storage (you back button and memory cache), 70MB on "heap-unclassified" whatever that is. JS seems to be the biggest consumer of memory.

  7. Re:not too bad by dbc · · Score: 4, Informative

    OS X here -- I usually have 8 to 12 tabs open. I almost never see memory usage below 500M, and it usually grows to about 1G after 2 or 3 hours. Firefox 4 for the Mac is seriously broken w.r.t. memory usage IMHO, and if they can't fix it fast, I'll probably be switching. I clobbers the performance of the whole system when it hogs that much memory. I'm tired of having to restart FF all the time.

  8. Yay by OverlordQ · · Score: 5, Insightful

    At long last, Mozilla developers are finally set to take this issue seriously

    Yay, it only took 5 years of bitching for them to actually look into it instead of blaming addons or your profile.

    --
    Your hair look like poop, Bob! - Wanker.
  9. Pinpoint by globalist · · Score: 3, Interesting

    This is easily reproduced in FF4 when you load a page with lots of images. The mem tends to grow proportionally to the size of the images on the page. But this is the only scenariou where the mem usage is different from other browsers and needs looking into.

  10. Problem of perception? by parlancex · · Score: 3, Insightful

    I think at least part of the problem is perception. Most people seem to have this mindset that using RAM is bad, and the more memory you have free and unused the faster your computer will be. These are the same people who think they're increasing their computer's performance by turning off superfetch, etc.The problem with this perception is that it's completely stupid.

    Programs load data into memory because memory is fast and your disk and the network are significantly slower; hundreds or thousands of times slower, and pointlessly unloading the data from memory increases the risk of having to go back to the slower disk or network to retrieve it later. If you still have RAM available, it is actually detrimental to your system performance to free this data.

    Now, when you're running out of RAM there is a problem, the operating system and applications should begin to free data that is the least likely to be useful in the near future to make room for whatever is needed at the moment. If Firefox has a problem it isn't RAM usage, it's that it isn't paying attention to global system memory levels and caching less aggressively when there is RAM pressure, and honestly I wouldn't know if that IS a problem because I have way more RAM than I've ever seen my computer manage to use.

    1. Re:Problem of perception? by mrnobo1024 · · Score: 4, Insightful

      Most people seem to have this mindset that using RAM is bad, and the more memory you have free and unused the faster your computer will be. These are the same people who think they're increasing their computer's performance by turning off superfetch, etc.The problem with this perception is that it's completely stupid.

      Programs load data into memory because memory is fast and your disk and the network are significantly slower; hundreds or thousands of times slower, and pointlessly unloading the data from memory increases the risk of having to go back to the slower disk or network to retrieve it later. If you still have RAM available, it is actually detrimental to your system performance to free this data.

      If it were possible for programs to allocate caches that work like the filesystem cache, where old items get discarded automatically to make room for anything more important, then this would make sense. But in real life, when a program written with that "unused memory is wasted memory" philosophy has filled up RAM and you start another program, the first program will have to go to the swapfile. Return to it later and it'll take forever to become usable again, while it gets re-loaded 4kB at a time. (I'll usually just kill the firefox.exe process and restart it when this happens, because that's actually faster)

    2. Re:Problem of perception? by mysidia · · Score: 4, Informative

      as this amount decreases Firefox can begin to proactively free memory used for the oldest cached data.

      So your suggestion is that each application running should be fine allocating a huge cache and changing its memory footprint according to how much memory it sees available on the entire system, instead of the OS making a decision?

      I am not sure I am convinced that the outcome of that methodology is optimal. If it were; I think i'd favor "number of pages swapped in/out per second" over amount of memory free, though.

      I am trying to imagine the interactions of 4 or 5 different applications all running with a huge cache, and the same behavior.... when memory usage is low, all 5 applications prepare a huge cache -- their huge cache causes the total memory free to drop, eventually to 1MB... now, suddenly, all 5 applications will use a bit of CPU time proactively freeing up large swaths of cache -- CPU will be 100% running for a couple seconds, as processes adjust their memory.

      After all 5 apps reduce their huge caches, suddenly there will now be a lot of memory free --- so much memory free, that one or more of the applications might immediately see an opportunity for increased caching.

      So what mechanism will protect fairness? Each application will believe its cache is important, but who's to say one of the applications isn't more important to the user, or having more requests more frequently made of it (so that the user's performance will best if application X's cache is bigger versus application Y).

      There seems a fundamental weakness here, involving each application trying to make their own memory management decisions about cache --- the application making the decision to expand its cache may be the one the user cares about the least, and the one whose cache is the least useful.

      The OS is in a position to make decisions and mediate in regards to the working set needs of processes, and the user's actual usage patterns. The OS knows which running process makes the most demand of its cache -- the other processes don't know much about each other.

    3. Re:Problem of perception? by Anonymous Coward · · Score: 5, Insightful

      Using ram is 'good' and 'superfetch' are good too huh?

      How did that info get into memory. Oh thats right by me waiting for it to load from either the internet or from disk.

      Back when I used vista turning superfetch off was the #1 way to make the computer usable. Superfetch was too aggressive in loading things. Turn computer on wait 15 mins (no kidding) before I could use the start button. It was even so aggressive it would keep memory full at all times. Meaning as programs freed and used memory the disk was thrashing ALL the time. It was noticeably faster and measurable too (I measured it many times thinking guys like you must be right). Win 7 is a whole different story (it actually works).

      As for using too much memory. Using less does mean better performance. *IF* done right. But how much of that information in this case is just 'leaks'? A leak is never going to be used. Meaning windows needs to keep track of it and page it out a some point (meaning less performance at some point).

      To bury your head in the sand and say using memory is bad is stupid. But so is using it all up is good. But think about this I have open 1 tab right now to type this in. It is using 250 meg (and that is just what is paged in meaning something in the program touched it). WTF is it doing... That is a seriously crazy amount of memory to be using to have this page open which about 30k. That is some serious overhead there. From this old 80s/90s programmer I look at that and think wtf dudes. You KNOW you can do better than that.

    4. Re:Problem of perception? by QuoteMstr · · Score: 5, Informative

      If it were possible for programs to allocate caches that work like the filesystem cache, where old items get discarded automatically to make room for anything more important, then this would make sense

      The system you describe is called malloc()!

      In a system with a unified buffer cache (essentially, every OS in wide use except OpenBSD), it makes little difference whether a page of memory comes from a private memory allocation (e.g., a heap allocation), a memory-mapped file, or the OS's disk cache. When a process needs a page not already present in memory, the kernel's memory manager tries to find an unused page. If one is available, it hands it to the program that requested memory.

      Otherwise, it looks for an in-use page, saves its contents, and hands the just-freed page to the program requesting memory. If that page is "dirty" --- i.e., it's backed by a file and somebody's written to that part of the file, or it's a private allocation backed by the page file --- the memory manager writes the page out to disk first. If the page isn't dirty, the memory manager can just discard its contents because it knows it can reconstruct it by reading back the original file.

      When the memory manager has to go to disk to satisfy a request for a new page, it's called a hard fault. The mission of the memory manager is to reduce the number of hard faults, because hard faults are slow. The fewer hard faults you have, the less time will be spent waiting for the disk, and the faster your system will run.

      The most important part of the memory manager is page replacement: i.e., how the memory chooses what page to evict in order to satisfy a memory allocation request. Most systems use an approximation of LRU (i.e., least recently used), throwing out pages that haven't been accessed in a while. It doesn't usually matter where a page came from. The only important factor is how recently it was accessed.

      So, you can see that there's no difference between a program mapping a file into memory and modifying it, reading and writing it using file APIs, and just manipulating an equal amount of information in buffers created with malloc. To the kernel, all memory is made up of pages.

      The "go away for a while" problem isn't caused by any particular memory strategy. It's an artifact of the memory manager's LRU approach. How does it know that the pages corresponding to Firefox are going to be used again? If some other program needs those pages, the older ones will be discarded. There is nothing applications can do.

      Instead, the OS itself has to be tweaked to preserve interactivity. Sometimes the memory manager will prefer disk cache pages to malloc-backed ones. Sometimes (e.g., for Windows SuperFetch) the OS will try to identify pages belonging to activate applications and try harder to keep those in memory. Some systems favor keeping executable pages over private allocations. You can tweak the page replacement algorithm, but the basic idea, that all memory is made up of pages subject to the same management scheme, applies.

      Ultimately, it's ridiculous to hear people talk about programs "keeping things in memory" like we were still dealing with DOS 6 and OS 9. The actual situation is a lot more subtle, and silly memory counters don't even come close to giving you a good picture of what's actually going on.

      In short, don't worry about fine-tuning what's "in memory". Don't change behavior based on total amount of memory in the system. Operating systems (OpenBSD aside) ALREADY DO THAT. Just let the memory manager do its job, and give it enough information (via interactivity information, memory priority, etc.) to do its job properly. Don't try to hack around problems at the wrong layers.

    5. Re:Problem of perception? by taosk8r · · Score: 3, Informative

      I use this little restart firefox extension, kind of a cool trick.. Also config trim on minimize (google it).

      --
      -taosk8r
    6. Re:Problem of perception? by adri · · Score: 4, Interesting

      Ah, the sounds of someone spewing the 80's virtual memory rhetoric.

      There's more to it then that. I could go into it, but I'm supposed to be studying for a psychology exam, so I'll be brief.

      You assume that the OS will make sensible paging decisions. You assume you can hint to the OS that you're going to make sensible paging decisions. You hope the application, which is likely big, multithreaded and such, is doing the sensible thing of not wrapping large accesses to "memory things" (eg big trees of data, as an example, or image caches, or whatever takes up more than a small bit of RAM) in mutexes. You assume that your application is using memory in a sensible fashion, and not simply using a few bytes here and there in each allocated chunk.

      The trouble is, application writers have been taught from an "early" age that hey, memory is cheap, the OS will handle paging out unused bits, so please go right ahead and use it without caring about how it's used. This is how you end up with application behaviours which include, but aren't limited to:

      * walking a tree requires a page in (ie, a random disk read) for each tree node touched. Because each node is malloc()'ed, and although on modern implementations small objects are packed into pages, your 10,000 tree node is going to likely be spread across multiple pages based on when and how often you allocated them ('temporal location');
      * this also means your memory use versus memory allocation isn't terribly efficient ('fragmentation');
      * your mutex protected data structures are suddenly now mutex'ing _page disk access_, so whilst the OS is busy paging in your data, all other threads currently trying to do stuff that requires that mutex (which may even not require paging in) suddenly has to stop and wait for your page-in to complete.

      It's a real shame that memory management has really stopped progressing since virtual memory systems were made. They're convenient, but they hide the worst case behaviours from unknowing coders. Then those worse case behaviours become _very_ worse case behaviours which can't be changed without a fundamental rearchitecture of your software. Likely what people are realising here.

      Enjoy!

    7. Re:Problem of perception? by arth1 · · Score: 5, Insightful

      it's not designed to be open for a week straight.

      Then it needs to be redesigned.
      I do not need a server to become sluggish because a user left on vacation with Firefox still running with a detached display.
      And I do not want to have to close my munin/bigbrother/mrtg windows running on the wall monitor. It should keep running.

      Remember how we ridiculed Microsoft for Windows 95 not being able to run more than a few days without a reboot? Well, it's time we laugh at the idiotic Firefox devs who repeat the same mistakes and invalid assumptions.

      Because of the severe bloat and WTF assumptions made by the newer generation of Firefox devs, I've started using other browsers more and more. Like Midori ("it's not easy being green").

  11. Needs to be all of Mozilla by binarybum · · Score: 4, Insightful

    I hope this isn't just targeted towards firefox. Thunderbird is an unwieldy beast of an email app as well. No good reason that checking my email should involve consuming 200Mb of memory.

    --
    ôó
  12. Re:Let's do a test by Dwedit · · Score: 3, Interesting

    Looks like the Url Classifier (worthless anti-phishing feature) is eating up a fair amount of resources for nothing.  You look in your user profile, and you see the urlclassifier3.sqlite file keep getting bigger and bigger for no real reason, and you see that Firefox needs 24MB of RAM just to have it loaded and working.

    Here's my list (sorted top 25, but may be misleading since some of the numbers look like sums of others)

    win32/workingset                             432,160,768
    win32/privatebytes                           420,995,072
    malloc/committed                             376,307,712
    DescriptionValuemalloc/allocated             354,080,976
    storage/sqlite/pagecache                     79,423,848
    js/gc-heap                                   66,060,288
    storage/places.sqlite/Cache_Used             52,400,928
    storage/urlclassifier3.sqlite/Cache_Used     24,250,888
    js/string-data                               7,282,684
    layout/all                                   6,261,185
    images/content/used/raw                      4,043,747
    images/content/used/uncompressed             3,444,586
    gfx/surface/win32                            3,398,208
    storage/sqlite/other                         1,589,512
    malloc/dirty                                 1,097,728
    js/mjit-code                                 913,507
    storage/cookies.sqlite/Cache_Used            528,376
    storage/formhistory.sqlite/Cache_Used        508,664
    storage/extensions.sqlite/Cache_Used         494,144
    storage/addons.sqlite/Cache_Used             362,512
    content/canvas/2d_pixel_bytes                360,000
    images/chrome/used/uncompressed              353,440
    shmem/allocated                              344,064
    shmem/mapped                                 344,064

  13. Re:All browsers are consuming more memory. by BZ · · Score: 3, Informative

    There are several things going on here:

    1) JS JITs. These optimize for speed of compilation and speed of generated code; small size of generated code is not really something being optimized for except insofar as it helps one of the other two metrics. In the case of Firefox, just deciding to JIT a little less aggressively late in the 4.0 cycle saved a good bit of memory when JS-heavy pages are open.

    2) Images. Sites are using more and more bigger images, in addition to larger and larger scripts. With images you have the options of decompressing on draw (slow, typically) or storing decompressed images in memory (uses lots of memory). Guess which one browsers are typically doing?

    3) Leaks in webpages. By this I mean web pages that allocate more and more JS objects and have them all reachable (e.g. by sticking things in an array that's hanging off the Window) so the web page uses more and more memory. gmail did this until recently; they were working on fixing it last I checked. This means that if one of your "few tabs" is gmail and you've had it open for a while a lot of that memory could be actually being used by gmail.

    about:memory in Firefox is being improved to make it easier to answer the "what's using the memory" question, at least....