Slashdot Mirror
Reducing Firefox's Memory Use
An anonymous reader writes "Many people have complained about Firefox's memory use. Federico Mena-Quintero has a proposal for reducing the amount of memory used to store images, which, in his proof of concept code, 'reduced the cumulative memory usage... by a factor of 5.5.'."
With CPU development practically stagnating, but RAM and HD storage still growing fast, it also strikes me as a pretty good solution. I wouldn't want to instance that CPU-hogging old JPEG algorithm 50 times just because I hit the back button.
There's always a tradeoff between memory use and CPU comsumption. If it's simple to do and has enough impact, maybe they should let the users decide?
Or even automatically configure it depending on the user's hardware.
He gives three possibilities: storing it uncompressed in the server, storing it uncompressed in the client, and storing it compressed in the client (and uncompressing it on the fly).
I wonder if it might be interesting and worthwile to have an X extension to store it compressed on the server? That way there's a lot less X traffic, and potentially a lot more applications could make use of it.
The only condition is that you don't need to decompress in Moz, and recompress it to send to the X server, but just pass along the compressed data (there's some security implications with that though, but I guess they could be dealt with).
Windows has something called GDI objects (graphics device interface objects), and firefox uses too damn many of them.
Somewhere after 5000 of them are in use windows slows down to a crawl and dies no matter how much memory you have, and with enough tabs and windows open firefox will be using 4000+ of them all by itself.
it's not just the amount of space it takes in RAM and HD, but also the amount of time it takes to transport something that's 5.5x bigger. Have you seen hard drives become 5.5x faster recently? Or even RAM?
At the same time, rendering/decompressing an image might be quite self-contained, and running 50 self-contained instances of that CPU-hogging old JPEG algorithm might parallellize quite easily, so would scale with the current trends in CPU development (more parallel - just throw more chips/cores/DSPs/GPUs at it).
Just imagine trying to take advantage of graphics hardware that can decode JPEGs in hardware.
I'm testing out browsers for use on some old machines as web kiosks. Basically, my choices are:
These machines (P1), and lots of machines like them, pretty much max out at about 64 megs of RAM. I could probably find more RAM, but it'd be costly, and there are usually hardware compatibility problems.
Although I'm leaning towards Opera at the moment, I was using Konqueror for a while. Linux does a great job of swapping, and Konqueror is quite snappy, so even with low memory it's a viable option. But, with all the libraries that Konqueror requires, 64 megs is kind of pushing it.
And there is a decided trend in hardware towards less memory and faster processors. It's not uncommon to find Pentium III's with only 128 megs of RAM. Unfortunately, many open source programs are written without limited memory requirements in mind.
It's kind of humbling to think that, as few as five years ago, a Pentium I with 64 megs of RAM would run an entire OS and web browser without so much as touching swap space. Today, you have to use apps designed for embedded machines to run in 64 megs of RAM, and you're lucky if you can run more than one app at a time.
From my testing, Firefox is barely outside the range of viable options for a machine with 64 megs of RAM. But as with any performance tuning, there are probably trade-offs. And having lots of options is usually the best strategy. But I think these improvements suggested for Firefox would be beneficial in almost any scenario. Avoiding I/O seems to be the best strategy on any system newer than, say, a Pentium I, when web browsing. So uncompressing images on the fly in exchange for less memory usage would doubtlessly be a good trade-off.
"I assumed blithely that there were no elves out there in the darkness"
That's fine when the X server and the application are on the same host, but it is less than ideal when the X server is on a different host (you really want to send the data just once in this case). It's probably better to have it both ways.
Possible outcomes:
This was a nifty piece of investigation, but doesnt address the largest cause of firefox memory usage. Namely, memory is not freed when tabs are closed.
5 6
See:
https://bugzilla.mozilla.org/show_bug.cgi?id=1314
Try a test. Fire up a clean FF and note memory usage. Go to somewhere like fark.com and open 50 links in tabs and note mem usage. Close every tab and see if mem usage goes down. It doesnt. Most people visit dozens of pages a day. Hundreds per week. After a while, the memory footprint of FF can grow to epic proportions (ie hundreds of megs) even with only a few tabs open because FF cannot release memory of closed tabs. I have to restart FF every week or so because I'm tired of it using 200MB for no good reason.
It doesnt bother me so much that FF stores uncompressed images for tabs which are active (ie. open, even if not visible). The article itself mentions a performance hit when storing compressed images. But why the f*** cant it free the memory when I close that tab? The fact that I explicity closed it should indicate that I dont want it anymore. FF developers have acknowledged the problem but have said that there is no easy fix. Probably a poor design in the underlying architecture, though no one associated with the project would state it that bluntly.
BTW, this article reminds me of one of the best reasons to use some sort of adblocking software. You save quite a bit of memory when you arent caching a dozen useless images with every new web page you visit. Especially in light of the above bug, you can significantly slow down the expanding memory footprint with adblocking.
When you have as few as 5 extensions, memory and CPU usage soar terribly. I realise this won't affect 80% of users like images would, but certainly 95% of FF-using slashdotters it will affect.
Surprisingly flock doesn't appear to suffer as badly from either problem, perhaps they've put it on a diet?
~HTP~ Hug that tux
One possible contributor to the memory issue here, and in some other programs, could be the way the memory is allocated. Memory is obtained from the kernel in chunks a multiple of a page size. These pages cannot be returned back to the kernel unless all usage of the entire page is gone. Memory usage for a typical object tends to be small pieces. If the pieces allocated for one page (in one tab) are interleaved with pieces allocated for another page (in another tab), then closing one of those tabs, even if the mainline code destructs all objects which correctly free all their underlying memory allocations, does not necessarily result in pages being released back to the kernel.
So how can memory allocations get scattered around like that? Consider that many objects need to persist as long as the page exists, but many others can be destructed because they are only needed when the page is being loaded or rendered. During loading and rendering, both sets of objects can be created in a mixture. Then the non-persistent ones would be destructed. Because of the order of allocation of underlying memory, the persistent objects tend to be interleaved with the non-persistent ones. That then means most pages may have some persistent object lying around, preventing it from being returned to the kernel.
Solutions to this problem would be difficult. But I also think this effort would be valuable for any and all large projects that can face this kind of memory issue. Some means is needed to control the memory alloction, and in particular to allow grouping of memory into contexts. The first kind of context would be a context for each tab or window being opened by the browser user. That way, if a tab is closed, it should substantially destruct objects grouped together. But this can also be wasteful because the non-persistent objects that do get destructed after rendering is done cannot have their memory recycled by other contexts. So another dimenstion of context needs to be on the basis of what is persistent vs. non-persistent, so that all non-persistent memory gets grouped together so it can be returned to the kernel as whole pages, which can then be recycled to other contexts (getting the pages from the kernel again).
This would require a much more involved memory allocation system. Further, it would also require major changes in many of the abstract programming classes used by such large programs ... in ways that tend to be counter to what the abstraction is all about in the first place. Abstraction is supposed to hide details about the underlying implementation so the programmer can/should concentrate on application logic. But this is not really an optimal way to program when dealing with limited resource issues that need to be managed, such as the memory issues seen here. In particular, the various classes themselves won't know whether they are persistent (in the context of what the browser application needs) or not. Many instances of the very same class may be created for both persistent and non-persistent intents, so the class itself could never be designed to make any such assumptions (e.g. think of hiding the details in reverse ... the class does not see the details of the application, e.g. which instances are to be long lived and which are not).
A concept that may help with this is one that would have to be applied to the whole of such object oriented programming, or even non-OO functions that also could allocate memory for such variant uses (this isn't fundamentally an OO problem ... OO merely exposes it due to the larger application scale that OO enables to be implemented). This concept is to create instance groups that can span laterally across all classes. It would require that each time an instance is created, that it be associated with a particular instance group. Then instead of destructing each instance individually, the group is destructed, which destructs all instances in the group. The implementation of all thi
now we need to go OSS in diesel cars
At work we have half a dozen PCs (of ~22) with the maximum amount of RAM installed that they support. The list includes my laptop. I am therefore for anything that reduces RAM use and against any lazy programming that requires more RAM.
This is exactly the sort of thing you handle with a slider in the options; "How would you like Firefox to handle images" with High CPU/low RAM on one end and Low CPU/High RAM on the other. The slider would determine how aggressively Firefox caches uncompressed images.
> What I do have a problem with is how it doesn't bother to remove raw images that are no longer
> needed. Essentially, it is a really bad memory leak that they haven't fixed for ages.
This is *partly* due to the way memory allocation and freeing work at the system call level. In a nutshell, memory that you free does not actually become free for other programs to use until your process exits. (As bad as this sounds, it's preferable to the situation wherein the system doesn't know what process owns the memory, so if an app has a leak the only way to reclaim it is to restart the whole system.) I am not sure which platforms have this (wait until the app exits) behavior, but I think it might be fairly common, although on some systems the not-really-free memory may be given to the *same* process again, next time it requests more memory, and in that case it shouldn't grow infinitely, as it would be only *other* processes that can't use it. Still, it's annoying that once you've loaded up too much at once and grown the process' memory usage to a large figure, the only way to reclaim that for other apps is to exit the process -- closing the document that caused all the usage won't do it. I *think* this is the behavior I see on Ubuntu, e.g., once I've had a really complex SVG image open in Inkscape (or a big document in OpenOffice with a lot of frames and especially images in it), and I then open a smaller, simpler image (or document) and close the big one, the memory doesn't get freed until I close Inkscape. However, if I've had a big document open, which I've closed, and then I open another big document, the memory usage doesn't go up any worse (unless the second document is bigger or more complex than the first one, of course), so it appears the same process is handed that same memory again that it freed earlier, even though another process can't have it.
I *suspect* the same is true on Win NT/2K/XP, although it's harder to tell there, due to the lack of facilities such as panel applets and good process management tools. There's the mem command, but I haven't spent a lot of time on tracking memory usage that way. On the other hand, part of the reason I haven't spent time figuring it out is because memory that's not in usage any more on Windows systems usually ends up just growing the swapfile automatically, which is really not such a big deal if you've got plenty of hard drive space. I *mostly* like the way Linux handles virtual memory better than Windows (in particular, it (since circa 2.4, or maybe it was in 2.2) seems to do a quite good job of knowing what to swap out or not to keep thrashing to a minimum), but I do like the auto-allocation that Windows does.
Cut that out, or I will ship you to Norilsk in a box.
Not true. Say that after that hour, your program needs to access all of its memory again (because you deiconified it, or whatever...). And that you have done other stuff during the last hour, that caused the rest of your gargantuan memory hog program to be paged out to disk. That means 1GB needs to be loaded from disk.
At about 18 MB per second (which is what I measured when trying "time cat somemovie.avi > /dev/null"), that would take about a minute.
Paging to disk is no longer a solution to the problem of running out of memory. With most users having >512MB RAM, it just takes too long to reload memory from disk. Swapspace is obsolete! Besides, linux paging and disk scheduling isn't that smart. Try to do something productive at the same time as updatedb is running.
In the future, we may be able to swap to some cheaper slower memory, a solid state diskdrive, or something else. But untill then, both memory usage and disk time is a scarce resource. On the other hand, multicore has already arrived. CPU is going to get cheap!
I don't know about your box, but mine (Athlon XP2000+) can decompress JPEGs at a rate of around only 3MB per second. My disk drives, OTOH, are a hell of a lot faster than that.
I agree you have a point here. I'm not advocating decompressing jpegs on the fly when scrolling. Memory is scarce, but not *that* scarce. But saving memory by discarding uncompressed images from closed windows and tabs, and "back" pages, would be a good start. You only need speed for common operations. Wasting tons of memory on things rarely done, will eventually slow the whole system down.
I understand your point, but how about something as simple as a "cleanup memory" button. I too often have several tabs open (4 instances right now with 24 tabs) but most of those tabs will sit idle most of the time, like my PHP/MySQL/M-W.com windows that are mainly used for occasional reference. Being able to force a cleanup would be good. I'd still save the time to reload the entire page, even if it does take a second to redraw when I flip back to the tab.
Great for people like us, who knows whats going on in the background. Many people never run into the firefox memory issues because they tend to single process with their machines, and switch their machines off regularly. The solution should not be a manual thing, it should be able to be solved with clever programming (perhaps an idle, lack of mouse movement detection?) its hard enough to train users to use computers to do their jobs, without adding extra load to their fragile little minds. M
Do the following really mean anything? SCSA MCP CCSA CCNA
--I'm not actually after an answer!