Slashdot Mirror
Reducing Firefox's Memory Use
An anonymous reader writes "Many people have complained about Firefox's memory use. Federico Mena-Quintero has a proposal for reducing the amount of memory used to store images, which, in his proof of concept code, 'reduced the cumulative memory usage... by a factor of 5.5.'."
Coming from me, this is sarcasm, but it's a depressingly prevalent real attitude in the industry.
John
A quick read through the article (sacrilege, I know) claimed a small amount of jerkiness on uncompressing images. Does anyone have any idea how this would affect various hardware? For instance I have a 333MHz Pentium with ~144MB of mem that I expect (WAG) would get extremely "jerky" using this.
-- The morphemes of your disquisition are ascertainable, but they have eschewed an ambit of transpicuous exposition.
He gives three possibilities: storing it uncompressed in the server, storing it uncompressed in the client, and storing it compressed in the client (and uncompressing it on the fly).
I wonder if it might be interesting and worthwile to have an X extension to store it compressed on the server? That way there's a lot less X traffic, and potentially a lot more applications could make use of it.
The only condition is that you don't need to decompress in Moz, and recompress it to send to the X server, but just pass along the compressed data (there's some security implications with that though, but I guess they could be dealt with).
Suppose you click a link to a huge (say 10 meg) image from a slow server or on dial up. Then you want to right click and save as. Since firefox might not keep a copy of the image in it's compressed form, you may very well wind up downloading that file from the server again.
It may or may not get cached, I've definitely noticed situations where it does not get cached and must download completely again. There's a bug report on this behavior but I think it's closed as not a bug.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Windows has something called GDI objects (graphics device interface objects), and firefox uses too damn many of them.
Somewhere after 5000 of them are in use windows slows down to a crawl and dies no matter how much memory you have, and with enough tabs and windows open firefox will be using 4000+ of them all by itself.
I'm testing out browsers for use on some old machines as web kiosks. Basically, my choices are:
These machines (P1), and lots of machines like them, pretty much max out at about 64 megs of RAM. I could probably find more RAM, but it'd be costly, and there are usually hardware compatibility problems.
Although I'm leaning towards Opera at the moment, I was using Konqueror for a while. Linux does a great job of swapping, and Konqueror is quite snappy, so even with low memory it's a viable option. But, with all the libraries that Konqueror requires, 64 megs is kind of pushing it.
And there is a decided trend in hardware towards less memory and faster processors. It's not uncommon to find Pentium III's with only 128 megs of RAM. Unfortunately, many open source programs are written without limited memory requirements in mind.
It's kind of humbling to think that, as few as five years ago, a Pentium I with 64 megs of RAM would run an entire OS and web browser without so much as touching swap space. Today, you have to use apps designed for embedded machines to run in 64 megs of RAM, and you're lucky if you can run more than one app at a time.
From my testing, Firefox is barely outside the range of viable options for a machine with 64 megs of RAM. But as with any performance tuning, there are probably trade-offs. And having lots of options is usually the best strategy. But I think these improvements suggested for Firefox would be beneficial in almost any scenario. Avoiding I/O seems to be the best strategy on any system newer than, say, a Pentium I, when web browsing. So uncompressing images on the fly in exchange for less memory usage would doubtlessly be a good trade-off.
"I assumed blithely that there were no elves out there in the darkness"
That's fine when the X server and the application are on the same host, but it is less than ideal when the X server is on a different host (you really want to send the data just once in this case). It's probably better to have it both ways.
Possible outcomes:
:-) Decreasing the momory footprint is fine. But for the start I would be happy to don't see the message "out of memory" when downloading 30kB XUL code for my administration ;-)
I prefer for the start bugfree Gecko that is consuming lot of memory. For me it is better to buy new memory then delay the development of my XUL CMS.
Well, I've got to get back to work. When I stop rowing, the slave ship just goes in circles.
This was a nifty piece of investigation, but doesnt address the largest cause of firefox memory usage. Namely, memory is not freed when tabs are closed.
5 6
See:
https://bugzilla.mozilla.org/show_bug.cgi?id=1314
Try a test. Fire up a clean FF and note memory usage. Go to somewhere like fark.com and open 50 links in tabs and note mem usage. Close every tab and see if mem usage goes down. It doesnt. Most people visit dozens of pages a day. Hundreds per week. After a while, the memory footprint of FF can grow to epic proportions (ie hundreds of megs) even with only a few tabs open because FF cannot release memory of closed tabs. I have to restart FF every week or so because I'm tired of it using 200MB for no good reason.
It doesnt bother me so much that FF stores uncompressed images for tabs which are active (ie. open, even if not visible). The article itself mentions a performance hit when storing compressed images. But why the f*** cant it free the memory when I close that tab? The fact that I explicity closed it should indicate that I dont want it anymore. FF developers have acknowledged the problem but have said that there is no easy fix. Probably a poor design in the underlying architecture, though no one associated with the project would state it that bluntly.
BTW, this article reminds me of one of the best reasons to use some sort of adblocking software. You save quite a bit of memory when you arent caching a dozen useless images with every new web page you visit. Especially in light of the above bug, you can significantly slow down the expanding memory footprint with adblocking.
One of the things I occaisionally do is set up older computers that would otherwise end up in an environmentally dangerous scrap heap so they can be used by people that otherwise be unable to afford a computer (at least one with all legal software loaded). A lean configuration of Slackware has worked well before. I'm considering using Ubuntu in the future, but it raises the lower limit on memory. And Firefox on any of these has posed problems.
But the real problem is these older machines are limited in how much memory can be added at all. I have machines around that can't go above 32 MB or 64 MB. My own desktop is an 800 MHz machine that is now at the maximum 512 MB of RAM (the Intel chipset won't go higher) already has Firefox issues on a few web sites that are heavy on images and such.
I do have the following suggestion I have made elsewhere which seems to be ignored by the Firefox developers. I can set Firefox to just not load images at all. But that is annoying. What would help a lot is to have a configuration setting to go ahead and load images, but just not run animations of animated images. In addition, a pair of buttons to turn on and off the image animations would be very valuable. Also, a right click menu item for each image individually that would allow turning it's animation on and off (present if that image is an animated one) would help. Same for Marquees.
now we need to go OSS in diesel cars
Consider: since my box has 1G of memory, I do want the X server to hang on the all those pixmaps, because that makes firefax run fast. The hack would make it waste CPU time re-uncompressing images, whether it's needed or not.
With the way Firefox works now, if memory does start to run short, well, that's when the kernel will start paging things out based on its clever working set algorithms. If a given pixmap area in the X server hasn't been accessed in quite a while, it'll get swapped out to disk and the memory reclaimed. If the pixmap is accessed later, it'll automatically page back in.
I don't know about your box, but mine (Athlon XP2000+) can decompress JPEGs at a rate of around only 3MB per second. My disk drives, OTOH, are a hell of a lot faster than that.
In other words, letting the OS do its job by tossing the images onto swap when necessary strikes me as a much better strategy than constantly sucking up CPU decompressing every image every time it's used just in case the memory might be needed.
People worry too much about VMEM, IMHO. If I write a program that allocates 1G of memory, but then spins around using only 10k for the next hour, it'll have basically zero impact on the OS. Only ~10k if real RAM is actually getting used.
I've switched to opening links in new windows a lot now. In part this is because I want to group a set of tabs together. And it's easier to just close the whole mess by closing that whole window (generally all on one site or about one topic). But it seems FF is not free-ing up memory in these cases, either.
I don't see why a tab or a new window should be different internally, though. It should only be a matter of associating the state of a loaded page with a given display context. The real issue, though, is the memory management issue. Apparently something fundamentally wrong in the browser architecture is preventing that. I highly suspect it is due to over-abstraction and/or the inability of some tools they are using to properly destruct objects that are no longer needed. It does seem that large complex software projects such as this do tend to suffer a lot of complexity issues that result in basic things like free-ing memory becoming impossible to do. I don't encounter these problems in my programs, but then, I don't do anything nearly as large as Firefox, nor do I use a team of developers, nor do I use all these abstract tools by ignoring their internal operation implementations. I'll be curious as to the actual, real cause.
now we need to go OSS in diesel cars
It is an interesting notion, but I see problems with the methodology.
The biggest problem is that there is a big hit to user experience. Changing tabs and scrolling faster than your wheel will suck.
I've got a couple of 16MB & 24MB Compaq laptops that are running the Damn Small Linux distro. :-(
I think the Firefox version shipped with DSL is 0.93
I guess I won't be upgrading to 1.5 anytime soon
This is a pretty neat idea, but JPEG compression tends to be rather CPU-intensive for various reasons. I'd recommend either recompressing to a more reasonable format - LZW, for instance - or perhaps going to an intermediate compression also based on JPEG but requiring less CPU to deal with. For instance, decompressing to 32- or 64-bit fixed-length tuplets rather than the LZ or numerical compression used in the files.
When you have as few as 5 extensions, memory and CPU usage soar terribly. I realise this won't affect 80% of users like images would, but certainly 95% of FF-using slashdotters it will affect.
Surprisingly flock doesn't appear to suffer as badly from either problem, perhaps they've put it on a diet?
~HTP~ Hug that tux
Firefox simply never releases memory, AFAIK. But it'll reuse the memory.
One possible contributor to the memory issue here, and in some other programs, could be the way the memory is allocated. Memory is obtained from the kernel in chunks a multiple of a page size. These pages cannot be returned back to the kernel unless all usage of the entire page is gone. Memory usage for a typical object tends to be small pieces. If the pieces allocated for one page (in one tab) are interleaved with pieces allocated for another page (in another tab), then closing one of those tabs, even if the mainline code destructs all objects which correctly free all their underlying memory allocations, does not necessarily result in pages being released back to the kernel.
So how can memory allocations get scattered around like that? Consider that many objects need to persist as long as the page exists, but many others can be destructed because they are only needed when the page is being loaded or rendered. During loading and rendering, both sets of objects can be created in a mixture. Then the non-persistent ones would be destructed. Because of the order of allocation of underlying memory, the persistent objects tend to be interleaved with the non-persistent ones. That then means most pages may have some persistent object lying around, preventing it from being returned to the kernel.
Solutions to this problem would be difficult. But I also think this effort would be valuable for any and all large projects that can face this kind of memory issue. Some means is needed to control the memory alloction, and in particular to allow grouping of memory into contexts. The first kind of context would be a context for each tab or window being opened by the browser user. That way, if a tab is closed, it should substantially destruct objects grouped together. But this can also be wasteful because the non-persistent objects that do get destructed after rendering is done cannot have their memory recycled by other contexts. So another dimenstion of context needs to be on the basis of what is persistent vs. non-persistent, so that all non-persistent memory gets grouped together so it can be returned to the kernel as whole pages, which can then be recycled to other contexts (getting the pages from the kernel again).
This would require a much more involved memory allocation system. Further, it would also require major changes in many of the abstract programming classes used by such large programs ... in ways that tend to be counter to what the abstraction is all about in the first place. Abstraction is supposed to hide details about the underlying implementation so the programmer can/should concentrate on application logic. But this is not really an optimal way to program when dealing with limited resource issues that need to be managed, such as the memory issues seen here. In particular, the various classes themselves won't know whether they are persistent (in the context of what the browser application needs) or not. Many instances of the very same class may be created for both persistent and non-persistent intents, so the class itself could never be designed to make any such assumptions (e.g. think of hiding the details in reverse ... the class does not see the details of the application, e.g. which instances are to be long lived and which are not).
A concept that may help with this is one that would have to be applied to the whole of such object oriented programming, or even non-OO functions that also could allocate memory for such variant uses (this isn't fundamentally an OO problem ... OO merely exposes it due to the larger application scale that OO enables to be implemented). This concept is to create instance groups that can span laterally across all classes. It would require that each time an instance is created, that it be associated with a particular instance group. Then instead of destructing each instance individually, the group is destructed, which destructs all instances in the group. The implementation of all thi
now we need to go OSS in diesel cars
Time is expensive. Ram is cheap. do the math.
Say you have 500 customers, if they each have to get 512 MB of extram ram to run your software, the cost of that would run about 512 * $50 = $25,000, give or take.
Now, say instead, you get your development team to spend 3-4 weeks chasing down memory issues and optimizing the code to be lean and mean. Even if the team is very small (10 people), that just cost you $40,000 to $50,000 in salaries, not to mention the lost time they could be working on something else.
See the point? You could *buy the ram for the customers yourself* and it woudl still be cheaper than paying your developers to chase after leaks.
with knoppix and some other live cds you can use a cheatcode (I think it's knoppix:toram or some such) and when it is booting it puts the entire OS into RAM if you have enough, like you have. I do this with a mini distro, austrumi, and it makes it *blazingly fast*, as in quick. Like a megaprocessor upgrade or something. Knoppix is I think 2 gigs compressed with the single CD version, so 4 gigs of ram would give you some decent cushion.
The memory in question is not taken by firefox process per se, it's memory resources in X11 or Windows that are never freed.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Or something perhaps a bit less sexist; that's all I could come with on short notice. Point is, the operating system is fine and dandy for general purpose computing, and has lots of tweaks, but at the end of the day the general purpose OS must to generalize it's algorithms for ALL programs by definition, wheras the application itself knows best which parts of its working set are nessecary and which are trivial.
Exokernels take this concept to the extreme and let applications decide where to allocate their resources and whatnot, while the OS simply "securely multiplexes hardware." I have yet to discover how this can possibly work safely, but my gut intuition is that even if this does work, most application developers aren't prepared to take on the burden traditionally handled by the OS.
But even without having super duper access to resource allocation primitives, you can still make some sensible choices. For example, a tradeoff between CPU cycles and memory utilization like compressing large images. If your computer consistantly triggers underclocking because it's not used much, it might make sense to compress images and free up more space for other applications or more file cache.
I Browse at +4 Flamebait
Open Source Sysadmin
This is exactly the sort of thing you handle with a slider in the options; "How would you like Firefox to handle images" with High CPU/low RAM on one end and Low CPU/High RAM on the other. The slider would determine how aggressively Firefox caches uncompressed images.
Try a test. Fire up a clean FF and note memory usage. Go to somewhere like fark.com and open 50 links in tabs and note mem usage. Close every tab and see if mem usage goes down. It doesnt.
Be sure to minimize every Firefox window in between, at least on Windows. (and turn on the "VM Size" tab) Tracking usage in Task Manager is a bad way to isolate leaks like this. In fact, I don't know of a good way on Windows, I've never worked on something as large as FF.
Sigh. You have a very poor understanding of memory usage patterns.
Applications will seldom free memory back to the system, even though it has been freed within the program's memory manager (malloc etc). Most Unix systems give applications a single contiguous chunk of virtual memory that typically only grows rather than shrinks (due to memory fragmentation). That is a terriblly ineffective way to diagnose a memory leak.
AC
A 512MB DDR2 stick can be easily bought for 30-40 and a 512MB PC3200 (=DDR1) stick for 39-49.
I recently upgraded my laptop with a 1GB DDR2 SODIMM module, total cost: 99.
$200? Nice joke buddy but no cigar!
Also, maybe pure MHz hasn't gone up more than 15-20% but pure 'speed', or performance/ has at least tripled. I got my AMD X2 4200+ cheaper than a P4/2533 only a few years ago and the difference in practice in LAME/Xvid/games is on a completely different scale.
Capitalization is the difference between "Helping your uncle jack off a horse" and "Helping your uncle Jack off a horse"
I have of course noticed that Firefox uses quite a lot of memory, but it is the CPU usage that is of a greater concern to me and actually causes me problems.
I have an Athlon 64 3000+ and a gig of RAM, and I regularly go to use an application that has a genuine need for a lot of CPU e.g. a game, to find it runs slowly and then upon investigation Firefox is found to be using 50% or more of my CPU!
I have investigated further and the problem seems to relate to tabs containing Flash, I have had to get into the habit of making sure I don't keep pages open that contain flash banner ads etc (which I already found to be an annoyance, and unfortunately are appearing on an ever-increasing number of sites). Has anybody else had a similar experience? Is it Flash using all the CPU, or an inefficient interfacing with Flash by Firefox? Having a few pages containing Flash open shows immeasurably small CPU usage, but add a few more and it skyrockets :(