Firefox Memory Leak is a Feature

SenseOfHumor writes "The Firefox memory leak is not a bug. It's a feature! The 'feature' is how the pages are cached in a tabbed environment." From the article: "To improve performance when navigating (studies show that 39% of all page navigations are renavigations to pages visited less than 10 pages ago, usually using the back button), Firefox 1.5 implements a Back-Forward cache that retains the rendered document for the last five session history entries for each tab. This is a lot of data. If you have a lot of tabs, Firefox's memory usage can climb dramatically. It's a trade-off. What you get out of it is faster performance as you navigate the web."

Comments

Total cached page limit.

[Short Circuit]

So there's a way to limit the number of cached pages per tab, but no way to limit the total number of cached pages, for those of us who have fifteen tabs open?

Whoops!

Re:Total cached page limit.

[Rolan]

Yeah, where can I turn off this "feature"? Or, better yet, why doesn't this "feature" release memory when the tab is closed? Either of those would make me much much happier with Firefox.

Re:Total cached page limit.

[bpd1069]

FTFA: "...For those who remain concerned, here's how the feature works. Firefox has a preference browser.sessionhistory.max_total_viewers which by default is set to -1."......If you set this preference to another value, e.g. 25, 25 pages will be cached for every tab. You can set it to 0 to disable the feature, but your page load performance will suffer.

Re:Total cached page limit.

[SmartSsa]

From the Tips & Tricks page:

Specify the memory cache usage

        Normally, Firefox determines the memory cache usage dynamically based on the amount of available memory. To specify a specific amount of memory cache, add the following code to your user.js file: // Specify the amount of memory cache: // -1 = determine dynamically (default), 0 = none, n = memory capacity in kilobytes
        user_pref("browser.cache.memory.capacity", 4096);

        To disable the memory cache completely, add the following code: // Disable memory cache:
        user_pref("browser.cache.memory.enable", false);

Re:Total cached page limit.

[Short Circuit]

I wouldn't call completely disabling the feature an optimal solution.

Me, I'd like to limit it to 25 cached pages, and have the old ones shuffled out as new ones are shuffled in.

Re:Total cached page limit.

[Anonymous Crowhead]

I have a little trick I use from time to time with firefox:

[user@localhost ~]$pkill -9 firefox

I use it once a day at least.

Re:Total cached page limit.

[Michalson]

Thank you thank you thank you (sorry ot no mod points, but you're already up to 5 anyway).

I browse with a lot of tabs in FireFox, and with FireFox 1.5 the performance when a lot of those tabs are loading has been beyond horrible. Like several seconds just to switch tabs, and then actually trying to scroll...

If you are feeling generous, perhaps you also know how to shutoff the new tab thumbnail "feature" when you've got images. 16x16 thumbnails of 4000x4000 images are nothing but a waste of CPU time and a visual distraction.

Re:Total cached page limit.

[masklinn]

Sorry? When I'm using Opera I often see it ramping above 150Mb of ram usage, that's equivalent to Firefox' RAM consumption on my machine.

Re:Total cached page limit.

[masklinn]

1. set browser.sessionhistory.max_total_viewers to "0"
2. It does (try opening a huge Fark photohop thread, huge as in multiple hundreds of pictures, see Firefox ramp up to 600 or 700Mb ram consumption, close the fark tab, see firefox' ram usage drop dramatically to regular ram usage levels)

Re:Total cached page limit.

[Anonymous Coward]

An operating systems class might help you understand why memory usage meters are completely unresponsive in the down direction.

See here's what happens:
Firefox allocates memory for a rendered page. You've got 20MB allocated already, all in 3 chunks. None have enough room for the single large allocation it needs so the OS sets aside a new chunk of memory for the firefox process.
Now it's using say 28MB of memory. And only 22MB of that is used. Well, it does a couple more allocations, some fairly permanent ones, and these get put in the newest block of memory.
Then you close the tab. Firefox frees the associated memory. The OS changes it tables around for that block to indicate so. But it still has some stuff in that block. So guess what? Firefox' memory usage remains exactly the same.

The solution? Use a GC system. Some Garbage Collectors (most) actually move objects to condense them in memory. This is one of the things that makes garbage collections noticeable if a lot has happened since the last one (it's gotta move a lot of RAM and change a bunch of links to said RAM). It becomes especially bad when you move into swap space ;).
The downside? While GC advocates will often amaze you with the fact that malloc is not an atomic operation (it has a lot of work to allocate, more or less depending on the current situation of your memory chunks and the free memory on the system), malloc is still not nearly as costly as a garbage collection cycle. And, free is atomic (at least, TMK all a good implementation does is remove something from a data structure, unless it's the last part in which case it also needs to mark that memory as free).

So, you see, no matter how few memory leaks firefox has, it still won't drop in RAM usage every time you click close.
If you want to prove memory leaks in firefox you can. Get yourself a memory debugger (such as valgrind) and run firefox under it. Now, I'll warn you that this is harder than it sounds:
1.) Memory debuggers are about 100x to 1000x slower than your machine natively.
2.) Firefox is a script, not a binary, it sets up a bunch of stuff for the binary to run.
3.) Everything you see on the memory debugger is not necessarily a leak. Some of the leaks aren't even really leaks (it's generally no big deal to leak when you're exitting because the kernel cleans that up for you).
4.) To get any useful information on the leaks (other than size) you'll need to have compiled with debug symbols and you'll need to have the source code.

Go ahead, post your list of firefox memory leaks. Then post your list of IE memory leaks. I bet both have some, but neither has anything major. And I bet it takes you a week to find them ;).

Re:Total cached page limit.

[Andrewkov]

Yep, type about:config in the address bar, then change browser.chrome.image_icons.max_size to 0.

Re:Total cached page limit.

[numark]

Type "about:config" (no quotes) into your address bar, then scroll down to the browser.sessionhistory.max_total_viewers setting, double-click on it, then change the number to 0 and hit OK. Any sort of Firefox setting like this is found in about:config.

Re:Total cached page limit.

[afidel]

And a detailed explanation of the feature and it's values can be found here.

Re:Total cached page limit.

[Edgewize]

You're confusing garbage collection with heap compaction.

People have written garbage collectors for C++, and they work just fine. But they do not help with fragmentation, which is the problem you're describing. That requires a heap-compacting allocator (aka a "handle" allocator). Many languages with garbage collection also use a heap-compacting allocator. C++ does not, because of a low-level language "feature": pointers, and specifically, pointer arithmetic.

If an object moves in memory, then people have to be notified that it has moved, or they won't know where to access it. Languages like Java handle this behind-the-scenes; the system library tracks objects for you, and your program never knows (or cares) whre an individual object is.

C++ allows direct access to system memory, and it tells you precisely where your objects are located. Programmers are then free to do all kinds of things like compute distance to other objects, or convert the location to a number and do arbitrary math operations on it.

When an object moves, anything that refers to it needs to be updated. Well, good luck figuring that out in a language with pointer arithmetic! The system would need to magically determine whether or not a numeric value was actually a memory locations. And what if a program computed the distance between two objects, and later on used that distance to get from one object to the other? The system has no idea of what can be safely moved, and what has to stay put. So nothing can ever be moved.

There are workarounds of course -- if you write a program with heap-compaction in mind, then you can use a "handle" system, where every object has an ID. You remember the ID, and to access the object, you ask the system for a temporary memory location. And as soon as you're done, you "forget" the memory location and let the system shuffle things around in memory. The next time you give that ID to the system, you might get back a different memory location, but you were already expecting that so your program doesn't mind.

But handle allocation is slower, less efficient, and more annoying to use than a traditional fixed-location allocator. You have to start your project with it in mind; retrofitting existing code to use a handle allocator is a giant timesink and prone to conversion errors. And if you don't mind the loss of performance due to using a handle allocator, why are you using C++ in the first place?

Re:Total cached page limit.

[kjots]

I'd rather ad a second gigabyte of memory before I tried upgrading my CPU.

Please hand in your geek card. You just failed.

(For those of you playing at home, the correct answer is "I would rather add a second gigabyte of memory and upgrade my CPU")

Re:Total cached page limit.

[Tony Hoyle]

You use double-pointers rather than handles. No need to notify anyone.

I used to use this years ago on machines like the archimedes that had little memory. In a modern paged system it's a nearly useless technique - the most you'll lose is a single page even if there are large 'gaps' in the virtual address space.

Re:Total cached page limit.

[Anonymous Coward]

Programmers are then free to do all kinds of things like compute distance to other objects, or convert the location to a number and do arbitrary math operations on it.

Actually, neither of these things are guaranteed to work by the C++ standard. Implementations usually make it work, because it's more easy than not and occasionally convenient.

I think you could do a heap-compacting implementation of C++. Unlike conservative garbage collection, however, you can't do it in a library. Pointers would double in size and become handle-offset pairs, plus an entry in a map for each heap block. Every pointer access would also take a second indirection. As a side benefit, this makes garbage collection and bounds checking a little easier.

Things like taking the difference of pointers to two objects not in the same array would not work, but C++ doesn't guarantee it anyway. Converting pointers to ints and back wouldn't either, but same deal.

But the problem is, someone needs to implement it. And that someone needs to also have a compiler that they can modify in terrible, terrible ways. It also would help to be an expert in designing thread-safe, interruptable heap compaction algorithms.

Now, if you were in a position to do that, would you bolt this on to C++, or would you put that effort into a different language?

Re:Total cached page limit.

[maxwell demon]

C++ does not, because of a low-level language "feature": pointers, and specifically, pointer arithmetic.

Pointer arithmetic is irrelevant here, because it's undefined behaviour to move a pointer to outside the object through pointer arithmetic (exception: The one-past-end pointer, but that can be easily resolved by just allocating an extra byte at the end).
Now pointers per se are relevant, because they are usually implemented as direct address of the object it points to. There is nothing in the C++ standard which forces this implementation, but I guess you would upset most low-level developers (e.g. OS or embedded) if you used another implementation. Now the implementation as direct address is problematic because there's no way the compiler could know what is a pointer and what isn't (you could have a normal integer which just happens to match the address of an allocated block; changing that integer on moving the block would of course be fatal).

However, a standard conforming pointer implementation could also consist of an index into an address table and an offset relative to that address (after all, the x86 protected mode segmented address is exactly that, just at processor level). Since now each access would go through that address table, moving blocks of memory around would be quite easy: You only would have to change that one entry (Ok, in multithreaded programs it would be more complicated because another thread could currently use the address; probably one would have to scan the stored register values of all threads for values matching the address range of the block to move, and don't move the block if it is found). The entries of the table would probably have to be GCed (without compacting, of course).

Of course that would slow down all memory use, even for programs where this wouldn't have a benefit, so it's unlikely to ever get implemented.

[Manual implementation through handle]

And if you don't mind the loss of performance due to using a handle allocator, why are you using C++ in the first place?

Not every part of a program needs the same performance. Indeed, in the high-level parts of your code it is often viewed as error to use raw pointers at all. Also note that if the handle is implemented the same way as the hypothetical pointer impementation above, but without the offset part (because in high-level code you usually won't use pointer arithmetics), and implement the index into the table just as normal pointer to the table entry, the overhead is just an extra indirection (basically you use **p instead of *p), which for high-level code isn't much of a performance hit. Of course the compaction pass itself costs performance, but you wouldn't do that if you wouldn't consider it worth that performance hit (after all, since you manually implement it, you have complete control over when this compaction pass is done). For freeing the table entries, you have the choice of using any memory management method you want, from explicitly freeing by function call through reference counting up to garbage collection. Of course for all objects managed this way, you should make sure that you can cheaply relocate them.

Re:Total cached page limit.

[Edgewize]

I was not necessarily talking about Firefox, just addressing (hah) a misconception in the parent post: fragmentation is not related to garbage collection. I didn't want to get into a lengthy discussion of virtual addressing, page faults, etc. as well ;)

But you're correct. Thanks to the rise of "smart" binned allocators like dlmalloc, fragmentation is no longer the huge concern that it used to be with (for example) the basic Win32 heap API. Modern allocators are now reasonably smart about reusing best-fit blocks and not leaving tiny holes scattered in your VM pages.

Anyway, fragmentation is certainly not the primary cause of Firefox's high memory use. I am merely pointing out that my original post's parent had a poor conception of how fragmentation could be dealt with in.

Re:Total cached page limit.

[Edgewize]

A handle can be defined as a double pointer, and the locking operation could just be dereferencing it (assuming a single-threaded model of operation). I was speaking metaphorically about notification, not describing an actual event model ;)

The technique is still very useful if you have a program that follows one of a few very specific allocation patterns. For example, consider a program that allocates many small blocks at once and then frees a large (but non-consecutive) percentage. The ability to compact the heap lets you regain all the space you lose when your free holes are smaller than your VM page size.

Will most applications have this problem? Probably not. But the ones that do have it, tend to have it in spades. A handle-based allocation sytem can be a real win in those cases.

Re:Total cached page limit.

[Mistshadow2k4]

Thank you. My machine has 512 mb RAM (planning ot upgrade soon) and I multi-task a lot, so when I'm running FF I'm almost running 1 or 2 other programs at the same time. That's why it's very high use of memory becomes a problem. And with a fast connection like I have now, I can't tell that my page load performance is suffering at all.

But... what they should do is to put this in the regular optios menu instead of about:config. Lots of users don't even know to use about:config. I really like FF but the "we know better than you" attitude of the Moz team is starting to bug me. No, it's my machine and I know what I want to do with it, so I think I just might know better than they do.

Re:Total cached page limit.

[luckystuff]

wow. first time in a long while I have read something /.'d and put it to *immediate* use. congrats.

What a small world

[Rude Turnip]

Did the Mozilla Foundation hire the same PR firm that Microsoft uses?

My boss doesn't agree..

[lbrandy]

My bos doesn't agree at all. I tried including this feature in several of my builds. My company is so regressive, we have alot to learn from the leaders like Firefox.

spyware with IE or memory leak with FF..

[madnuke]

Hard choice but I'm seriously looking at Opera after seeing this I have a gig of RAM and its still laggy, I was wondering why the 'leak' was so high theres no way you could put that much bad programing to make a programe eat memory like a fat kid in a pie shop.

In other news...

[GillBates0]

...scientists have determined that the human appendix is not an evolutionary anomaly as previously thought, but an intelligent design feature aimed at keeping the humans guessing as to it's actual function.

And in totally unrelated news, the Mozilla foundation recently announced that their flagship browser Firefox shall soon be renamed to Bigfoot, to reflect the software's large memory footprint.

More breaking news on these topics at 11.

Doesn't seem to be true

[amliebsch]

If this is true, then why is so little memory freed after the tab is closed, compared with how much it consumed when it was created?

about:config

[__aaitqo8496]

how about a configurable option to not take up 200mb of ram? keep it as-is by default, but let power users toggle it off

So I'll be the first to say it....

[JordanL]

Why does Opera do the same thing faster without the memory penalties?

Re:So I'll be the first to say it....

[JordanL]

I suspect that some of it may be due to talent. Perhaps Opera programmers are just more talented on average than the typical FF dev. As well, I also suspect that its simple goal orientation. The Opera company works on building a new browser that's better with new features. The MozDevs collaborate on whose got which tickets and how they'll be integrated into the source and such.

This is proof positive, I think, that OSS != the best option in all scenarios. Opera consistently beats FF out on features, security, and speed... and it does it without having to download "extensions".

Re:So I'll be the first to say it....

[masklinn]

It doesn't. (or Process Explorer lies when it tells me that Opera is using 160Mb of ram after some surfing, which is roughtly what Firefox also maxes at)

Re:So I'll be the first to say it....

[diegocgteleline.es]

and it does it without having to download "extensions"

Well, you just picked up the worst reason - opera is great when it comes to performance, but firefox + extensions consistently beat opera and IE when it comes to features. I can have the features I want, there're way more extensions that features than opera has, and if I don't want them, I don't need to keep the extra UI involved in those features.

Re:So I'll be the first to say it....

[hkmwbz]

The extra features in Opera don't waste UI space either.

And guess what, Opera can be extended in many different ways:

http://virtuelvis.com/archives/2005/01/opera-and-f irefox-extensions
http://virtuelvis.com/archives/2005/09/opera-and-f irefox-extensions-ii

Re:So I'll be the first to say it....

[dzfoo]

>> Why does Opera do the same thing faster without the memory penalties?

Magic!

            -dZ.

Quick Fix

[I_am_Rambi]

about:config then search for browser.sessionhistory.max_total_viewers and set it to 0. This will be 0 pages in the cache per tab. You will get a reload slow down since FF will be going out to the web. You can manually set this to 2 or whatever you want. By default FF will cache upto 8 pages per tab with 1 gig of memory or more.

Re:My pet peeve!

[nxtw]

It's not the most ideal solution, but I can drag and drop the favicon (the icon in between the Home button and address in the address bar, with default toolbar settings) to my tabbar to effectively get a duplicate of the current page. (Tab Mix Plus might be the cause of this feature). I don't have a Firefox that isn't loaded with Tab Mix Plus around, but I don't think you need the extension to do this.

Tab Mix Plus also has an option to always open the current page in a new tab.

Re:My pet peeve!

[Rude Turnip]

"Why can't I hit ctrl-T and get a new tab with the same page I'm currently on, then hit reply and anything I want to quote I can just switch tabs instead of screwing around with back/forward and scrolling."

I just replied to your post using Firefox. I middle clicked on "Reply to This," which brought up your post appearing by itself in its own tab. I just copied and pasted some of your post into my reply, hit "Submit" and went on my merry way. Isn't that simple enough? Although I would like to see a /. equivalent of the Farkit extension.

Re:My pet peeve!

[TERdON]

At least in the OS X version, commandclicking (and probably middleclicking as well, but I haven't got a mouse connected) the back button solves this problem nicely. I would guess that middleclicking the back button works under other operating systems as well. So just click "reply" first, then middleclick the back button. :)

Re:My pet peeve!

[pete6677]

That would be a nice user-configurable option. As it is, I don't find it too much trouble to hit shift-tab a couple of times, ctrl-C, ctrl-T, ctrl-V, enter to get the same page in a new tab, but it would save a few steps to have this choice. But I definitely wouldn't want new tabs opening up with the current page and no option to turn this feature off.

NOT per tab

[savala]

Ben was mistaken, it's cached globally.

See this comment by Boriz Zbarsky:

Ben, those numbers are NOT per tab. The bfcache is global; there are never more than 8 pages total in bfcache (and you need to have 1GB of RAM for this to happen). Most users have 3 or 5 pages in bfcache at any given time.

and this comment by David Baron:

The point of bug 292965 was that the pref should be global, not per-tab. Is that not working correctly?

(Boris and David are back-end developers; they have much more working knowledge of this than Ben does.)

Also, there are actual memory leaks in Firefox. See this weblog post about progress on that. However, as that weblog post says as well, most excessive memory usage that people are seeing is entirely due to faulty extensions.

Re:My pet peeve!

[JuzzFunky]

Looks like you need the duplicate tab extension...
https://addons.mozilla.org/extensions/moreinfo.php ?id=28&application=firefox

Re:My pet peeve!

[Alystair]

I found that pressing Ctrl+Z after Ctrl+T brings up the URL from the last tab you were on. Now you just need to press Enter.

Re:My pet peeve!

[SaturdayNight]

This extension, tab mix plus, has that functionality and a lot more built in (Duplicate Tab). Not sure if you can make it a keyboard shortcut though. Very handy tool for me.

What I'd like Mozilla devs to do

[A beautiful mind]

Let them release 2.0, but then try to focus on:

• fixing the practically fixable bugs (not the design decisions)
• making code performance improvements (faster, with less memory!)
• security auditing

...for a half year or a year. I don't need new features, I'm currently happy with the ones I have and I'd prefer the current features working securely, in a speedy fashion and mostly without bugs. This time period would also give enough time for extensions to mature more.

Before someone jumps at my throat, it's just a description what I'd like to see, but of course its all up to the developers, they decide what to code and do with their time. It is just simple user feedback.

Firefox is the most unstable program in common use

[Futurepower(R)]

See Firefox is the most unstable program in common use.

The Firefox CPU hogging bug makes a computer unusable until all Firefox windows and tabs are closed. Basically, Firefox uses first maybe 10%, then maybe 20% of the CPU, and, as Firefox windows and tabs are opened and closed, continues taking more of the CPU time until Firefox is closed. This CPU usage is with NO Firefox activity, or any activity of any program.

This bug is more than 3 years old. It is extremely difficult to characterize; no one has succeeded yet. Here are some clues:

Somehow Thunderbird and Mozilla share this bug. Sometimes when Firefox is taking say, 94% of the CPU, and Firefox is closed completely, Thunderbird or Mozilla will begin using a lot of CPU time. Very weird, but it often happens.

Firefox 1.5.0.1 is much worse than 1.5, which is worse than earlier versions. This suggests that there is some resource in Firefox that is being more overused as features are added.

The CPU hogging bug continues unchanged when Firefox 1.5.0.1 is installed with a clean profile and no extensions.

Too many mouse clicks too closely spaced will often increase Firefox's CPU usage, or sometimes cause it to crash.

--
Before, Saddam got Iraq oil profits & paid part to kill Iraqis. Now a few Americans share Iraq oil profits, & U.S. citizens pay to kill Iraqis. Improvement?

seems snappier

[miyako]

firefox's memory usage has always been a thorn in my side. I tend to average around 20 to 25 tabs open, usually while I'm running other ram hungry applications. Firefox generally was eating up about 200-250 megs of ram on my machine (and I've seen it go as high as 600 megs). After changing the browser.sessionhistory.max_total_viewers to 0 and running "top" firefox seems to be using about 46 megs of ram right now. It also doesn't feel particularly slower than it did before. I have a feeling that the benefit of caching so much was actually having a negative return after a certain point because the machine was so starved for ram.
On a side note, if anyone is like me and looks in about:config for browser.sessionhistory.max_total_viewers and doesn't see it, you have to actually add the line. Right click and choose "new" then type in "browser.sessionhistory.max_total_viewers" and then 0 (or whatever you like).

Re:My pet peeve!

[HTH NE1]

I regularly use middle-click to open a link in a new tab in the first place. However, the Mac OS X version of Mozilla lacks this option, expecting me to configure my mouse to do a command-click on middle-click instead to get the same functionality I enjoy on Linux.

Usually the only time I use a browser under Windows is for Windows Update.

And just testing right now, middle-clicking on the Back button does nothing for me under Linux. It has a visual reaction but otherwise does nothing else. Maybe it is another one of those Firefox features not found in Mozilla?

releasing memory

[DreadSpoon]

The answer to that is pretty simple:

The heap, where dynamic allocations occur, is only allowed to grow or to be truncated. An application cannot release memory in the middle of the heap without also releasing the memory at the end of the heap.

So let's say Firefox makes 10 one-page allocations, and frees the first 9. The memory layout might look something like:
XXXXXXXXXU (X- unused, U- used)

Those 9 pages worth of memory aren't being used, but it's impossible to release them back to the OS.

Thankfully, there is some good news: when Firefox needs to allocate more memory, it can and will just reuse those 9 unused pages instead of allocating more memory from the OS and growing the heap.

The best solution to this problem is to use a compacting garbage collector. Which is something that Java and C# and other higher-level langauges can easily make use of (and many do use them), but which C and C++ can't really make use of given the complete lack of compiler support. That's one reason why a Java or C# app can actually out-perform a similar C/C++ app, especially with a good native-code compiler and an library implementation with a modern GC.

Re:releasing memory

[Anonymous Coward]

An application cannot release memory in the middle of the heap without also releasing the memory at the end of the heap.

That's not strictly true. If the hole is larger than 4k (or 8 if it's not page-aligned), there are two things that can be done.

You can decommit the page, but keep it reserved. This frees RAM, decreasing the process's memory usage, but still takes up some of its address space. You MUST coalesce free heap blocks in this case, because all data in those pages is lost. This also requires extra housekeeping.

Or you can keep it committed, and the OS will push that memory out to the page file. This is done pretty much automatically, but it's more lazy. In Windows, you can somewhat force this by minimizing all windows owned by the process.

The best solution to this problem is to use a compacting garbage collector.

Unless you've done actual research, I wouldn't be so quick to say that. It's certainly the easiest solution if you're using a language that doesn't use real pointers, but there are numerous possible solutions.

Eg, you could allocate pages for large objects like this directly from the OS rather than using a heap. Then freeing objects in the middle is no problem at all.

Re:releasing memory

[mrsbrisby]

You can decommit the page, but keep it reserved. This frees RAM, decreasing the process's memory usage, but still takes up some of its address space. You MUST coalesce free heap blocks in this case, because all data in those pages is lost. This also requires extra housekeeping.

mmap() can do this, but on many systems [s]brk() cannot. brk() is also alot faster than mmap().

This is really moot on most systems; don't do a lot of little allocations that you're going to keep around for a while and DO use pooled allocators that can use mmap(). It requires planning, but it really does pay off for applications that need an awful lot of memory in stages.

This is (by the way), why many C compilers are implemented in "stages", so that they don't have to worry about crap like this. Allocate as needed, and the next stage exec() will automatically compact and garbage collect any pages used in the previous stage that aren't needed here (that weren't passed to the next process using mmap or pipes).

Re:releasing memory

[Bloater]

> Wouldn't all pointer references then have to go through some kind of lookup table, so that the objects could be relocated by the runtime without breaking them?

Only on a computer without virtual memory. In a PC (which *has* virtual memory), you just punch holes in the memory.

What happens is a process gets an "address space", into which pointers can point, but any given address may not map onto some real storage. The process asks the operating system to map a range of addresses onto real storage which the operating system will try to map to real fast memory when it thinks it will be used at any moment. When the OS figures the memory wont be needed for a while, and something else needs some memory, the OS copies the data to disk and redirects the mapping to a proxy that will pull the data back into memory when the process tries to use it again.

When a process knows that it won't need a section of that real storage, it can tell the operating system to unmap it from the address space.

There are various other things that go on, but that's the simple story. From a figure posted in an earlier message, it seems that opera does pretty damned well (in comparison to most modern programs) with just the simple story, not having to rely much on nasty unreliable heuristics. Of that I am impressed.

Re:releasing memory

[xenocide2]

That's one way, but you'd have to somehow instruct the compiler to use that for every pointer dereference. The easier method is to go in and change the values during compaction. Compaction is also known as stop-and-copy; it starts with a live set, everything you can reference from the stack, then copies over only the live objects while modifiying every pointer that uses it. It's messy but it works. Allocation is dead simple and fast. There's no fragmentation. And the runtime is limited by the live set rather than the heap size. There is a huge downside, however.

I wouldn't recommend mixing anything resembling C pointer maths with compaction, since its incredibly difficult to tell what's a pointer and what isn't (in fact, without modifying the compiler, it can't be done in C or C++). For this reason, the Boehm collector (a collector that replaces new and delete) goes for the Mark-and-Sweep method instead of compaction. Because you dont move objects, you don't have to worry about figuring pointers. Boehm's collector is also called conservative, not only because it doesn't modify live objects, but also in that it treats any data on the stack or in the heap as a potential pointer. If the data points inside the heap, the object containing that address is marked. This can lead to false positives on occasion, but there's no helping that without any support from the compiler (again contradicting the grandparent). The good news is that a false positive isn't going to cause direct harm in mark and sweep. All that happens is that space that could be used isn't; Boehm claims this is irrelevant in today's operating systems with virtual memory, although I doubt you'd see an entire page's worth of false positives. Certainly, I can't do any better than him.

In language R&D labs where people are paid quite well to think hard and long about things, they tend to use both approaches in what's called a "generational" collector. Young objects can be copied or collected as needed, while older objects are mark/swept away as needed. This works because old objects much more likely to stay than new ones. Last I knew, both Java and C# use generational techniques, because it makes sense in most nearly every case. However, as I described above, C++ doesn't have that, and even those libraries that replace new and delete have conventions and costs associated with it. I certainly wouldn't try to take Boehm and pidgeonhole it into Mozilla. And even if you did, it still wouldn't solve the compaction problem. All you can do is hope the virtual memory manager is doing it's job well. Even though the application and garbage collector is more likely to know what's useful than the VM manager.

Re:releasing memory

[Klootzak]

Excellent post, well worth the +5.

But you pointed out the flaw in the wording of the article - this IS NOT a memory leak, just inefficient use of the heap.

I thought the definition of a memory leak was an application that kept allocating memory from the OS as it ran, not an application that asked for a chunk of memory and just reused it inefficiently?
(If I'm wrong, someone please correct me).

Re:releasing memory

[angel'o'sphere]

I wouldn't know about C, but this statement is utterly false as applied to C++.
No, its not false, its true.

Replacing the default new and delete routines is perhaps not for the inexperienced C++ programmer, but to say that there's an complete lack of compiler support is simply wrong.

And? What do you mean with compiler support for heap compacting (or GC)?

Q: What has replacing new and delete with your own implementations to do with garbage collection?
A: Nothing

Q: How would new and delete of class A be able to compact a heap by moving allocated instances of class B down?
A: Difficult!

Q: So if you now add a class C you like to rewrite A::new and B::delete to also cope with class C instances?
A: I assume you understand that EVERY delete of EVERY class needs to know EVERY other class to be able to compact the heap, yes?

It is true that out-of-the-box C++ does not have a compacting garbage collecter, but one can certainly be written (and used, of course) with any conformant compiler.

Indeed, but not by merly only by replacing operator new and delete.
Existing C++ garbage collectors are very limited to more or less conservative garbage collecting. See e.g. Boehms c++ / C garbage collector.
And, the mere point of garbage collecting if you want to start nitpicking is: you don't ever call delete.

angel'o'sphere

Re:releasing memory

[Profound]

>> That's one reason why a Java or C# app can actually out-perform a similar C/C++ app

Maybe in theory, but in practice 99.9% of the world uses C++ browsers because the Java ones suck.

Re:releasing memory

[julesh]

The good news is that a false positive isn't going to cause direct harm in mark and sweep. All that happens is that space that could be used isn't

Or an allocated page leaks. While the collector's "blacklisting" approach minimises the amount of this that happens, it does still happen from time to time.

Boehm claims this is irrelevant in today's operating systems with virtual memory, although I doubt you'd see an entire page's worth of false positives.

I'm not entirely sure I agree with him. Consider, for example, applications that store large arrays of seemingly random data on the stack (e.g. compressed data). As the amount of this data tends towards (2^32/PAGESIZE)*4 bytes (= 2^24 bytes, or 16Mb), the number of pages blacklisted should approach half of available virtual memory, reducing to 2Gb the memory available for the application on a 32 bit address system (i.e. todays standard). 2Gb is a lot, but many applications are starting to get close to that. And doubling the amount of randomised data on the stack will halve it, I think.

Boehm does most of his work on 64 bit platforms these days. I'm not surprised that he's unconcerned about virtual address space usage.

That said, it's very simple to avoid this pitfall: use GC_MALLOC_ATOMIC to allocate any space that will store apparently-random data. You just have to understand how the garbage collector works, and code around it, to avoid such problems. I guess we're looking at a leaky abstraction here, as always.

Huh?

[countach]

Uh, using a lot of memory is not the same as a memory leak.

Don't bug me

[joeytsai]

I think this submission is confusing two points. First of all, is this really a memory leak? A program that uses a lot of memory is not necessarily a leaking program. A memory leak is a programmatic error where memory is allocated but never freed, even when there's no way to use that object again. As the program continues to allocate memory, the heap size of the process increases until eventually the OS terminates the process (eg., the OOMKiller). Actually, many applications you normally use leak memory - but as long as they don't waste a ridiculous amount of memory most people don't care, especially since most process lifetimes are relatively short (compared to a daemon process like apache), and after termination the OS reclaims all the program's memory, leaked or not.

What is being described here sounds much more like a cache of recent pages, which in my opinion is perfectly sane for a browser. Sure, maybe the cache is a bit overzealous, but even if that's the case, just disable it - worse case scenario, you edit the source. But otherwise, this is definitely a feature - I can promise you it's much more programming effort to save old pages for a quick redraw than to free the old page and replace it with the new.

So I guess the discussion here is, "is it right for firefox to use so much memory?" My answer is yes. It is not a memory leak, it seems like a very valid design decision. But if you disagree, old versions of firefox still work great (I still haven't upgraded myself).

Re:My pet peeve!

[emmetropia]

For the sake of knowledge, you *don't* need the extension.

Why does Opera work well, and not Firefox?

[Futurepower(R)]

More clues:

• Opera has none of these problems. So, the quote from the Mozillazine blog shown below, although it is typical, is not supported by the facts.
  
  
• Whatever causes the CPU hogging bug is definitely associated with extreme memory use. No doubt there are leaks, but this is not a leak, since it is not necessarily associated with greater use of Firefox.
  
  
• Users often report that just leaving Firefox open overnight causes CPU hogging and extreme memory use.
  
  
• The problems are the same in Mozilla browser.
  
  
• It's good to test Firefox with a laptop in a quiet environment. When you hear the laptop fan begin to run while there is no activity, you know Firefox has begun to suck CPU cycles.
  
  
• Putting a computer into standby or hibernation often makes the CPU hogging bug much worse. That's why Firefox users sometimes just leave their computers on.
  
  
• When a computer takes a long, long time to start from standby, you know Firefox is taking CPU cycles. What about coming out of standby makes Firefox unstable? No other program has that problem.
  
  

Quote from the blog linked in this Slashdot story About the Firefox "memory leak": "A lot of people complain about the Firefox "memory leak(s)". All versions of Firefox no doubt leak memory - it is a common problem with software this complicated."

No other program in common use is so buggy. The problems in Firefox are not "common".

Another quote from the linked Mozillazine blog: "What I think many people are talking about however with Firefox 1.5 is not really a memory leak at all. It is in fact a feature."

That's not what the technical magazines, newsletters, web sites entirely devoted to Firefox problems, and even the mainstream media say. They say it is a serious problem.

Mozilla developers have been denying that there is a serious problem for more than 3 years. It seems that it would be less work to fix the problem than to undertake a cottage industry of trying to convince people they aren't having problems. Mozilla developers have been impeding characterization by marking Bugzilla bug reports of these problems invalid.

However, it is clear that it would take a serious scientific investigation; this is not an easy bug to characterize.

Re:Why does Opera work well, and not Firefox?

[geekoid]

I am using:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12)

And I can not duplicate any of those issues.
When clicking forwadr and back buttons quickyl, I manged to spike at 60% for a brief moment.
I have ahd it running all day.
I do not use the feature that lets some of it stay resident so opening it up is quicker, so maybe the problem is there.
Any clues on ther things I can try to duplicate this issue?

to address this specific issue, more memory does not equal memory leak. Yes, the cahching mechanizim may be too agressive, but if it ahs been designed that way, then it is, in fact, a feature.
Some other poster suggest setting changes that seem to take care of this issue.

Clone Window

[rakslice]

The Clone Window extension.

Look it up, dingus. There's no reason that every web browser should behave exactly like IE out of the box. That's what the extension feature is for. =)

Re:A limitation of the C library

[tepples]

memory is released upon calling free()

Too many C libraries' implementation of free() release memory back to the application, not to the operating system.

There is another serious Firefox bug as well

[cecom]

Firefox crashes when two browser windows are making synchronous XMLHttpRequests. I have experienced this under Linux - I have no idea whether it is the same under Windows. Basically under Lunux all Firefox windows are running in the same thread utilizing a scheme of cooperative multitasking.

So far so good. The bug appears when two separate Firefox windows are making periodic synchronous XMLHttpRequest-s. When such a potentially lengthy task has to be executed synchronously, Firefox creates a new "nested" event queue. If two (or more) browser windows are doing it at the same time, new event queues are created all the time and eventually (within 5 minutes) the application core-dumps.

I found this by recompiliging Firefox with debug information and debugging it. Even if my interpretation of what happens is not completely correct, the fact remains - a simple JavaScript can crash Firefox causing all open browse windows to be closed.

The solution is to always use asynchronous XMLHttpRequest (which is a better practice anyway) and to hope that the same problem doesn't appear in other places. Still, it is troublesome.

two words: "totally" "wrong"

[Mini-Geek]

The person that sent in this article is mistaken. In Firefox 1.0 the Memory Leak still existed without caching previous pages completely as is currently done. Do I need to say anything else? It may be possible that the new caching system worsens the problem slightly, but it's not the cause of it.

Re:spyware with IE or memory leak with FF..

[crabpeople]

How can you trust the CEO after he didnt swim across the ocean?

i could never stand behind a company like that and refuse to use opera products untill he makes good on his word. You cant just throw statements like that around. Browsers designed by liars are dishonorable browsers.

Re:Firefox is the most unstable program in common

[crabpeople]

firefox currently sitting at 0% cpu usage. perhaps you should upgrade your pentium 133 :)\

seriously, the only thing i could think of is that if firefox ran out of ram and had to start using the pagefile, that would eat up tonnes of CPU. This would also effect other programs on the system. Are you sure you have enough ram in the machine? I assume you can replicate this bug on more than one machine right?

Ive had some sites crash firefox repeatidly but i cant think of any examples off hand.

Firefox developers don't "get it"

[Vellmont]

Memory isn't an unlimited resource you just hoard whenever you think you need it. Right now my instance of firefox is taking up 128 megs! I've seen it up to 256 megs before. This is just simply insane. I've seen people who's computer performance has gone down the tubes because firefox is taking up all the memory (and these are machines with 512 megs of memory, not exactly tiny). What I'd like to convey to the firefox devs is this: Your application isn't the only one running on the system. Play nice and don't be a hog.

With the number of people complaining about this (and the number of people that don't even KNOW to complain) isn't it a safe bet that you've made a mistake in the amount of cached pages?

Re:Firefox developers don't "get it"

[Nimey]

You could always close the browser when you're not using it. I close any non-daemon I'm not using, and it makes my system more responsive.

Re:sounds good to me

[jlarocco]

Why is everyone bitching about this? I hate waiting for any refetch or rerendering when I use the Back button; I want it to be instantaneous. That page was fetched and rendered aslready, so having the browser keep it around for when I go back to it is exactly what I'd want it to do.

You've totally missed the point. People aren't bitching because the back and forward buttons are faster. They're bitching because the memory used for the fast back/forward is never released. Because Opera implements the same feature even faster and doesn't use >85% of physical memory after 20 minutes. Because a web browser should not use >1 GB of RAM because it's left open over night.

Re:My pet peeve!

[RalphSleigh]

Firefox 1.5.0.1 under windoze, middle clicking the back button brings up the page in a new tab, and I dont have any tab related extensions installed.

Simple fix

[Cryolithic]

Go to about:config Create a new boolean name it config.trim_on_minimize set it to true Next time firefox is using too much ram, quickly minimize it, then bring it back up. Done.

Doesn't this just seem silly?

[robertjw]

The cache feature is nice, but why distribute it out to every tab? If I have 20 tabs open I'm not going to be constantly clicking the back button on each of them. Why not clear the cache on tabs that haven't been accessed recently and only keep cache on tabs actively being used. Often when I open new tabs I just want to be able to quickly access that page, or use it as a temporary bookmark - not navigate back through the path that got me there.

Re:bfd?

[Sigma 7]

Virtual memory, look into it!

Virtual memory is not a carte blanche for memory hogging. As you should know, memory hogging will result in degraded performance.

Assuming that users have unlimited resources is exactly how Mozilla is barely usable on Windows 95-ME - especially when you have Slashdot Moderator access.

Who cares if Firefox or any app is a bit of a memory hog???

As long as it is just using the memory as cache space and not accessing the memory randomly, it'll be paged out into virtual memory as needed.

In an ideal situation, that would be correct.

However, the operating system does not know which memory is currenly "in use" and which ones are "in cache" - in fact, it's quite easy for an "in use" to be physically sandwiched between two "in cache" entries. Because of this, you will have a sudden loading time if you do plenty of other tasks in the background and suddenly switch back to Mozilla.

Small applications, being small, do not generally have to wait 1/2 seconds to recover from being pages in or out. Since Mozilla allocates the cache in memory, it will have to wait those two seconds.

assuming you're using an OS with decent vmem support.

An OS with decent vmem support would allow you to map files to memory. This results in no swapping at all - only writing perodic output to the hard drive, and loading the file into memory as required. If another application needs more memory, the memory map is discarded with no need to write the contents of memory.

An application that doesn't exploit the usage of memory maps is as good as an OS with shoddy vmem support. (Of course, it can simply use it's disk cache for the same effect.)

Re:Obviously there is a memory penalty with Opera

[dreemernj]

Opera uses disk cache and ram cache. It uses as much ram cache as it can for the speed benefit, and on older systems with less memory it tones down the ram cache usage to avoid stepping on toes. In any case how much disk and ram cache it uses can both be controlled manually from the preferences. Ram is set to auto by default meaning it will take a percentage of the free memory for itself. On my 1gb system it tends to use about 90-120megs of ram if I've been browsing for a while, but it does keep a cap even on that system so it never gets so large that it gets bogged down. Then on systems like the one I am on right now that has 128mb ram, it relies mostly on disk cache and doesn't really suffer any hits in performance for it.

Cluecheck!

[Burz]

I just loaded FF 1.5.0.1 and Opera 8.5 on Mac OSX 10.3.9 (iBook G4 1.2GHz 768M) each with identical nine tabs:

Firefox: 54.15M (Real) 190.07M (VM) ; 2.1% idle

Opera : 59.36M (Real) 239.66M (VM) ; 0.4% idle

Assessment: This Firefox outperforms both Opera and Safari in memory usage, and is faster than Opera on challenging pages. However it has the least favorable idling habits, starting at 2% here and would climb to 4% after several days of intensive use. FF 1.5.0.1 memory use would climb to about 100M for the same pages over the same period, indicating the cache grows somewhat but not wildly the way FF 1.5 did.

The test of also rather unfair, as I have 7 FF extensions running.

R-RTFA

[Chris Snook]

The article has been corrected. Note that the maximum number of cached pages, regardless of the number of tabs, defaults to 8, and that's only if you have at least 1 GB of RAM. RTFSC:

http://lxr.mozilla.org/seamonkey/source/docshell/s history/src/nsSHistory.cpp#161

If you're unhappy with the memory usage with 50 tabs open, I advise the following workaround:

DON'T DO THAT.

Rewritting history.

[SeaFox]

Firefox 1.5 implements a Back-Forward cache that retains the rendered document for the last five session history entries for each tab. This is a lot of data. If you have a lot of tabs, Firefox's memory usage can climb dramatically. It's a trade-off. What you get out of it is faster performance as you navigate the web.

The only problem is there were bugs filed for memory leaks long before Firefox 1.5 and the Back-Forward cache were implemented. Maybe this feature does contribute to Firefox's large memory footprint, but to say that this feature is the only reason and that there are no leaks is simply false.

POST data?

[Quixote]

If Firefox is caching these pages, why doesn't it cache POST results? When I hit back to go back to a page obtained via POST, FF refuses to show it to me, asking me to either cancel the action or resubmit the form. JUST SHOW ME THE GODDAMMN PAGE, DAMMIT!. Once the page lands in my machine, regardless of how I obtained it (i.e. via GET, POST or whatever), then just show it, or at the very least give me the option of seeing the possibly expired page. Let it be my decision.

Comment removed

[account_deleted]

Comment removed based on user account deletion

This explanation makes no sense

[Siguy]

How does this explain why firefox eats more and more and more memory when left alone.

If I leave my computer on and come back the next morning, with no new pages having been loaded, the browser is suddenly taking up like 400 megs of RAM.

That can't be explained by caching tabs.

I'm definitely insane

[try_anything]

Every time I close all the tabs in my browser session except two or three, then check a few hours later and see that Firefox is sluggish and hogging a few hundred megabytes, I go to the police and ask them to take me into protective custody. I'm obviously a danger to myself and others. When I'm not responsible enough to seek psychiatric help, I just stare at my monitor and tell myself, "You only see three tabs there, but that's because you're crazy. You still have all those dozens of porn tabs open. You just can't see them because you went blind masturbating."

Seriously, what's with all the song and dance? Firefox obviously has at least one problem, probably several, that leads to bad performance for many users, under certain circumstances. Call it a UI problem, call it a documentation problem, I don't care, just call it a problem. Don't call it a feature or a misunderstanding. Don't pick a feature that can't account for many of the reported problems and say, "Aha! This is THE Firefox memory leak that's bothering everyone. See? It's a feature!" The denials and talk-arounds on this issue are what you would expect from a political party, not an open-source software project.

Of course, I only know all this because I use Firefox. It's the best. The memory problems would only be a minor annoyance if I didn't have to constantly read about how I'm crazy or stupid.

Re:Firefox performance slowed to a crawl

[cruachan]

Give up and use Opera. Firefox is profoundly broken in combination with (a) memory leak being discussed and (b) memory leaks in plugins. The second seems to even include Flash, where some flash pages appear to be cache in active state and sit there using CPU cycles as well as memory.

I think we've no got to the state where Firefox can be seen as a nice try, but no cigar. Opera on the other hand just works - and increadibly it's quick and lean too.

I've no connection with Opera, just like many I've been through the "Dump IE, Use Firefox, Think Firefox is wonderful, Find Firefox's dreadful memory/cpu cycle leaks, Dump Firefox" cycle!

You call it a bug we call it a feature

[Arimus]

First off I like Firefox and its my primary browser of choice. HOWEVER this calling a bug a feature doesn't half remind me of a certain other company I could mention (and several Dilbert cartoons).

If this feature is for my benifit then let me decide whether to use it or not. Apart from that it does not explain why when I leave firefox idle with only one window open on a simple HTML page over time my memory useage goes up...

Stop hiding behind feable excuses and actually work on reducing the footprint firefox uses... FF is suposed to be a lightweight browser alternative to the usual browser bloatware - it is failing at the moment (rather like my spelling ;) ).

Are you confused or stupid?

[Anonymous Coward]

Using a lot of memory is not a memory *leak*. A memory leak is where memory is allocated but the pointer to it is lost so the memory doesn't get freed.

Fasterfox's Clear Cache function helps

[klui]

I find that if I set browser.sessionhistory.max_total_viewers to 2 it uses less memory if it's left at the default (I have 1GB RAM). If I use Fasterfox's Clear Cache function, it compacts the memory usage even more. There is still some leakage but not as bad as the default -1.

It may be a feature, but it's still undesirable

[OneSmartFellow]

I find firefox to be one of the worst browsers - for my use anyway - available. I have tried to use it over an X session, and basically gave up; and that was with my X client running on a P4 1.5 GHz 500M RAM linux server, and my X server (running on my client machine accessing the server) a similar spec machine. The connection was wired 100MB ethernet.

Considering the hooplah that goes along with it, firefox underperforms on basic tasks when compared to KDE's Konqueror.

What's worse is that this command firefox ./index.html tries to open http://index.html/ rather than file://index.html. Meanwhile Konqueror behaves correctly.

Competition

[Britz]

First everyone complains that they should be as fast as possible to compete with IE, which is pretty fast.

When they do it the get slapped for using too much resources?

But I see a good point. I also would like to see new developement halted for some time to catch bugs and security problems. This would also help plugin developers to catch up. New features could be developed in plugins anyways.

Per Session, not per Tab

[dzfoo]

Its a non-issue. As explained on a note at the end of the article, its a per session setting, not per tab, so the entire article misrepresented the "feature".

"Edit: In the comments, Boris and David pointed out that I misread the code, and that this is a global preference so that there are no more than 8 cached pages for the entire session, not per tab. My initial posting had claimed that it was per-tab. Oops!"

If Firefox has memory leaks (and I think it does), this is not what is causing it. If it were, however, per tab, as the article originally claimed, then it would have been a problem, because the more tabs you open, the memory usage increases at an alarming rate, if it has to keep up to 8 history pages cached.

Nothing to see here. Move along.

          -dZ.

Fixed in Firefox 1.5

[Kelson]

At least, it has been according to the unofficial 1.5 changelog. The list of Mac-specific bugs fixed includes:

151249 - [Mac] Middle click on link does nothing on Mac OS X (should open link in new tab).