Slashdot Mirror
Mozilla MemShrink Set To Fix Firefox Memory
darthcamaro writes "If you're like a lot of Firefox 4 users out there, you've probably noticed that Firefox has a serious memory problem — it uses more than it really should. At long last, Mozilla developers are finally set to take this issue seriously with a dedicated team called MemShrink that are focused on the problem. 'It's pretty clear by now that this is a much bigger problem than any one person can likely tackle,' Mozilla Developer Johnny Stenback said."
64 bits is only 8 bytes. An unsigned 64-bit number can take Firefox all the way up to version 18,446,744,073,709,551,615.
So that should take them at least until through next Thursday.
There's no -1 for "I don't get it."
A lot of it's got to be the increasing size of web pages in general. Now that most folks have higher bandwidth connections, web designers don't focus on keeping the download size small.
Multiply that increase by the size of your cache (how many times can you click "back" without hitting the disk?) and you can see the full scope of the problem.
There's no -1 for "I don't get it."
My experience has been that both leaks and overall memory use have gone down between 3.0 and 4.0.
At the moment, Firefox is at about 375 megabytes, with 16 tabs open. It has been open for 3 weeks. I do have browser.sessionhistory.max_total_viewers set to 3 and the anti-malware databases disabled though.
Nerd rage is the funniest rage.
A useful new feature in latest Nightly versions from Mozilla is about:memory. It gives you a full tree view of where the memory is being used. Of my 360MB which the browser is currently using, 101MB is JS, 46MB is storage (you back button and memory cache), 70MB on "heap-unclassified" whatever that is. JS seems to be the biggest consumer of memory.
OS X here -- I usually have 8 to 12 tabs open. I almost never see memory usage below 500M, and it usually grows to about 1G after 2 or 3 hours. Firefox 4 for the Mac is seriously broken w.r.t. memory usage IMHO, and if they can't fix it fast, I'll probably be switching. I clobbers the performance of the whole system when it hogs that much memory. I'm tired of having to restart FF all the time.
Yay, it only took 5 years of bitching for them to actually look into it instead of blaming addons or your profile.
Your hair look like poop, Bob! - Wanker.
I hope this isn't just targeted towards firefox. Thunderbird is an unwieldy beast of an email app as well. No good reason that checking my email should involve consuming 200Mb of memory.
ôó
If it were possible for programs to allocate caches that work like the filesystem cache, where old items get discarded automatically to make room for anything more important, then this would make sense. But in real life, when a program written with that "unused memory is wasted memory" philosophy has filled up RAM and you start another program, the first program will have to go to the swapfile. Return to it later and it'll take forever to become usable again, while it gets re-loaded 4kB at a time. (I'll usually just kill the firefox.exe process and restart it when this happens, because that's actually faster)
as this amount decreases Firefox can begin to proactively free memory used for the oldest cached data.
So your suggestion is that each application running should be fine allocating a huge cache and changing its memory footprint according to how much memory it sees available on the entire system, instead of the OS making a decision?
I am not sure I am convinced that the outcome of that methodology is optimal. If it were; I think i'd favor "number of pages swapped in/out per second" over amount of memory free, though.
I am trying to imagine the interactions of 4 or 5 different applications all running with a huge cache, and the same behavior.... when memory usage is low, all 5 applications prepare a huge cache -- their huge cache causes the total memory free to drop, eventually to 1MB... now, suddenly, all 5 applications will use a bit of CPU time proactively freeing up large swaths of cache -- CPU will be 100% running for a couple seconds, as processes adjust their memory.
After all 5 apps reduce their huge caches, suddenly there will now be a lot of memory free --- so much memory free, that one or more of the applications might immediately see an opportunity for increased caching.
So what mechanism will protect fairness? Each application will believe its cache is important, but who's to say one of the applications isn't more important to the user, or having more requests more frequently made of it (so that the user's performance will best if application X's cache is bigger versus application Y).
There seems a fundamental weakness here, involving each application trying to make their own memory management decisions about cache --- the application making the decision to expand its cache may be the one the user cares about the least, and the one whose cache is the least useful.
The OS is in a position to make decisions and mediate in regards to the working set needs of processes, and the user's actual usage patterns. The OS knows which running process makes the most demand of its cache -- the other processes don't know much about each other.
Using ram is 'good' and 'superfetch' are good too huh?
How did that info get into memory. Oh thats right by me waiting for it to load from either the internet or from disk.
Back when I used vista turning superfetch off was the #1 way to make the computer usable. Superfetch was too aggressive in loading things. Turn computer on wait 15 mins (no kidding) before I could use the start button. It was even so aggressive it would keep memory full at all times. Meaning as programs freed and used memory the disk was thrashing ALL the time. It was noticeably faster and measurable too (I measured it many times thinking guys like you must be right). Win 7 is a whole different story (it actually works).
As for using too much memory. Using less does mean better performance. *IF* done right. But how much of that information in this case is just 'leaks'? A leak is never going to be used. Meaning windows needs to keep track of it and page it out a some point (meaning less performance at some point).
To bury your head in the sand and say using memory is bad is stupid. But so is using it all up is good. But think about this I have open 1 tab right now to type this in. It is using 250 meg (and that is just what is paged in meaning something in the program touched it). WTF is it doing... That is a seriously crazy amount of memory to be using to have this page open which about 30k. That is some serious overhead there. From this old 80s/90s programmer I look at that and think wtf dudes. You KNOW you can do better than that.
The system you describe is called malloc()!
In a system with a unified buffer cache (essentially, every OS in wide use except OpenBSD), it makes little difference whether a page of memory comes from a private memory allocation (e.g., a heap allocation), a memory-mapped file, or the OS's disk cache. When a process needs a page not already present in memory, the kernel's memory manager tries to find an unused page. If one is available, it hands it to the program that requested memory.
Otherwise, it looks for an in-use page, saves its contents, and hands the just-freed page to the program requesting memory. If that page is "dirty" --- i.e., it's backed by a file and somebody's written to that part of the file, or it's a private allocation backed by the page file --- the memory manager writes the page out to disk first. If the page isn't dirty, the memory manager can just discard its contents because it knows it can reconstruct it by reading back the original file.
When the memory manager has to go to disk to satisfy a request for a new page, it's called a hard fault. The mission of the memory manager is to reduce the number of hard faults, because hard faults are slow. The fewer hard faults you have, the less time will be spent waiting for the disk, and the faster your system will run.
The most important part of the memory manager is page replacement: i.e., how the memory chooses what page to evict in order to satisfy a memory allocation request. Most systems use an approximation of LRU (i.e., least recently used), throwing out pages that haven't been accessed in a while. It doesn't usually matter where a page came from. The only important factor is how recently it was accessed.
So, you can see that there's no difference between a program mapping a file into memory and modifying it, reading and writing it using file APIs, and just manipulating an equal amount of information in buffers created with malloc. To the kernel, all memory is made up of pages.
The "go away for a while" problem isn't caused by any particular memory strategy. It's an artifact of the memory manager's LRU approach. How does it know that the pages corresponding to Firefox are going to be used again? If some other program needs those pages, the older ones will be discarded. There is nothing applications can do.
Instead, the OS itself has to be tweaked to preserve interactivity. Sometimes the memory manager will prefer disk cache pages to malloc-backed ones. Sometimes (e.g., for Windows SuperFetch) the OS will try to identify pages belonging to activate applications and try harder to keep those in memory. Some systems favor keeping executable pages over private allocations. You can tweak the page replacement algorithm, but the basic idea, that all memory is made up of pages subject to the same management scheme, applies.
Ultimately, it's ridiculous to hear people talk about programs "keeping things in memory" like we were still dealing with DOS 6 and OS 9. The actual situation is a lot more subtle, and silly memory counters don't even come close to giving you a good picture of what's actually going on.
In short, don't worry about fine-tuning what's "in memory". Don't change behavior based on total amount of memory in the system. Operating systems (OpenBSD aside) ALREADY DO THAT. Just let the memory manager do its job, and give it enough information (via interactivity information, memory priority, etc.) to do its job properly. Don't try to hack around problems at the wrong layers.
Ah, the sounds of someone spewing the 80's virtual memory rhetoric.
There's more to it then that. I could go into it, but I'm supposed to be studying for a psychology exam, so I'll be brief.
You assume that the OS will make sensible paging decisions. You assume you can hint to the OS that you're going to make sensible paging decisions. You hope the application, which is likely big, multithreaded and such, is doing the sensible thing of not wrapping large accesses to "memory things" (eg big trees of data, as an example, or image caches, or whatever takes up more than a small bit of RAM) in mutexes. You assume that your application is using memory in a sensible fashion, and not simply using a few bytes here and there in each allocated chunk.
The trouble is, application writers have been taught from an "early" age that hey, memory is cheap, the OS will handle paging out unused bits, so please go right ahead and use it without caring about how it's used. This is how you end up with application behaviours which include, but aren't limited to:
* walking a tree requires a page in (ie, a random disk read) for each tree node touched. Because each node is malloc()'ed, and although on modern implementations small objects are packed into pages, your 10,000 tree node is going to likely be spread across multiple pages based on when and how often you allocated them ('temporal location');
* this also means your memory use versus memory allocation isn't terribly efficient ('fragmentation');
* your mutex protected data structures are suddenly now mutex'ing _page disk access_, so whilst the OS is busy paging in your data, all other threads currently trying to do stuff that requires that mutex (which may even not require paging in) suddenly has to stop and wait for your page-in to complete.
It's a real shame that memory management has really stopped progressing since virtual memory systems were made. They're convenient, but they hide the worst case behaviours from unknowing coders. Then those worse case behaviours become _very_ worse case behaviours which can't be changed without a fundamental rearchitecture of your software. Likely what people are realising here.
Enjoy!
it's not designed to be open for a week straight.
Then it needs to be redesigned.
I do not need a server to become sluggish because a user left on vacation with Firefox still running with a detached display.
And I do not want to have to close my munin/bigbrother/mrtg windows running on the wall monitor. It should keep running.
Remember how we ridiculed Microsoft for Windows 95 not being able to run more than a few days without a reboot? Well, it's time we laugh at the idiotic Firefox devs who repeat the same mistakes and invalid assumptions.
Because of the severe bloat and WTF assumptions made by the newer generation of Firefox devs, I've started using other browsers more and more. Like Midori ("it's not easy being green").