Slashdot Mirror
Firefox Too Big To Link On 32-bit Windows
An anonymous reader writes "Firefox has gotten so large that it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB. This problem had happened last year with 2 GB, which was worked around by adding a/3GB switch to the Windows build servers. Now the problem is back, and things aren't quite that simple anymore."
This only affects the inbound branch, but from the looks of it new code is no longer being accepted until they can trim things from the build to make it work again. The long term solution is to build the 32-bit binaries on a 64-bit system.
And this would be the point where the fact that the Firefox devs have been trying to do too much with a "browser" becomes beyond blatantly obvious. Firefox team: get your stuff together or you will die a slow death of attrition.
E.g. where's the status bar in recent firefoxes?
Size of the Firefox codebase is one factor of course, but the amount of RAM needed by Visual Studio to compile code with all optimizations turned on (especially PGO, which is extra RAM-intensive at the compilation stage) is also a major factor. Notice that this only happens in the 32-bit Visual Studio builds specifically.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Firefox devs requesting immediate RAM bailout.
It doesn't mean much now, it's built for the future.
It takes 16GB to compile Android.
"When information is power, privacy is freedom" - Jah-Wren Ryel
Some of us actually think that using a web browser is more important than compiling a web browser.
Seriously. My resource usage rarely goes about 1 GB with multiple applications open. These days, the hard drive is a far bigger bottle neck than RAM. Well, unless you're compiling Firefox it appears.
The long term solution is to build the 32-bit binaries on a 64-bit system.
No, the long-term solution involves freezing the 32-bit version as an eternal final-state "stable" branch, and moving on to the 64-bit world.
Yes, most people still run 32-bit hardware. And yes, most likely someone would maintain 32-bit backports of FireFox for years to come. But the Mozilla foundation needs to direct their efforts on the reality of the present, not cripple themselves in the interests of backward compatibility.
FWIW, I write this as someone who would primarily use that 32-bit backport at the moment. But I don't expect Mozilla to support my aging XP boxen any more than I expect them to support the Timex Sinclair 1000 or Atari ST I have stashed in the bottom of a closet somewhere.
"First tests indicate that, for example, moving parts of the WebGL implementation to one side could save 300 KB. In a test run, the newer version of Visual Studio required less memory than the one that was previously used, and 64-bit Windows offers 4 GB of address space."
So, first of all, saving 300KB on WebGL seems like a pittance. Then, there's what appears to be the blatantly incorrect statement of 64-bit windows offering 4GB of address space - shouldn't that be way bigger, or am I stupid?
Sheesh. I can remember when we had to keep the size of the completed application small enough to fit on a 360K bootable floppy.
He was just trying to say that 1080p gives him a big endian.
No, no it doesn't, but I'm impressed, turning this against MS instead of complaining about the real culprit, kudos.
I guess that America's obesity problem has extended to software.
To offset political mods, replace Flamebait with Insightful.
Speaking of someone who regularly does large C++ builds, MSVS is nowhere near a bad culprit here. The linker is essentially doing code generation -- link time optimization. Why? Because LTO gives a substantial performance benefit. The profile-guided optimization mentioned in the summary gets them about 10% over even doing non-profile-guided LTO.
One project I've worked on has single files which cause GCC to take over 6 GB to compile when you compile with -O2. Who's bloated now?
(Takeaway: broadly speaking, MSVS is actually very competitive, at least compared with GCC, when comparing similar settings.)
It sounds like a build-tool chain problem, not an issue with the eventual binary produced for Firefox.
Why not just run the 64-bit tools on a 64-bit platform and have them output 32-bit code, the same as can be done by virtually any cross-platform compiler system. I can't imagine worrying about the fact that I can't run the builds on an outdated 32-bit OS as long as I can produce the binaries for such platforms.
I shudder to think what it would have been like to develop for some of the military communications systems I worked on for my first job if we had to run the compilers on that pathetically slow mil-spec Sperry AN-UYK system (magnetic core memory -- talk about slowing down the CPU! But it's radiation hard!) We only tested on the Sperry -- all builds and editing were done on a VAX.
In modern terms, could you imagine having to run your editors and compilers on an iDevice instead of OS/X?
I do not fail; I succeed at finding out what does not work.
There's something seriously wrong when a browser hits a 3GB linker limit.
Imho, the only sane solution is to delete most of the code. It's been crap for years anyway.
Did you ever try compiling Chromium before you made that statement?
You're right. As anyone who's read C++ FQA by Yossi Kreinin might guess, C++ itself could be one of the culprits.
Summary should read: Visual Studio is too teh suck to link Firefox on Windows with PGO.
Firefox links just fine with VS, if you don't use PGO. The problem is that Visual Studio's PGO routine loads all our object files in at once, then uses a ton of memory on top of that. And the linker for 32-bit programs is itself a 32-bit program; if there were a 64-bit x86-32 linker, we wouldn't have this problem. But so far, Microsoft has not given any indication that they will release a 64-bit to x86-32 cross-compiler.
Note that Chrome doesn't build with PGO at all, last time I checked.
http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/533e94237691e2f6
Note: Visual Studio 2010 seems to help a bit, but not much. We use VS 2005 because it's the last version whose CRT supports Windows 2000 and Windows XP before Service Pack 2.
Comment removed based on user account deletion
"One project I've worked on has single files which cause GCC to take over 6 GB to compile when you compile with -O2. Who's bloated now?"
emm ... those single project files?
Seems ironic that the FF team is using stuff from seven years and two major versions ago while at the same time bemoaning that anybody might want to keep a version of Firefox for more then 6 weeks - especially enterprise users.
Interesting how they don't practice what they preach.
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
Nope, the issue is still present in Visual C++ 2010. We're planning on switching to that soon, but we either have to work around its lack of support for Windows 2000 / Windows XP pre-SP2, or declare those platforms (and a few million users) unsupported.
Let's guess Firefox for Windows is about the size of Google Chrome for Windows because it does about as much. But an HTML5 browser almost has to be an operating system by itself, containing a GUI layout engine (for CSS), a JIT-compiling virtual machine (for JavaScript), a window manager (tabbed document interface), a memory manager (RAM and disk caches), and a task scheduler (for scripts in one tab and scripts in another tab). Is Chrome a plug-in, a single application, or an operating system?
I don't know if the blame is so easy to place. It is kind of like watching one fat guy explain to another why they can't sit on the same bench.
The software product I work on also used PGO at one point. Our software is roughly 2-3 million lines of C++ code, and links with 40-50 external libraries. Under Window XP, VS2008 could not link our product with PGO turned on.
Before you complain about the bloat of FF, please understand that PGO uses many many times more memory than just compiling a regular release build. This is a Visual Studio linker problem, not FF problem.
Yes, but you can get a 32->64 cross-GCC toolchain easily. You can't get one of those from Microsoft, so you're stuck with a maximum of 4GB of address space if you run the 32-bit toolchain on a 64-bit Windows system.
I still play 8-bit video games, and the memory card adapter I use to play them has an 8-bit OS.
What, exactly, would be wrong with just using gcc for all platforms, like an awful lot of projects do? What's VC++ doing for Firefox that can't be done any other way?
Back in my day we didn't have gigabyte memory and disk space was at a premium.
It might seem a bit strange now, but back in the good 'ol days we used to have to break up a project into separate components, just in order to compile it!
This is where your interface and API design skills came in handy. If you could partition some piece of the project off into it's own DLL, you could effectively break up the project into smaller pieces that could be individually compiled.
That's where the name DLL came from originally: "Dynamic link library". You didn't need to have all the code read into memory when you first executed the application - less commonly used features wouldn't get loaded until they were actually asked for.
It's not like it is nowadays, where you actually need all the code to be available all the time. "Rich user experience" they call it.
I suppose it's just the future overtakin' us. Them good old days is gone forever.
there aren't any other than Acrobat that support layers that can be turned on or off at runtime (which have been supported by Adobe Acrobat since, I believe, v6 or so).
Why would you want to do that? I've never seen this feature used (I'm not saying it doesn't have a use, just that I can't conceive of one).
"First they came for the slanderers and i said nothing."
Can we turn this into a competitive sport? Please?
"Lack of speed can be overcome. In the worst case by patience." --Znork
Requirements to run Firefox and resources required to compile Firefox aren't the same thing at all. Your attempt to conflate the two suggests you're stupid and don't know what you're talking about.
I'd say it nicer but then I'd risk not hitting +5 Funny.
"Oh no... he found the
No, it has nothing to do with running Firefox. It has everything to do with running Visual Studio's linker.
This matters only to Firefox developers.
Not that they shouldn't care, mind you, as that is some seriously monolithic code. But it won't make any difference to Joe Sixpack.
John
Templates out the wazoo. Don't blame me, I didn't design it. (Or write most of it; I'm only tangentially related nowadays.)
Point being, there are features which, while valuable, are costly. I blame our source, our fondness of templates, and the compilation consequences that templates almost necessitate way more than I blame GCC for having a poor implementation of templates.
But similarly, going "stupid bloaty MSVC, it shouldn't support this feature which can give double-digit percentage speedup because it takes a lot of memory" is stupid. LTO is costly -- period. It's just very worth it.
Old gripe about window-to-the-whole-world is old. And we're talking about builds, not in-use resource consumption... where it behaves just like every other browser.
I can't speak to Firefox specifically, but in my hands, for my project, VC++ produces code that is VASTLY superior to gcc. With gcc, I can often get significant speedup by hand-optimizing code; with VC++, my bog-simple code gets automatically optimized better than my most aggressive manual efforts. Like it or not, Microsoft has the currently best compiler.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
Point taken.
But FYI: We use VS2005 because it's the last version whose CRT supports Windows XP before SP2 and Windows 2000.
We would love to upgrade, and are in fact devoting a lot of engineering time towards figuring out if we can upgrade while maintaining compatibility.
Or just, yknow, stop running a bloated resource hog of an INTERNET BROWSER.
Read TFA agin. Oh, I know this is /.
So read the summary again.
it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB
It is the compiler which is having ressource problems. The profile-guided optimiser needs more than 3GB to be able to do its optimisations. And apparently, the Windows its running on can't do PAE to use more than 3GB neither.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
And how is this connected to the amount of memory required to link Firefox?
It's insane that a compiler, any compiler, needs 3+GB to "just" link a whatever big number of object files into an executable.
It's insane that a single piece of software cannot be linked in a 32bit environment. That is, a Windows32 environment, as it can be linked into a 32bit Linux environment.
How much insanity nowadays!
Sent as ripples into the electromagnetic field. No single photon has been harmed in the process.
Then I find that more worrying than anything. The firefox code is literally so bad that not only do they have to do profile-guided optimisation but must have an optimal compiler with all the options turned up in order to get it to run like the stunned sloth that it does on my systems?
It just reeks of horrendous code. Makes me wonder what the hell all those other large open-source projects are doing that's so much better than the Firefox code that they can outperform it using "only" the sub-optimal gcc.
You mean some people still run a 32-bit OS?
Not only that, but apparently Windows cannot use PAE - Physical Address Extension to address more than 4GB (according to the WP entry, PAE is supported, but the 4GB limit is still enforced - due to some obscure licensing problems).
The problem is with virtual memory. A process still uses 32bit memory addresses to reference memory. This means that a process can still only address 4GB of memory. If you were to use more than 32bit memory addresses to get more memory, suddenly you aren't a 32-bit OS anymore.
PAE only helps the OS be able to manage more than 4GB of memory.
He was gonna answer that question but he took and arrow to the knee.
People keep saying stuff like this, but in my experience it's just not true. Right now I'm clocking Firefox at 596,900 kB commit + 75,000 kB for the plugin-container.
I have never seen it above 1 GB, although I've only sampled maybe once/month.
Are you using another measuring technique than I, or are you just trolling?
What?
This isn't about RUNNING Firefox, but compiling/linking it. I guarantee Skyrim takes more than 500GB under that context.
However, yes, Firefox has become exceedingly bloated.
It is more productive to voice thoughtful opinions (reply) than to judge (moderate) others.
Folks are crawling all over each other to show how ignorant they are. "I ditched firefox for Chrome cause it's lighter!" only to ignore the fact that Chrome also has the same problem with the PGO thing running out of RAM so they don't even bother with trying those optimizations anymore. "Geez Firefox needs more RAM than the kernel to compile? Something's wrong!" Yes if the Linux kernel was built with PGO on VS 32-bit it probably would run out of RAM there too. Then there's the guy that claims this PGO problem is evidence that the Firefox devs need to go back to remedial school. I'm sure he could do it far better (and avoid the PGO linking optimization running out of memory too!).
Hilarious reading. At least I choose to laugh rather than cry at people's inability to read and understand the issues here.
You need to go re-read your source on PAE (or tell them to reread their source).
PAE does not increase the memory space available to a single process, so your statement that "So GCC *could* use 6GB on a 32bits machine" is absolutely false. What PAE allows is multiple processes to, in total, take more than 4 GB. (So you could have a 32-bit machine with 6 GB of RAM and have, e.g., two GCC processes, each taking 3 GB with no paging.)
One of the biggest problems with newer versions is the runtimes using EncodePointer/DecodePointer, which aren't available pre-SP2.
Try this:
* Install VS2010
* In your project configuration(s), under Configuration Properties->General, set "Platform Toolset" to "v90".
I can use this configuration with VS2010 and the latest Windows SDK and get binaries that work on XP pre-SP2.
> 1) What the hell are you doing with your code to be
> that large?
How large? It's a few million lines of C++ code, just like every other browser. What it's doing is implementing all the stuff people want to do on the web.
> 2) What the fuck is your linker doing to do that?
Please read up on link-time code generation.
> 3) Why the hell didn't you see this coming and
> prune LONG before you hit the 3Gb limit if you
> already hit the 2Gb limit once already?
This is an excellent question that I too am asking.
> 4) What's the problem with compiling on 64-bit
> computers only,
None, except updating a large build farm from 32-bit to 64-bit can't happen overnight. Needs some staging, testing, etc.
> You're honestly telling me that Firefox is more
> complicated and needs more memory to compile
> than, say, LibreOffice?
Have you tried to compile LibreOffice with LTO and PGO turned on?
> The Linux kernel?
Quite possibly. The kernel is C, not C++; C++ is a lot more of a pain for compilers to deal with.
The whole point of LTO is that you optimize your entire binary as a single object, no matter how your code is structured. It requires more memory, but can produce faster code because the compiler is able to make optimizations it can't make otherwise.
Whether that's "crappy" is a matter of your priorities, of course. 10-25% performance improvement tends to be a high priority for web browsers, though perhaps not for KDE or LibreOffice.
Not only that, but apparently Windows cannot use PAE - Physical Address Extension [wikipedia.org] to address more than 4GB
Sure it can, you just have to either pay for a server edition or hack the restriction out of the kernel.
But more physical address space doesn't help here, the problem here is virtual address space for running an effective but memory hungry profile guided whole program optimisation process. Nromally 32-bit windows has a maximum of 2GB virtual memory per process (and this is one big process we are dealing with). This can be increased to 3GB at the cost of reducing the kernel address space to 1GB.
Going to a 64-bit OS (which allows 4GB of virtual address space for 32-bit processes) will buy them a little bit of time but it's not a long term soloution. Really they need a 64 to 32 cross toolchain (which according to other posts here MS do not offer) if they want to keep using profile guided optimisation as the codebase grows.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Even more unfortunate that 3/4 of the whole discussion will be "Re: Wow" as everybody dogpiles unrelated replies onto the frist post, because we all know that for mod points, it's not what you post but where you post it.
Why would you choose slackware and then seek technical help? Are you sure you're their target user?
You insensitive clod, I only have a 286 with 640kb, they told me I didn't need any more than that!
This is basic stuff to anyone who actually maintains a build, but Slashdot hasn't been a forum mostly populated by engineers for a number of years, now.
This appears to be due to link-time optimization blowing up the resident memory size of the linker, taking it past 3GB (which is already a non-standard hack the 32 bit build has had to do). Firefox is large, yes, but this has nothing to do with the final binary - which appears to be about 100MB total including all libraries in the Aurora builds.
I used to routinely run out of 32 bit address space compiling executables for a 64MB embedded ARM platform. This was due to symbol bloat, not executable size (which was 8MB). I also ran out of space compiling for a DSP with 288KB of RAM and 1MB flash, but that was mostly piss poor tools (Tasking). In fact, doesn't Chrome and even the Android sources already require building on a 64 bit host?
Blame MS.
The sane option for windows from vista on would have been to deny users the choice and install a 64bit os if the hardware supports it.
Personally I have run 64 bit since xp, although you can only call that a test, vista 64 was ready for all.
Yes I know, the 1 in 16384 people that insist on having some old POC device. Well they can keep an old machine around for it if it is that important.
The better option would have been to deny users the choice and only offer 64-bit builds of Vista and 7.
"But it doesn't work with my 14 year old scanner!!!" Then keep using Windows XP?
There were already growing pains with Vista (shitty GPU drivers from AMD and Nvidia, a new audio framework, and users not being administrator by default), so it would have been the perfect time to force everyone to move to x64. They couldn't force people to switch with 7 because they wanted to get people off of Vista ASAP (even though Vista itself was fine). Oh well, hindsight is 20/20 and what not.
It would be cool if there was a solution to properly handle a 32-bit driver in a 64-bit host environment, but there isn't. (Is there? Even if you use a 32-bit VM on a 64-bit host, you can't do shit.)
I think what he was trying to say -- in the meanest possible way -- is that the setting you chose tells VS to use the old compiler and linker. It doesn't switch out the CRT -- it switches out the whole toolchain. So using that setting is no different from where we are now, afaik.
Iceweasel, 35 tabs open, 1 week uptime, 352MB of resident memory ;)
Dilbert RSS feed
Damn kids and their PGO giving double digit perf increases! Get off my lawn!
-1 overrated isn't the same thing as "I disagree".
It would be cool if there was a solution to properly handle a 32-bit driver in a 64-bit host environment, but there isn't. (Is there? Even if you use a 32-bit VM on a 64-bit host, you can't do shit.)
In the beginning, Microsoft used Thunking to be able to use 16-bit drivers (and other DLLs) on a 32-bit system (Windows 95). This halted in Windows 98 (driver wise) due to horrible stability problems. Eventually, Microsoft moved to Windows on Windows (WOW) which provided an adequate environment for 16-bit apps to run on 32-bit Windows NT. However, if you have a 64-bit version of Windows, you don't have WOW, but WOW64, which runs 32-bit apps on 64-bit Windows, and has no support for 16-bit Windows applications.
"When life gives you lemons, don't make lemonade. Make life take the lemons back!" -- Cave Johnson
I have an idea: why not fork Firefox and create a leaner/faster just-a-browser and cut out all the excess junk.
And it could be named after some sort of animal or bird that comes out of ashes maybe ?
Ah, that's why Firefox is so slow but everyone insists it's not that bad. They're all Windows users!
Making the usage of Firefox fast is the whole point of PGO, a process that uses up orders of magnitude more when compiling and linking an executable than without PGO. So, in order to get the fastest speeds out of the Firefox binary, the Firefox build process optimizes and uses lots of RAM in the process. The problem is that the MSVC linker is 32-bits, so the *linker* is running out of memory ONLY WHEN FIREFOX IS COMPILING AND OPTIMIZING. The executable that is generated is not as large, does not use as much memory, and is faster than if Firefox was built with a "straight" boring old build process.
The whole point of this exercise is to save assholes like you more time by making your browser faster and more responsive so you can post snarky criticisms of Firefox (without understanding the problem) faster.
Your welcome, asshole.
You can't. Microsoft simply doesn't offer a 64-bit linker that can produce 32-bit code.
Microsoft doesn't offer a 64-bit compiler that cross-compiles to 32-bit. Fine, I can accept that. But there are other compilers one could use:
http://software.intel.com/en-us/articles/intel-compilers/ is still reputed to produce the tightest, fastest WinXX executables available.
http://www.embarcadero.com/products/cbuilder (formerly Borland C++)
The venerable http://gcc.gnu.org/
http://www.ghs.com/products/optimizingC++EC++Compilers.html (though it looks like GHC might no longer target Windows at all)
I do not fail; I succeed at finding out what does not work.
And the trolly troll is trolling.
Unless you regularly compile optimized builds of Chrome, Opera, Safari, or IE, kindly STFU.
I still get calls to support industrial processes driven by DOS based machines.
I am an old guy, raised with these beasts. I know the innards of these things like the back of my hand.
The harbinger of death seemed to be a dearth of disk drives compatible with old DOS machines. Even my stream of disk drives from the recycler is drying up. But it looks like SanDisk and Syba have pulled me and my customer's arse out of the fire again.
Its hard to throw away an enormous expensive piece of machinery because its controller ( which has been doing exactly what it needed to do for 25 years ) isn't supported anymore.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
You want scary? At $CLINIC we use, they run Windows NT 4.0 on their ultrasonic diagnostic equipment, because the system was certified only with that OS at whatever the patch level was when they tested it. While that may seem innocuous, the damn thing is on the freakin' network, because the doctors want to email images to their patients!
I sure don't want me or my family to be the ones to have to file the lawsuit when a "zombie" actually causes someone to die from a faulty diagnosis, or because some machinery failed due to a DDoS attack originating on their network.
John
Not that they shouldn't care, mind you, as that is some seriously monolithic code.
They're talking about using link-time optimisation (LTO). That means that you compile each compilation unit to an intermediate representation and then run optimisations on the whole program. This takes a staggering amount of memory (which is why no one bothered with it off high-end workstations with very expensive compilers until very recently), but can sometimes be worth it. It actually helps modularity, because you can keep your source code nicely separated into independent components without worrying about efficiency, and you can do better data hiding.
As a trivial example, consider an accessor method. Something like getFoo() { return foo; }. Without LTO, you'd want to put the declaration of the class and this method in the header so that every compilation unit could separately inline it. This reduces modularity, but it saves you the cost of a function call just to access a field in a structure. With LTO, you can make the class opaque (if you're a C++ junkie, using the Pimpl pattern, if you're a C programmer by just not declaring the struct in the header). You'll get a single copy of the function emitted, and then the link-time optimiser will inline it everywhere.
I am TheRaven on Soylent News
There is no advantage to running 64bit LINUX....FTFY
Not true at all. There are several advantages to running code compiled for x86-64 instead of for x86. If you're on *NIX, then one of the big ones is that x86-64 has rip-relative addressing, while x86 lacks eip-relative addressing. In position-independent code, this means that any call to a static function or access to a static variable is cheaper. Windows doesn't use position-independent code for DLLs on 32-bit (it tries to load them at a fixed address and performs load-time relocations if it can't), so that's not a problem, but on *NIX.
There are other advantages. One of the big ones is that anything targeting x86-64 will default to using SSE for floating point instead of x87. This is generally at least slightly faster in the silicon, but it's also much easier to generate good SSE code for the register-register SSE coprocessor (with lots of registers!) than for the stack-based x87 (with only a handful).
Then, of course, there's the fact that you have a lot more general-purpose registers with x86-64, so you need far fewer stack spills to do the same thing.
I am TheRaven on Soylent News