Firefox Too Big To Link On 32-bit Windows

An anonymous reader writes "Firefox has gotten so large that it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB. This problem had happened last year with 2 GB, which was worked around by adding a/3GB switch to the Windows build servers. Now the problem is back, and things aren't quite that simple anymore." This only affects the inbound branch, but from the looks of it new code is no longer being accepted until they can trim things from the build to make it work again. The long term solution is to build the 32-bit binaries on a 64-bit system.

Comments

interaction of two things

[Trepidity]

Size of the Firefox codebase is one factor of course, but the amount of RAM needed by Visual Studio to compile code with all optimizations turned on (especially PGO, which is extra RAM-intensive at the compilation stage) is also a major factor. Notice that this only happens in the 32-bit Visual Studio builds specifically.

Re:The code gets larger, and yet things dissapear!

[cruff]

I immediately added the Status-4-Evar addon to my Firefox installations, works great.

Re:whose bloat

[EvanED]

Speaking of someone who regularly does large C++ builds, MSVS is nowhere near a bad culprit here. The linker is essentially doing code generation -- link time optimization. Why? Because LTO gives a substantial performance benefit. The profile-guided optimization mentioned in the summary gets them about 10% over even doing non-profile-guided LTO.

One project I've worked on has single files which cause GCC to take over 6 GB to compile when you compile with -O2. Who's bloated now?

(Takeaway: broadly speaking, MSVS is actually very competitive, at least compared with GCC, when comparing similar settings.)

The problem is VS's PGO architecture

[jlebar]

Summary should read: Visual Studio is too teh suck to link Firefox on Windows with PGO.

Firefox links just fine with VS, if you don't use PGO. The problem is that Visual Studio's PGO routine loads all our object files in at once, then uses a ton of memory on top of that. And the linker for 32-bit programs is itself a 32-bit program; if there were a 64-bit x86-32 linker, we wouldn't have this problem. But so far, Microsoft has not given any indication that they will release a 64-bit to x86-32 cross-compiler.

Note that Chrome doesn't build with PGO at all, last time I checked.

http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/533e94237691e2f6

Note: Visual Studio 2010 seems to help a bit, but not much. We use VS 2005 because it's the last version whose CRT supports Windows 2000 and Windows XP before Service Pack 2.

Re:The problem is VS's PGO architecture

[askldjd]

Parent is right. The software product I work on also used PGO at one point. Our software is roughly 2-3 million lines of C++ code, and links with 40-50 external libraries. Under Window XP, VS2008 could not link our product with PGO turned on. Before you complain about the bloat of FF, please understand that PGO uses many many times more memory than just compiling a regular release build. This is a Visual Studio linker problem, not FF problem.

Re:Last paragraph in the TFA is... confusing

[EvanED]

Also MSVC only has 32-bit binaries

Clarification: the only linker for 32-bit targets is, itself, 32-bits.

The linker which targets 64-bit Windows is still 64 bits. If they had a 64-bit-to-32-bit cross compiler (they have the reverse, but not 64-to-32) that would solve Mozilla's problem. (Well, for some definition of "their problem.")

Re:Trying to do too much

[jlebar]

Chrome doesn't even try to build with PGO, last time I checked.

http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/533e94237691e2f6

Re:VS 2005?

[jlebar]

Point taken.

But FYI: We use VS2005 because it's the last version whose CRT supports Windows XP before SP2 and Windows 2000.

We would love to upgrade, and are in fact devoting a lot of engineering time towards figuring out if we can upgrade while maintaining compatibility.

Its the compiler, stupid.

[DrYak]

Or just, yknow, stop running a bloated resource hog of an INTERNET BROWSER.

Read TFA agin. Oh, I know this is /.
So read the summary again.

it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB

It is the compiler which is having ressource problems. The profile-guided optimiser needs more than 3GB to be able to do its optimisations. And apparently, the Windows its running on can't do PAE to use more than 3GB neither.

Re:Its the compiler, stupid.

[cjb909]

It is the compiler which is having ressource problems. The profile-guided optimiser needs more than 3GB to be able to do its optimisations. And apparently, the Windows its running on can't do PAE to use more than 3GB neither.

PAE allows 32-Bit computers to use more than 4GB of ram, but it doesn't allow Windows to assign more than 3GB to any single process.

Re:I don't understand the issue

[EvanED]

There is no 64-bit version of the MSVC linker

Clarification (I've posted this a few times): there's no 64-bit version of MSVC that targets 32 bits. There is a 64-bit version which targets 64 bits, but that doesn't help the current situation.

Re:Trying to do too much

[BZ]

Define "too much"?

It's been over a year since Chrome had to turn off PGO altogether and move to 64-bit builders even without PGO, because they ran into this same problem.

So maybe your issue is with the fact that all "browsers" as you call them are trying to do too much? They should drop the fast graphics and jits and video and audio support and all that jazz, right?

Re:Time to move on, perhaps?

[plover]

No, you missed a fact.
Visual Studio 2005 is a 32-bit app on any Windows platform.
Visual Studio 2010 is a 32-bit app when running on a 32-bit platform.
Mozilla builds on 32-bit platforms can no longer support the PGO linker.

Visual Studio 2010 is a 64-bit app on a 64-bit platform.
Mozilla builds on 64-bit platforms can PGO link just fine.
When Visual Studio 64-bit compiles a 32-bit app, that app can run only on XP SP2 or later.
Mozilla has millions of users on pre-XP SP2 platforms.

So Mozilla has a choice: change nothing but stop PGO linking the 32-bit versions (sub-optimum for 32-bit users), go forward on a 64-bit only path and disenfranchise the old users from getting any new functions, abandon them completely (which is irresponsible in terms of security), cut back on new features for all, or take an axe to the existing code. Only one is an easy choice.

Re:Trying to do too much

[maztuhblastah]

I think it's still indicative of the problem GP mentions. The more code you are trying to pull in, the larger the footprint during the build process. You don't see a 'Hello world' program requiring a 3GB+ build footprint do you? No, because it's not doing enough to warrant that. Likewise, Firefox apparently *is* trying to do a lot. More than it used to at any rate.

Well you're right in that Firefox does need a hell rather large amount of RAM to build... but it's not just them; all browsers are trying to do a lot nowadays.

Chrome doesn't exactly have light build requirements either. In fact, the Chromium project already seems to have dropped 32-bit build environments:

A 64 bit OS is highly recommended as building on 32 bit OS is constantly becoming harder, is a lot slower and is not actively maintained.

(From "Build Instructions (Windows) - Build Environment")

That's why I think that the parent poster's implication that it's due to Firefox becoming "bloated" is basically hogwash. Browsers are more complex than they were in the mid-90s. That's what happens when you add 10+ years of new formats and technologies that must be supported for a browser to be considered "usable". Directing one's ire at Firefox is unwarranted, IMHO.

Re:Trying to do too much

[BZ]

> And that is to make lots of parts be separately loaded
> modules

This actually leads to pretty serious performance penalties because of the way that web specs tend to interdepend on each other. It also loses you optimization opportunities.

Pieces of functionality that are not interdependent with other stuff (e.g. audio and video backends, webgl) are in fact either in separate libraries or being moved there.

For the other stuff, there did use to be more modules. It turned out that in practice most users needed them at startup (to show the browser itself and restore their tabs), so the only thing multiple modules got you was a slower startup and slower runtime code.

It's not an accident that both Firefox and Chrome ship almost all their code in a single library (binary in the case of Chrome). It turns out that for web browsers specifically this works somewhat better than the alternatives.

Oh, I just checked and looks like Opera also also links all its code into a single library, at least on Mac.

And Safari links all the core WebKit code into a single library.

Not sure what IE does nowadays, but last I checked mshtml.dll in fact included all the actual browser bits.

Now it may be that all the people involved in all these projects can't design worth anything. Or maybe they did some measurements that you haven't done and found that this approach works better....

Re:Time to move on, perhaps?

[jlebar]

I am a Mozilla guy.

There's an official 64-bit version for Linux. We've been shipping that since before I can remember. There are also nightly builds for 64-bit Windows, but we're not shipping these even as Aurora at the moment.

64-bit Linux isn't listed on most of our download pages. I'd argue it should be there, but I'm not in charge. :)

Anyway, here are links to get all the builds we produce:

Nightly builds: http://nightly.mozilla.org/ (has win-64 builds)
Aurora builds: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/
Beta builds: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/9.0b6-candidates/build1/ (I don't know the directory for the latest beta build, unfortunately, so you'll have to update this URL each time you go looking.)
Release builds: http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/

Re:Trying to do too much

[Tolkien]

PGO = Profile Guided Optimization. It allows the optimizing compiler/linker to figure out for itself exactly what the critical execution path(s) of the program (the code that runs most frequently) is/are by requiring the user to execute the code with this special pgo-instrumentation of the compiled code as many times as required to cover typical use-cases, and optimize the code based around the execution paths taken. Profile guided optimization is a strictly optional step which can be used to improve program performance (potentially) more or better than the generalized optimizer (/Ox compiler option with MSVC) can. Fuck. Posting this is undoing my moderations. Oh well.