Slashdot Mirror
Firefox Too Big To Link On 32-bit Windows
An anonymous reader writes "Firefox has gotten so large that it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB. This problem had happened last year with 2 GB, which was worked around by adding a/3GB switch to the Windows build servers. Now the problem is back, and things aren't quite that simple anymore."
This only affects the inbound branch, but from the looks of it new code is no longer being accepted until they can trim things from the build to make it work again. The long term solution is to build the 32-bit binaries on a 64-bit system.
And this would be the point where the fact that the Firefox devs have been trying to do too much with a "browser" becomes beyond blatantly obvious. Firefox team: get your stuff together or you will die a slow death of attrition.
E.g. where's the status bar in recent firefoxes?
Size of the Firefox codebase is one factor of course, but the amount of RAM needed by Visual Studio to compile code with all optimizations turned on (especially PGO, which is extra RAM-intensive at the compilation stage) is also a major factor. Notice that this only happens in the 32-bit Visual Studio builds specifically.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
Firefox devs requesting immediate RAM bailout.
It doesn't mean much now, it's built for the future.
It takes 16GB to compile Android.
"When information is power, privacy is freedom" - Jah-Wren Ryel
He was just trying to say that 1080p gives him a big endian.
I guess that America's obesity problem has extended to software.
To offset political mods, replace Flamebait with Insightful.
Speaking of someone who regularly does large C++ builds, MSVS is nowhere near a bad culprit here. The linker is essentially doing code generation -- link time optimization. Why? Because LTO gives a substantial performance benefit. The profile-guided optimization mentioned in the summary gets them about 10% over even doing non-profile-guided LTO.
One project I've worked on has single files which cause GCC to take over 6 GB to compile when you compile with -O2. Who's bloated now?
(Takeaway: broadly speaking, MSVS is actually very competitive, at least compared with GCC, when comparing similar settings.)
It sounds like a build-tool chain problem, not an issue with the eventual binary produced for Firefox.
Why not just run the 64-bit tools on a 64-bit platform and have them output 32-bit code, the same as can be done by virtually any cross-platform compiler system. I can't imagine worrying about the fact that I can't run the builds on an outdated 32-bit OS as long as I can produce the binaries for such platforms.
I shudder to think what it would have been like to develop for some of the military communications systems I worked on for my first job if we had to run the compilers on that pathetically slow mil-spec Sperry AN-UYK system (magnetic core memory -- talk about slowing down the CPU! But it's radiation hard!) We only tested on the Sperry -- all builds and editing were done on a VAX.
In modern terms, could you imagine having to run your editors and compilers on an iDevice instead of OS/X?
I do not fail; I succeed at finding out what does not work.
You're right. As anyone who's read C++ FQA by Yossi Kreinin might guess, C++ itself could be one of the culprits.
An individual 32 bit process (IE the visual studio linker) is limited to 4GB of addressable memory.
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx#memory_limits
-1 overrated isn't the same thing as "I disagree".
Summary should read: Visual Studio is too teh suck to link Firefox on Windows with PGO.
Firefox links just fine with VS, if you don't use PGO. The problem is that Visual Studio's PGO routine loads all our object files in at once, then uses a ton of memory on top of that. And the linker for 32-bit programs is itself a 32-bit program; if there were a 64-bit x86-32 linker, we wouldn't have this problem. But so far, Microsoft has not given any indication that they will release a 64-bit to x86-32 cross-compiler.
Note that Chrome doesn't build with PGO at all, last time I checked.
http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/533e94237691e2f6
Note: Visual Studio 2010 seems to help a bit, but not much. We use VS 2005 because it's the last version whose CRT supports Windows 2000 and Windows XP before Service Pack 2.
Also MSVC only has 32-bit binaries
Clarification: the only linker for 32-bit targets is, itself, 32-bits.
The linker which targets 64-bit Windows is still 64 bits. If they had a 64-bit-to-32-bit cross compiler (they have the reverse, but not 64-to-32) that would solve Mozilla's problem. (Well, for some definition of "their problem.")
Seems ironic that the FF team is using stuff from seven years and two major versions ago while at the same time bemoaning that anybody might want to keep a version of Firefox for more then 6 weeks - especially enterprise users.
Interesting how they don't practice what they preach.
-- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
Let's guess Firefox for Windows is about the size of Google Chrome for Windows because it does about as much. But an HTML5 browser almost has to be an operating system by itself, containing a GUI layout engine (for CSS), a JIT-compiling virtual machine (for JavaScript), a window manager (tabbed document interface), a memory manager (RAM and disk caches), and a task scheduler (for scripts in one tab and scripts in another tab). Is Chrome a plug-in, a single application, or an operating system?
I don't know if the blame is so easy to place. It is kind of like watching one fat guy explain to another why they can't sit on the same bench.
No, the long-term solution involves freezing the 32-bit version as an eternal final-state "stable" branch, and moving on to the 64-bit world.
Um.
Some 90% of our users are on 32-bit Windows. Just because *you're* not one of them doesn't mean that they don't matter.
It's nice that you don't expect us to support your aging XP boxes, but I think you'd find you're the minority in this respect.
(Also, all phones are 32-bit, and will be for at least the next few years.)
The software product I work on also used PGO at one point. Our software is roughly 2-3 million lines of C++ code, and links with 40-50 external libraries. Under Window XP, VS2008 could not link our product with PGO turned on.
Before you complain about the bloat of FF, please understand that PGO uses many many times more memory than just compiling a regular release build. This is a Visual Studio linker problem, not FF problem.
32 bit programmes (with the large address aware flag set) get 4GB of address space on 64 bit Windows, compared to 2GB / 3GB on 32 bit Windows.
10 PRINT "LOOK AROUND YOU ";
20 GOTO 10
Back in my day we didn't have gigabyte memory and disk space was at a premium.
It might seem a bit strange now, but back in the good 'ol days we used to have to break up a project into separate components, just in order to compile it!
This is where your interface and API design skills came in handy. If you could partition some piece of the project off into it's own DLL, you could effectively break up the project into smaller pieces that could be individually compiled.
That's where the name DLL came from originally: "Dynamic link library". You didn't need to have all the code read into memory when you first executed the application - less commonly used features wouldn't get loaded until they were actually asked for.
It's not like it is nowadays, where you actually need all the code to be available all the time. "Rich user experience" they call it.
I suppose it's just the future overtakin' us. Them good old days is gone forever.
Can we turn this into a competitive sport? Please?
"Lack of speed can be overcome. In the worst case by patience." --Znork
Requirements to run Firefox and resources required to compile Firefox aren't the same thing at all. Your attempt to conflate the two suggests you're stupid and don't know what you're talking about.
I'd say it nicer but then I'd risk not hitting +5 Funny.
"Oh no... he found the
No, it has nothing to do with running Firefox. It has everything to do with running Visual Studio's linker.
This matters only to Firefox developers.
Not that they shouldn't care, mind you, as that is some seriously monolithic code. But it won't make any difference to Joe Sixpack.
John
I can't speak to Firefox specifically, but in my hands, for my project, VC++ produces code that is VASTLY superior to gcc. With gcc, I can often get significant speedup by hand-optimizing code; with VC++, my bog-simple code gets automatically optimized better than my most aggressive manual efforts. Like it or not, Microsoft has the currently best compiler.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
Point taken.
But FYI: We use VS2005 because it's the last version whose CRT supports Windows XP before SP2 and Windows 2000.
We would love to upgrade, and are in fact devoting a lot of engineering time towards figuring out if we can upgrade while maintaining compatibility.
Or just, yknow, stop running a bloated resource hog of an INTERNET BROWSER.
Read TFA agin. Oh, I know this is /.
So read the summary again.
it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB
It is the compiler which is having ressource problems. The profile-guided optimiser needs more than 3GB to be able to do its optimisations. And apparently, the Windows its running on can't do PAE to use more than 3GB neither.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
And how is this connected to the amount of memory required to link Firefox?
He was gonna answer that question but he took and arrow to the knee.
No, you missed a fact.
Visual Studio 2005 is a 32-bit app on any Windows platform.
Visual Studio 2010 is a 32-bit app when running on a 32-bit platform.
Mozilla builds on 32-bit platforms can no longer support the PGO linker.
Visual Studio 2010 is a 64-bit app on a 64-bit platform.
Mozilla builds on 64-bit platforms can PGO link just fine.
When Visual Studio 64-bit compiles a 32-bit app, that app can run only on XP SP2 or later.
Mozilla has millions of users on pre-XP SP2 platforms.
So Mozilla has a choice: change nothing but stop PGO linking the 32-bit versions (sub-optimum for 32-bit users), go forward on a 64-bit only path and disenfranchise the old users from getting any new functions, abandon them completely (which is irresponsible in terms of security), cut back on new features for all, or take an axe to the existing code. Only one is an easy choice.
John
Folks are crawling all over each other to show how ignorant they are. "I ditched firefox for Chrome cause it's lighter!" only to ignore the fact that Chrome also has the same problem with the PGO thing running out of RAM so they don't even bother with trying those optimizations anymore. "Geez Firefox needs more RAM than the kernel to compile? Something's wrong!" Yes if the Linux kernel was built with PGO on VS 32-bit it probably would run out of RAM there too. Then there's the guy that claims this PGO problem is evidence that the Firefox devs need to go back to remedial school. I'm sure he could do it far better (and avoid the PGO linking optimization running out of memory too!).
Hilarious reading. At least I choose to laugh rather than cry at people's inability to read and understand the issues here.
One of the biggest problems with newer versions is the runtimes using EncodePointer/DecodePointer, which aren't available pre-SP2.
Try this:
* Install VS2010
* In your project configuration(s), under Configuration Properties->General, set "Platform Toolset" to "v90".
I can use this configuration with VS2010 and the latest Windows SDK and get binaries that work on XP pre-SP2.
> 1) What the hell are you doing with your code to be
> that large?
How large? It's a few million lines of C++ code, just like every other browser. What it's doing is implementing all the stuff people want to do on the web.
> 2) What the fuck is your linker doing to do that?
Please read up on link-time code generation.
> 3) Why the hell didn't you see this coming and
> prune LONG before you hit the 3Gb limit if you
> already hit the 2Gb limit once already?
This is an excellent question that I too am asking.
> 4) What's the problem with compiling on 64-bit
> computers only,
None, except updating a large build farm from 32-bit to 64-bit can't happen overnight. Needs some staging, testing, etc.
> You're honestly telling me that Firefox is more
> complicated and needs more memory to compile
> than, say, LibreOffice?
Have you tried to compile LibreOffice with LTO and PGO turned on?
> The Linux kernel?
Quite possibly. The kernel is C, not C++; C++ is a lot more of a pain for compilers to deal with.
The whole point of LTO is that you optimize your entire binary as a single object, no matter how your code is structured. It requires more memory, but can produce faster code because the compiler is able to make optimizations it can't make otherwise.
Whether that's "crappy" is a matter of your priorities, of course. 10-25% performance improvement tends to be a high priority for web browsers, though perhaps not for KDE or LibreOffice.
Not only that, but apparently Windows cannot use PAE - Physical Address Extension [wikipedia.org] to address more than 4GB
Sure it can, you just have to either pay for a server edition or hack the restriction out of the kernel.
But more physical address space doesn't help here, the problem here is virtual address space for running an effective but memory hungry profile guided whole program optimisation process. Nromally 32-bit windows has a maximum of 2GB virtual memory per process (and this is one big process we are dealing with). This can be increased to 3GB at the cost of reducing the kernel address space to 1GB.
Going to a 64-bit OS (which allows 4GB of virtual address space for 32-bit processes) will buy them a little bit of time but it's not a long term soloution. Really they need a 64 to 32 cross toolchain (which according to other posts here MS do not offer) if they want to keep using profile guided optimisation as the codebase grows.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
I am a Mozilla guy.
There's an official 64-bit version for Linux. We've been shipping that since before I can remember. There are also nightly builds for 64-bit Windows, but we're not shipping these even as Aurora at the moment.
64-bit Linux isn't listed on most of our download pages. I'd argue it should be there, but I'm not in charge. :)
Anyway, here are links to get all the builds we produce:
Nightly builds: http://nightly.mozilla.org/ (has win-64 builds)
Aurora builds: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/
Beta builds: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/9.0b6-candidates/build1/ (I don't know the directory for the latest beta build, unfortunately, so you'll have to update this URL each time you go looking.)
Release builds: http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/
Not that they shouldn't care, mind you, as that is some seriously monolithic code.
They're talking about using link-time optimisation (LTO). That means that you compile each compilation unit to an intermediate representation and then run optimisations on the whole program. This takes a staggering amount of memory (which is why no one bothered with it off high-end workstations with very expensive compilers until very recently), but can sometimes be worth it. It actually helps modularity, because you can keep your source code nicely separated into independent components without worrying about efficiency, and you can do better data hiding.
As a trivial example, consider an accessor method. Something like getFoo() { return foo; }. Without LTO, you'd want to put the declaration of the class and this method in the header so that every compilation unit could separately inline it. This reduces modularity, but it saves you the cost of a function call just to access a field in a structure. With LTO, you can make the class opaque (if you're a C++ junkie, using the Pimpl pattern, if you're a C programmer by just not declaring the struct in the header). You'll get a single copy of the function emitted, and then the link-time optimiser will inline it everywhere.
I am TheRaven on Soylent News