Slashdot Mirror

← Back to Stories (view on slashdot.org)

Firefox Too Big To Link On 32-bit Windows

Posted by Unknown on from the ate-too-many-pizzas dept.

An anonymous reader writes "Firefox has gotten so large that it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB. This problem had happened last year with 2 GB, which was worked around by adding a/3GB switch to the Windows build servers. Now the problem is back, and things aren't quite that simple anymore." This only affects the inbound branch, but from the looks of it new code is no longer being accepted until they can trim things from the build to make it work again. The long term solution is to build the 32-bit binaries on a 64-bit system.

36 of 753 comments (clear)

  1. Trying to do too much by bobcat7677 · · Score: 5, Insightful

    And this would be the point where the fact that the Firefox devs have been trying to do too much with a "browser" becomes beyond blatantly obvious. Firefox team: get your stuff together or you will die a slow death of attrition.

    1. Re:Trying to do too much by dn15 · · Score: 5, Funny

      And this would be the point where the fact that the Firefox devs have been trying to do too much with a "browser" becomes beyond blatantly obvious.

      Firefox is a great operating system, if only it included a decent browser :P

    2. Re:Trying to do too much by jlebar · · Score: 5, Informative

      Chrome doesn't even try to build with PGO, last time I checked.

      http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/533e94237691e2f6

    3. Re:Trying to do too much by lpp · · Score: 5, Insightful

      I think it's still indicative of the problem GP mentions. The more code you are trying to pull in, the larger the footprint during the build process. You don't see a 'Hello world' program requiring a 3GB+ build footprint do you? No, because it's not doing enough to warrant that. Likewise, Firefox apparently *is* trying to do a lot. More than it used to at any rate.

    4. Re:Trying to do too much by Unoriginal_Nickname · · Score: 5, Insightful

      Thank you for posting this. Filling the virtual address space of the linker probably does indicate some problems with the Firefox source code - crazy big translation units, for example - but it doesn't imply anything about the size or quality of the resulting binary.

      I thought people on this site were supposed to know something about computers.

    5. Re:Trying to do too much by BZ · · Score: 5, Informative

      Define "too much"?

      It's been over a year since Chrome had to turn off PGO altogether and move to 64-bit builders even without PGO, because they ran into this same problem.

      So maybe your issue is with the fact that all "browsers" as you call them are trying to do too much? They should drop the fast graphics and jits and video and audio support and all that jazz, right?

    6. Re:Trying to do too much by BZ · · Score: 5, Insightful

      The "translation unit" involved here is the whole binary. We're talking about link-time code generation with profile-guided optimization, not regular compiles.

      So it doesn't indicate much of anything about the Firefox source code other than general overall quantity of code being compiled...

    7. Re:Trying to do too much by maztuhblastah · · Score: 5, Informative

      I think it's still indicative of the problem GP mentions. The more code you are trying to pull in, the larger the footprint during the build process. You don't see a 'Hello world' program requiring a 3GB+ build footprint do you? No, because it's not doing enough to warrant that. Likewise, Firefox apparently *is* trying to do a lot. More than it used to at any rate.

      Well you're right in that Firefox does need a hell rather large amount of RAM to build... but it's not just them; all browsers are trying to do a lot nowadays.

      Chrome doesn't exactly have light build requirements either. In fact, the Chromium project already seems to have dropped 32-bit build environments:

      A 64 bit OS is highly recommended as building on 32 bit OS is constantly becoming harder, is a lot slower and is not actively maintained.

      (From "Build Instructions (Windows) - Build Environment")

      That's why I think that the parent poster's implication that it's due to Firefox becoming "bloated" is basically hogwash. Browsers are more complex than they were in the mid-90s. That's what happens when you add 10+ years of new formats and technologies that must be supported for a browser to be considered "usable". Directing one's ire at Firefox is unwarranted, IMHO.

      --
      The real litigious bastards...
    8. Re:Trying to do too much by BZ · · Score: 5, Informative

      > And that is to make lots of parts be separately loaded
      > modules

      This actually leads to pretty serious performance penalties because of the way that web specs tend to interdepend on each other. It also loses you optimization opportunities.

      Pieces of functionality that are not interdependent with other stuff (e.g. audio and video backends, webgl) are in fact either in separate libraries or being moved there.

      For the other stuff, there did use to be more modules. It turned out that in practice most users needed them at startup (to show the browser itself and restore their tabs), so the only thing multiple modules got you was a slower startup and slower runtime code.

      It's not an accident that both Firefox and Chrome ship almost all their code in a single library (binary in the case of Chrome). It turns out that for web browsers specifically this works somewhat better than the alternatives.

      Oh, I just checked and looks like Opera also also links all its code into a single library, at least on Mac.

      And Safari links all the core WebKit code into a single library.

      Not sure what IE does nowadays, but last I checked mshtml.dll in fact included all the actual browser bits.

      Now it may be that all the people involved in all these projects can't design worth anything. Or maybe they did some measurements that you haven't done and found that this approach works better....

    9. Re:Trying to do too much by Tolkien · · Score: 5, Informative

      PGO = Profile Guided Optimization. It allows the optimizing compiler/linker to figure out for itself exactly what the critical execution path(s) of the program (the code that runs most frequently) is/are by requiring the user to execute the code with this special pgo-instrumentation of the compiled code as many times as required to cover typical use-cases, and optimize the code based around the execution paths taken. Profile guided optimization is a strictly optional step which can be used to improve program performance (potentially) more or better than the generalized optimizer (/Ox compiler option with MSVC) can. Fuck. Posting this is undoing my moderations. Oh well.

      --
      how is babby formed?
  2. interaction of two things by Trepidity · · Score: 5, Informative

    Size of the Firefox codebase is one factor of course, but the amount of RAM needed by Visual Studio to compile code with all optimizations turned on (especially PGO, which is extra RAM-intensive at the compilation stage) is also a major factor. Notice that this only happens in the 32-bit Visual Studio builds specifically.

    --
    10 PRINT CHR$(205.5+RND(1)); : GOTO 10
  3. Too big to link by Pope · · Score: 5, Funny

    Firefox devs requesting immediate RAM bailout.

    --
    It doesn't mean much now, it's built for the future.
    1. Re:Too big to link by TheGratefulNet · · Score: 5, Funny

      Occupy Swap Space, 2011!

      be there, or be nullified!

      --

      --
      "It is now safe to switch off your computer."
  4. Re:The code gets larger, and yet things dissapear! by cruff · · Score: 5, Informative

    I immediately added the Status-4-Evar addon to my Firefox installations, works great.

  5. Big deal by GameboyRMH · · Score: 5, Funny

    It takes 16GB to compile Android.

    --
    "When information is power, privacy is freedom" - Jah-Wren Ryel
  6. Re:Wow by Anonymous Coward · · Score: 5, Funny

    He was just trying to say that 1080p gives him a big endian.

  7. Re:Are you serious? by masternerdguy · · Score: 5, Funny

    I guess that America's obesity problem has extended to software.

    --
    To offset political mods, replace Flamebait with Insightful.
  8. Re:The code gets larger, and yet things dissapear! by webmistressrachel · · Score: 5, Insightful

    If Seamonkey adopt this silly fade-in fade-out floating toolbar-as-status-bar-replacement, that'll be the end of the Mozilla line for me. Seamonkey has always been the sensible browser for Netscape-heads from way back when since Mozilla Suite died a death and SeaMonkey came into being.

    --
    This tagline was transcoded to result in at least one smirk. If you experience failure to smirk, please consult your Gen
  9. Re:whose bloat by EvanED · · Score: 5, Informative

    Speaking of someone who regularly does large C++ builds, MSVS is nowhere near a bad culprit here. The linker is essentially doing code generation -- link time optimization. Why? Because LTO gives a substantial performance benefit. The profile-guided optimization mentioned in the summary gets them about 10% over even doing non-profile-guided LTO.

    One project I've worked on has single files which cause GCC to take over 6 GB to compile when you compile with -O2. Who's bloated now?

    (Takeaway: broadly speaking, MSVS is actually very competitive, at least compared with GCC, when comparing similar settings.)

  10. I don't understand the issue by msobkow · · Score: 5, Insightful

    It sounds like a build-tool chain problem, not an issue with the eventual binary produced for Firefox.

    Why not just run the 64-bit tools on a 64-bit platform and have them output 32-bit code, the same as can be done by virtually any cross-platform compiler system. I can't imagine worrying about the fact that I can't run the builds on an outdated 32-bit OS as long as I can produce the binaries for such platforms.

    I shudder to think what it would have been like to develop for some of the military communications systems I worked on for my first job if we had to run the compilers on that pathetically slow mil-spec Sperry AN-UYK system (magnetic core memory -- talk about slowing down the CPU! But it's radiation hard!) We only tested on the Sperry -- all builds and editing were done on a VAX.

    In modern terms, could you imagine having to run your editors and compilers on an iDevice instead of OS/X?

    --
    I do not fail; I succeed at finding out what does not work.
    1. Re:I don't understand the issue by EvanED · · Score: 5, Informative

      There is no 64-bit version of the MSVC linker

      Clarification (I've posted this a few times): there's no 64-bit version of MSVC that targets 32 bits. There is a 64-bit version which targets 64 bits, but that doesn't help the current situation.

  11. Re:whose bloat by tepples · · Score: 5, Interesting

    You're right. As anyone who's read C++ FQA by Yossi Kreinin might guess, C++ itself could be one of the culprits.

  12. The problem is VS's PGO architecture by jlebar · · Score: 5, Informative

    Summary should read: Visual Studio is too teh suck to link Firefox on Windows with PGO.

    Firefox links just fine with VS, if you don't use PGO. The problem is that Visual Studio's PGO routine loads all our object files in at once, then uses a ton of memory on top of that. And the linker for 32-bit programs is itself a 32-bit program; if there were a 64-bit x86-32 linker, we wouldn't have this problem. But so far, Microsoft has not given any indication that they will release a 64-bit to x86-32 cross-compiler.

    Note that Chrome doesn't build with PGO at all, last time I checked.

    http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/533e94237691e2f6

    Note: Visual Studio 2010 seems to help a bit, but not much. We use VS 2005 because it's the last version whose CRT supports Windows 2000 and Windows XP before Service Pack 2.

    1. Re:The problem is VS's PGO architecture by askldjd · · Score: 5, Informative

      Parent is right. The software product I work on also used PGO at one point. Our software is roughly 2-3 million lines of C++ code, and links with 40-50 external libraries. Under Window XP, VS2008 could not link our product with PGO turned on. Before you complain about the bloat of FF, please understand that PGO uses many many times more memory than just compiling a regular release build. This is a Visual Studio linker problem, not FF problem.

  13. Re:Last paragraph in the TFA is... confusing by EvanED · · Score: 5, Informative

    Also MSVC only has 32-bit binaries

    Clarification: the only linker for 32-bit targets is, itself, 32-bits.

    The linker which targets 64-bit Windows is still 64 bits. If they had a 64-bit-to-32-bit cross compiler (they have the reverse, but not 64-to-32) that would solve Mozilla's problem. (Well, for some definition of "their problem.")

  14. Re:VS 2005? by Tridus · · Score: 5, Insightful

    Seems ironic that the FF team is using stuff from seven years and two major versions ago while at the same time bemoaning that anybody might want to keep a version of Firefox for more then 6 weeks - especially enterprise users.

    Interesting how they don't practice what they preach.

    --
    -- "So they told me that using the download page to download something was not something they anticipated." - Bill Gates
  15. Re:whose bloat by ohnocitizen · · Score: 5, Funny

    I don't know if the blame is so easy to place. It is kind of like watching one fat guy explain to another why they can't sit on the same bench.

  16. Re:whose bloat by Talderas · · Score: 5, Funny

    Can we turn this into a competitive sport? Please?

    --
    "Lack of speed can be overcome. In the worst case by patience." --Znork
  17. Re:Oh, now we admit it is getting bloated by PsychoSlashDot · · Score: 5, Insightful

    Requirements to run Firefox and resources required to compile Firefox aren't the same thing at all. Your attempt to conflate the two suggests you're stupid and don't know what you're talking about.

    I'd say it nicer but then I'd risk not hitting +5 Funny.

    --
    "Oh no... he found the .sig setting."
  18. Re:Wow by plover · · Score: 5, Insightful

    No, it has nothing to do with running Firefox. It has everything to do with running Visual Studio's linker.

    This matters only to Firefox developers.

    Not that they shouldn't care, mind you, as that is some seriously monolithic code. But it won't make any difference to Joe Sixpack.

    --
    John
  19. Re:VS 2005? by jlebar · · Score: 5, Informative

    Point taken.

    But FYI: We use VS2005 because it's the last version whose CRT supports Windows XP before SP2 and Windows 2000.

    We would love to upgrade, and are in fact devoting a lot of engineering time towards figuring out if we can upgrade while maintaining compatibility.

  20. Its the compiler, stupid. by DrYak · · Score: 5, Informative

    Or just, yknow, stop running a bloated resource hog of an INTERNET BROWSER.

    Read TFA agin. Oh, I know this is /.
    So read the summary again.

    it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB

    It is the compiler which is having ressource problems. The profile-guided optimiser needs more than 3GB to be able to do its optimisations. And apparently, the Windows its running on can't do PAE to use more than 3GB neither.

    --
    "Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
    1. Re:Its the compiler, stupid. by cjb909 · · Score: 5, Informative

      It is the compiler which is having ressource problems. The profile-guided optimiser needs more than 3GB to be able to do its optimisations. And apparently, the Windows its running on can't do PAE to use more than 3GB neither.

      PAE allows 32-Bit computers to use more than 4GB of ram, but it doesn't allow Windows to assign more than 3GB to any single process.

  21. Re:Time to move on, perhaps? by plover · · Score: 5, Informative

    No, you missed a fact.
    Visual Studio 2005 is a 32-bit app on any Windows platform.
    Visual Studio 2010 is a 32-bit app when running on a 32-bit platform.
    Mozilla builds on 32-bit platforms can no longer support the PGO linker.

    Visual Studio 2010 is a 64-bit app on a 64-bit platform.
    Mozilla builds on 64-bit platforms can PGO link just fine.
    When Visual Studio 64-bit compiles a 32-bit app, that app can run only on XP SP2 or later.
    Mozilla has millions of users on pre-XP SP2 platforms.

    So Mozilla has a choice: change nothing but stop PGO linking the 32-bit versions (sub-optimum for 32-bit users), go forward on a 64-bit only path and disenfranchise the old users from getting any new functions, abandon them completely (which is irresponsible in terms of security), cut back on new features for all, or take an axe to the existing code. Only one is an easy choice.

    --
    John
  22. Too funny reading these comments by caseih · · Score: 5, Insightful

    Folks are crawling all over each other to show how ignorant they are. "I ditched firefox for Chrome cause it's lighter!" only to ignore the fact that Chrome also has the same problem with the PGO thing running out of RAM so they don't even bother with trying those optimizations anymore. "Geez Firefox needs more RAM than the kernel to compile? Something's wrong!" Yes if the Linux kernel was built with PGO on VS 32-bit it probably would run out of RAM there too. Then there's the guy that claims this PGO problem is evidence that the Firefox devs need to go back to remedial school. I'm sure he could do it far better (and avoid the PGO linking optimization running out of memory too!).

    Hilarious reading. At least I choose to laugh rather than cry at people's inability to read and understand the issues here.

  23. Re:Time to move on, perhaps? by jlebar · · Score: 5, Informative

    I am a Mozilla guy.

    There's an official 64-bit version for Linux. We've been shipping that since before I can remember. There are also nightly builds for 64-bit Windows, but we're not shipping these even as Aurora at the moment.

    64-bit Linux isn't listed on most of our download pages. I'd argue it should be there, but I'm not in charge. :)

    Anyway, here are links to get all the builds we produce:

    Nightly builds: http://nightly.mozilla.org/ (has win-64 builds)
    Aurora builds: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/
    Beta builds: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/9.0b6-candidates/build1/ (I don't know the directory for the latest beta build, unfortunately, so you'll have to update this URL each time you go looking.)
    Release builds: http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/