The search team at Google doesn't consider FF a competitor, probably. Other teams at Google (e.g. the ones trying to create Chrome-only content silos or actively creating and maintaining WebKit-only web content), it's not clear.
I wouldn't treat "Google" as a terribly monolithic entity for purposes of claims of what is or is not considered.
> Chrome has moved to pushing HTML5 > implementation quality and performance
Chrome has moved to pushing certain things Google cares about and labeling those as "HTML5". Which is not quite the same thing. There are plenty of things that Chrome is not implementing and not planning to implement that other browsers support and that might be good for the web platform. But Google happens to not care about them.
> not to lock people into Chrome, but, again, to > make the web a better platform
Seriously? Then why is the Angry birds "chrome app" being marketed as Chrome-only? Why is Chromebook-specific content being encouraged? Why do Google engineers create Google web apps that only work in WebKit?
I think you described "Google's" general viewpoint on web stuff as of 2 years ago or so. It's changed a lot since then, sadly.
> The only surprising thing about this move was that > MS didn't outbid Google
The default browser setting in Firefox depends on what users want, not on who pays the most money.
This is why Google is not default in the Russian localization of Firefox, for example (Yandex is), and why Google wasn't the default in the Chinese localization for a while until their Chinese search results got better.
For what it's worth, people seem to consistently underestimate how many people it takes to build a web browser. Opera had over 700 employees as of Feb 2011 (see http://my.opera.com/haavard/blog/2011/02/01/decade ), for example. Other browser makers are at similar or bigger headcounts, almost certainly.
> I think you'll find we measure compatibility by how > much special care we need to coax a given > browser to do what it's told.
But that measure presumes a certain set of things that you're telling the browser, and in practice that set is most affected by the marketing that browser developers do and the things they push web developers to try. Web developers only think to try things that they've been told work in some browser; if a browser implements a useful feature that doesn't get sufficiently hyped, web developers tend to not find out about it and tend to not try using it and so don't realize other browsers are missing it. Gecko's support for SVG filters on non-SVG content is an interesting example.
So what you're telling me is that WebKit has the best marketing department at the moment. You're right; it does. That doesn't make it most-compatible; it just means that web developers don't even realize how limited they're being by WebKit's limitations and what they could do if they could do things other browser support but WebKit doesn't.
Or in other words, by your definition the most "compatible" browser is always the one with the best advertising department and the best state for the web is for that browser to have a monopoly so you can just target it (using the canned snippets the advertising department provides, more often than not) and not have to worry about anything else....
I can see how that situation might make web developers' lives easier in the short term, and indeed it did back in 2000-2001 or so. The browser they targeted then was IE6, and it supported all sorts of things others did not (while lacking support for other things, but the advertising was heavily slanted IE-wards). As I said, easy short-term, but long-term life starts to suck.
I have no idea what your issue was with sentences, but looks like WebKit has finally fixed enough of the bugs specifically in the test suite to be ahead of Presto, but still behind Gecko and Trident.
Let me know when they fix their broken selector handling, ok?
And the worst CSS2.1 implementation out of the 4 major browser engines, if you look at the W3C CSS2.1 test suite.
I'm not sure how you're measuring "compatibility" here. Are you counting claimed-to-be-implemented features or are you counting correctly-implemented features?
You keep missing the fact that there are benefits to having as much code as possible in a single library in this case. So the first time, the address space available was increased. The second time, that will happen as well, and some code was moved out of the library to give time for that to happen.
Long-term, a different solution is needed, of course. At that point, the costs of splitting the code out may have to be borne after all (not the cost of doing the work, but the cost of the resulting code being slightly slower).
> Does it really need PGO to speed up a function call?
Depends on how often it has to be called. PGO can in fact speed up function calls (e.g. by inlining; MSVC with PGO will inline virtual methods in many cases).
> Or even if the display routines store up all links in a > page then send them to a history function which > returns all the ones that have been visited
That doesn't work for handling dynamic style changes (which includes style recomputation on any mutation to the DOM). It might be OK for completely static pages, but those are very rare on the web today.
> Of course, if such a small modification does make > a large performance hit, then you have to wonder > what the code paths are like.
You don't have to wonder. The consistent thing with any style system related code is that it will get run a _lot_, because the number of times it's run is O(number of nodes * number of mutations * number of selectors on the page). Those numbers are typically order of "thousands", "thousands", and "thousands" for normal pages, and all three end up in the tens to hundreds of thousands for complicated pages. Browsers do what they can to avoid running this code when possible, but even then it's still very hot code.
> Why in God's name would you need more than one > computer to be automatically doing the nightly > build
First of all, there are 4 different nightly builds on Windows (optimized and debug, 32-bit and 64-bit builds).
But we're not talking just nightly builds. There are per-checkin builds and test runs, for all major browsers; otherwise at the end of the day you'd probably be in a world of hurt with test failures.
So for each checkin, there are two Windows builds created (debug and optimized) then run through a variety of test suites. And this happens not just for the main integration branch but also for various development branches. It also happens for changes people just want to test (there's a service where you can submit a patch and have it build the patch for you and run tests).
So the net result is that as of March 2011, there were about 44 build jobs being kicked off per hour according to http://oduinn.com/blog/2011/05/24/infrastructure-load-for-march-2011/ -- that's across Win32, WIn64, Mac32, Mac64, Linux32, Linux64, and Android, so figure about 18 builds per hour on Windows. Each build takes more than an hour, so you need more than 18 build machines to keep up. In practice, there are about 40 at the moment for Windows; you can see the list near the bottom of http://build.mozilla.org/builds/last-job-per-slave.txt
That's assuming that only the build machines, not the test machines, need to be upgraded, which is a good assumption in this case, I think.
> I don't suppose the huge web-display system needs > to be split into pieces
The web-display system is what's being linked here.
> but that's not that big an optimisation step as you'll > still get it inside each module
Historical performance data from back when there were in fact multiple modules involved says otherwise.
> The code that displays a window doesn't have to > be in the same module as the code that figures out > how to display the data stream.
Firefox uses the web display system to draw its window already. There is no separate code for the window to pull out.
> The bookmarks system could be a separate > module that's linked in at the end
This may be possible, yes. Though keep in mind that bookmarks and history is a single system in Firefox to some extent, and history is actually a high-performance part of the main html rendering pipeline due to the need to implement visited link coloring and selectors.
Well... when two different programs trying to implement the same set of functionality have the same issue with the linker, you _could_ assume that both development teams are incompetent. Or you could assume that the functionality involved takes a lot of code and the linker doesn't scale well. Your choice.
You _really_ need to actually go read about the issue here.
This is about needing that much ram with a particular set of very memory-hungry compiler options. You can build the browser in much less ram than that if you don't enable those particular compiler optimizations.
So as far being selectively literate... are you sure that's not you?
> And that is to make lots of parts be separately loaded > modules
This actually leads to pretty serious performance penalties because of the way that web specs tend to interdepend on each other. It also loses you optimization opportunities.
Pieces of functionality that are not interdependent with other stuff (e.g. audio and video backends, webgl) are in fact either in separate libraries or being moved there.
For the other stuff, there did use to be more modules. It turned out that in practice most users needed them at startup (to show the browser itself and restore their tabs), so the only thing multiple modules got you was a slower startup and slower runtime code.
It's not an accident that both Firefox and Chrome ship almost all their code in a single library (binary in the case of Chrome). It turns out that for web browsers specifically this works somewhat better than the alternatives.
Oh, I just checked and looks like Opera also also links all its code into a single library, at least on Mac.
And Safari links all the core WebKit code into a single library.
Not sure what IE does nowadays, but last I checked mshtml.dll in fact included all the actual browser bits.
Now it may be that all the people involved in all these projects can't design worth anything. Or maybe they did some measurements that you haven't done and found that this approach works better....
Except Chrome uses more resources, in my experience. And may or may not be faster.
Though I'm looking at it on Mac, not on Windows, where they're just compiled completely differently (e.g. Chrome is 32-bit while Firefox is 64-bit, etc).
You do realize that Chrome had this same issue a few years back, right?
Maybe you need to do some reading about how to engineer software and about the specific compiler options being used here and why they might cause a lot of memory usage?
> 1) What the hell are you doing with your code to be > that large?
How large? It's a few million lines of C++ code, just like every other browser. What it's doing is implementing all the stuff people want to do on the web.
> 2) What the fuck is your linker doing to do that?
Please read up on link-time code generation.
> 3) Why the hell didn't you see this coming and > prune LONG before you hit the 3Gb limit if you > already hit the 2Gb limit once already?
This is an excellent question that I too am asking.
> 4) What's the problem with compiling on 64-bit > computers only,
None, except updating a large build farm from 32-bit to 64-bit can't happen overnight. Needs some staging, testing, etc.
> You're honestly telling me that Firefox is more > complicated and needs more memory to compile > than, say, LibreOffice?
Have you tried to compile LibreOffice with LTO and PGO turned on?
> The Linux kernel?
Quite possibly. The kernel is C, not C++; C++ is a lot more of a pain for compilers to deal with.
The whole point of LTO is that you optimize your entire binary as a single object, no matter how your code is structured. It requires more memory, but can produce faster code because the compiler is able to make optimizations it can't make otherwise.
Whether that's "crappy" is a matter of your priorities, of course. 10-25% performance improvement tends to be a high priority for web browsers, though perhaps not for KDE or LibreOffice.
The "translation unit" involved here is the whole binary. We're talking about link-time code generation with profile-guided optimization, not regular compiles.
So it doesn't indicate much of anything about the Firefox source code other than general overall quantity of code being compiled...
It's been over a year since Chrome had to turn off PGO altogether and move to 64-bit builders even without PGO, because they ran into this same problem.
So maybe your issue is with the fact that all "browsers" as you call them are trying to do too much? They should drop the fast graphics and jits and video and audio support and all that jazz, right?
Slashdot Mirror
> Google doesn't consider FF a competitor.
The search team at Google doesn't consider FF a competitor, probably. Other teams at Google (e.g. the ones trying to create Chrome-only content silos or actively creating and maintaining WebKit-only web content), it's not clear.
I wouldn't treat "Google" as a terribly monolithic entity for purposes of claims of what is or is not considered.
> Chrome has moved to pushing HTML5
> implementation quality and performance
Chrome has moved to pushing certain things Google cares about and labeling those as "HTML5". Which is not quite the same thing. There are plenty of things that Chrome is not implementing and not planning to implement that other browsers support and that might be good for the web platform. But Google happens to not care about them.
> not to lock people into Chrome, but, again, to
> make the web a better platform
Seriously? Then why is the Angry birds "chrome app" being marketed as Chrome-only? Why is Chromebook-specific content being encouraged? Why do Google engineers create Google web apps that only work in WebKit?
I think you described "Google's" general viewpoint on web stuff as of 2 years ago or so. It's changed a lot since then, sadly.
> The only surprising thing about this move was that
> MS didn't outbid Google
The default browser setting in Firefox depends on what users want, not on who pays the most money.
This is why Google is not default in the Russian localization of Firefox, for example (Yandex is), and why Google wasn't the default in the Chinese localization for a while until their Chinese search results got better.
For what it's worth, people seem to consistently underestimate how many people it takes to build a web browser. Opera had over 700 employees as of Feb 2011 (see http://my.opera.com/haavard/blog/2011/02/01/decade ), for example. Other browser makers are at similar or bigger headcounts, almost certainly.
> I think you'll find we measure compatibility by how
> much special care we need to coax a given
> browser to do what it's told.
But that measure presumes a certain set of things that you're telling the browser, and in practice that set is most affected by the marketing that browser developers do and the things they push web developers to try. Web developers only think to try things that they've been told work in some browser; if a browser implements a useful feature that doesn't get sufficiently hyped, web developers tend to not find out about it and tend to not try using it and so don't realize other browsers are missing it. Gecko's support for SVG filters on non-SVG content is an interesting example.
So what you're telling me is that WebKit has the best marketing department at the moment. You're right; it does. That doesn't make it most-compatible; it just means that web developers don't even realize how limited they're being by WebKit's limitations and what they could do if they could do things other browser support but WebKit doesn't.
Or in other words, by your definition the most "compatible" browser is always the one with the best advertising department and the best state for the web is for that browser to have a monopoly so you can just target it (using the canned snippets the advertising department provides, more often than not) and not have to worry about anything else....
I can see how that situation might make web developers' lives easier in the short term, and indeed it did back in 2000-2001 or so. The browser they targeted then was IE6, and it supported all sorts of things others did not (while lacking support for other things, but the advertising was heavily slanted IE-wards). As I said, easy short-term, but long-term life starts to suck.
I have no idea what your issue was with sentences, but looks like WebKit has finally fixed enough of the bugs specifically in the test suite to be ahead of Presto, but still behind Gecko and Trident.
Let me know when they fix their broken selector handling, ok?
> have deep HTML5/CSS3 compatibility
And the worst CSS2.1 implementation out of the 4 major browser engines, if you look at the W3C CSS2.1 test suite.
I'm not sure how you're measuring "compatibility" here. Are you counting claimed-to-be-implemented features or are you counting correctly-implemented features?
Can you link to the relevant bug, please? That sort of thing is not OK...
Well, sure. Except the code is compiling just fine as of yesterday, after a few changes to it. Tempest in a teapot and all.
The XUL user interface is written in JS and XUL, not C++, so isn't part of the library in question to start with.
The extension infrastructure is almost all written in JS, so isn't part of the library either.
You keep missing the fact that there are benefits to having as much code as possible in a single library in this case. So the first time, the address space available was increased. The second time, that will happen as well, and some code was moved out of the library to give time for that to happen.
Long-term, a different solution is needed, of course. At that point, the costs of splitting the code out may have to be borne after all (not the cost of doing the work, but the cost of the resulting code being slightly slower).
The first of those is what's being done, of course. What made you think it's not?
> Does it really need PGO to speed up a function call?
Depends on how often it has to be called. PGO can in fact speed up function calls (e.g. by inlining; MSVC with PGO will inline virtual methods in many cases).
> Or even if the display routines store up all links in a
> page then send them to a history function which
> returns all the ones that have been visited
That doesn't work for handling dynamic style changes (which includes style recomputation on any mutation to the DOM). It might be OK for completely static pages, but those are very rare on the web today.
> Of course, if such a small modification does make
> a large performance hit, then you have to wonder
> what the code paths are like.
You don't have to wonder. The consistent thing with any style system related code is that it will get run a _lot_, because the number of times it's run is O(number of nodes * number of mutations * number of selectors on the page). Those numbers are typically order of "thousands", "thousands", and "thousands" for normal pages, and all three end up in the tens to hundreds of thousands for complicated pages. Browsers do what they can to avoid running this code when possible, but even then it's still very hot code.
> In practice, there are about 40 at the moment for
> Windows
Looks like my information is out of date. From one of the people who maintain these machines:
catlee> we have ~70 win32 builders
catlee> and ~30 win64
> so figure about 18 builds per hour on Windows.
Actually, more like 12.5. Arithmetic fail on my part.
> Why in God's name would you need more than one
> computer to be automatically doing the nightly
> build
First of all, there are 4 different nightly builds on Windows (optimized and debug, 32-bit and 64-bit builds).
But we're not talking just nightly builds. There are per-checkin builds and test runs, for all major browsers; otherwise at the end of the day you'd probably be in a world of hurt with test failures.
So for each checkin, there are two Windows builds created (debug and optimized) then run through a variety of test suites. And this happens not just for the main integration branch but also for various development branches. It also happens for changes people just want to test (there's a service where you can submit a patch and have it build the patch for you and run tests).
So the net result is that as of March 2011, there were about 44 build jobs being kicked off per hour according to http://oduinn.com/blog/2011/05/24/infrastructure-load-for-march-2011/ -- that's across Win32, WIn64, Mac32, Mac64, Linux32, Linux64, and Android, so figure about 18 builds per hour on Windows. Each build takes more than an hour, so you need more than 18 build machines to keep up. In practice, there are about 40 at the moment for Windows; you can see the list near the bottom of http://build.mozilla.org/builds/last-job-per-slave.txt
That's assuming that only the build machines, not the test machines, need to be upgraded, which is a good assumption in this case, I think.
> I don't suppose the huge web-display system needs
> to be split into pieces
The web-display system is what's being linked here.
> but that's not that big an optimisation step as you'll
> still get it inside each module
Historical performance data from back when there were in fact multiple modules involved says otherwise.
> The code that displays a window doesn't have to
> be in the same module as the code that figures out
> how to display the data stream.
Firefox uses the web display system to draw its window already. There is no separate code for the window to pull out.
> The bookmarks system could be a separate
> module that's linked in at the end
This may be possible, yes. Though keep in mind that bookmarks and history is a single system in Firefox to some extent, and history is actually a high-performance part of the main html rendering pipeline due to the need to implement visited link coloring and selectors.
Well... when two different programs trying to implement the same set of functionality have the same issue with the linker, you _could_ assume that both development teams are incompetent. Or you could assume that the functionality involved takes a lot of code and the linker doesn't scale well. Your choice.
You _really_ need to actually go read about the issue here.
This is about needing that much ram with a particular set of very memory-hungry compiler options. You can build the browser in much less ram than that if you don't enable those particular compiler optimizations.
So as far being selectively literate... are you sure that's not you?
> And that is to make lots of parts be separately loaded
> modules
This actually leads to pretty serious performance penalties because of the way that web specs tend to interdepend on each other. It also loses you optimization opportunities.
Pieces of functionality that are not interdependent with other stuff (e.g. audio and video backends, webgl) are in fact either in separate libraries or being moved there.
For the other stuff, there did use to be more modules. It turned out that in practice most users needed them at startup (to show the browser itself and restore their tabs), so the only thing multiple modules got you was a slower startup and slower runtime code.
It's not an accident that both Firefox and Chrome ship almost all their code in a single library (binary in the case of Chrome). It turns out that for web browsers specifically this works somewhat better than the alternatives.
Oh, I just checked and looks like Opera also also links all its code into a single library, at least on Mac.
And Safari links all the core WebKit code into a single library.
Not sure what IE does nowadays, but last I checked mshtml.dll in fact included all the actual browser bits.
Now it may be that all the people involved in all these projects can't design worth anything. Or maybe they did some measurements that you haven't done and found that this approach works better....
File a bug on Microsoft to do that? Might happen.... sometime.
Ah, indeed. Thanks for the clarification!
Except Chrome uses more resources, in my experience. And may or may not be faster.
Though I'm looking at it on Mac, not on Windows, where they're just compiled completely differently (e.g. Chrome is 32-bit while Firefox is 64-bit, etc).
You do realize that Chrome had this same issue a few years back, right?
Maybe you need to do some reading about how to engineer software and about the specific compiler options being used here and why they might cause a lot of memory usage?
> 1) What the hell are you doing with your code to be
> that large?
How large? It's a few million lines of C++ code, just like every other browser. What it's doing is implementing all the stuff people want to do on the web.
> 2) What the fuck is your linker doing to do that?
Please read up on link-time code generation.
> 3) Why the hell didn't you see this coming and
> prune LONG before you hit the 3Gb limit if you
> already hit the 2Gb limit once already?
This is an excellent question that I too am asking.
> 4) What's the problem with compiling on 64-bit
> computers only,
None, except updating a large build farm from 32-bit to 64-bit can't happen overnight. Needs some staging, testing, etc.
> You're honestly telling me that Firefox is more
> complicated and needs more memory to compile
> than, say, LibreOffice?
Have you tried to compile LibreOffice with LTO and PGO turned on?
> The Linux kernel?
Quite possibly. The kernel is C, not C++; C++ is a lot more of a pain for compilers to deal with.
The whole point of LTO is that you optimize your entire binary as a single object, no matter how your code is structured. It requires more memory, but can produce faster code because the compiler is able to make optimizations it can't make otherwise.
Whether that's "crappy" is a matter of your priorities, of course. 10-25% performance improvement tends to be a high priority for web browsers, though perhaps not for KDE or LibreOffice.
The "translation unit" involved here is the whole binary. We're talking about link-time code generation with profile-guided optimization, not regular compiles.
So it doesn't indicate much of anything about the Firefox source code other than general overall quantity of code being compiled...
Define "too much"?
It's been over a year since Chrome had to turn off PGO altogether and move to 64-bit builders even without PGO, because they ran into this same problem.
So maybe your issue is with the fact that all "browsers" as you call them are trying to do too much? They should drop the fast graphics and jits and video and audio support and all that jazz, right?