IE9, FF4 Beta In Real-World Use Face-Off

An anonymous reader writes "Most browser benchmarks are isolated, artificial tests that can be gamed by browser vendors optimizing those specific cases. With only those benchmarks to go on, the folks at LucidChart were skeptical that the IE9 beta would actually outperform other modern browsers in real-world applications. To separate hype from reality, they built their first browser benchmarking tool, based in LucidChart itself. This benchmark is to SunSpider what a Left4Dead 2 benchmark is to 3Dmark Vantage. Product specs don't matter, only real-world performance on a real-world application. The results were surprising. IE9 held its own pretty well (with a few caveats), and the latest Firefox 4 beta came in dead last."

Comments

Too late for a film at 11 joke...

[LostCluster]

Browser makers design for tests, real world data shows exact opposite results to what you expect. We've got a crew working on the story overnight and will have a full update for you on the weekend edition of Wicked Early News, we start before normal people wake up.

Re:Too late for a film at 11 joke...

[beelsebob]

Actually, these results look very similar to the ones the WebKit guys (both chrome and safari teams) publish. They're almost always saying chrome and safari are similarly fast in the lead, firefox lags slightly, and IE8 is way slower... These are the same results as here. It just appears that IE9 is now added to the pile, and added at the top.

So what have we learned
1) Mozilla are good at lying about benchmarks (actually, we already knew that, they've been claiming the next big firefox release would be faster than everything for a while now)
2) IE9 is quick

The question is... is IE9 correct. I'll take works correctly but takes time over doin it rong quickly any day.

Re:Too late for a film at 11 joke...

[h4rm0ny]

I'm not so quick to forget why I dropped IE in the first place. To get away from ActiveX and general apathetic browser security. Where's that represented in benchmark?

Well, that's more specifically why you dropped IE 5.5 and IE6. By the time you get to IE8, these aren't really issues and Firefox has traded its early security-consciousness for usability. And it looks from those charts that IE9 runs pretty fast. That lines up with my own experiences of the beta. Though it's worth keeping in mind that the IE9 is a beta so it's not 100% fair to draw conclusions until it goes to release.

The most interesting things in the article aren't in TFS, though. One is that javascript processing is now apparently so fast that it's dwarfed by the time it actually takes the browser to render an updated page. Another is that in order to get the results from Firefox that they did, they actually had to drop a number of anomalous results where it ran vastly more slowly for unknown reasons. I'm not surprised as Firefox has been getting fatter and fatter ever since 3. The third is that Chrome blows everything else out of the water. They used an old version of Opera which is a shame as I have the newer one and my anecdotal impression is that it's snappier than the previous one so it would have been worth using the latest Opera. But still, Chrome apparently renders far faster than both IE9 and FF. I'd love to know why. Is it just that it's a new project designed from scratch without the cruft that other browsers come with? Does it lack support for significant functionality? Has there been a lot of low-level optimization?

I think in a years time, the real battle is going to be between IE and Chrome. Even today I mainly only use FF for web-development due there being some excellent development add-ons for it that IE can't compete with. Opera is a nice general browser and my default, but its cookie management is shit. Much of FF's funding comes from Google, who I presume would want to push Chrome and who I guess fund Firefox as a means of keeping Microsoft from re-establishing browser-dominance. But that may not be sufficient reason to really push it to be the best it can be, just to keep it "good enough".

I don't wish to be disrespectful to the FF developers. I know how complicated a code-base that size is and they're to be commended on producing a browser that serves well. But it's become a big, unwieldy beast in comparison to the leanness of Chrome and the new IE.

Re:Too late for a film at 11 joke...

[FunPika]

And on this very site the exact article you linked to was dismissed as being pro-Microsoft FUD. And I ran a couple Google searches and I still saw little more than FUD from Microsoft sponsored research or security companies grouping vulnerabilities from all Gecko based browsers as "Mozilla" (which as we all know the average person will read as meaning "Firefox").

Re:Too late for a film at 11 joke...

[Bert64]

The prevalence of firefox has helped the security of IE considerably, for a number of reasons...
Before they started losing market share, MS had no intention of improving IE at all.. Also now that no browser has over 90% market share they become far less attractive targets for malware authors, who instead now either target specific areas (eg ie6 is still huge in corporate settings) or other software which has a huge market share (adobe flash/pdf, msoffice).
So long as there are a handful of browsers out there competing with each other.
Were it not for firefox, ie7 would be a slightly warmed over ie6, and would still be getting attacked on a daily basis and users would have nowhere to go.

Incidentally, those benchmarks show ie9 coming in 3rd from last, only firefox 4 beta (by a small margin) and ie8 (by a huge and laughable margin) are slower. Actually released browsers such as firefox 3.6, safari 5, chrome 6 and opera 10 are all ahead of both betas of ie9 and firefox 4.

Now as for why the firefox 4 beta is slower than 3.6, it could be compiled in a debug mode (as betas often are), its new javascript engine has only just been integrated and needs tweaking etc...

The fact that current beta versions of both ie9 and firefox 4 are way behind current non beta versions of other browsers is rather poor, and you would hope that both will be fixed by the time their final releases come around.

As for Google, they make their money from the web and generally don't care which browser you use to access it, so long as that browser is fast enough to deliver a reasonable experience and standard enough so it doesn't force them to spend a lot of time kludging around. Older versions of IE were a threat to google as they made their web based apps appear slow, new versions appear better however IE is still controlled by google's biggest competitor who, given the chance, would use it against google in any way they can. When it comes to firefox/opera/chrome/safari i doubt google really cares which you use.

Re:Too late for a film at 11 joke...

[Anonymous Coward]

And on this very site the exact article you linked to was dismissed as being pro-Microsoft FUD.

You are kidding, right? On this site everything remotely positive to Microsoft is routinely dismissed as pro-Microsoft FUD and paid astroturfing.

Not that it will matter, but here is another report, this time from Secunia: http://blogs.computerworld.com/report_firefox_is_the_worlds_most_vulnerable_browser

and a newer one from Secunia: http://www.infoworld.com/t/browsers/internet-explorer-deemed-least-vulnerable-browser-593

Real test?

[haystor]

Unless the test contains porn and all the accompanying popups, it's not a real world test.

Re:Real test?

[gerardrj]

Isn't a popup the point of porn?

no sir, i didn't like it.

[gandhi_2]

No native Gopher support?

From my cold, dead hands!

Re:Maybe time to move to Chrome?

[FutureDomain]

Try one of the Chrome forks, such as ChromePlus, SRWare Iron, Comodo Dragon, or a pure Chromium build. They're just like Chrome, but without the questionable Client-ID and RLZ modules that Google put in Chrome. I typically use ChromePlus since it has several features that I like builtin, but I've been trying the IE 9 beta and I like that as well. It's faster than Firefox in my opinion and I absolutely love the UI layout.

Speaking as someone that switched to OS X

[aussersterne]

from Linux this month after using Linux since 1993, I think this applies to all of FOSS.

Somehwere around 2000-2006 FOSS was basically head-and-head with commercial software in practical usability and maintainability, with its own distinct advantages and a relatively small learning curve.

Then there was this veer into "if you ever want all the Windows users to switch..." thinking, and in an effort to eliminate the learning curve FOSS threw away pretty much all of its advantages as well. If FOSS is just Windows/Mac OS/IE by another name, why choose FOSS?

Particularly when Windows/Mac OS/IE win on the polish, compatibility, and accessibility fronts by virtue of their being cathedral-built software?

With Firefox slow and cumbersome, Thunderbird choking on Gmail IMAP continuously while Apple's Mail.app sails along happily, and KDE4/GNOME3 being emblematic of the many ways in which FOSS has lost its way, I just decided I'd had enough of the nonsense. I'm ready to be able to walk into Best Buy, purchase any device, and expect that it will work seamlessly with the current generation of computing devices, without options, without Bugzilla (and condescendingly dismissive developer retorts), and without lots of consulting Google to find out how the gconf infrastructure has changed in the last two years or how HAL has been replaced by DeviceKit or policies moved from /etc tree A to uneditable dynamic filesystem B (but just use this easy command line management tool to set options...)

It just plain saves me a boatload of time and headache to use something else, like OS X plus Google apps plus Chrome. The pending desktopization of FOSS has fizzled thanks to the politics of the bazaar.

Re:Speaking as someone that switched to OS X

[Moridineas]

Hear, hear. I'm right there with you.

As an example along similar lines, a user at my office reported a bug in Thunderbird to me. I tested and found it was definitely a Thunderbird bug. I made a test case file and submitted it to Bugzilla. A few days later my reported bug is deleted, to be merged with the same bug report from *2005*

Nobody who works on Thunderbird felt like working on the bug. It's not a sexy bug, probably doesn't hit too many people, and has work arounds...so it's stayed in the software for ~6 years.

And yeah I know, I should go in and fix it myself. Maybe one day I will. In the meanwhile I'll keep using Mail.app and I'll move more users over to new versions of Outlook that actually seem somewhat decent, and we'll go from there.

Is there any quality email app for Windows??

Re:Speaking as someone that switched to OS X

[guanxi]

I made a test case file and submitted it to Bugzilla. A few days later my reported bug is deleted, to be merged with the same bug report from *2005*

Nobody who works on Thunderbird felt like working on the bug. It's not a sexy bug, probably doesn't hit too many people, and has work arounds...so it's stayed in the software for ~6 years.

And yeah I know, I should go in and fix it myself. Maybe one day I will. In the meanwhile I'll keep using Mail.app and I'll move more users over to new versions of Outlook that actually seem somewhat decent, and we'll go from there.

How have Apple and Microsoft handled your bug reports for Mail.app and Outlook? Did the handle them like Mozilla, where you enter the bug directly in their internal bug databases, monitor the progress, participate in discussions with the developers, and even contribute development yourself? Or do you have no idea what the status is, no influence on the outcome, and no ability to contribute at all? Were the bugs even submitted to development? Were you able to find a way to submit them to Apple and Microsoft at all -- could you communicate with anyone beyond level 1 end user support technicians?

Every application has bugs as old as its first release-- have you seen the age of some Windows security vulnerabilities, going back over a decade? -- and your particular concern won't necessarily get fixed. But if you compare the experience of handling end user bugs at Mozilla with the same thing at Apple or Microsoft, well, there really is no comparison.

Re:Speaking as someone that switched to OS X

[Kjella]

Let me see if I understand your complaint - a rare bug with workarounds is given extremely low priority and that's indicative of a general problem with FOSS software? What do you propose as a change to the FOSS model of development to improve the engineering?

Well, first here we have a user. He sure as hell wouldn't have found the workaround. Then it gets reported to an administrator, who can't find any known bug (assuming he searched, but I would first) who then spends time building a test case. Then it goes to a bug triager who knows about five year old bugs and can identify this as a dupe and provide a workaround.

1. 99% of users would never find that workaround
2. After that effort by three people, nothing will really be done to fix it

Even better are the projects that drive down bug count by attrition, WINE is rather notorious for this I've noticed. Every few WINE versions - who are on a biweekly schedule - they'll ask you to retest even though there's been no patches towards the bug, so when you grow tired of that shit the bug is "solved".

As for rarity, any bug that doesn't happen on a developer's machine for a developer's use case is by assumption rare unless a real shit storm of complaints prove otherwise. Don't get me wrong, it's pretty much central to OSS that people fix the stuff they wanted fixed, no bug is too obscure if you're willing to fix it yourself. But it also very much creates an A-list and a B-list of bugs to get fixed, and you go in B. It's actually slightly easier with commercially paid support who don't really have an agenda of their own, what the customers most want fixed is their priority.

Of course no, a single anecdote from single project doesn't prove anything. Sometimes I've had good experiences, but also many bad. Most annoying ar those where a bug report only leads to more and more work until it far outweighs my interest in fixing the bug. I'm guessing that's often the case for the OSS developer on the other side of the table too, which is why he is pushing 90% of the work right back at me. Often the choices then end up: a) keep using buggy software, b) spend way too much time getting it fixed or c) buy something that works.

I wish there was more of an organized bounty system, you'd pledge towards some bug fix or feature request, then developers could take it on. The pledges would be kept by a trust until the developer claims to be done (or withdraw), then the pledgers vote to confirm / claim incomplete / reject that it has been done. You'd probably need to have some sort of arbitration system to deal with formal disputes, who could take a processing fee off the pledge. Combine that with an eBay-like reputation system and I think it should work out well without too much hassle. I know it sounds a little like rent-a-coder and I don't mean it like that, more of an organized way of doing small custom work inside existing projects.

"Real World"?

[farnsworth]

I'm not sure what is "real world" about spinning a UML box around another UML box in a giant (presumably) canvas-based javascript app.

For me, "real-world" means: is gmail fast enough? is opening a new tab fast? is image rendering fast enough? is html video fast enough? is the occasional embellished html5 animation fast enough? is typing into the address bar fast enough?

I'm sure their diagramming app is cool and everything, but I don't think I've ever seen anyone use anything like it, so I'm not sure what is "real world" about using it for a benchmark.

They even said that they altered the test in the middle to fix IE's performance problem. Come on.

Broken time measurements of the inter-frame time

[BZ]

The way this benchmark measures "intra-frame time" is broken. In particular, it uses a setInterval with a 1ms delay. No browser actually respects that 1ms. Chrome clamps it to 5ms; others clamp it to 10ms, all to avoid the website thrashing the CPU pointlessly.

The upshot is that Chrome's interframe delay in the graph is about 5ms and Firefox 3.6's interframe delay is aboug 10ms. Which this particular benchmark can't tell apart from "no delay at all", given its methodology.

Firefox 4 beta, IE9 beta, Safari, and Opera seem to have delays greater than 10ms, so they're clearly doing some work they can't finish in 10ms.... or have slightly buggy timer implementations. Or both.

Of course in practice frame rates above 60fps or so are pointless since the screen doesn't redraw that often. ;)

On the other hand, on Mac, on modern hardware, I get 4.5fps in Chrome 7 dev on a random trial document I just tried, with JS render tiems on the order of 7ms (with a 7ms standard deviation) and "intra-frame time" of 224ms with a 900ms standard deviation (yes, those numbers are nuts). Firefox 4 beta comes in at about 11s for the JS (with 3ms stddev) and 125ms for the "intra-frame time" (with a claimed stddev of 0, which looks really suspicious).

It'd be nice if there were non-obfuscated source for this benchmark so its number-crunching could be evaluated; that 0 stddev is ... highly improbable.

Re:dev IE9 and dev FF vs release Chrome?

[icebike]

Me too.

Its odd how both the summary above and the linked article sort of over look the fact that Chrome just blew the doors off of every other browser and the compared the production version to the latest and greatest of the others.

Chrome really deserves top billing, but the story is about the who is going to come in dead last.

Yawn.

Bullshit Slashvertisment

[phantomcircuit]

This benchmark can be run by anyone in LucidChart. First, sign up for a free account here.

Nuff said

Odd definition of "dead last."

[whoever57]

TFA (yes, I actually read it) says: "Firefox 4.0 Beta 6 came in behind all other browsers except for IE8". That's quite different from "dead last".

Re:oh, come on.

[feepness]

(gmail passwords are now randomly rejected in FF, but work in any other browser).

Yeah, that would be the phishing malware screwing with you.

Who cares?

[Goonie]

I use Firefox and IE regularly, have played with Chrome, and occasionally use Safari on the Macs at work.

I honestly can't notice any difference between any of them in rendering speed.

99.99% of the time, web browsing performance is network-limited anyway.

Surely standards support and browser stability are more important features, at least on platforms with more grunt than an iphone?

Re:dev IE9 and dev FF vs release Chrome?

[moderatorrater]

Modern browsers are so fast that the difference is miniscule. If you're looking at using IE6 or using Chrome, then obviously Chrome needs to be praised. If you're comparing several browsers that are all fast enough that there's no strong difference between them in real world use, then mocking the loser is just more fun :D

Re:dev IE9 and dev FF vs release Chrome?

[Tumbleweed]

They forever screwed themselves because they used to charge people to use it. When there's a ton of competition, and it's free? Yeah.

So did Netscape, which eventually turned into Firefox. I don't think that has anything to do with it. Opera has great technology, and no understanding of what most users want in a (default) user interface. They also seem to have no clue that most users never change default settings, so it doesn't matter how configurable it is.

useless benchmark, horrible summary

[macshit]

I'm not sure how they get off calling this a "real world benchmark", as it seems to bear almost no resemblance to what people normally use web browsers for: "The benchmark works by simply dragging a part of the diagram around the page for five seconds." WTF?

It certainly doesn't seem to be any more useful than the other browser benchmark being touted these days, and arguably it's much less useful, because it measures a single very narrow aspect of browser operation, one which has little connection with typical browser usage.

Moreover, the slashdot summary seems to go to great lengths to emphasize how "badly" FF4 did on this (useless, remember) benchmark, and to pump up IE9: "The results were surprising. IE9 held its own pretty well (with a few caveats), and the latest Firefox 4 beta came in dead last" -- but if you actually look at the results that emphasis is misplaced: almost all the browsers were quite close to each other, with a few outliers, but in no cases was FF4 an outlier, and indeed was pretty much identical to IE9 (on this test).

The only clear result I can see is: When doing a certain very specific type of javascript rendering, most modern browsers have pretty much identical performance, though chrome's particularly fast, and IE8 particularly slow.

Of course, that isn't very interesting to anybody except LucidChart users, of course, nor very likely to generate any controversy...

Fuck this shit

[Massacrifice]

Really, fuck it. I've had it with corporate-sponsored dick-fighting contest about which browser is the fastest. I really, really couldn't care any less. Features, openness, security, standards compliance, yeah. But If I want a fast app, I'll go native, thank you. Maybe I'm too old, but I've always thought HTML sucked as a programming paradigm. As an information distribution mechanism, sure. But for interactivity? Please. It's about time somebody called bullshit on this. Hell, a goddamn Visual Basic app from fifteen years ago kicked the butt of most modern web sites in usability, performance and ease of maintenance. The only thing that makes the web so attractive is the barrier to entry : free, nothing to install, immediate access to the average brains of millions. Just like TV. No thanks for dumbing it down to this. And now you wanna make it faster? Piss off. Go write real code that does something, not just another abstraction layer.

Re:Fuck this shit

[Gamer_2k4]

Meanwhile, in the real world, people actually want the source of all their information to be fast. Go figure.

Re:Maybe time to move to Chrome?

[satuon]

Alternatively just go to the Options page in google's Chrome version, and uncheck "Use a suggestion service to help complete searches and URLs typed in the address bar". Crome stops the 'snooping' then.

... and even older Opera?

[TeXMaster]

The versions used for tests were quite debatable. For IE and FF they put up both the final version and the current beta, while for the other browsers they only show the results for the current release ... mostly: the latest released Opera version is 10.60. A fair comparison should have included Chrome 7 and Opera 10.70 and replaced Opera 10.53 with 10.60.

I guess the actual selection of versions shows how the point of the article was more about bashing FF4 compared to IE9 (in which it also failed, given the very small difference between them) rather than doing a honest comparison of all of the browsers.