IE9, FF4 Beta In Real-World Use Face-Off
An anonymous reader writes "Most browser benchmarks are isolated, artificial tests that can be gamed by browser vendors optimizing those specific cases. With only those benchmarks to go on, the folks at LucidChart were skeptical that the IE9 beta would actually outperform other modern browsers in real-world applications. To separate hype from reality, they built their first browser benchmarking tool, based in LucidChart itself. This benchmark is to SunSpider what a Left4Dead 2 benchmark is to 3Dmark Vantage. Product specs don't matter, only real-world performance on a real-world application. The results were surprising. IE9 held its own pretty well (with a few caveats), and the latest Firefox 4 beta came in dead last."
Browser makers design for tests, real world data shows exact opposite results to what you expect. We've got a crew working on the story overnight and will have a full update for you on the weekend edition of Wicked Early News, we start before normal people wake up.
I'd like to see Chrome 7 results in there...
Unless the test contains porn and all the accompanying popups, it's not a real world test.
t
to be honest, firefox, the darling of the FOSS movement is starting to get a little tiring.
it is lagging wayyyy behind...and it's a pain in the ass to manage in fleets. honestly, there is no point to installing it in the enterprise anymore.
the old argument...unpatched days, well who gives a fuck. if users run unprivileged, firefox can't update anyway (and don't tell me to make the directory writable, what about registry edits?)
it's just worth it anymore.
from ff beta 6 of course
I have been thinking about using Chrome for some time, it seems faster and already has a respectable community around it. But I also would like to avoid google stuff and I'm already used to Firefox, not sure if it is worth the trouble.... Any opinions?
No native Gopher support?
From my cold, dead hands!
THL phish sticks
Frames per second seems like pretty much the opposite of "real-world" for how 99% of users use their browsers.
from Linux this month after using Linux since 1993, I think this applies to all of FOSS.
Somehwere around 2000-2006 FOSS was basically head-and-head with commercial software in practical usability and maintainability, with its own distinct advantages and a relatively small learning curve.
Then there was this veer into "if you ever want all the Windows users to switch..." thinking, and in an effort to eliminate the learning curve FOSS threw away pretty much all of its advantages as well. If FOSS is just Windows/Mac OS/IE by another name, why choose FOSS?
Particularly when Windows/Mac OS/IE win on the polish, compatibility, and accessibility fronts by virtue of their being cathedral-built software?
With Firefox slow and cumbersome, Thunderbird choking on Gmail IMAP continuously while Apple's Mail.app sails along happily, and KDE4/GNOME3 being emblematic of the many ways in which FOSS has lost its way, I just decided I'd had enough of the nonsense. I'm ready to be able to walk into Best Buy, purchase any device, and expect that it will work seamlessly with the current generation of computing devices, without options, without Bugzilla (and condescendingly dismissive developer retorts), and without lots of consulting Google to find out how the gconf infrastructure has changed in the last two years or how HAL has been replaced by DeviceKit or policies moved from /etc tree A to uneditable dynamic filesystem B (but just use this easy command line management tool to set options...)
It just plain saves me a boatload of time and headache to use something else, like OS X plus Google apps plus Chrome. The pending desktopization of FOSS has fizzled thanks to the politics of the bazaar.
STOP . AMERICA . NOW
...to ragequit again.
I'm not sure what is "real world" about spinning a UML box around another UML box in a giant (presumably) canvas-based javascript app.
For me, "real-world" means: is gmail fast enough? is opening a new tab fast? is image rendering fast enough? is html video fast enough? is the occasional embellished html5 animation fast enough? is typing into the address bar fast enough?
I'm sure their diagramming app is cool and everything, but I don't think I've ever seen anyone use anything like it, so I'm not sure what is "real world" about using it for a benchmark.
They even said that they altered the test in the middle to fix IE's performance problem. Come on.
There aint no pancake so thin it doesn't have two sides.
... it took Microsoft to finally produce a web browser that performs as well as other browsers were doing in their previous generations, and meanwhile the one they actually have in production (IE8) sucks eggs. So what else is new? Have they got a modern filesystem yet?
Netcraft has to confirm it first.
Firefox 4 beta 6 doesn't have the new JavaScript engine in it. Beta 7 will have it. But there's no particular need to wait for beta 7 as they could benchmark a nightly now. They also don't mention what kind of video card they've got in that laptop. IE9 and Firefox 4 can take better advantage of a good video card on Windows 7 than the other browsers tested and that may significantly influence a charting benchmark like this one.
I love how a difference of a few milliseconds (looks to be 5ms) means a browser "tanks" and its position, when compared to other browsers, can be described as "dead last." Oh no, we're not painting a bias picture here.
The results were surprising. IE9 held its own pretty well (with a few caveats), and the latest Firefox 4 beta came in dead last.
No. Not surprising at all... but I still prefer Firefox for everyday use.
"You want to know how to help your kids? Leave them the fuck alone." -George Carlin
It's a beta. Don't waste your breath.
Also, 2 charts?
I'd be much more interested to see it being done with the builds of FF 4 that have jaegermonkey enabled. Though that should be merged into the main branch fairly soon with any luck.
http://www.conceivablytech.com/2673/products/first-look-firefox-4-jaegermonkey/
Your country functions by having a huge cheap workforce of illegal immigrants so of course nobody in Federal Government is really going to do anything to stop that - not Democrat and not Republican. Also whoever wrote that stuff needs to learn how to use English and what a dictionary is for - "communist" means something completely different instead of just being a generic swear word.
The way this benchmark measures "intra-frame time" is broken. In particular, it uses a setInterval with a 1ms delay. No browser actually respects that 1ms. Chrome clamps it to 5ms; others clamp it to 10ms, all to avoid the website thrashing the CPU pointlessly.
The upshot is that Chrome's interframe delay in the graph is about 5ms and Firefox 3.6's interframe delay is aboug 10ms. Which this particular benchmark can't tell apart from "no delay at all", given its methodology.
Firefox 4 beta, IE9 beta, Safari, and Opera seem to have delays greater than 10ms, so they're clearly doing some work they can't finish in 10ms.... or have slightly buggy timer implementations. Or both.
Of course in practice frame rates above 60fps or so are pointless since the screen doesn't redraw that often. ;)
On the other hand, on Mac, on modern hardware, I get 4.5fps in Chrome 7 dev on a random trial document I just tried, with JS render tiems on the order of 7ms (with a 7ms standard deviation) and "intra-frame time" of 224ms with a 900ms standard deviation (yes, those numbers are nuts). Firefox 4 beta comes in at about 11s for the JS (with 3ms stddev) and 125ms for the "intra-frame time" (with a claimed stddev of 0, which looks really suspicious).
It'd be nice if there were non-obfuscated source for this benchmark so its number-crunching could be evaluated; that 0 stddev is ... highly improbable.
This benchmark can be run by anyone in LucidChart. First, sign up for a free account here.
Nuff said
I think if it were truly a test of "real world" use, they would find that all of the browsers have the same performance level and none of this matters.
TFA (yes, I actually read it) says: "Firefox 4.0 Beta 6 came in behind all other browsers except for IE8". That's quite different from "dead last".
The real "Libtards" are the Libertarians!
Firstly, the only thing including more samples in a test does is give you clearer results. It doesn't cost the tester that much time or money to simply run a web browser.
Furthermore, even if someone were to accept your claims and assertions, the matter is simply that the selection of browsers in the article covers all the actively developed rendering engines currently in use. No one would argue to include Seamonkey, Flock, or Galeon, even if they had a higher usage share than Opera, since Firefox already represents Gecko.
If anything, Safari or Chrome should be dropped, since they are both based on Webkit.
The thing with "real-life" benchmarks is that "based on Webkit" only gets you so far. Safari and Chrome use totally different JavaScript engines, for example. They use totally different drawing libraries. Heck, Chrome on Windows uses totally different drawing library than Chrome on Mac (which makes a difference in "real-life" benchmarks, since drawing is anywhere from 30% to 80% of the total benchmark time).
The less synthetic the benchmark the more silly details (exact browser, not just rendering engine, exact graphics driver version, exact Xorg version on Linux, exact graphics hardware, etc) start to matter...
Now, will IE9 scale with a faster GPU? I'd like to see some benchmarks with different video cards.
Life is not for the lazy.
I use Firefox and IE regularly, have played with Chrome, and occasionally use Safari on the Macs at work.
I honestly can't notice any difference between any of them in rendering speed.
99.99% of the time, web browsing performance is network-limited anyway.
Surely standards support and browser stability are more important features, at least on platforms with more grunt than an iphone?
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
He seem to have used the most recent build of all browser, but he uses a 6 month old built of Opera when 10.62 was release just a couple of weeks ago.
Doesn't seem fair to me.
But then again maybe he just didn't want Opera to best the fastest
The story doesn't mention the GPU in use, but it does mention it's an i7 processor. So I assume it's using an i965-class GPU. These aren't exactly known for speed or stability on linux. I believe FF4 uses Cairo, which in turn uses XRender, and my experience with integrated Intel GPUs and XRender is that pure software ( ie X on FBDev ) is faster. I would have liked to have seen a system used which could actually accelerate the drawing operations.
I read the article. I thought, "Hey, I should try Chrome". I try to install. I get "unknown installer error".
Color me impressed.
I don't respond to AC's.
The results were surprising. IE9 held its own pretty well (with a few caveats), and the latest Firefox 4 beta came in dead last."
Why is this surprising? Firefox has been the slowest of the current crop of browsers (IE8 excepted) for quite a while now.
... and then they built the supercollider.
According to TFA, FF4 was not "dead last", IE8 was. From thier charts, IE8 was far inferior to all the other browsers. Of course, you might be able to argue that IE9 is the new IE, so that wouldn't be fair, but IE8 is still the one being used. If you want to use the latest browser, what about the FF nightly builds? Once again, even with a bias, MS products don't beat the free ones.
And once again I see no mention of lag or latency behind everyday controls, one of the real factors which affect a user's perception of speed and responsiveness. The kind which gets them to say, "It feels faster, but I don't know how". I'm talking about switching between tabs, closing tabs, clicking the browser's back or forward button, and general UI navigation. You want hundredths of a second or less for these kind of actions.
Why OpalCalc is the best Windows calc
I'm not sure how they get off calling this a "real world benchmark", as it seems to bear almost no resemblance to what people normally use web browsers for: "The benchmark works by simply dragging a part of the diagram around the page for five seconds." WTF?
It certainly doesn't seem to be any more useful than the other browser benchmark being touted these days, and arguably it's much less useful, because it measures a single very narrow aspect of browser operation, one which has little connection with typical browser usage.
Moreover, the slashdot summary seems to go to great lengths to emphasize how "badly" FF4 did on this (useless, remember) benchmark, and to pump up IE9: "The results were surprising. IE9 held its own pretty well (with a few caveats), and the latest Firefox 4 beta came in dead last" -- but if you actually look at the results that emphasis is misplaced: almost all the browsers were quite close to each other, with a few outliers, but in no cases was FF4 an outlier, and indeed was pretty much identical to IE9 (on this test).
The only clear result I can see is: When doing a certain very specific type of javascript rendering, most modern browsers have pretty much identical performance, though chrome's particularly fast, and IE8 particularly slow.
Of course, that isn't very interesting to anybody except LucidChart users, of course, nor very likely to generate any controversy...
We live, as we dream -- alone....
Chrome is faster because it massively favors speed over customization and features. FF is slower because it favors customization, and assumes, correctly, that no one actually actually gives a flying fark if it needs slightly more than a 1/50th of a second to render a page that Chrome can do in 1/100th of a second. This isn't a problem, nor is it news. Now of course, you may do the occasional task where those milliseconds actually matter because your browser is processing something enormous, but then just install both browsers, use Chrome in those rare instances, and use FF for a primary browser.
Frame rate isn't really an issue either. You can point out that the human eye sees at roughly 60 FPS, so going under 60 is undesired, but let's be realistic. Those Flash games are usually built at 20-25 FPS, because running at 60 would make them freaking huge. Video on the web likewise runs at 60 FPS roughly never, because it needs to stream. Downloaded video WILL run at 60, but your browser isn't playing that.
This doesn't mean there isn't a couple of very specific tasks that FF is abnormally slow at and could use a code cleanup on, but for the most part, FF's speed difference vs. Chrome is utterly negligible in actual use.
Really, fuck it. I've had it with corporate-sponsored dick-fighting contest about which browser is the fastest. I really, really couldn't care any less. Features, openness, security, standards compliance, yeah. But If I want a fast app, I'll go native, thank you. Maybe I'm too old, but I've always thought HTML sucked as a programming paradigm. As an information distribution mechanism, sure. But for interactivity? Please. It's about time somebody called bullshit on this. Hell, a goddamn Visual Basic app from fifteen years ago kicked the butt of most modern web sites in usability, performance and ease of maintenance. The only thing that makes the web so attractive is the barrier to entry : free, nothing to install, immediate access to the average brains of millions. Just like TV. No thanks for dumbing it down to this. And now you wanna make it faster? Piss off. Go write real code that does something, not just another abstraction layer.
-- Home is where you eat your heart out.
few timers can really measure 2 or 3 ms accuracy (you have to use high performance counters). Your standard dev of 0 probably means the resolution of the timer was less than the actual std dev.
Stability and web standards implementation is what matters most.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
I guess the actual selection of versions shows how the point of the article was more about bashing FF4 compared to IE9 (in which it also failed, given the very small difference between them) rather than doing a honest comparison of all of the browsers.
"I'm never quite so stupid as when I'm being smart" (Linus van Pelt)
I've said this before. I think it's time that Mozilla release a stripped down, lean, mean version of Firefox without all of the bloat.
They can call it something like Firebird, or Phoenix, to distinguish it from the main branch.
"What kind of music do pirates listen to?" -Paul Maud'dib
"Yeeeaaarrrrr n' Bee!!" -Stilgar, Leader of Sietch Tabr
Real world people use flashstop, noscript, adblock plus. I use firefox only because of those. If the other browser had the same functionality as easily accessible built in, I would maybe use them. As it stands, only firefox interrest me. As for performance... Talk to me when the majority of the people bandwidth and access time stops being *the* limiting factor.
C. Sagan : A demon haunted world:
http://www.amazon.com/gp/product/0345409469/
visit randi.org
I'm surprised none of the browser vendors hired the author of this virtual machine, which beats both V8 and TraceMonkey in the shootout benchmarks by a wide margin.
I'm hoping Google or Mozilla does that eventually, it'd be a pity if he got hired by a closed source browser vendor.
The abbreviation of Firefox is not "FF", but "Fx".
3) is now obsolete in the latest Firefox 4 nightlies - http://blog.zpao.com/post/1140456188/cascaded-session-restore-a-hidden-bonus
It has 1-2% market share in the English-speaking world. It has a much larger share (over 20% in some countries) in Eastern Europe and Russia.
I am TheRaven on Soylent News
The Firefox beta does not even have the new Method JIT. So they are comparing oranges and apples, not nearly finished products. Maybe Mozilla shouldn't have called it Beta, maybe that is the problem. But rerunning the tests in a few weeks will give very different results.
New things are always on the horizon
(Grumpy old codger ahead)
'Real world torture test'. Yeah, right. Spoiled broadband using whippersnappers have no idea what a real-world browser torture test is. Try this one on for size, you spoiled basement dwellers. Here's torture test for your 'real world' that I run in every day.
Bandwidth: Max of 32K shared on a 30+ computer network with 4 analog phones using the same bandwidth.
Connection: 357ms ping time to the DMZ. >600ms ping to the 'real world'.
Connection Type: Throttled Inmarsatt B.
Browser: IE 6
Yet somehow I can still manage to download Slashdot in 1 second over this connection.
http://www.google.com/gwt/n kicks ass over all your snooty high-faluting bells-and-whistle browsers!
Not quite as fast as good old Lynx though. I miss that. 300 baud.. monochrome... keyboard navigation... those were the days. Could hold all your opcodes on a three by five, and write your machine code by hand.
How does the timer in firefox "kick off"? It's obviously not on the branch callback anymore; surely you guys aren't firing an operation callback every 10ms?
Do you know how often in the tracing JIT checks operation callback? How about JM? I'm wondering if this style of benchmark will become increasingly innaccurate as the JS JITs get tighter. Is a super-accurate 10ms timer a goal @moz?
Do daemons dream of electric sleep()?
For me speed is secondary, both deliver adequate speed subjectively. If I don't care that Google, Yahoo and all Ad companies know that I'm ah.... say date online, then IE is OK. When my privacy is important to me, FF+noscript, hands down.
In the former Soviet republics, Opera is #1 in many cases.
..and this is just a short sampling of the former Soviet republics. Its #1 in more of them, and where it isn't #1 its often a close #2.
#1 in Uzbekistan
#1 in Ukraine
#1 in Georgia
#1 in Belarus
"His name was James Damore."
I hadn't realised it was quite so popular - for those too lazy to click the link, or who don't have Flash, it has 30-50% market share in those countries. The original poster's comment reminds me of the poster in the last iPhone story claiming he'd never seen a Nokia Smartphone in the wild. It's important to remember that the US market is often not reflective of the global market.
I am TheRaven on Soylent News
and seeing how many of these fruits of your labor actually end up in future releases. (Hint: none, even if the bug does things like solve massive NTFS corruption or critical on-screen corruption in the ATI 2D driver, both of which are real attempts of mine.)
They say "U Want? U Fix!"
Then you do and they say it doesn't:
Fit with project goals
Adhere to project style or standards
Offer regression data about other use cases
Solve a big enough problem to justify effort to include
Or they'll just ignore the hell out of you and eventually (as you noted) mark the bug as "solved" simply because bug submitters stop responding to repeated nonsense bug labor/reply requests after 2-3 years... even if there are replies (SOMETIMES DOZENS OF THEM) in the Bugzilla threads linking to WORKING PATCHES.
God I've had it with having to rebuild half of my packages from .src.rpm each release using hacked and rehacked patches, version after version, from Bugzilla discussions that never, ever seem to make it into the code year after year and release after release because arrogant maintainers have their heads up their asses.
THIS is why I'm done with FOSS. Just done. I loved it and the community in the '90s. Now it's mostly arrogant young hotshots with no particular interest in getting actual work done apart from the work of coding for coding's sake, implementing new experimental unstable (if not useful) features at the expense of old, stable, useful ones that most users rely on.
After all, u want... u fix! (But not in my project -- build your own codebase from scratch!)
STOP . AMERICA . NOW
Looks like IE8 came in dead last on both charts provided... How does that make FF4 dead last?
Timers run from the event loop. They have all along; they didn't fire via the branch callback at all. So to get your timers to fire you have to stop whatever you're doing and return to the event loop.
I don't know how often the operation callback runs, sorry.
Having a 10ms timer that actually runs after 10ms (accurate to under a ms) is definitely a goal. But again, you'll have to go to the event loop to get the timer, and if something else is running when the timer is supposed to fire (e.g. if you have 1000 timers all scheduled for the same millisecond), chances are your timer will run late.
I can't make enough sense of the benchmark's obfuscated code so far to say anything useful about its accuracy past what I already said.
I just tested the latest beta of IE 9 with our application and it beats out all the other browsers. Our web application heavily depends on image display and canvas and the difference is likely due to IE9's rendering integration with DirectX. If you are looking at writing web apps that heavily use Canvas, IE9 is going to be more than relevant.
I can't recall when my "user experience" has been limited by anything other than download speed (and the abysmal quality of most Web sites, of course). So why should I care about browser speed?
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
In my tests, on the average, Ie9 was infinitely slower. Of course, I knew that if I just posted raw cross-platform results, people would object that Windows has a disproportionate market share, so I took that into account. In my final numbers, Windows counts for a generous 99% of the result, and other OSes for only 1%. Of course, 1% of infinity is still infinity, so, on the average, any given user will experience an infinite slowdown with IE9. It's all there in the numbers! :)
Try opening a generic website .... lets say CNN.com or even Slashdot.org on FF and IE at the same time. Then tell me if there is no difference in speed.
The real benchmark between browsers should be the time taken to perform a search on a site like isoHunt. The Flash adds alone are an IE8 killer even on a Core2duo, and I'm sure they'll only get worse.
.
Microsoft Software? No.
.
Microsoft Web Browsers? No.
.
So who cares? I don't.
.
Voting up, Voting down - If I really gave a fuck about your approval or not, I'd come and ask you.
Bah. I'll wait until it's ported to linux.
Chrome runs in the background when you start your PC, so naturally it loads faster - it has slowed down the rest of your computer to make sure of it.
Also even after you uninstall Chrome, the program continues to connect to Google periodically, slowing down your computer and wasting your time even more as you try to figure out how to fully remove it.
What's the practical difference between 6.3 and 5.5 seconds to get to 100 km/h?
.GE. 6 versus Netscape Navigator I would probably want to trash Navigator all over again. The weakest link gets kicked off the island, even if the difference is measured in fractions.
Big difference. As a commuter I value the acceleration (along with other properties) of my vehicle. If everyone else takes 6.3 and I take 5.5, this difference will help me numerous times per commute to merge more easily or to be able to take advantage of the hole created in front of some grey beard, or in some cases to make the next light due to dumb configuration of light sequence. So it is not 6.3 vs 5.5 with car acceleration, it is relative acceleration that matters. In the dog-eat-dog world of commuting, acceleration matters quite a bit. For me, more acceleration equals less stress. My brother's "zero to 50 in 30" two-cylinder Toyota has no place on today's roads.
And FWIW this is at least partially true with other things -- if browser x is the slowest, whenever we run it we will notice. For me, IE v6, 7, 8 are all so slow vs Opera & Firefox that I simply never run them. Yet if I did an absolute comparison of IE
I come here for the love
The MAFIAA is a bunch of mindless jerks who will be the first up against the wall when the revolution comes