Slashdot Mirror
Google Kills Off Octane JavaScript Benchmark Due To 'Diminishing Returns and Over-Optimization' (betanews.com)
Google has announced that its widely used Octane JavaScript benchmark is being retired, with Google saying that it's no longer a useful way for browser developers to determine how best to optimize their JavaScript engines. From a report: Google goes as far as saying that developers were essentially cheating the system. It says that compiler optimizations needed to achieve high benchmark scores have become common and, in the real world, these optimizations translate into only very small improvements in webpage performance. In fact, in some instances it was found that tactics used to boost benchmark performance actually had a detrimental effect on real-world performance. Developers exploited known bugs in Octane to achieve higher scores than were warranted, and Google believes the time has now come to retire the system completely.
Chrome isn't the best at this benchmark so we've retired it
Across the industry, benchmarks become a double edged sword when the industry embraces it too much and matures.
There are certain benchmarks that drive technology to make choices that can get them 2-3% wins compared to other things on the market, but that translates into real world performance that can be 50% slower in pretty much anything but the particular benchmark.
XML is like violence. If it doesn't solve the problem, use more.
A recruiter called yesterday about wanting to "octane my particular skill set" for a job.
https://twitter.com/cdreimer/status/852671049942446081
Solution: Obsolete the test and insist that the latest benchmarks are not "real performance". :P
You mean before the dot com bust? That was before you were born.
No it is just our very own APK. As a long time /.er you should be well familiar that if you point out any failings with his hosts file engine or happen to criticize him in any way expect an even more incoherent rant and personal attacks. Each one gets more vulgar and towards the end he is usually threatening to cause grievous bodily harm. He will also fly off the handle at just about anyone so you don't really need to slight him or draw his ire. Don't try arguing with him and in most cases don't respond because he seems to watch his posts like a hawk and will provide some circular arguments as the thread progresses. Then, as the legend in his own mind he is, he will declare that he showed you and the world the genius that is his own personal madness. He will respond to my post and will also claim that I have a fake online identity yet he hides behind good old Anonymous Coward but at least signs his garbage rants but even then who knows if they are from the same person as it seems it wouldn't be that hard to impersonate him.
It is fun winding him up and I do it from time to time although I usually arrive too late in the thread for it to get any real notice. He is really no better than the golden girls guy, the apps guy, or the GNAA guy.
Time to offend someone
Because that would be very un-google of them. Google rarely fixes known bugs, but frequently retires platforms.
OP is obviously older than the dot com bubble since OP can conceive of a world without Google.
I didn't read it that way.
How fucking stupid are you, fat ugly stupid troll.
I made the president's list for maintaining a 4.0 GPA when I got my A.S. degree in computer programming while working 60+ hours as a video game tester and teaching Sunday school after the dot com bust.
As a professional developer: if you are coding around a performance benchmark, you are doing it wrong.
These kinds of tools exist for developers to evaluate the performance of their own code. Anyone who uses them in any other fashion (like to evaluate the performance of someone else's code, for example) needs to take the results with a grain of salt, and that will always be the case for any automated software benchmark.
That Google feels the need to retire Octane over this is almost unbelievable... there must be some ulterior motivation.
That Google feels the need to retire Octane over this is almost unbelievable... there must be some ulterior motivation.
Why that assumption? Google explained its reasons quite clearly:
Investigations into the execution profile of running Octane versus loading common websites (such as Facebook, Twitter, or Wikipedia) revealed that the benchmark doesn’t exercise V8’s parser or the browser loading stack the way real-world code does. Moreover, the style of Octane’s JavaScript doesn’t match the idioms and patterns employed by most modern frameworks and libraries (not to mention transpiled code or newer ES2015+ language features). This means that using Octane to measure V8 performance didn’t capture important use cases for the modern web, such as loading frameworks quickly, supporting large applications with new patterns of state management, or ensuring that ES2015+ features are as fast as their ES5 equivalents.
In addition, we began to notice that JavaScript optimizations which eked out higher Octane scores often had a detrimental effect on real-world scenarios.
If you think about the above, consider also that every JavaScript engine in use today that I can think of is open source. That means the projects accept contributions from independent developers all over the world. Many of those developers may be submitting patches designed to improve the performance of the engine. It may even be that most of the patches are designed to improve performance. But if the "proof" that the patches increase performance is the Octane benchmark suite, and the Octane suite doesn't model real-world web scenarios, then some of those performance "enhancements" may actually decrease real-world performance.
Google is retiring the benchmark suite so that good-intentioned open source developers will not be able to use it as a proof point for why their patches improve performance, when in fact they don't.
P.S. It seems one other group is disappointed that the benchmark is going away, though, and that's Chromebook fans. They've been using Octane to benchmark the performance of hardware from different vendors running the same version of Chrome OS. That still seems like a legit use case to me.
Breakfast served all day!
People are smarter than computers.
It little behooves the best of us to comment on the rest of us.
Solution: Obsolete the test and insist that the latest benchmarks are not "real performance". :P
That would be insightful if it weren't for the fact that Chrome is one of the best benchmarked browsers in Octane.
But the difference between browsers is just so minimal that quite frankly no one should give a crap about these benchmarks anymore. It's nothing more than e-penis measurement at this point.
Chrome no longer gets top score.... This benchmark is irrelevant and time to be decommissioned. I am sure this is just pure coincidence!
"When a measure becomes a target, it ceases to be a good measure".
The fact that you said "one of the best" instead of "the best" is the point here. Any company that makes a benchmark (or other test) for a product they make never intends for a competitor to come out as #1.
Dumb. Chrome goes through Canary, Developer, and Beta channels before the general public ever sees a new build. You can use Octane on any of those. By your reckoning, of course no version of Chrome ever comes out behind. But what actually happens is the benchmark score bounces up and down with ongoing development of the engine.
Breakfast served all day!
While I don't wish to lessen your earlier achievements, you place a great deal of emphasis on looking for some sort of approval that you believe you're due for early life achievement, but you don't follow up with "and since then I have accomplished...".
That's because I'm not here looking for approval.
Don't look for approval from people on Slashdot, you have to post something truly exceptional to be greater with anything but flames.
The purpose of Slashdot is to keep me amuse while I'm waiting for a script to finish at work.
Don't cheapen it to the point that people actually begin to remember you as "The guy who posts his high school resume on every thread"
This is Slashdot, not LinkedIn. No one gives a shit about what I do on Slashdot. On LinkedIn, I have connections to 800+ recruiters. So you won't see me post anything on LinkedIn.
1) video game tester: ridiculous joke of a job for stupid people
As a video game tester for six years, I wrote 30,000+ bug reports. As a lead video game tester, I was responsible for leading 10 titles through QA, trained three assistants to become lead video game testers, and worked extensively with developers all over the world. Stupid people don't last in this job if they don't figure out the difference between playing and testing video games.
2) associates degree in computer programming: you went to Devry because you couldn't get into college.
I transferred to university and got kicked out in my junior year for playing too many games of Magic: The Gathering. When I went back to community college a decade later to learn computer programming, it was paid for with a $3,000 tax credit signed into law by George W. after 9/11.
3) Sunday school. you're religious = moron
I got kicked out of a church that I went to for 13 years because I accused the local leadership of being morally corrupt—and they proved my point by making it personal ("you made me look bad") instead of spiritual ("God forgive for we are sinners").. I'm no longer a fan of organized religion. The only time religion makes you a moron is when you stop thinking.
Now go find someone else to play with, troll.
1) yeah man. that's what smart people with marketable skills want to do. write 30k bug reports. you're a fucking video game tester. you have no skills. testing video games is one step up for data entry. you are literally below helpdesk monkeys.
I was a help desk technician for nearly six years. I closed 300 to 700 tickets per month with a 98.8% SLA rate. I was always number three the department because the phone guys got all the five-minute tickets.
2) so... you got kicked out of college and got an a non-degree from the shithole where secretaries learn msoffice. gotcha.
Spreadsheet manipulation wasn't something I learned at college but on the job. One of my coworkers recently found a neat DOS trick for merging CSV files into one CSV file:
copy *.cvs single.cvs
Instead of working off of multiple spreadsheets, we can merge them together into one spreadsheet with a minimal amount of clean up.
3) morons never think they're morons. it's not my job to change their mind, and I like my morons to stay dumb, like you. they're more entertaining that way.
Says the moronic troll.
dipshit, you literally said Zero to counter any of the claims made.
Why would I? I'm posting facts that dispute your moronic ASSumptions.