Slashdot Mirror
Google Kills Off Octane JavaScript Benchmark Due To 'Diminishing Returns and Over-Optimization' (betanews.com)
Google has announced that its widely used Octane JavaScript benchmark is being retired, with Google saying that it's no longer a useful way for browser developers to determine how best to optimize their JavaScript engines. From a report: Google goes as far as saying that developers were essentially cheating the system. It says that compiler optimizations needed to achieve high benchmark scores have become common and, in the real world, these optimizations translate into only very small improvements in webpage performance. In fact, in some instances it was found that tactics used to boost benchmark performance actually had a detrimental effect on real-world performance. Developers exploited known bugs in Octane to achieve higher scores than were warranted, and Google believes the time has now come to retire the system completely.
A recruiter called yesterday about wanting to "octane my particular skill set" for a job.
https://twitter.com/cdreimer/status/852671049942446081
No it is just our very own APK. As a long time /.er you should be well familiar that if you point out any failings with his hosts file engine or happen to criticize him in any way expect an even more incoherent rant and personal attacks. Each one gets more vulgar and towards the end he is usually threatening to cause grievous bodily harm. He will also fly off the handle at just about anyone so you don't really need to slight him or draw his ire. Don't try arguing with him and in most cases don't respond because he seems to watch his posts like a hawk and will provide some circular arguments as the thread progresses. Then, as the legend in his own mind he is, he will declare that he showed you and the world the genius that is his own personal madness. He will respond to my post and will also claim that I have a fake online identity yet he hides behind good old Anonymous Coward but at least signs his garbage rants but even then who knows if they are from the same person as it seems it wouldn't be that hard to impersonate him.
It is fun winding him up and I do it from time to time although I usually arrive too late in the thread for it to get any real notice. He is really no better than the golden girls guy, the apps guy, or the GNAA guy.
Time to offend someone
Because that would be very un-google of them. Google rarely fixes known bugs, but frequently retires platforms.
That Google feels the need to retire Octane over this is almost unbelievable... there must be some ulterior motivation.
Why that assumption? Google explained its reasons quite clearly:
Investigations into the execution profile of running Octane versus loading common websites (such as Facebook, Twitter, or Wikipedia) revealed that the benchmark doesn’t exercise V8’s parser or the browser loading stack the way real-world code does. Moreover, the style of Octane’s JavaScript doesn’t match the idioms and patterns employed by most modern frameworks and libraries (not to mention transpiled code or newer ES2015+ language features). This means that using Octane to measure V8 performance didn’t capture important use cases for the modern web, such as loading frameworks quickly, supporting large applications with new patterns of state management, or ensuring that ES2015+ features are as fast as their ES5 equivalents.
In addition, we began to notice that JavaScript optimizations which eked out higher Octane scores often had a detrimental effect on real-world scenarios.
If you think about the above, consider also that every JavaScript engine in use today that I can think of is open source. That means the projects accept contributions from independent developers all over the world. Many of those developers may be submitting patches designed to improve the performance of the engine. It may even be that most of the patches are designed to improve performance. But if the "proof" that the patches increase performance is the Octane benchmark suite, and the Octane suite doesn't model real-world web scenarios, then some of those performance "enhancements" may actually decrease real-world performance.
Google is retiring the benchmark suite so that good-intentioned open source developers will not be able to use it as a proof point for why their patches improve performance, when in fact they don't.
P.S. It seems one other group is disappointed that the benchmark is going away, though, and that's Chromebook fans. They've been using Octane to benchmark the performance of hardware from different vendors running the same version of Chrome OS. That still seems like a legit use case to me.
Breakfast served all day!