Slashdot Mirror
Internet Explorer 9 Caught Cheating In SunSpider
dkd903 writes "A Mozilla engineer has uncovered something embarrassing for Microsoft – Internet Explorer is cheating in the SunSpider Benchmark. The SunSpider, although developed by Apple, has nowadays become a very popular choice of benchmark for the JavaScript engines of browsers."
This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.
Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.
what can be attributed to stupidity.
1) Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2)Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
I see no reason why explanation number one is more likely than explanation number two.
Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.
Some things to consider: 1) I'm not doing business with the 8th grader. Nor am I relying on his understanding and memorization of history to run Javascript that I write for clients. 2) You are giving Microsoft a pass by building an analogy between their javascript engine and an 8th grade history student.
Just something to consider when you say we shouldn't be shocked by this.
My work here is dung.
Everything in italics is unsupported opinion by the author, yet is treated as fact in the summary and title by CmdrTaco and Slashdot. Perhaps if Slashdot would stick to actual news sites (you know NEWS for nerds and all that), this would be a balanced report with a good amount of information. Instead, it is just another Slashdot supported hit piece against MicroSoft.
FTFA:
There are three possible explanation for this weird result from Internet Explorer:
1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.
I'm not saying if what they have done is right or wrong, but this is a sensationalist headline that offers two other "less evil" alternatives to the outcome.
They have shown their Sunspider results quite a few times on http://blogs.msdn.com/b/ie/
This is my signature. There are many like it, but this one is mine.
Another misleading tabloid headline from Taco et al.
Short story: Someone notices a perhaps too-fast result for a particular benchmark test with IE 9 and modifies the benchmark code which then throws IE 9 performance the other way. One *possible* conclusion is that MS have done some sort of hardcoding/optimisation for this test, which has been thrown out by the modifications.
Optimisations done purely for use only on a benchmark to achieve far better results than normal is the exact definition of cheating. Benchmarks are meant to test the browser with some form of real performance measure and not how good the programmers are at making the browser pass that one test. If the thing is getting thrown off by some very simple instructions to the tune of 20 times longer then it is seriously broken. Optimization or not.
It is like when ATI/Nvidia made their drivers do some funky shit on the benchmarks to make their products seem way better; This was also called cheating at the time.
Did you look at the diffs? The addition of the "true;" operation should make absolutely no difference to the output code. It's a NOP. The fact that it makes a difference indicates that either something fishy is going on, or there is a bug in the compiler that fails to recognise "true;" or "return (at end of function)" as being deadcode to optimise away, and yet the compiler can apparently otherwise recognise the entire function as deadcode. Just to be clear, we are talking about a compiler that can apparently completely optimise away this whole function:
function cordicsincos() {
var X;
var Y;
var TargetAngle;
var CurrAngle;
var Step;
X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ /* AG_CONST * sin(0) */
Y = 0;
TargetAngle = FIXED(28.027);
CurrAngle = 0;
for (Step = 0; Step CurrAngle) {
NewX = X - (Y >> Step);
Y = (X >> Step) + Y;
X = NewX;
CurrAngle += Angles[Step];
} else {
NewX = X + (Y >> Step);
Y = -(X >> Step) + Y;
X = NewX;
CurrAngle -= Angles[Step];
}
}
}
but fails to optimise away the code when a single "true;" instruction is added, or when "return" is added to the end of the function. Maybe it is just a bug, but it certainly is an odd one.
This shows the dangers of synthetic non-realistic benchmarks. I was amused to read Microsoft's comments on SunSpider: "The WebKit SunSpider tests exercise less than 10% of the API’s available from JavaScript and many of the tests loop through the same code thousands of times. This approach is not representative of real world scenarios and favors some JavaScript engine architectures over others." Indeed.
btw the Hacker News discussion is more informative.
Everyone uses because it is a fairly objective benchmark.
For the record, I caught wind of this a month or two ago and posted it here in a firefox performance article. I was trolled and troll moderated despite pointing to the Mozilla team's own experiments.
The ONLY reasonable explanation, assuming you actually understand the implications of what is it you're (generalized readers, not your specifically) reading, which based on previous happenings is questionable, is that Microsoft is cheating their asses off by identifying the exact benchmark and returning a pre-computed value. Either that, or this is indicative of a horrible optimization bug which would negatively effect all javascript in their browser and it would be impossible for them to be competitive in the least. Given there is no evidence to support the later, the only reasonable conclusion is they are cheating their asses off in these benchmarks.
Socially liberal does not mean "spend money".
Any other definition necessarily requires taking my money and giving it to someone else.
Ah, the "anti-tax" argument. I'm happy with taxes. Honestly. Do I wish they were lower - of course. Do I think that we spend money on stupid things? Yep.
Put taxes are still cheaper than having my own private doctor and hospital, my own roads, my own water towers and power generation, my own private library, swimming pool, and so on. Governments should do these things, because it's cheaper for everyone to pitch in.
Why don't you try reading it before you make that claim? The article is a few simple benchmark results and mild speculation as to what caused them. The summary may be inflammatory, the article goes out of its way not to be.
1) Microsoft beats everyone else by a factor of 10.
2) Making any of a number of effectively cosmetic changes to the function results in Microsoft taking twice as long as everyone else.
3) Making the inner loop 10x longer makes everyone else take 10x longer, except MS, who takes 180x longer.
Sorry, but if that counts as an optimization "bug", I have a bridge to sell you.
> And their 10-Q definitely indicates that they're not losing money.
Microsoft added 'optimization' for the 10-Q results as well.
boycott slashdot February 10th - 17th check out: altSlashdot.org