Slashdot Mirror
Internet Explorer 9 Caught Cheating In SunSpider
dkd903 writes "A Mozilla engineer has uncovered something embarrassing for Microsoft – Internet Explorer is cheating in the SunSpider Benchmark. The SunSpider, although developed by Apple, has nowadays become a very popular choice of benchmark for the JavaScript engines of browsers."
This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.
Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.
what can be attributed to stupidity.
1) Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2)Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
I see no reason why explanation number one is more likely than explanation number two.
Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.
Some things to consider: 1) I'm not doing business with the 8th grader. Nor am I relying on his understanding and memorization of history to run Javascript that I write for clients. 2) You are giving Microsoft a pass by building an analogy between their javascript engine and an 8th grade history student.
Just something to consider when you say we shouldn't be shocked by this.
My work here is dung.
Everything in italics is unsupported opinion by the author, yet is treated as fact in the summary and title by CmdrTaco and Slashdot. Perhaps if Slashdot would stick to actual news sites (you know NEWS for nerds and all that), this would be a balanced report with a good amount of information. Instead, it is just another Slashdot supported hit piece against MicroSoft.
FTFA:
There are three possible explanation for this weird result from Internet Explorer:
1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.
I'm not saying if what they have done is right or wrong, but this is a sensationalist headline that offers two other "less evil" alternatives to the outcome.
Meh I think claiming they are cheating with no evidence seems a little too out there. I've never seen MS brag about how fast their browser is on this particular benchmark, and frankly seems more like a bug than a cheat.
did you forget to take your meds?
The article clearly states:
There are three possible explanation for this weird result from Internet Explorer:
1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.
So, what proof do we have that Microsoft actually cheated?
He who knows best knows how little he knows. - Thomas Jefferson
Another misleading tabloid headline from Taco et al.
Short story: Someone notices a perhaps too-fast result for a particular benchmark test with IE 9 and modifies the benchmark code which then throws IE 9 performance the other way. One *possible* conclusion is that MS have done some sort of hardcoding/optimisation for this test, which has been thrown out by the modifications.
Thanks for someone pointing this out. I mean really, if they were going to throw this test why would they throw it quite this much? And is this the ONLY portion of this test that seems to act this way? If so then why in the world would they throw only this portion and why this much? The original result was uber fast, the result on the modified test pretty slow - if they were going to try and hide something why make it uber fast and not just slightly better?
Something is weird, possibly hinky, but to outright declare cheating based just on this? Really? O_o
Build it, Drive it, Improve it! Hybridz.org
It is actually a couple of months old. The thing that makes me doubt the claims of cheating is that nobody has been able to find other examples of performance variations in this benchmark in all the time since this came to light. If they were going to cheat, why limit it to the cordic test? Nobody would base their browser choice on this obscure test.
I don't have the beta installed yet, but what I would like to see is the actual calculation changed and then run the tests again. Don't just put in weird code like "true;" but make the javascript plausible. It could be that the addition of these unusual statements are enough to confuse the optimiser so that it resorts back to a completely unoptimised version.
Socially liberally means allowing people to do whatever they want. In their bedrooms, in their homes, in their personal lives, so long as their actions don't physically harm another.
It does Not mean politically-or-fiscally liberal (using gov't to steal money from workers and redistribute it).
"I disapprove of what you say, but I will defend to the death your right to say it." - historian Evelyn Beatrice Hall
Optimisations done purely for use only on a benchmark to achieve far better results than normal is the exact definition of cheating. Benchmarks are meant to test the browser with some form of real performance measure and not how good the programmers are at making the browser pass that one test. If the thing is getting thrown off by some very simple instructions to the tune of 20 times longer then it is seriously broken. Optimization or not.
It is like when ATI/Nvidia made their drivers do some funky shit on the benchmarks to make their products seem way better; This was also called cheating at the time.
Really? Are you sure about that that?
Second paragraph of the article:
The parent is not insightful, it is merely a troll.
The real "Libtards" are the Libertarians!
Did you look at the diffs? The addition of the "true;" operation should make absolutely no difference to the output code. It's a NOP. The fact that it makes a difference indicates that either something fishy is going on, or there is a bug in the compiler that fails to recognise "true;" or "return (at end of function)" as being deadcode to optimise away, and yet the compiler can apparently otherwise recognise the entire function as deadcode. Just to be clear, we are talking about a compiler that can apparently completely optimise away this whole function:
function cordicsincos() {
var X;
var Y;
var TargetAngle;
var CurrAngle;
var Step;
X = FIXED(AG_CONST); /* AG_CONST * cos(0) */ /* AG_CONST * sin(0) */
Y = 0;
TargetAngle = FIXED(28.027);
CurrAngle = 0;
for (Step = 0; Step CurrAngle) {
NewX = X - (Y >> Step);
Y = (X >> Step) + Y;
X = NewX;
CurrAngle += Angles[Step];
} else {
NewX = X + (Y >> Step);
Y = -(X >> Step) + Y;
X = NewX;
CurrAngle -= Angles[Step];
}
}
}
but fails to optimise away the code when a single "true;" instruction is added, or when "return" is added to the end of the function. Maybe it is just a bug, but it certainly is an odd one.
This shows the dangers of synthetic non-realistic benchmarks. I was amused to read Microsoft's comments on SunSpider: "The WebKit SunSpider tests exercise less than 10% of the API’s available from JavaScript and many of the tests loop through the same code thousands of times. This approach is not representative of real world scenarios and favors some JavaScript engine architectures over others." Indeed.
btw the Hacker News discussion is more informative.
Everyone uses because it is a fairly objective benchmark.
For the record, I caught wind of this a month or two ago and posted it here in a firefox performance article. I was trolled and troll moderated despite pointing to the Mozilla team's own experiments.
The ONLY reasonable explanation, assuming you actually understand the implications of what is it you're (generalized readers, not your specifically) reading, which based on previous happenings is questionable, is that Microsoft is cheating their asses off by identifying the exact benchmark and returning a pre-computed value. Either that, or this is indicative of a horrible optimization bug which would negatively effect all javascript in their browser and it would be impossible for them to be competitive in the least. Given there is no evidence to support the later, the only reasonable conclusion is they are cheating their asses off in these benchmarks.
Well, yes, taking some of your money. But since the only way that you can make money is because the wider society sees that as a benefit, suck it up and pay your goddamned taxes. This illusion that somehow the money in your pocket came to you by yourself alone is the greatest lie of Libertarianism.
The world's burning. Moped Jesus spotted on I50. Details at 11.
Not unusual, but it makes the function 1 statement longer..
If you knew anything about JIT compilers, you would know that they have simple heuristics on purpose (compile speed is a strict constraint.) Making something 1 statement longer could remove it as a candidate for quite a few optimizations (inlining, static loop evaluation, loop unrolling, dead code elimination, etc..)
These simple heuristics use quickly evaluated metrics once the source is translated into an abstract syntax tree. The number of nodes in the tree.. the depth of the tree.. the number of conditional nodes..
JIT's are not simply compilers that try to produce the best code possible. JIT's make tradeoff decisions between compile time and the resulting code quality.
"His name was James Damore."
Socially liberal does not mean "spend money".
Any other definition necessarily requires taking my money and giving it to someone else.
Ah, the "anti-tax" argument. I'm happy with taxes. Honestly. Do I wish they were lower - of course. Do I think that we spend money on stupid things? Yep.
Put taxes are still cheaper than having my own private doctor and hospital, my own roads, my own water towers and power generation, my own private library, swimming pool, and so on. Governments should do these things, because it's cheaper for everyone to pitch in.
Why don't you try reading it before you make that claim? The article is a few simple benchmark results and mild speculation as to what caused them. The summary may be inflammatory, the article goes out of its way not to be.
1) Microsoft beats everyone else by a factor of 10.
2) Making any of a number of effectively cosmetic changes to the function results in Microsoft taking twice as long as everyone else.
3) Making the inner loop 10x longer makes everyone else take 10x longer, except MS, who takes 180x longer.
Sorry, but if that counts as an optimization "bug", I have a bridge to sell you.
I think by "socially liberal" he means "nothing should be illegal unless it harms someone else." But as to fiscal conservatism, that's the Libertarian Party, not libertarians in general. Personally, I agree that anything that doesn't trample someone else's rights should be legal, but at the same time I'd like to see universal health care, and have the poor taken better care of. "There but for the grace of God go I".
And I'd like to see corporate power reigned in, and I'd like the corporates to stop getting government welfare and getting away without paying taxes. So I seldom vote for a Libertarian candidate.
Free Martian Whores!
First of all, I (and I'll not speak for all left libertarians here, as there is some debate on the matter, but I think I'm expressing the majority opinion) distinguish "personal property" (that is, what you own and use) from "private property" (that is, property owned privately, usually by an organization, but not owned or used by an individual, and leveraged to extract profit). This distinction is important before any discussion of any wealth-distribution theory.
Personal property is personal property. I can think of scarce few real leftists (which is to say, true socialists, communists, anarcho-communists, left-libertarians, etc) who include personal property when they say "property is theft". Private property, on the other hand, being the spoils of a great and sustained theft from the public, belong to the public and should be returned to the public.
Note here that I do not mean the state when I say public. Which is to say that I'm not an advocate for systems like the Soviet Union, but I am an advocate for movements like the worker takeover of factories we see in some Latin American countries.
In short, I advocate taking back what was stolen from us.
> And their 10-Q definitely indicates that they're not losing money.
Microsoft added 'optimization' for the 10-Q results as well.
boycott slashdot February 10th - 17th check out: altSlashdot.org
The benchmark in question can be considerably optimized by dead code elimination, since a computationally expensive function in there (one that loops computing stuff) does not have any observable side effects, and does not return any values - so it can be replaced with a no-op. It is a perfectly legitimate optimization technique, but the one which tends to trip up naively written benchmark suites because they assume that "for(int i=0; i < 1000000; ++i) {}" is going to be executed exactly as written.
Thee was actually a similar case with artificial tests in the past - Haskell (GHC) scores on the Programming Language Shootout. Most tests there were also written as loops with some computations inside and no side-effects, on the presumption that compilers will leave the computations intact even though their results are never used. Well, one thing that GHC has always had is a particularly advanced dead code eliminator, and it compiled most of those tests down to what is essentially equivalent to "int main() { return 0; }" - with corresponding benchmark figures. Once they've changed the tests to print out the final values, this all went back to normal.
In this case it's not quite that simple, because seemingly trivial changes to benchmark code - changes which do not change the semantics of the code in any way - trip off the dead code elimination analyzer in IE9 JS engine. However, it is still an open question on whether it is deliberate, or due to bugs in the analyzer. One plausible explanation was that analyzer is written to deal with code which at least looks plausible, and neither of the suggested optimizer-breaking changes (inserting an extra statement consisting solely of "false;" in the middle of the function, or "return;" at the end of it) make any sense in that context. Any dead code elimination is necessarily pessimistic - i.e. it tries to guess if the code is unused, but if there are any doubts (e.g. it sees some construct that it doesn't recognize as safe) it has to assume otherwise.
The only true way to test this is to do two things:
1. Try to change the test in other ways and see if there are any significant diffs (such that they are not reasonably detectable as being the same as the original code) which will still keep the optimizer working.
2. Write some new tests specifically to test dead code elimination. Basically just check if it happens on completely different test cases.
By the way, the guy who found the discrepancy has opened a bug in MS bug tracker regarding it, in case you want to repro or track further replies.
The return statement was "return;" i.e. a return statement that did not return anything. Looking at the other JavaScript engines, that line added at most 1ms, while with the IE engine it added 19ms. If the IE9 JS engine is handling this function in a super-efficient way that is not due to cheating, the optimisation must be highly sensitive to variance.
One way to check if the IE9 engine is doing some sort of special casing (e.g. hashing the text for the function) would be to change the name of a variable. This should not change the behaviour of the engine as it is the same code (there are no extra elements in the tree, like additional returns). If the IE9 engine is cheating, this should jump from 1ms to 20ms like the other variances. If it is an optimisation bug, the performance should be 1ms for both cases.
Hi, we've posted an official response and explanation at the bottom of this post:
http://blogs.msdn.com/b/ie/archive/2010/11/17/html5-and-real-world-site-performance-seventh-ie9-platform-preview-available-for-developers.aspx
Bottom line - we built an optimizing compiler into the Chakra JavaScript engine that eliminates dead code where it finds it.
It's a fun conspiracy theory that we'd somehow 'game' SunSpider in this way, but it doesn't make sense that we'd go to all that risk to cheat on just one test out of 26.
Best wishes, Tim Sneath | Microsoft