Internet Explorer 9 Caught Cheating In SunSpider

dkd903 writes "A Mozilla engineer has uncovered something embarrassing for Microsoft – Internet Explorer is cheating in the SunSpider Benchmark. The SunSpider, although developed by Apple, has nowadays become a very popular choice of benchmark for the JavaScript engines of browsers."

Comments

Embarassing?

[intellitech]

I would think Microsoft would be used to embarassing by now..

Re:Embarassing?

[Pojut]

They're kinda like the rich fat cat who constantly puts his foot in his mouth. He knows he should shut up, but then again why should he care...he's rich, bitch!

Re:Embarassing?

[Anonymous Coward]

I would think Microsoft would be used to embarassing by now..

Embarassment requires a sense of shame.

Re:Embarassing?

[commodore64_love]

Not for long . . . I've been hearing that MS is losing tons of money and heading towards IBM territory.

Re:Embarassing?

[Sponge Bath]

Why would Microsoft be embarrassed by business as usual?

Re:Embarassing?

[Mushdot]

Another misleading tabloid headline from Taco et al.

Short story: Someone notices a perhaps too-fast result for a particular benchmark test with IE 9 and modifies the benchmark code which then throws IE 9 performance the other way. One *possible* conclusion is that MS have done some sort of hardcoding/optimisation for this test, which has been thrown out by the modifications.

Re:Embarassing?

[Anonymous Coward]

They're called Bidenisms

Re:Embarassing?

[eleuthero]

um... I'll admit they've not been doing as well as they used to, but their stock would seem to indicate that they are not losing money. Individual divisions might be (they did get in the news yesterday for playing games with sales figures in one division--but not in a way that the money didn't come from somewhere in the company), but as a whole, their P/E is better than Apple's (even if they do have a lower market cap).

Re:Embarassing?

[BLKMGK]

Thanks for someone pointing this out. I mean really, if they were going to throw this test why would they throw it quite this much? And is this the ONLY portion of this test that seems to act this way? If so then why in the world would they throw only this portion and why this much? The original result was uber fast, the result on the modified test pretty slow - if they were going to try and hide something why make it uber fast and not just slightly better?

Something is weird, possibly hinky, but to outright declare cheating based just on this? Really? O_o

Re:Embarassing?

[Richard_at_work]

I distinctly remember this being heavily discussed in the last few IE9 benchmark stories as well, so its not new and its not necessarily cheating.

Re:Embarassing?

[Attila Dimedici]

Not for long . . . I've been hearing that MS is losing tons of money and heading towards IBM territory.

Why would heading towards "IBM territory" be a bad thing? IBM is a profitable, successful company that has been around for longer than most companies of its size.

Re:Embarassing?

[Nutria]

Libertarian==fiscally-conservative/socially-liberal

How can one be fiscally conservative (spend less money) and simultaneously be socially liberal (spend more money on those sad, pitiful poor people)?

Re:Embarassing?

[nedwidek]

And their 10-Q definitely indicates that they're not losing money. http://biz.yahoo.com/e/101028/msft10-q.html

The stock price is a meaningless indicator unless you are using it indirectly with their P/E ratio.

Re:Embarassing?

[poetmatt]

anyone who says libertarian doesn't understand the conflict in the term. it's basically republicans who don't want to be called republicans or someone who hates both parties but still leans republican. if someone declares themselves independent libertarian, then they're more acknowledging that they don't necessarily align with "libertarian" views. I have a friend like this, and it's basically republican but he doesnt' want to admit it.

Re:Embarassing?

[commodore64_love]

Socially liberally means allowing people to do whatever they want. In their bedrooms, in their homes, in their personal lives, so long as their actions don't physically harm another.

It does Not mean politically-or-fiscally liberal (using gov't to steal money from workers and redistribute it).

Re:Embarassing?

[nschubach]

We're not... true libertarians (small "L") believe people should be responsible for themselves and their own actions. Fiscal conservatism is a means to the end... where people can pay their own bills and take care of their own kids.

Re:Embarassing?

[BubbaDave]

Libertarian==fiscally-conservative/socially-liberal

How can one be fiscally conservative (spend less money) and simultaneously be socially liberal (spend more money on those sad, pitiful poor people)?

Socially liberal does not mean "spend money".

Dave

Re:Embarassing?

[poetmatt]

uh, hate to break it to ya, but ballmer selling 87 million in shares is not necessarily a good sign for the company given current losses.

Re:Embarassing?

[Anonymous Coward]

yeah, because we all know that stock price is a perfect reflection of business performance and has nothing to do with public opinion :rolleyes:

Re:Embarassing?

[k_187]

I usually self-identify as libertarian and describe my politics as "I believe in tax cuts and gay marriage". Although I will admit that as time passes I'm moving toward the American left on the tax cut side.

Re:Embarassing?

[Anonymous Coward]

Optimisations done purely for use only on a benchmark to achieve far better results than normal is the exact definition of cheating. Benchmarks are meant to test the browser with some form of real performance measure and not how good the programmers are at making the browser pass that one test. If the thing is getting thrown off by some very simple instructions to the tune of 20 times longer then it is seriously broken. Optimization or not.

It is like when ATI/Nvidia made their drivers do some funky shit on the benchmarks to make their products seem way better; This was also called cheating at the time.

Re:Embarassing?

[eleuthero]

87 million shares is only a quarter of what he owns in the company (I know, he has a ridiculous amount of money in MSFT stock). Is it a lot? probably so, but he owns the third highest number of MSFT shares (second highest individual owner) and is unlikely to want to rock the boat. As a case in point, his announcement to sell was not something that did much to the price of the shares over the long term. People are unconcerned. Further, selling from time to time is expected - if the company is doing badly, he might sell, but if it is doing well, he will sell some shares for precisely the reason he has claimed--diversity. Gates owns only ~15 billion dollars' worth of MSFT shares and yet has a net worth over 40 billion.

Re:Embarassing?

[poetmatt]

please. it's almost trivial to sell shares in a company that puts anything above $1b in revenue a year, even if they're on the verge of bankruptcy.

Re:Embarassing?

[Nutria]

Then you're looking for the phrase laissez-faire, which means let do.

Re:Embarassing?

[commodore64_love]

The 70s and 80s IBM used to be much, much bigger than it is now.

You'll find numerous buildings scattered across the US that once belonged to IBM, but no longer do, because of IBM's shrinking profits (mainly due to their loss of the IBM PC business, but also other downsizing). Another analogy is that Microsoft might end-up like Kmart, who was once the #1 retailer but is now increasingly irrelevant.

Re:Embarassing?

[Nutria]

Socially liberal does not mean "spend money".

Any other definition necessarily requires taking my money and giving it to someone else.

Re:Embarassing?

[mwvdlee]

The problem is that that is the most logical conclussion.

The other possible conclussions are both more unrealistic and worse for MicroSoft.

The benchmark modifications were trivial and non-functional; they shouldn't have made that big of a difference.

Re:Embarassing?

[dkh2]

Still, the old addage stile holds...

Noone ever got fired for buying IBM.

Re:Embarassing?

[BubbaDave]

No, socially liberal means (one example) 'You want to marry whom? None of my business'.

Dave

Re:Embarassing?

[Nutria]

As I mentioned in a previous post, "None of my business" == "laissez-faire".

Re:Embarassing?

[Attila Dimedici]

The Kmart analogy is apt. Ending up like Kmart would be a bad thing. While Kmart may pull off a comeback, right now they are an also ran that is currently (and has been for a few years now) one bad decision from going out of business.
However, While IBM was once the computer company and is no longer, they are still a dominant force in computers and no smart company in the computer business intentionally picks a fight with them.

Re:Embarassing?

[iserlohn]

Ok I took the bait -

First, social liberalism is about liberalism in social values (not liberal socialism). Stuff like universal suffrage, civil rights for everyone and basically equal opportunity with advancement in society being based on merit as opposed to birth and circumstance. This is also classic liberalism which does not require a large state per se.

Second, the problem with fiscal irresponsibility is somewhat beyond the left/right continuum. Fiscal responsibility should be encouraged, but the issue is that markets tend to go into a positive feedback loop in the good times and vice versa, leading to the boom and bust cycle. The idea of Keynesian and neo-Keynesian policies is to provide a counter-balance to that.

This is a problem that the Austrian School fails to address. They believe that getting rid of credit and the related money supply expansion you will get rid of boom-and-bust. However, there is not one shred of evidence that this is the case, and by looking at the big all of crashes in markets throughout history, it appears that credit only aids in the boom and bust but does not cause it.

Re:Embarassing?

[Anonymous Coward]

All I know Kmart for is that it's the only department store (and only major retailer) in Canada where you can find cheap, regular AV cables. Everywhere else sells Monster cables and similar overpriced shit.

Re:Embarassing?

[gabebear]

Yep, I agree, I think IE9 javascript just sucks.

It's not that they cheat, it's that ANY little thing can create HUGE performance problems for no good reason. Completely unusable.

Re:Embarassing?

[imakemusic]

You'll find numerous buildings scattered across the US that once belonged to IBM[...]

There was one around the corner from where I work (in the UK) until a month or two ago. It is now a pile of rubble.

Re:Embarassing?

[gordguide]

Insiders selling company stock is always a Red Flag for investors. Whether that is justified or not in this case is up to the individual to decide, but it's a Red Flag for a reason ... very often it means problems within the company and bad news follows.

There are perfectly good reasons for insider selling, and it helps to be very straightforward about it ... it's not like you can hide it anyway, it's reported and watched vigorously.

Not being a Microsoft investor, and not particularly interested in their area of the market, it's not really important to me. But, if it were, I'd be very wary of the level of stock he's liquidating ... 25% is a massive sell by the usual standards. 5% is probably not going to raise too many eyebrows ... even 10% might be OK under the right circumstances.

But, since he can always pledge stock to back up any personal investment, this level is worrisome, I would think. It really means he's moving his money out of tech. And he's the CEO of a huge tech player. This is an unusual development by any investor standard.

Re:Embarassing?

[GameboyRMH]

Not to defend the statement but what you call "socially liberal" is actually "fiscally liberal."

"Socially liberal" means you keep any bigotry you may have to yourself and you don't want to legislate morality, to sum it up in as few words as possible.

Re:Embarassing?

[chrb]

Did you look at the diffs? The addition of the "true;" operation should make absolutely no difference to the output code. It's a NOP. The fact that it makes a difference indicates that either something fishy is going on, or there is a bug in the compiler that fails to recognise "true;" or "return (at end of function)" as being deadcode to optimise away, and yet the compiler can apparently otherwise recognise the entire function as deadcode. Just to be clear, we are talking about a compiler that can apparently completely optimise away this whole function:

function cordicsincos() {
        var X;
        var Y;
        var TargetAngle;
        var CurrAngle;
        var Step;

        X = FIXED(AG_CONST); /* AG_CONST * cos(0) */
        Y = 0; /* AG_CONST * sin(0) */

        TargetAngle = FIXED(28.027);
        CurrAngle = 0;
        for (Step = 0; Step CurrAngle) {
                        NewX = X - (Y >> Step);
                        Y = (X >> Step) + Y;
                        X = NewX;
                        CurrAngle += Angles[Step];
                } else {
                        NewX = X + (Y >> Step);
                        Y = -(X >> Step) + Y;
                        X = NewX;
                        CurrAngle -= Angles[Step];
                }
        }
}

but fails to optimise away the code when a single "true;" instruction is added, or when "return" is added to the end of the function. Maybe it is just a bug, but it certainly is an odd one.

This shows the dangers of synthetic non-realistic benchmarks. I was amused to read Microsoft's comments on SunSpider: "The WebKit SunSpider tests exercise less than 10% of the API’s available from JavaScript and many of the tests loop through the same code thousands of times. This approach is not representative of real world scenarios and favors some JavaScript engine architectures over others." Indeed.

btw the Hacker News discussion is more informative.

Re:Embarassing?

[zach_the_lizard]

Using that term could also mean you want to not interfere with the economy, bu you could also want to do things like ban gay marriage, drugs, etc. Socially liberal is the term I've always heard used. Welfare and the like would be economically liberal.

Re:Embarassing?

[Creepy]

Well some companies pick fights with them while collaborating in other areas - I know, and I've dealt with their 6 month hardware delays and red tape because they didn't want to sell to competitors (incidentally, they sold hardware to our contractors in India during this time, so they really didn't delay us with their wallowing).

I have worked for direct competitors that got killed in the market, however - Control Data Systems and EDS - both of which had similar downward spirals of spinning off profitable divisions to infuse them with cash to sustain the rest of the company. In EDS's case, they bought my company, took the excess cash, and spun us off. They also technically fired me (and everyone else they spun off), so yes, I have hard feelings - hope HP handles their employees better.

Re:Embarassing?

[GameboyRMH]

Yes it requires socialistic wealth redistribution to support any-consenting-adult-parties marriage (or to just keep government out of marriage), non-discrimination by gender/race/ethnicity, people's rights to do as they please with their own bodies, etc.

(Now I'm going to take a Panadol because you're so fucking wrong it hurts).

Re:Embarassing?

[Anonymous Coward]

If I think the government shouldn't tell people they can't do things that only directly affect themselves (drink, smoke, do drugs, have sex, make poor decisions, give all their money away, not use their seatbelts, engage in personal rocketry projects to attempt to fly to the moon, whatever - the government shouldn't have the right to prevent me from making bad personal decisions or ruining my life), but I also don't think that Robin Hood was a contributing member of society, where does that leave me? I'm clearly not "Basically Republican", I don't want their church-run nanny state protecting me from myself. But no way in hell am I a democrat, I don't want their government-run nanny state protecting me from life. Yet, I'm not a card carrying member of the Libertarian party, because I don't want their anarchy dismantling the military.
As near as I can tell, the Libertarian ideals are closest to my views, but no party currently chooses to represent me.

Re:Embarassing?

[cobrausn]

I call myself a libertarian. I hate republican war hawking and moral litigation (this coming from a Catholic US Navy veteran), and I hate democratic over-taxation and nanny-state tendencies (I hope to, one day, be rich, and I'd like to keep my money after all that hard work and not be labeled a 'fat cat', thanks). Don't lump me in with either of these idiotic parties, please.

Re:Embarassing?

[realityimpaired]

That's actually why there's blackout periods for insiders buying/selling shares, as part of the Sarbanes-Oxley rules. When I was at Dell, I wasn't allowed to buy/sell within 30 days (either way) of any public statement regarding earnings or future plans. These rules exist specifically to prevent insiders from gaming the market by buying up a large amount of stock right before an expected rise (higher than expected earnings announced, for example), or selling right before an expected fall (lower than expected earnings, for example). It's also why there was an obligation to announce any large sale by upper management as a matter of public record.

I'd be *very* surprised if Microsoft didn't play by the same rules, and I sincerely doubt that Ballmer's decision to sell had anything to do with Microsoft's performance as a company. MS could get in a *lot* of trouble with the SEC if it did, as they do take Sarbox seriously.

Re:Embarassing?

[Anonymous Coward]

The driver stuff was damning because if you renamed the test executable performance would suffer. They were special-casing doing things fast-but-wrong only if you were a known benchmark. Haven't seen anything saying that IE is treating the test differently - maybe they just optimized specific tested functionality in general, which isn't nearly as bad. That's not cheating, that's just teaching the test.

Re:Embarassing?

[Anonymous Coward]

It's still even cheaper to just order from Monoprice.

Re:Embarassing?

[Targon]

The purpose of a benchmark is to try to show how performance will be in the real world. If a given application has been programmed to do very well in a given benchmark yet does not do as well with a real-world situation, then the benchmark results are flawed. The idea of coding an application just to have good benchmark numbers that would not be seen in the real world is considered cheating. In this case, we are talking about JavaScript speeds, so you would be very surprised if you believed that IE 9 was really fast due to the benchmark results, yet was really slow going to other sites that use JavaScript.

Re:Embarassing?

[javaxjb]

but as a whole, [Microsoft's] P/E is better than Apple's (even if they do have a lower market cap).

Better is in the eye of the beholder. Apple's earnings growth rate is much higher than Microsoft's. Since Q1 2006, Apple has averaged a nearly 62% earnings growth rate (and last year was almost 155%). Over the same period, Microsoft had an average earnings growth rate a of 23% (55% last year).

Investors give a higher P/E to companies with higher sustained growth (so long as nothing brings doubt about the future prospects of growth). Those earnings could be paid out as dividends, and thus the investor "owns" a stake in those earnings (plus any cash, property, etc less debt) proportional to their stock holdings and thus will pay more in order to make more. If Apple can sustain that average growth rate for another 5+ years, the current P/E will look like a bargain. In a better economy (less perceived uncertainty about future earnings), Microsoft's P/E might look more like Apple's (but then Apple's would be higher than it is now, too).

Re:Embarassing?

[MightyMartian]

Well, yes, taking some of your money. But since the only way that you can make money is because the wider society sees that as a benefit, suck it up and pay your goddamned taxes. This illusion that somehow the money in your pocket came to you by yourself alone is the greatest lie of Libertarianism.

Re:Embarassing?

[gordguide]

" ... Socially liberal does not mean "spend money".

Any other definition necessarily requires taking my money and giving it to someone else. ..."

Libertarians do have a political philosophy that straddles Democratic and Republican party lines. This is somewhat complicated by the fact that neither Republicans nor Democrats follow either liberal or conservative political philosophy strictly; each borrows some from both, despite which brush people tend to paint them with.

Libertarians support lowering taxes, eliminating taxes, and would be against any new taxes. If that sounds Republican, note that they would also support eliminating government programs and spending on anything except essential services. That might mean no welfare, no unemployment insurance, no spending on drug rehab centres, but it also means no money for renewable energy, no farm subsidies, no foreign aid (1), and get the Defence Department out of the highway building and waterway dredging business. Let the airlines and passengers pay for the airports, let the sports teams pay for the stadiums, let the universities fund their own research.

Socially, there is really no difference ... it's "get government out of it" again. No restrictions on who can carry guns, no speed limits, no meddling in abortion debates or stem cell research, no teacher led school prayer but no restrictions on individual student religious expression. No DRM legislation, possibly no or very limited copyright and patent law. Let the gays in the military if that's what they want to do. Whatever, as long as it doesn't infringe on others rights to security and enjoyment of private property.

In other words, smaller government, period. No exceptions.

I'm not sure where the Tea Party fits in all this ... they started out acting Libertarian but then the issues started coming up and many of their answers were just regular Republican policies rehashed a bit.

The Republican Party supports spending on areas that Libertarians would be stridently against, just like they would be against most Democratic spending initiatives.

Re:Embarassing?

[ArcherB]

Libertarian==fiscally-conservative/socially-liberal

How can one be fiscally conservative (spend less money) and simultaneously be socially liberal (spend more money on those sad, pitiful poor people)?

Simple... Smaller Government, period!

Libertarians want the federal government to get out of everything that someone else can do. If the states can do the job, let the states do it. If the private sector can do the job, let the private sector do it. The only things the federal government should be doing is what is spelled out in The Constitution, per the 10th Amendment. If there is a job the Feds should be doing that is not spelled out in the Constitution, then the Constitution should be amended to give the Feds that power. The government should never stand in the way of your success. The government should never stand in the way of your failure. You should be free to do either. There can be no success without the possibility of failure. This is what is meant by fiscally conservative.

As for things like smoking pot or being gay or women being allowed to walk around in public topless, Libertarians don't care. As long as your actions don't endanger or strip the rights of another, then you should be free to do it. This is what is meant by socially liberal. It has nothing to do with money. It has everything to do with freedom.

Re:Embarassing?

[gordguide]

[Missed my footnote] ... but it also means no money for renewable energy, no farm subsidies, no foreign aid (1) ..."

(1) Foreign Aid is always money transferred from taxpayers to corporations or citizens. When the Federal Government offers Foreign Aid to another country, what really happens is the Government spends all the money on US jobs and products. If it's food, they buy the food from US growers and give that away. If it's technology or some building initiative, all the companies involved are US companies with American employees. If it's development aid, they hire US citizens to do the liason. If it's military aid, they buy the planes from Boeing or weapons from Raytheon and then give them to the foreign government. Foreign Aid is simply just another government transfer to Americans.

Re:Embarassing?

[ArcherB]

anyone who says libertarian doesn't understand the conflict in the term. it's basically republicans who don't want to be called republicans or someone who hates both parties but still leans republican. if someone declares themselves independent libertarian, then they're more acknowledging that they don't necessarily align with "libertarian" views. I have a friend like this, and it's basically republican but he doesnt' want to admit it.

I like to think of a Libertarian as a Republican that smokes pot and/or downloads porn. It could also be a Democrat who hates paying taxes to a federal government that either wastes the money or gives to someone who does not deserve it.

Re:Embarassing?

[realityimpaired]

You don't seem to understand what social liberalism is, that's how. :)

Being fiscally conservative means not spending money we don't have, and not spending money on things that aren't needed. Being socially liberal means supporting things like equal rights and progressive legislation, and not *just* spending on social welfare programs. Even including spending on social welfare programs, it's entirely possible to be both socially liberal and fiscally conservative: it's just a question of priorities. Rather than spending billions of dollars on scanners at airports that let perverted TSA agents look at naked pictures of kidlets, for example, we could have spent that money on raising the standard of life for inner-city youth, which, in turn, would reduce the statistical likelihood of their turning to drugs and/or crime. (not to say that all inner-city youth do this, or that it's limited to inner-city youth, but it's been shown by many studies that quality of life and wealth *significantly* reduces the chance that somebody will turn to crime).... of course, by helping to reduce inner-city crime rates, we would in turn reduce the amount of money that needs to be spent on crime prevention, on drug/substance abuse treatment programs, and also increase the amount of tax revenue available as these individuals find paying jobs.

Of course, I don't identify as a libertarian, but I do identify as a social progressive while wanting to remain fiscally conservative. :)

Re:Embarassing?

[snowraver1]

Yea, but usually when I want a cable, I want it 10 minutes ago, not in 4-6 business days. I usually pay the idiot tax and take home my steel braided, EMF sheilded optical cable that day.

Re:Embarassing?

[anyGould]

Libertarian==fiscally-conservative/socially-liberal

How can one be fiscally conservative (spend less money) and simultaneously be socially liberal (spend more money on those sad, pitiful poor people)?

By using better definitions. I'm fiscally conservative (we shouldn't spend more than we make), and socially liberal (we shouldn't legislate morality, and helping neighbors is helping ourselves).

Putting the two together means that we help as much as we can, and there's plenty of money we give away to big corps that we can cut to balance the budget without hacking at actual necessities.

(Local example: city gave tax breaks and subsidies to Dell to build a new call centre out here. Usual "new jobs" rhetoric. First downturn, Dell packs up and leaves down, before the tax breaks had even finished. Turns out that the city didn't bother to put any guarantees into the deal, and we ended up losing money and not getting any jobs either.)

Re:Embarassing?

[anyGould]

Socially liberal does not mean "spend money".

Any other definition necessarily requires taking my money and giving it to someone else.

Ah, the "anti-tax" argument. I'm happy with taxes. Honestly. Do I wish they were lower - of course. Do I think that we spend money on stupid things? Yep.

Put taxes are still cheaper than having my own private doctor and hospital, my own roads, my own water towers and power generation, my own private library, swimming pool, and so on. Governments should do these things, because it's cheaper for everyone to pitch in.

Re:Embarassing?

[realityimpaired]

Not true. Laissez-faire would support leaving the status quo as it is. The status quo, as it is, includes idiotic policies like DADT in the military, and the DOMA. (as pertains to the gay marriage issue Dave mentionned). If Laissez-faire had been allowed to continue, then we'd still have segregation, and it'd be illegal for a white girl to marry a black guy.

A social liberal wants to challenge the status quo when it's repressive, and support the status quo when it's progressive.

Re:Embarassing?

[Nutria]

Laissez-faire would support leaving the status quo as it is.

Eh? The status quo is DADT; how can the status quo be laissez-faire?

A "social laissez-faire" would allow anything (gay marriage, taking drugs, carrying guns, eating trans-fats, etc) as long as it doesn't directly impact someone else.

Re:Embarassing?

[Nutria]

I'm not sure where the Tea Party fits in all this

They fit into the "I want to have my cake and eat it, too" camp of idiots who don't realize that deep cuts means cutting Medicare.

Re:Embarassing?

[Nutria]

taxes are still cheaper than having my own private doctor and hospital, my own roads, my own water towers and power generation, my own private library, swimming pool, and so on. Governments should do these things, because it's cheaper for everyone to pitch in.

Honest question: where does it stop?

Is it cheaper for the gov't to grow the crops, thresh the wheat, mill the grain, bake the bread, haul it all around, distribute it to the peasants?

The Soviet Union tried that and failed miserably.

Re:Embarassing?

[Nutria]

Read the whole thread and know that I am not a libertarian.

Re:Embarassing?

[Stradivarius]

by "socially liberal" they really mean live-and-let-live, i.e. stop trying to use the government to coerce people into following your own cultural/religious/social views.

It's not really accurate to label this "socially liberal", as many liberals in the US are as fond of using government force as social conservatives. The liberal nanny-state folks in the US tend to focus more on food and lifestyle, whereas conservative nanny-staters focus on sex and religion. But the latter seems to be a more recent phenomenon, so folks seeking the opposite of socially conservative nanny statism tend to say they're socially liberal.

"Libertarian" is probably more accurate, but the term is not so widely known.

Re:Embarassing?

[Stradivarius]

I need to proofread better. I meant to say the *former* (food/lifestyle judgmentalism) is more recent.

Re:Embarassing?

[mcgrew]

So THAT'S why Jasper Noone killed all those people!

Dude, it's two words. No one. Not "noone".

Re:Embarassing?

[toastee]

A libertarian is a republican that is uncomfortable lying about their porn, drugs, and being nice to other humans habits.
Republicans are just as into drugs and porn, they just lie about it, and act offended.

Re:Embarassing?

[budgenator]

Liberals generally want to give a man a fish, Libertarians would rather teach him to how to fish.

Re:Embarassing?

[Nutria]

we shouldn't legislate morality

Then what are the rules against "hate" speech and discrimination against gays/blacks/women?

They are legislation against the kind of morality that Progressives disagree with.

Re:Embarassing?

[bonch]

Actually, it was probably the original article's headline:

We have changed the title of the story as some commentators have rightly pointed out the previous one was too biased.

Re:Embarassing?

[Anonymous Coward]

Microsoft is arguably the best at optimizing compilers. Visual Studio compilers have always had better optimization that anything the FOSS community has put out.

Re:Embarassing?

[Anonymous Coward]

I believe in tax cuts and gay marriage

I don’t believe in square circles. That said, I think marriage should be strictly religious in nature and the government should have nothing to say about it whatsoever.

Re:Embarassing?

[Nutria]

Finally, someone who makes sense!

Re:Embarassing?

[Nutria]

What do Conservatives want? To sell him the fish?

Or steal it from the Amerindians?

Re:Embarassing?

[jefe7777]

libertarians are reluctant to spend other people's money.

Republicans and Democrats do not.

libertarians warn of the dangers of an authoritarian state.

Republicans and Democrats embrace the authoritarian state. It helps with their warfare/welfare agenda.

libertarians don't want to save the world, just their neighbors.

Republicans and Democrats each are 100% sure they know best how to save the world either from terror or poverty or inquality, at gun point if necessary. They don't give a rat's ass about their neighbors though.

real libertarians don't want to even be called Libertarian or libertarian, in fact they find the idea of collectivist labeling the domain of left brain sheeple.

when I was a kid, I was a democrat, then I grew up and became a republican, then I realized that both parties were filled with lying sons of bitches, and became Libertarian, then found that card carrying Libertarians are full of hot air, so I became libertarian (small L), then realized that I didn't fit that mold either.

Now I find most politicians, partisans, fox news, cnn, msnbc, etc, and a surprisingly large number of slashdotters to be lying, self deluded sacks of shit.

Don't be surprised if you fall into that category.

I didn't make the world. I just live in it.

Re:Embarassing?

[LordThyGod]

Yea, but come on, this *is* MS we are talking about. It is a perfectly legitimate conclusion based on 20 years+ of history. Why would anybody give them the benefit of the doubt when it comes to integrity and such?

Re:Embarassing?

[BitZtream]

I'm not the kind of person who works on GCC, but I've done a little bit of compiler type work for my own 'languages' stuffed into apps I've written.

Personally, I can totally see how that kind of a performance hit would happen, handling a function entirely differently when it returns a value versus when it doesn't return a value.

If it never returns a value, and never touches non-local vars (or objects), its easy to just nop the whole thing, when values get returned then you have to wonder what the caller is doing with the data, though you could just turn the function into a static returned value if it still touches no external vars, which would as fast as not calling it for practical purposes.

I think Microsoft should have people that would have written tests to catch this sort of thing, so I'm not ruling out the possibility of cheating by any means, but I can understand exactly how this got missed.

Re:Embarassing?

[poetmatt]

pretty much. however, considering how our government is basically a two party system, being libertarian means nothing in the big picture right now.

either you have seats in congress and influence or you're more likely to be able to change the direction the earth rotates.

Re:Embarassing?

[Carewolf]

No, economically liberal, would be letting people do whatever they want with their money, lower taxes and smaller govenment. Social liberal -> the government should not interfere in our social lives. Economically liberal -> the government should not interfere in our economic lives.

  Economical liberalism is sometimes also known as libertarianism in the US, because the term "liberal" in general have been contaminated with the politics of a single US party. This is not the case outside the US, and could be argued to be a misuse of the term.

Re:Embarassing?

[mcgrew]

I think by "socially liberal" he means "nothing should be illegal unless it harms someone else." But as to fiscal conservatism, that's the Libertarian Party, not libertarians in general. Personally, I agree that anything that doesn't trample someone else's rights should be legal, but at the same time I'd like to see universal health care, and have the poor taken better care of. "There but for the grace of God go I".

And I'd like to see corporate power reigned in, and I'd like the corporates to stop getting government welfare and getting away without paying taxes. So I seldom vote for a Libertarian candidate.

Re:Embarassing?

[omfgnosis]

Well, I can't speak for right libertarians but for left libertarians it's easy. We don't think the solution to political problems is to throw money at them, we think there are underlying systemic problems that need to be addressed, in a fundamental way rather than a patchwork way.

Poverty doesn't exist because the government doesn't have adequate social programs to funnel money into the hands of poor people; poverty exists because wealth is power and is used to leverage more wealth and power. The genesis of wealth is the conversion of public property into private, which is to say theft from society. And this wealth was not created in a market, but in feudal societies with brutal enforcement of class. It was preserved when the "enlightened" transitioned to markets.

All of these shenanigans are shielded by even the most "liberal" governments, not surprisingly run by the same wealth interests. Nuts to the lot of them.

Re:Embarassing?

[Zak3056]

That's actually why there's blackout periods for insiders buying/selling shares, as part of the Sarbanes-Oxley rules. When I was at Dell, I wasn't allowed to buy/sell within 30 days (either way) of any public statement regarding earnings or future plans.

Is that really correct? I'm generally in favor of laws and rules that prevent insiders from profiting on confidential information at the expense of the public, but the above seems really, really, harsh. The company does, at a minimum, four such statements a year (quarterly earnings), not counting other possible announcements (forecasts, Jobs doing his "one more thing" bit, etc). I mean, even just looking at quarterly announcements, assuming that they were equally spread across the year, you would prohibit anyone classified as an insider from trading for 244 calendar days, or 2/3rds of the year! Add on anything outside of this, and you are basically locked in or locked out of the market. completely.

It's like one of those laws that says sex offenders can't live within 1000' of a school--good luck finding anywhere that actually meets this definition in any US city.

Re:Embarassing?

[Nutria]

The genesis of wealth is the conversion of public property into private, which is to say theft from society.

Interesting.

So what do Left-Libertarians say that we do with private property?

Re:Embarassing?

[omfgnosis]

The problem is that the term "liberal" carries a few different contradictory meanings. It's generally safe to assume that:

- "socially liberal" is usually meant to be taken literally;
- "economically liberal", in (and perhaps some places outside) the US, is usually taken to mean something like a hybrid of corporatist and social democratic economic policies;
- "economically liberal", taken literally, is meant more or less to mean the same as the colloquial "economically conservative".

Re:Embarassing?

[dave562]

30 days is nothing. A person like Ballmer has insights into where Microsoft is going five plus years from now. If he looks down the road and sees tough times ahead, a 30 day delay on divestiture is a minor inconvenience. Like the OP said, when the CEO divests 25% of his holdings, that is not a good sign. He's basically admitting that in the long run, Microsoft is not the best investment.

Re:Embarassing?

[BuckaBooBob]

If there was not Foul Play to fake out the benchmarks then it goes to show that the robustness of the changes they did to their JavaScript engine are far from robust when minor changes have such a dramatic effect. So now that this is brought to light when the fix comes out you will see what direction it will go..

If they fix the 20ms and they run at 1ms.. and no-one can find a way to tweak the test to get inconsistent results then you will see that MS did some admirable work.. but if the inital testing goes south to 20ms then the QA team at MS should be fired for not suspecting something in amiss when the results from a benchmark are simply too good to be true. (Or it will confirm that MS did meddle and find a way to fake out the Benchmark.. Which benchmarks should be adjusted so that real world performance would be reflexed by the results )

Re:Embarassing?

[afidel]

Uh, the CEO diversifying his portfolio is perfectly normal, in fact a CEO who doesn't is probably too financially reckless to lead a bluechip.

Re:Embarassing?

[ATMAvatar]

There is no objective, hard line in the sand for where to stop. That's the hard part.

The same could be asked of going the other way with it. If you start cutting government expenditures to pave the way for lower/eliminated taxes, where does it stop? Do we cut education? Social security? Infrastructure? Emergency services? Defense?

You have to weigh pros and cons as a group and determine what the majority wants, which can also change over time.

For example, with any kind of infrastructure, you pay taxes to have a network of interconnected nodes of varying quality versus a private network which may be excellently maintained but with no guarantee of coverage.

Re:Embarassing?

[hairyfeet]

Not if he was going to get bit in the ass by taxes. Sorry I can't find the link ATM, but I'm sure someone else can. What I read was come Jan 2011 Ballmer would have seriously gotten bit in the ass if he would have sold those stocks and since he had just defeated the Washington State sales tax he went ahead and got some cash when he could. IIRC the numbers they showed was something like 180million+ it would have cost him in taxes if he sold those shares just two months from now.

As for TFA, is anybody surprised? News Flash: Damned near EVERYBODY cheats in some way or another. Now I don't use IE and get my customers off it ASAP thanks to the bad taste IE 6 left, but everything from "quick starters" to Quack.exe to only rendering half the scene (Geforce 5xxx) everybody has put in cheats it seems on these benchmarks at one time or another, hence why nobody should really be picking a product based on benchmarks. IMHO the only real test there is for anything is real world tests. Just like passing ACID doesn't make you the most perfect browser so does getting good grades on sunspider (which IIRC is made by Apple and thus probably leans towards Webkit) make you the fastest. And it isn't like there aren't a bazillion sites where they have run real world tests like Gmail, Youtube, FaceBook, the apps people use every day, and timed them.

While MSFT needed to be called out for the underhanded BS, basing your choices on benchmarks is as stupid as those customers I have that base all their PC purchases on how high they can get on some benchmark leaderboard, while ignoring the fact that other than a couple of hundred $$$ the difference between 100FPS and 110FPS IRL is pretty much zilch when it comes to gameplay. And it isn't like these browsers cost money folks. Try ALL the good ones, even the fringe ones like Comodo Dragon (pretty badass based on Chromium) or Kmeleon CCF ME (ultra fast on older hardware with a nice UI) and then judge for yourself.

Re:Embarassing?

[omfgnosis]

First of all, I (and I'll not speak for all left libertarians here, as there is some debate on the matter, but I think I'm expressing the majority opinion) distinguish "personal property" (that is, what you own and use) from "private property" (that is, property owned privately, usually by an organization, but not owned or used by an individual, and leveraged to extract profit). This distinction is important before any discussion of any wealth-distribution theory.

Personal property is personal property. I can think of scarce few real leftists (which is to say, true socialists, communists, anarcho-communists, left-libertarians, etc) who include personal property when they say "property is theft". Private property, on the other hand, being the spoils of a great and sustained theft from the public, belong to the public and should be returned to the public.

Note here that I do not mean the state when I say public. Which is to say that I'm not an advocate for systems like the Soviet Union, but I am an advocate for movements like the worker takeover of factories we see in some Latin American countries.

In short, I advocate taking back what was stolen from us.

Re:Embarassing?

[bl8n8r]

> And their 10-Q definitely indicates that they're not losing money.

Microsoft added 'optimization' for the 10-Q results as well.

Re:Embarassing?

[shutdown -p now]

anyone who says libertarian doesn't understand the conflict in the term. it's basically republicans who don't want to be called republicans or someone who hates both parties but still leans republican.

A republican, as in, not a monarchist?

(the world does not end at the border of US)

Re:Embarassing?

[Aerorae]

And if you think that there is any hardware or software manufacturer out there that DOESN'T optimize for benchmarks, you're frighteningly naive.

Re:Embarassing?

[shutdown -p now]

The benchmark in question can be considerably optimized by dead code elimination, since a computationally expensive function in there (one that loops computing stuff) does not have any observable side effects, and does not return any values - so it can be replaced with a no-op. It is a perfectly legitimate optimization technique, but the one which tends to trip up naively written benchmark suites because they assume that "for(int i=0; i < 1000000; ++i) {}" is going to be executed exactly as written.

Thee was actually a similar case with artificial tests in the past - Haskell (GHC) scores on the Programming Language Shootout. Most tests there were also written as loops with some computations inside and no side-effects, on the presumption that compilers will leave the computations intact even though their results are never used. Well, one thing that GHC has always had is a particularly advanced dead code eliminator, and it compiled most of those tests down to what is essentially equivalent to "int main() { return 0; }" - with corresponding benchmark figures. Once they've changed the tests to print out the final values, this all went back to normal.

In this case it's not quite that simple, because seemingly trivial changes to benchmark code - changes which do not change the semantics of the code in any way - trip off the dead code elimination analyzer in IE9 JS engine. However, it is still an open question on whether it is deliberate, or due to bugs in the analyzer. One plausible explanation was that analyzer is written to deal with code which at least looks plausible, and neither of the suggested optimizer-breaking changes (inserting an extra statement consisting solely of "false;" in the middle of the function, or "return;" at the end of it) make any sense in that context. Any dead code elimination is necessarily pessimistic - i.e. it tries to guess if the code is unused, but if there are any doubts (e.g. it sees some construct that it doesn't recognize as safe) it has to assume otherwise.

The only true way to test this is to do two things:

1. Try to change the test in other ways and see if there are any significant diffs (such that they are not reasonably detectable as being the same as the original code) which will still keep the optimizer working.

2. Write some new tests specifically to test dead code elimination. Basically just check if it happens on completely different test cases.

By the way, the guy who found the discrepancy has opened a bug in MS bug tracker regarding it, in case you want to repro or track further replies.

Re:Embarassing?

[shutdown -p now]

All JS functions return values. If no value is specified in the "return" statement, or if the return happens due to reaching the end of the function, "undefined" is returned. So adding a "return;" at the end of the function which does not otherwise return anything does not change its meaning in any way.

That said, it is quite possible that the optimizer does not know this, and treats any "return" as a signal that the function returns a meaningful value. Which then indicates a bug in the optimizer.

That doesn't adequately explain why "false;" is not optimized away consistently, though.

Re:Embarassing?

[msclrhd]

The return statement was "return;" i.e. a return statement that did not return anything. Looking at the other JavaScript engines, that line added at most 1ms, while with the IE engine it added 19ms. If the IE9 JS engine is handling this function in a super-efficient way that is not due to cheating, the optimisation must be highly sensitive to variance.

One way to check if the IE9 engine is doing some sort of special casing (e.g. hashing the text for the function) would be to change the name of a variable. This should not change the behaviour of the engine as it is the same code (there are no extra elements in the tree, like additional returns). If the IE9 engine is cheating, this should jump from 1ms to 20ms like the other variances. If it is an optimisation bug, the performance should be 1ms for both cases.

Re:Embarassing?

[NotBornYesterday]

Being socially liberal does not necessarily mean paying for all the world's wants. Libertarians generally speaking are concerned with the rights and liberties of individuals, and keeping the government's nose out of such affairs. For example, supporting gay marriage is perceived as a "socially liberal" stance, but in fact, it falls in the realm of individual rights, and the government should butt out (ie, accept the rights of couples to marry, refrain from passing quasi-religious laws interfering). As such, libertarians I know are generally supportive. Same with legalizing marijuana. Obviously not all libertarians agree on this, of course, but I would suggest looking out for the many so-called libertarians who are more like conservatives in libertarians' clothing.

Re:Embarassing?

[NotBornYesterday]

Please don't smear and presume to speak for those whom you a) don't understand, and b) don't represent. There is no conflict in being libertarian and supporting causes like marijuana legalization and gay marriage that are traditionally "socially liberal" positions. Small, effective, efficient government that mostly stays out of people's lives does not conflict with these causes either, and in fact is the goal of true libertarians. Asking your friend why he does not consider himself a Republican (assuming you haven't already) might be enlightening.

Re:Embarassing?

[jeff4747]

Where's the "-1 you can't possibly be this dumb" moderation?

Re:Embarassing?

[omfgnosis]

Well one reason is that at least some of their claims about huge improvements in IE 9 are true, whereas similar claims in the past required all sorts of mental acrobatics. The IE 9 development process doesn't really have an analogue in IE's past.

Re:Embarassing?

[clone53421]

Conservatives agree with the Libertarians, on this particular issue. Other issues, not always so much.

Re:Embarassing?

[losfromla]

are you then stating that the US is a police (military) controlled state and thus dismantling the military would result in or is equivalent to anarchy? Help me here, cause I don't see the tie-in between dismantling the military and anarchy. If the military is the only thing preventing anarchy, then the country has fallen further than I am aware of and our basic liberties nothing more than talking points.

Re:Embarassing?

[clone53421]

No, that’s not what he meant at all. He meant that the libertarian party is full of anarchists who want to decimate the military, for whatever reason.

Re:Embarassing?

[losfromla]

(I hope to, one day, be rich, and I'd like to keep my money after all that hard work and not be labeled a 'fat cat', thanks).

Try this, put all your hope in one hand, shit into the other, check which one fills up first.
 

Don't lump me in with either of these idiotic parties, please.

Will just chalk you up as a teabagger then, happy?

It is your irrational hope of one day being rich that makes you vote alongside those whose peer you wish you could one day be, never realizing that it will never ever happen. They (the rich) are happy to string you along since your vote counts as much as any other idiot's vote who votes against his own economic interests.

Re:Embarassing?

[Carewolf]

No. I would argue your definitional of "economically conservative" is also contaminated by US politics. Economically conservative means not changing the economic status quo, keeping the money with "the powers that be". Economical liberalism and libertarianism has the basic idea that the free market would challenge old structures, constantly keeping the market healthy through competition. Competition is however the exact opposite of what a convervative wants. So if you believe in this mechanism of a free market a conservative would oppose economical liberalism (which they did 100 years ago). As you may have observed though, non-intervention seems to benefit existing econmic powers more than it benefits entrepreneurs, which is why the same idea of economic freedom is often embraced by both conservatives and liberals.

Re:Embarassing?

[am 2k]

With the caveat that those guys are almost certainly not the same ones who wrote the js interpreter.

Re:Embarassing?

[omfgnosis]

Sorry, I should rephrase that: "economically liberal", taken literally, is meant more or less to mean the same as the colloquial "economically conservative" *is meant*, irrespective of implementation.

The rhetoric of "economic conservatives" and actual existing economic conservative policy are drastically different. Even as a left libertarian, a lot of "economic conservative" talk appeals to me; but I know better.

Re:Embarassing?

[v1]

it's pretty clear that the engine is looking for an exact match for a known routine (in oh say, a benchmarking application) that it knows a way to optimize, or even just plain cheat its way around doing the task and get away with it. and adding a line or two of actual code (as opposed to say, comments) causes it to not recognize what it can optimize, and thus fail like it should.

wouldn't surprise me if there's a lot of that going on in the world of benchmarking. benchmark apps need to be more dynamic... not just present a range of different kinds of test, but actually generate pseudorandom things that on the average will always produce very close to the same results, but that cannot be "pre-optimized" for as easily as SunSpider is.

at least they're not flaming obvious over it. I recall awhile ago someone got busted for detecting the benchmark app's process running on the computer and modifying its behavior accordingly. Probably some talk went around like "wow our test on xxx really stinks... what can we DO about that? not in general cases eh? any way we can say, look for their test and do it some other way?" I bet if you know what test is being thrown at you, this specifically, you can really optimize it well. They just never thought to nerf their optimized code a bit to make it more realistic. Problem here is it's an evolutionary process. They're not going to stop cheating, they're just going to learn to cheat better.

Re:Embarassing?

[h4rm0ny]

Being fired by EDS is like being dumped by Margaret Thatcher. It's pretty insulting in that all things considered, it should be the other way around, but let's face it, being together is not a nice thought.
EDS - Money goes in. Nothing comes out.

Re:Embarassing?

[thasmudyan]

I agree. While the addition of a simple true or return statement at the end should not make any (significant) difference in a straight-forward execution scheme, it might very well convince a simplistic "dead code" optimizer to not prevent the execution of the for loop. I say simplistic, because the optimizer (if it indeed exists in IE) should still recognize that none of the operations in the for loop have any meaning and a constant value is returned in any case (=null), but instead it probably just checks if anything happens after the loop. By the way, it would be trivial to run a few tests to investigate if this is only happening with SunSpider code. My guess would be that it works the same way for any code that exhibits this pattern.

Re:Embarassing?

[CheerfulMacFanboy]

No, economically liberal, would be letting people do whatever they want with their money, lower taxes and smaller govenment. Social liberal -> the government should not interfere in our social lives. Economically liberal -> the government should not interfere in our economic lives.

Ohh, so someone who wants to help support the poor with a soup kitchen out of his own free will instead of giving that responsibility to the state, who wants to pay less taxes and doesn't like the state interfering with their business is a Libertarian? Someone like Al Capone?

Re:Embarassing?

[DragonWriter]

First of all, I (and I'll not speak for all left libertarians here, as there is some debate on the matter, but I think I'm expressing the majority opinion) distinguish "personal property" (that is, what you own and use) from "private property" (that is, property owned privately, usually by an organization, but not owned or used by an individual, and leveraged to extract profit). This distinction is important before any discussion of any wealth-distribution theory.

Personal property is personal property. I can think of scarce few real leftists (which is to say, true socialists, communists, anarcho-communists, left-libertarians, etc) who include personal property when they say "property is theft".

IME, a more common distinction that the weird (and somewhat problematic, since when the owner is an individual leveraging it for profit, it clearly is being "used" by an individual) one you propose between "personal" and "private" property is between personal property in almost the exact sense that is known in legal property theory and real property. There are certainly brands of leftists that are opposed in theory to private ownership of real estate (though, usually, these leftists don't favor actual abolishment of property rights in land, just focussed ad valorem taxes) such as georgists and geolibertarians.

Re:Embarassing?

[CheerfulMacFanboy]

Laissez-faire would support leaving the status quo as it is.

Eh? The status quo is DADT; how can the status quo be laissez-faire?

A "social laissez-faire" would allow anything (gay marriage, taking drugs, carrying guns, eating trans-fats, etc) as long as it doesn't directly impact someone else.

Eh? Rand Paul is the poster child for laissez-faire segregation: http://www.salon.com/news/opinion/feature/2010/05/20/rand_paul_maddow_segregation_open2010

Paul argued that the parts of the Civil Rights Act of 1964 that require private businesses serving the general public to serve all customers without regards to their race, gender, religion, or national origin need further "discussion." He insisted that he agreed with the parts of the act that required publicly owned facilities like public transportation to serve everyone regardless of race, but that private businesses should have been exempt.

According to his beliefs, it should still be legal to send Rosa Parks to the back of any privately operated bus. Okay, bad example, as she is dead - doesn't make him right though.

Re:Embarassing?

[losfromla]

these are self-labeled anarchists? The only anarchists I'm familiar wear Doc's and mohawks and enjoy elbowing friends to loud music.

Anarchists really? I want to decimate the military, actually more, I'd like to cut it to 1/10th it's size at most and I think I'm not an anarchist. I think there are a lot of people who don't consider themselves anarchists who would like to see a smaller military.

Again, what makes them anarchists? The fact that they want a smaller military, or the fact that they are libertarians?

I would have thought the anarchists would not belong to any party, sort of by definition, knowhatImean? ;-)

what I question at root is how does: dismantling military = anarchy? Can you read ACs mind and answer or maybe you can explain how you equate the two?

sorry to ramble I wanted to really get clear what I am asking to be explained as I find such strident opinions fascinating and wish to understand their roots.

Re:Embarassing?

[Score Whore]

Being socially liberal means that you don't give a crap who other people are sleeping with, what they are smoking and other questions of personal morals. It has nothing to do with giving money to poor people or paying for their milk or their health care.

Re:Embarassing?

[anyGould]

Honest question: where does it stop?

Is it cheaper for the gov't to grow the crops, thresh the wheat, mill the grain, bake the bread, haul it all around, distribute it to the peasants?

The Soviet Union tried that and failed miserably.

It stops at the point where the benefits of private ownership outweigh the benefit of community ownership. The exact line will vary from community to community.

Re:Embarassing?

[Blink Tag]

The blackout window rules pre-dated SarbOx, but to answer a sibling post, yes, those rules are in place for employees of all publicly traded companies. The rules don't prevent trading other companies (although you'd be walking a fine line if you traded shares of a company your employer does business with), so one is not "locked out of the market" as one poster put it. Employees of public companies are also prohibited from shorting their company's stock.

One of the reasons Balmer's stock sale is such a big deal is that the top tier of employees in public corporations must publicly pre-announce their intention to sell (again, part of the same trading rules, another hedge against insider trading). I haven't looked at the announcement, but it may be something as simple as him cashing in the options CEOs are often paid with. Even if he just wished to convert his options to stock, some selling would necessarily be involved.

Re:Embarassing?

[drsmithy]

Is that really correct? I'm generally in favor of laws and rules that prevent insiders from profiting on confidential information at the expense of the public, but the above seems really, really, harsh. The company does, at a minimum, four such statements a year (quarterly earnings), not counting other possible announcements (forecasts, Jobs doing his "one more thing" bit, etc). I mean, even just looking at quarterly announcements, assuming that they were equally spread across the year, you would prohibit anyone classified as an insider from trading for 244 calendar days, or 2/3rds of the year! Add on anything outside of this, and you are basically locked in or locked out of the market. completely.

You're only restricted from trading if you're high enough up the org chart to have access to important information - most people aren't. Further, you're only restricted from trading in the stock of that particularly company, so you're in no way "locked out of the market".

Re:Embarassing?

[drsmithy]

Liberals generally want to give a man a fish, Libertarians would rather teach him to how to fish.

Liberals are happy to give a man a fish when he's starving. Libertarians will tell him if he'd only tried harder to catch more fish, he wouldn't be starving, even though he can't afford to buy a rod and reel.

Re:Embarassing?

[drsmithy]

The purpose of a benchmark is to try to show how performance will be in the real world. If a given application has been programmed to do very well in a given benchmark yet does not do as well with a real-world situation, then the benchmark results are flawed.

If a benchmark is so easily gamed, then its ability to return meaningful results at all is basically nil.

Re:Embarassing?

[Anonymous Coward]

I don't want a swimming pool - why should I pay taxes for it......

Re:Embarassing?

[Lord of the Fries]

Nicely put. Agree for me 100%

Re:Embarassing?

[ooshna]

So you would rather pay all those taxes now when your middle or lower class?

Re:Embarassing?

[Richard_at_work]

The purpose of a benchmark is to try and show performance when running the benchmark - if you think its anything else, then you are deluding yourself.

If you want to see real world results, then use the real world.

That said, everyone is presenting a case here without any real evidence, this is nothing more (currently) than hype on the articles part.

Re:Embarassing?

[chrb]

Sounds like you're talking about the quack.exe fiasco.

Re:Embarassing?

[clone53421]

Yeah, and how on earth did all those natives catch fish before the white people introduced fishing poles?

Re:Embarassing?

[thejynxed]

So very true. I am always grateful to the Fates that those EDS suit & tie folks gave me the heebie-jeebies when they came recruiting at my school. Out of my class of 50, only two people took on jobs there, with many of my fellow classmates picking up on the same vibe and totally NOT digging the ridiculous blue/blue/tie dress code they had at the time.

Re:Embarassing?

[gangien]

This illusion that somehow the money in your pocket came to you by yourself alone is the greatest lie of Libertarianism.

Which is why almost no libertarian would say that(except maybe in primitive examples). Go read/listen to milton friedman's explanation on the making of a pencil.

What libertarians say is individuals making their own choices, builds society. Not government making choices for you.

Re:Embarassing?

[gangien]

No, liberals will force me to give the man a fish. libertarians will say, it's up to me, whether I give him a fish.

Re:Embarassing?

[MightyMartian]

This is another issue. Governments are not separate from societies, they are an integral part of a society.

Re:Embarassing?

[gordguide]

I did not once mention the SEC. Of course the transaction is legal.

I'm talking as an investor. I play with my own money and I've learned some lessons along the way. An investor who does not perform due diligence when trading gets what he deserves.

I can assure you I never trade without checking the [perfectly legal] trading activity of insiders. This is stock market 101 stuff.

Re:Embarassing?

[lonecrow]

The original "Liberal Revolutions" that hit Europe a century and a bit ago were all about assigning sovereignty to the individual rather then to church or king. (http://en.wikipedia.org/wiki/Revolutions_of_1848, http://en.wikipedia.org/wiki/John_Stuart_Mill)

I believe at some point in the English parliamentary tradition, to be a conservative meant you were in favor of using monetary policy instead of fiscal policy to administer the economy.

In any case the fact that even the democrats consider the label "Liberal" to be a bad word is seriously disappointing. The Dem's have to grew a pair and stop caving in to Republican propaganda. The USA was the model for western liberalism and now its a dirty words? WTF?

Microsoft cheating?

[Anonymous Coward]

Inconceivable!

Re:Microsoft cheating?

[lavacano201014]

I do not think that word means what you think it means

I'm sure there's no hyperbole in this article

[Anonymous Coward]

No, none what-so-ever.

Welcome to your daily two minutes hate, Slashdot.

Re:I'm sure there's no hyperbole in this article

[stewbacca]

*although it was developed by Apple*

Doesn't this only make sense if there is reason to suspect Apple doesn't develop things well?

Re:I'm sure there's no hyperbole in this article

[Anonymous Coward]

Why don't you try reading it before you make that claim? The article is a few simple benchmark results and mild speculation as to what caused them. The summary may be inflammatory, the article goes out of its way not to be.

Re:I'm sure there's no hyperbole in this article

[king neckbeard]

No, it's just stating that it has basically become a standard benchmark despite Apple not being a standards organization.

Re:I'm sure there's no hyperbole in this article

[beelsebob]

But then, that's how most things become standards

A, who is not a standards organisation develops it.
B, who is also not a standards organisation uses it.
If sufficient numbers of Bs use it, it becomes a de-facto standard
Sometimes C, who is a standards organisation says it's a standard.
Then it becomes a de-jure standard.

Re:I'm sure there's no hyperbole in this article

[Jesus_666]

Well, it's a browser benchmark and Apple makes a browser. It's not far-fetched to assume that the benchmark might be skewed towards those areas where Safari performs especially well in. The fact that it's used industry-wide would indicate that Apple resisted the temptation and actually made a fairly objective benchamrk.

Re:I'm sure there's no hyperbole in this article

[Jesus_666]

Actually, the article's original headline was inflammatory. They changed it after someone pointed out that it doesn't fit the story.

Re:I'm sure there's no hyperbole in this article

[GooberToo]

Everyone uses because it is a fairly objective benchmark.

For the record, I caught wind of this a month or two ago and posted it here in a firefox performance article. I was trolled and troll moderated despite pointing to the Mozilla team's own experiments.

The ONLY reasonable explanation, assuming you actually understand the implications of what is it you're (generalized readers, not your specifically) reading, which based on previous happenings is questionable, is that Microsoft is cheating their asses off by identifying the exact benchmark and returning a pre-computed value. Either that, or this is indicative of a horrible optimization bug which would negatively effect all javascript in their browser and it would be impossible for them to be competitive in the least. Given there is no evidence to support the later, the only reasonable conclusion is they are cheating their asses off in these benchmarks.

Re:I'm sure there's no hyperbole in this article

[Sleepy]

No, none what-so-ever.

Welcome to your daily two minutes hate, Slashdot.

Wow, you got +4 for not reading the article (or for deliberately trolling).... we could live without you coming here to trash the forum, you know.

You did do us a service however... by exposing cheating by MODERATORS (those who gave you "+1").

See - It takes 2 seconds to skim through this article and get to the diff, and see the bare "true;" instruction (which is being used as a no-op in this context). This benchmark change does not significantly alter the benchmark (it doesn't change what it measures) but it DOES change the code signature of the benchmark.

It is unreasonable to expect a browser to change score just because the benchmark code signature changes... this strongly suggests that the browser is "looking" for the benchmark.
You wouldn't need to be a nerd to make the connection, because the article explains this in plain English.

So at the very least, you helped expose a few fake/dummy/abusive accounts which were set up for destructive moderation, or those who simply allocate mod points without reading the article (something moderators are required to do). I expect these accounts will be banned from future moderation.

Thank you for inadvertently helping improve Slashdot moderation. :-)

Re:I'm sure there's no hyperbole in this article

[Burb]

Let's review that argument, shall we?

Statement 1: The ONLY reasonable explanation is that they are cheating. Or there is an optimisation bug which screws performance.

Comment: I think you need to look at meaning of "ONLY" as you have used it, and the way the rest of the world uses is.

Statement 2: there is no evidence of an optimisation bug, therefore it must be cheating

Comment: One could plausibly argue that there is no actual evidence of cheating, therefore it's an optimisation bug. Since there is no internal evidence of any kind.

Yes, it COULD be cheating, but you can't argue that it must be so based on the information in the article. There are enough weasel phrases in your analysis to populate a weasel world theme park.

Re:I'm sure there's no hyperbole in this article

[pla]

Why don't you try reading it before you make that claim? The article is a few simple benchmark results and mild speculation as to what caused them. The summary may be inflammatory, the article goes out of its way not to be.

1) Microsoft beats everyone else by a factor of 10.
2) Making any of a number of effectively cosmetic changes to the function results in Microsoft taking twice as long as everyone else.
3) Making the inner loop 10x longer makes everyone else take 10x longer, except MS, who takes 180x longer.

Sorry, but if that counts as an optimization "bug", I have a bridge to sell you.

Re:I'm sure there's no hyperbole in this article

[clone53421]

Microsoft is cheating their asses off by identifying the exact benchmark and returning a pre-computed value.

Do they even have to return something? That function uses nothing but local variables and has no return value! For all we know, IE 9 cleverly optimizes it to function cordicsincos() { return; }...

Re:I'm sure there's no hyperbole in this article

[hrimhari]

To summarize, there's not enough evidence of either one thing or another (yet). So taking conclusions would be premature. Calling it cheating is amusing, tho.

Re:I'm sure there's no hyperbole in this article

[Anonymous Coward]

Statement1 has a formal error. Don't grab for straws.

The alleged evidence for the lack of an optimisation bug was that it would have "negatively effect all javascript in their browser and it would be impossible for them to be competitive in the least"

So, run again.

Re:I'm sure there's no hyperbole in this article

[GooberToo]

Statement 1: The ONLY reasonable explanation is that they are cheating. Or there is an optimisation bug which screws performance.

Dumb statement since that was already addressed. If placing a null op completely destroyed performance, they would be completely unable to perform - ever. If placing an unused variable completely destroyed performance, they would be completely unable to perform ever. Both of which completely ignore that both of these optimizations are some of the most basic and most obvious which ever get coded. Which brings us full circle. The ONLY likely explanation is they are cheating. Period.

Its so frustrating when people who have absolutely no idea what they are talking about attempt to dispute something based on the fact they have absolutely no idea what they are talking about. Defacto on slashdot these days...

Re:I'm sure there's no hyperbole in this article

[Anonymous Coward]

Whatever. No accounts will be shut down over this. You've done nothing and the moderation has done nothing.

Re:I'm sure there's no hyperbole in this article

[Your.Master]

What's also frustrating is people who claim that just because two different sequences of code can be proven semantically equivalent, that they must therefore behave equivalently in the presence of a just-in-time optimizing compiler.

A dead-code elimination heuristic should not be comprehensive -- even if something is in principle provably dead-code, it's not necessarily worth it to undergo the analysis to try to prove it. For instance, you might find that code blocks of X instructions of can be eliminated Y% of the time if you spend Z milliseconds to prove it, or YY% of the time if you spend ZZ milliseconds. So if you take a guess at the run-tome of those X instructions (including things like cache impact) and multiply by Y% and the result is greater than Z, you shouldn't even bother trying. This analysis could come out as "try dead code elimination on any function of exact same bug that would affect it without cheating. In both cases, an algorithm (either legitimate dead code elimination or the hypothesized Sunspider code elimination) is running against code that has not been preprocessed to remove no-ops (that is, an initial dead code elimination pass that only runs at the single-statement level). It's really unlikely that Microsoft would have cheated in such a poor way that only affects one microbenchmark.

All this proves is that this microbenchmark is a poor benchmark. It's not even a fantastic test of dead code elimination, because this isn't really a realistic piece of dead code.

Benchmarks

[Mongoose Disciple]

This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.

Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.

Re:Benchmarks

[Lunix Nutcase]

This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.

Except that the article writer tries to claim that that couldn't possibly be the case and thus claims that Microsoft is "cheating" instead. Basically this is an invented controversy.

Re:Benchmarks

[Anonymous Coward]

Optimize != cheat

Re:Benchmarks

[QuietLagoon]

This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.

It shows that Microsoft is more concerned about getting a good score on the benchmark than they are about providing a good customer experience.

Re:Benchmarks

[Anonymous Coward]

Why not create a benchmark which boots the operating system and launches a web browser. Then I wouldn't care if they cheat!

Re:Benchmarks

[Lunix Nutcase]

And there is no actual evidence that they are actually cheating. The article writer puts forth 2 other highly likely situations but then dismisses them for no good reason for the "cheating".

Re:Benchmarks

[Jugalator]

It shows that Microsoft is more concerned about getting a good score on the benchmark than they are about providing a good customer experience.

Could the same be said about the numerous bugs issued for Firefox about optimizing TraceMonkey's SunSpider performance?

Re:Benchmarks

[gl4ss]

read the article. their js performance is quite suspect if their results are "too good to be true" when the benchmark is unmodified and then too bad to be true when it's very slightly modified. some more 3rd party testing should be done.. and actually it would be pretty easy to do.

Re:Benchmarks

[aliquis]

They also have one more thing in common.

They're both utterly useless ;)

Nostalgic purpose / dreams / cool numbers, yeah, but what matter is today, and actual perceived performance.

Re:Benchmarks

[hey]

Shows a problem with benchmarks in general. Too easy to game.

Re:Benchmarks

[Rockoon]

Fear not, Slashdot, for I have read the fucking article!

The unnamed "Mozilla Engineer" didn't even bother adding his 'true' and "return' to non-sunspider code, but has the balls to declare that the only possible reason for the negative performance when added to the sunspider code is that Microsofts JavaScript engine is cheating.

Now, maybe Microsoft is cheating here and maybe they arent, but in this article the conclusion obviously preceded the result, that this unnamed mozilla hack didnt even do any of the extremely simple things that might nullify his hypothesis.

Re:Benchmarks

[aliquis]

Many customers make the benchmark result their experience of the soft- or hardware.

Re:Benchmarks

[Mongoose Disciple]

Nostalgic purpose / dreams / cool numbers, yeah, but what matter is today, and actual perceived performance.

Absolutely.

Except in the sense that you can get a lot of good press / free advertising by stomping a mudhole in the other guy's performance in a benchmark. There's a clear incentive to improve your actual performance, because when real people get ahold of your benchmarked piece of software/hardware/whatever they're going to notice that actual performance -- but there's also some incentive to improve you benchmark performance for cheap advertising. The former is more valuable than the latter, and I'm sure it gets more time/money/effort proportionately.

Re:Benchmarks

[99BottlesOfBeerInMyF]

Shows a problem with benchmarks in general. Too easy to game.

Benchmarks are great, for improving the performance of your code. Benchmarks are terrible, as soon as they start to get press and companies try to deceive users by gaming them. That's why it is important that we call out when they are caught so they get more bad press and maybe think twice about gaming the benchmark in the first place.

Re:Benchmarks

[Anonymous Coward]

They did, its called ChromeOS ;)

Re:Benchmarks

[Mongoose Disciple]

It shows that Microsoft is more concerned about getting a good score on the benchmark than they are about providing a good customer experience.

For that to be true, you'll need to demonstrate that they put more effort into scoring well on the benchmark than they did in improving performance in general. I don't think you can.

Improving performance in general is worth doing and I'm sure it's being done, but it's hard. Improving performance on a benchmark dramatically is often not that hard, and it's worth doing if it gets your product noticed.

I'm sure all browser makers are doing the exact same thing on both counts -- anonymous Mozilla guy is just bitter because he did a shittier job of the less-important task of benchmark performance.

Re:Benchmarks

[TheRaven64]

There is a difference between optimising for a benchmark and cheating at a benchmark. Optimising for a benchmark means looking at the patterns that are in a benchmark and ensuring that these generate good code. This is generally beneficial, because a well-constructed benchmark is representative of the kind of code that people will run, so optimising for the benchmark means that common cases in real code will be optimised too. I do this, and I assume that most other compiler writers do the same. Cheating at a benchmark means spotting code in a benchmark and returning a special case.

For example, if someone is running a recursive Fibonacci implementation as a benchmark, a valid optimisation would be noting that the function has no side effects and automatically memoising it. This would turn it into a linear time, rather than polynomial time, function, at the cost of increased memory usage. A cheating optimisation would be to recognise that it's the Fibonacci sequence benchmark and replaces it with one that's precalculated the return values. The cheat would be a lot faster, but it would be a special case for that specific benchmark and would have no impact on any other code - it's cheating because you're not really using the compiler at all, you're hand-cmpiling that specific case, which is an approach that doesn't scale.

The Mozilla engineer is claiming that this is an example of cheating because trivial changes to the code (adding an explicit return; at the end, and adding a line saying true;) both make the benchmark much slower. I'm inclined to agree. The true; line is a bit difficult - an optimiser should be stripping that out, but it's possible that it's generating an on-stack reference to the true singleton, which might mess up some data alignment. The explicit return is more obvious - that ought to be generating exactly the same AST as the version with an implicit return.

That said, fair benchmarks are incredibly hard to write for modern computers. I've got a couple of benchmarks that show my Smalltalk compiler is significantly faster than GCC-compiled C. If you look at the instruction streams generated by the two, this shouldn't be the case, but due to some interaction with the cache the more complex code runs faster than the simpler code. Modify either the Smalltalk or C versions very slightly and this advantage vanishes and the results return to something more plausible. There are lots of optimisations that you can do with JavaScript that have a massive performance impact, but need some quite complex heuristics to decide where to apply them. A fairly simple change to a program can quite easily make it fall through the optimiser's pattern matching engine and run in the slow path.

Re:Benchmarks

[Nutria]

Fear not, for I have RTFA and the original article that the digitizor article is based on.

Fortunately for the ethics of Mozilla, the named Mozilla engineer (Rob Sayre) never claimed that IE9 cheated. Instead, he diplomatically refers to it as a "oddity" and "fragile analysis" and filed a bug w/ MSFT.

http://blog.mozilla.com/rob-sayre/2010/09/09/js-benchmarks-closing-in/M
http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug-on-a-fragile-analysis/

So, blame Digitizor and ycombinator for putting words in Rob Sayre's mouth.

Re:Benchmarks

[hedwards]

Most likely they are cheating. The other possibilities are far less plausible. Even if you discount that possibility, either they're not competent at optimization or they're not competent at writing a robust engine.

In none of the cases is MS doing something legitimately. Optimizing to one test is invariably a bad idea, no matter how well designed, and quite honest at this point they should be able to code an engine that's a lot more resilient than that.

Re:Benchmarks

[Anonymous Coward]

Seriously, how have you not been modded up? I'm a bit better for having read this comment.

Re:Benchmarks

[hedwards]

No it couldn't. Firefox has for a long time lagged on pretty much all the tests, including that stupid ACID test. They lagged specifically because they were more focused on real improvements over faking it or optimizing for conditions that one is unlikely to encounter.

Or, it could be that they're just incredibly incompetent at cheating. I suppose that's possible. But given the degree to which the real speed has improved with the 4.0b7, I think we can largely rule out that level of incompetence.

Re:Benchmarks

[hedwards]

That's not a problem with benchmarks per se, that's a problem with the idiots that insist that benchmark performance is the same thing as good performance in general.

It really depends how the benchmark is set up, certain things are known to be costly in terms of IO, RAM and processing time. And a benchmark which measures things like that and gives some meaningful indication where the time is being spent is definitely valuable.

Re:Benchmarks

[Anonymous Coward]

And there is no actual evidence that they are actually cheating. The article writer puts forth 2 other highly likely situations but then dismisses them for no good reason for the "cheating".

Yeah, we just have to take everything that monopoly says as true. Like moving Office to open standards, preferring HTML5 for SL and so on...
Somehow I have always trusted more on companies which compete side by side for my attention and do not have OS-monopoly to protect.

Re:Benchmarks

[bluefoxlucid]

explain how this non-breaking modification suddenly means "slow." Is it following more (if(false-cond...)) etc and doing more processing than necessary just to find out there's nothing more to do? i.e. broken short circuiting?

Re:Benchmarks

[BZ]

1) If you actually read the article, you may have noticed that the engineer is named. It's
        right there there at the beginning of paragraph 2: "While Mozilla engineer Rob Sayre"
2) The "cheating" stuff is all from the Hacker News thread and the fucking articl. I
        suggest you further read item 1 under "Further Readings" on the fucking article, which
        is what Rob actually wrote. The link is: http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug-on-a-fragile-analysis/

Just to save you the trouble of reading it, if don't want to, it's pretty clear that IE9 is eliminating the heart of the math-cordic loop as dead code. It _is_ dead code, so the optimization is correct. What's weird is that very similar code (in fact, code that compiles to identical bytecode in some other JS engines) that's just as dead is not dead-code eliminated. This suggests that the dead-code-elimination algorithm is somewhat fragile. In particular, testing has yet to turn up a single other piece of dead code it eliminates other than this one function in Sunspider. So Rob filed a bug about this apparent fragility with Microsoft and blogged about it. The rest is all speculation by third parties.

Re:Benchmarks

[headLITE]

Sure, but the suspicious thing about this particular optimization is that adding a no-op statement that merely expresses something that is otherwise implicit (the return at the end of the function) disables it. This makes it look like they are optimizing for code that looks exactly like the source code of this function... which is not a very useful thing to do unless you want to cheat at a benchmark.

Re:Benchmarks

[Anonymous Coward]

Read that article again then because he said there were 3 possible explainations of what happened. One is cheating by deliberately making IE9 ace the test, the second is unintentional optimization by using Sunspider for testing the engine, the last is a broken engine on those simple instructions.

Also I checked the code, he added the "true" and "return".

As for why he didnt do anything to nullify his hypothesis - He works for Mozilla and isnt going to want to get MS off the hook when they can release these type of sensationalist reports showing IE9 to be shit (which it is and will be over time).

Re:Benchmarks

[hedwards]

There was a good reason for dismissing those possibilities. MS ought to be smart enough at this point to avoid those possibilities. If anything, he was overly generous in assuming that MS has the competence to know to use more than one benchmark when assessing speed and able to put together an engine which is resilient enough to deal with what should be a non-change.

Strictly speaking neither of those lines should even appear when run, that's supposed to be more or less stripped out before the engine starts doing any calculations. And in the case of the explicit return that shouldn't have any impact at all.

So, it's not without good reason that he's suggesting that it's cheating. MS at this point ought to be hiring people that are competent, at least if this isn't an effort to get themselves booted from even more people's computers.

Re:Benchmarks

[Anonymous Coward]

So being a slow ass browser is now evidence of a good customer experience. Gotcha.

Re:Benchmarks

[Jesus_666]

Actually, the three possibilities are discussed fairly neutrally.

Possibility one: Microsoft cheated. Presented as highly likely.
(I tend to agree that it's quite conceivable - other corporations have been caught doing similar things (like the NVIDIA/FutureMark debacle) and JavaScript execution speed is currently the most-hyped performance metric in the browser market.)

Possibility two: Microsoft have relied entirely on SunSpider when testing their JavaScript engine and over-optimized it to a point where it's now a SunSpider VM that happens to run JavaScript and doesn't work well with anything that isn't SunSpider. This is declared unlikely.
(Although I wouldn't put such a blunder past Microsoft, I do think that their tests extend beyond "how fast is SunSpider".)

Possibility three: The engine is legitimately ten times as fast as everyone else in this test but badly-written and so fragile that it experiences major slowdowns on code that meets currently-unknown criteria. Presented as unlikely.
(Note that in the Hacker News analysis the general consensus now seems to be that IE indeed does something with the code that it shouldn't; an earlier theory of broken dead code analysis couldn't stand up to the fact that any change that causes the bytecode to look differently, even if functionally equivalent, causes slowdowns).

Re:Benchmarks

[Omnifarious]

Oh, so this is acceptable behavior now because "everybody's doing it"? When did that start being a valid excuse?

Re:Benchmarks

[Omnifarious]

You richly deserve the +5 you got. :-)

Re:Benchmarks

[KingMotley]

Actually, there is other examples of dead-code being eliminated in that thread. One uses adding foo and setting it in and out of the loop, one of which does get eliminated.

Actually, from the look of what I am seeing, and I haven't done that much an analysis yet, it appears to me that the loop is getting eliminated as dead code, but only in certain cases. On the other hand, the two "simple" examples that shouldn't affect the performance may in many cases, and is not quite so "simple". First of all, no one SHOULD be putting 'true;' as a statement in javascript it just doesn't make sense to do so, so it's quite possible that statement would make the dead code analyzer fail to see the rest of the loop as being dead code.

Also functions that have a return statement may also be excluded from dead-code analysis, as then you have to analyze what is being returned and any local variables in the function are being returned.

A third possibility is that since sunspider actually uses code taken from real live libraries, that Microsoft did optimize this type of loop. That would include the real usage of the loop not only in sunspider but any other application that uses it. Sorry, but I don't know exactly where this piece of code came from, so I can't be more exact than that. If that is the case, that would make sense since Microsoft does optimize for the top x% of REAL websites, and it's possible sunspider duplicates part of that.

Re:Benchmarks

[KingMotley]

So you automatically trust the underdog more? Quite interesting. I suppose you naturally believe all the crackpots that try and sue famous people for child support because they had a kid and it's obviously theirs.

Re:Benchmarks

[Mongoose Disciple]

Oh, so this is acceptable behavior now because "everybody's doing it"? When did that start being a valid excuse?

It's a valid reason whenever we're being pragmatic or dwelling in the real world instead of happy magic unicorn fairy land.

If people optimize to benchmarks (and they do, they always have, for every benchmark ever), does a person grounded in reality

A) Yell "Cheatzors!!!!", or

B) Try to come up with a means to gauge performance that isn't so trivially gamed.

I mean, when the RIAA attempts to blatantly ignore reality to make an outdated business model that cannot continue to function work, we point out how stupid this is -- but if we're trying to get JavaScript benchmarks that cannot work for their theoretically intended purpose to work it somehow becomes a matter of morality and not reality?

Re:Benchmarks

[BZ]

Ah, yes. The Hacker News thread has the var foo example. It's just Mozilla's internal testing that didn't find other examples that are still DCEd.

So yeah, looks like just a very fragile dead code eliminator.

The cordic test didn't come from any web libraries, as far as I can tell.

Re:Benchmarks

[ArsenneLupin]

Just to save you the trouble of reading it, if don't want to, it's pretty clear that IE9 is eliminating the heart of the math-cordic loop as dead code. It _is_ dead code, so the optimization is correct.

So, it's really a problem with the benchmark then. What they should have done is to "use" the result of each iteration for something (maybe sum all results together?), return the sum, and display it somewhere (using document.write, for example). That way, a correct JS engine could not optimize it away.

Re:Benchmarks

[BZ]

Well, sure. If you read carefully, the vast majority of Sunspider is actually dead code; one of the many reasons it's a bad benchmark. More modern benchmarks are more careful to not do that....

Re:Benchmarks

[Mitsoid]

actually this makes an improvement, I think

MS optimized their JS code to *not* do pointless loops that have no use in the rest of the code... Great, that is an optimization.

I wouldn't say MS "Cheated" (as this is, as best i can tell from the high level summaries, a valid, beneficial, improvement to the JS engine) -- Instead, this should be a 'new' test for all broswers, and the old test should be re-created in a way to force engines to do it (making the values relevant to something)

Again, just my understanding from the article

Re:Benchmarks

[sartin]

This is the nature of benchmarks... whenever people start caring about them enough, software/hardware designers optimize for the benchmark.

It is possible to work really hard to improve the benchmarks, while still being ethical and working to help the customer. Way back when I worked at Hewlett-Packard (which I left in the early 90s), I was next to the PA-RISC compiler lab. One day, a sign went up announcing a celebration that they'd gotten the inner loop of a certain benchmark down to 18 instructions. They hadn't done it with hand optimization and rewriting the compiler to recognize the benchmark. They had done it by researching and implementing a series of general purpose optimizations and measuring how they impacted the various benchmarks and real code. As I recall, this particular benchmark was using some LINPACK (or maybe BLAS) code so the results were generally useful to real customers.

Re:Benchmarks

[shugah]

Do you mean to say that MS optimized their JS compiler to "not do pointless loops that have no use in the rest of the code" only when the "pointless loop" is contained in a function that matches a very specific code signature?

I fail to see how that would be of any use to anyone using their browser to execute real code. The likelihood of taking advantage of this particular optimization is effectively zero.

Re:Benchmarks

[bonch]

Hacker news has evidence that it is cheating, actually. One might initially assume that IE9 is doing dead code analysis, but the behavior doesn't occur when run on trivial variations of the SunSpider test (diffs provided in the link), which is at least suspicious.

Re:Benchmarks

[mounthood]

This suggests that the dead-code-elimination algorithm is somewhat fragile. In particular, testing has yet to turn up a single other piece of dead code it eliminates other than this one function in Sunspider. So Rob filed a bug about this apparent fragility with Microsoft and blogged about it. The rest is all speculation by third parties.

Isn't it speculation that this is an innocent "apparent fragility" in the code? This might be cheating. All we can do to assess the situation is look at the evidence of this incident, and consider the character of Microsoft and Mozilla. Honest people may differ in their assessment.

Re:Benchmarks

[cedrick12]

There are lies, damn lies, and benchmarks.

Re:Benchmarks

[clone53421]

Optimizing for a benchmark is cheating.

Re:Benchmarks

[BZ]

Well, the apparent fragility of the code is not speculation. Whatever is going on here is certainly very fragile.

Whether it's innocent or purposeful is certainly speculation. Rob isn't doing such speculation in public (though I bet he has a private opinion), nor am I willing to, precisely to uphold the character of Mozilla.

Re:Benchmarks

[Blakey Rat]

Am I having major deja-vu or was this discussed on Slashdot before?

I'm Googling it, but I'm not finding anything... but I swear this exact blog posting has been discussed before. And yes, I came to the same conclusion before:
1) IE removing dead-code = good (whether or not you call it "cheating".)
2) IE not removing dead-code for a near-identical function = bad
3) The most likely explanation is that the algorithm for removing dead-code was written based on the SunSpider tests
4) I don't see that as constituting "cheating", since... well, see #1

Re:Benchmarks

[blind monkey 3]

Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.

This sounds more like the 8th grader got a copy of the answers and memorised them. Unfortunately for the student, one of the questions got changed.

Re:Benchmarks

[roca]

Good comment, but data alignment has nothing to do with dead code elimination. The "true;" example is pretty much a dead giveaway that the optimization is strongly tuned to math-cordic and nothing much else.

Re:Benchmarks

[TheRaven64]

You're assuming that DCE is actually working properly. If it isn't, then true; will be compiled to a load of the true singleton (or a constant value if it's not implemented as an object). This would result in some register churn, which (especially on x86) would cause some spills to the stack.

If they're not properly aligning stuff on the stack, then spilling true; could mean that every other spill is not word aligned anymore, which could case some serious performance problems, especially if one of the values is now spanning two cache lines. You need to be impressively incompetent to do that, but I wouldn't put that level of incompetence past the IE team...

Re:Benchmarks

[WolfWithoutAClause]

They've more or less admitted it at: http://blogs.msdn.com/b/ie/archive/2010/11/17/html5-and-real-world-site-performance-seventh-ie9-platform-preview-available-for-developers.aspx

"The benchmark runs an expensive loop, and then does nothing with the results; the benchmark is written exactly in a way that triggers this general optimization [dead code elimination].

Of course, the benchmark could be rewritten to avoid triggering this optimization, which would bring our performance on this specific benchmark in line with other browsers.

The interest in this issue is a great example of why these microbenchmarks fail to represent the real world web. Webkit Sunspider uses an expensive JavaScript loop to approximate sine and cosine. Real world sites would actually use the much faster and CPU-optimized functions already available in JavaScript engines.

These optimizations are relatively new to the world of JavaScript runtimes even though there are many examples of dead code in real-world JavaScript on the Web. These optimizations often require significant flow analysis of the code, and in a real world site, spending too much time analyzing code can reduce the responsiveness of the page. The Chakra engine picks the right balance between code quality and analysis time, and only performs a small set of dead code optimizations. We continue to tune this for IE9, and bugs reported via Microsoft Connect are examples of where the optimization could do more. We continue to tune these and other optimizations for the final release."

In other words, they don't think dead code elimination is really worth it, so they only do a bit, and when they do it, it 'just happens' to optimise this benchmark. They've done no general optimisations.

That's corporate speak for: "Yup, we cheated! Doesn't everyone? The test was stupid anyway!"

Re:Benchmarks

[Your.Master]

Dead code elimination is only worth it sometimes, and you have to guess before trying whether it was worth it. That's not cheating. That's the only reasonable way to do dead code elimination, especially just-in-time.

That doesn't mean that they are correct that they chose the right balance. But you do have to strike a balance.

Re:Benchmarks

[WolfWithoutAClause]

Yeah, but the 'balance' they've chosen is to put the code segment of the freaking benchmark into their peephole optimizer, and then published it as if that's representative of their compiler.

It's basically corrupt and fraudulent.

Re:Benchmarks

[mounthood]

Well, the apparent fragility of the code is not speculation. Whatever is going on here is certainly very fragile.

Fragile in the sense that it doesn't cover all cases or even a reasonable majority, right? If that's what you mean by "fragile" then what would you call code to cheat on the test, which is targeted at just one or very few cases? Maybe you're just saying "fragile" to avoid any rude accusations, but that doesn't make it more accurate.

Whether it's innocent or purposeful is certainly speculation.

We do have some basis for that speculation: the character and history of the actors involved.

I wish...

[joe2tiger]

I could still use Internet Explorer 8. I have 9 right now, but certain websites that my job requires me to go to locks me out of some content. Autotask.net and some other sites that my school has access to is nonfunctional. I know most pages comply, but the few that don't - why are we forced to use Microsoft software?

Level playing field

[QuietLagoon]

Microsoft knows that they cannot win on a level playing field. Microsoft knows they are unable to compete without having their finger on the scale. So they cheat.

Re:Level playing field

[Anonymous Coward]

And /. article submitters know that /. readers never bother to read the fucking articles. So we get bullshit articles like this one and knee-jerk reactions like yours.

Re:Level playing field

[QuietLagoon]

A troll or the truth?

Do not attribute to malice ...

[Tar-Alcarin]

what can be attributed to stupidity.

1) Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2)Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.

I see no reason why explanation number one is more likely than explanation number two.

Re:Do not attribute to malice ...

[Spad]

Only because you could argue that Microsoft has a vested interest in doing #1 - I guess it depends on how malicious you think Microsoft is :)

Re:Do not attribute to malice ...

[Lunix Nutcase]

Because explanation number 2 doesn't feed into the hatred of Microsoft and create all sorts of controversy in order to drive more ad revenue?

Re:Do not attribute to malice ...

[The MAZZTer]

Let's check out some other benchmarks/parts of Sunspider IE9 does good on and tweak them similarly to see if the performance suddenly suffers.

Re:Do not attribute to malice ...

[MobyDisk]

Never attribute to malice that which is adequately explained by stupidity.

Re:Do not attribute to malice ...

[Tar-Alcarin]

Are you calling me stupid? Because I'm sure you're not implying that I misquoted that on purpose.

Re:Do not attribute to malice ...

[Inda]

Accuse someone of something when phishing for information. Watch the reactions, watch people back-peddling, listen for lies, listen for an overly reactive explaination, watch for the ultra-defensive, nose scratching, bullshitters, beads of sweat...

Does no on else use this trick in life? I doubt I've invented it; I'm sure it's taught somewhere and there's probably a fancy name for it.

Accuse Microsoft of cheating and see what information flows back.

Re:Do not attribute to malice ...

[LordKronos]

I see no reason why explanation number one is more likely than explanation number two.

I do. Given the nature of the changes that were used to uncover this, to me (as a programmer) it seems very unlikely that such over-optimization could happen in such a way that it would degrade so severely with those changes. Here is what was changed (look at the 2 diff files linked near the bottom of the article):

1) A "true;" statement was added into the code. It was not an assignment or a function call, or anything complex. Just a simple true statement. Depending on the level of optimization by the interpreter/compiler, this should turn into a noop at best, or simply loading a constant into a register or memory location (and then doing nothing more with it) at worst..
2) A "return" added at the end of a function, instead of using the implicit return. This also should have minimal impact. In the best case, the code generated should be identical to the return code already generated by the implicit return. In the worst case, the compiler might be sloppy and generate 2 return instructions (one for the explicit return, and another unreachable instruction for the implicit return).

So from my experience, it seems EXTREMELY unlikely that this could have happened by accident just through optimization. I'm not going to go so far as to say it CAN'T happen, because I know from experience that things can get really complex and things not so obvious can happen (like cache issues as mentioned in the article). However, in all likelihood, I suspect this is going to be just like when video card drivers detect the filename of the executable and optimize specifically for that (I think it was quake3.exe that we saw that happen with before).

Re:Do not attribute to malice ...

[sbates]

The second example--a general optimization of the engine that over-optimizes a corner case--is very difficult to pull off in these engines. When you consider that the over-optimization only occurs when the source is aligned a certain way things become very suspicious, because the engine isn't running against the source but an abstract representation of the code.

In today's engines there are also routines to strip out unnecessary and unreachable code, which is relevant because the code snippets added by the tester were extraneous and (slashdotted, so going from memory) unreachable. This means the bytecode would either have been generated and stripped or not generated at all.

Then consider that these engines are not actually running the bytecode but machine-optimizing it. So now you have a case where:

1. Extraneous and unreachable code is added
2. Extraneous and unreachable code is removed by the compiler to bytecode
3. The bytecode is further optimized to machine code
4. The code is executed

Step 4 is where the optimization is lost. This is why it's extremely unlikely that someone checked in code to attempt to optimize the engine which resulted in an over-optimization of the corner case.

Re:Do not attribute to malice ...

[Tar-Alcarin]

Troll? Seriously?
Come on! It was a direct reference to the nature of the quote being discussed.
I would have thought the ";)" was implied.

Re:Do not attribute to malice ...

[jason.sweet]

In my opinion, a useful benchmark reflects standard usage patterns. Therefore, optimizing for the benchmark can only benefit the end user. If shuffling the "return" and "True" is just as valid an option, perhaps the benchmark should include both approaches.

Maybe I'm a bit naive, but when I change my code, I expect the results to change as well.

Re:Do not attribute to malice ...

[joeyblades]

The keyword in number two is "unintentionally". Happy accidents do happen, but they rarely go unrecognized and once recognized they should be reconciled. If recognized but not reconciled then you can't say it was unintentional and therefore I have to agree that number two seems unlikely.

Re:Do not attribute to malice ...

[hedwards]

It doesn't work against the people that you most want to use it against. The people that are really tricky will have a pretty good idea that you're full of it and make you give details.

Re:Do not attribute to malice ...

[hedwards]

Plus, you can break most people with an extreme unflinching, unblinking stare. Did that to the branch manager of the company I used to work for and he just about shit his pants he was so scared.

Re:Do not attribute to malice ...

[Anonymous Coward]

there's probably a fancy name for it.

Infotrolling?

Re:Do not attribute to malice ...

[geoskd]

At the risk of getting an off-topic; I don't understand that moderation either, unless it was meant in the same spirit as your original post...

-=Geoskd

Re:Do not attribute to malice ...

[epine]

One needs to take into consideration that significant performance suites are often included in a compiler's regression suite. It's only the cheating by the most extreme standard for a compile team to ensure the compiler puts its best face forward on major benchmarks. We all cheat in the front of the mirror, don't we?

This practice can result in strange fragility on benchmark code. So can more nefarious intentions.

Better benchmark suites are engineered not to have dead code blocks. It's just begging for trouble.

The oldest benchmark cheating I recall (dimly) was ATI somewhere around 1990 cheating on a video benchmark to draw the same icon at the same place a bazillion times by only drawing it once.

Before that there was also a Taiwanese company IIRC who slowed the PC clock down by 10% on one of their turbo XT systems for a trade show. People at the trade show ran their own benchmark programs, but the timing result was liposuctioned by the wonky system clock.

The great thing about that machine if purchased as tweaked is that your work day now has 25 terrestrial hours so your productivity will be better than ever.

On a side note, I used to have an OpenBSD firewall named malice. When my network had issues, it was usually correct not to attribute to malice what could be attributed to vi. Malice was a Pentium Pro 200 with 128 MB of EDO that lasted 13 years with a couple of fresh disk drives. Seriously, if it doesn't break, don't fix it. Eventually I decided that scp really ought to saturate a 100 mbit link, so I finally pitched it out. Tellingly, it wasn't my broadband provider which forced the issue. If I waited for my ISP to exceed the machine's SSL crypto speed, I could have kept it another decade I think.

Re:Do not attribute to malice ...

[am 2k]

I suspect this is going to be just like when video card drivers detect the filename of the executable and optimize specifically for that (I think it was quake3.exe that we saw that happen with before).

Actually, that still happens, and not just for Quake. It's even configurable in nVidia's control panel.

Re:Do not attribute to malice ...

[StuartHankins]

Or maybe because Microsoft has rarely been accused of over-optimizing anything. However they have been caught many times using code to degrade competitors' products scores or experience on the Windows and DOS platforms, so it's not a stretch to imagine them "cheating" in code to help their benchmark scores.

It all goes back to setting expectations through behavior. When you get a reputation, it's difficult to change people's perceptions. Sometimes it's a good reputation, in which that's great for you, but other times the "problem child" is sometimes blamed simply because they have set up a pattern of misbehavior.

Re:Do not attribute to malice ...

[pegr]

But yet sufficiently advanced stupidity is indistinguishable from malice...

Re:Do not attribute to malice ...

[Anonymous Coward]

Troll? Seriously?

Hey, never attribute to malice that which is adequately explained by stupidity. You were modded troll simply because the mod was too stupid to notice you were good enough to have a laugh at your own mistake (with a joke that was pretty funny, too).

Re:Do not attribute to malice ...

[Anonymous Coward]

I agree with Tar-Alcarin, Lets think about this MSFT browser is hardly used these days so they are trying to get the public to see that OH LOOK our browser is super fast now we have magically changed the code and its just as fast or faster than Chrome,Firefox.

Really?

MSFT is just bending the truth so they can seem like they can still code a decent browser that will compete with the current offerings. OPTION 1 is correctly stated.

Re:Do not attribute to malice ...

[Anonymous Coward]

Given the nature of the changes that were used to uncover this, to me (as a programmer) it

If you bother to read the actual report the guy who did it filed, instead of basing suppositions on a crap summary and slanted editorial piece, what he discovered is that IE is detecting those extra statements as junk code that doesn't do anything so it optimizes them out. As it should, because they are dead code, and should be optimized out.

What I'm unclear on is why the IE optimizer doesn't work better in other situations, and more importantly why the other two browsers did NOT detect the garbage code and ignore it.

Or to put it another way, this article would have been titled "IE 10x better than other browsers when optimizing out dead code" had it been a MS fanboy, rather than a rabid MS-hater.

personally, I'm going back to bed. wake me up when something is demonstrated in the real world.

Re:Do not attribute to malice ...

[bonch]

Here you go.

Re:Do not attribute to malice ...

[shugah]

Neither explanation looks very good on Microsoft.

In scenario 1 MS is devious and cynical. In scenario 2 MS is incompetent. Choose your poison.

Re:Do not attribute to malice ...

[Rufty]

I see no reason why explanation number one is more likely than explanation number two.

See point 1), word #1.

Re:Do not attribute to malice ...

[Anonymous Coward]

Does your experience include working on a compiler at all? This is pretty easy to see how this happens. You have an algorithm for performing dead code elimiation, a common optimization for most languages but apparently not very common in the JavaScript world yet. Let's say this algorithm is implemented by performing an analysis of the AST (even if they first transform to a DAG nothing really changes). When you go to implement this algorithm you do so in a conservative fashion - that is you opt-in the specific nodes of the AST you understand (assignment, loops, conditionals, etc...) and as you walk the AST you check that all of the nodes you understand have no external side effects. For nodes that you don't understand, or that you know have side effects, you give up and compile the code. Because none of your benchmarks with dead code include return statements or expression statements you never write the code to understand these nodes and the DCE optimization becomes disabled. Viola, you have the exact behavior seen here - no cheating necessary.

Re:Do not attribute to malice ...

[shutdown -p now]

The optimization in this case is dead code elimination, so it's not a question of what native opcodes those particular statements turn into. It's a question of whether the JIT makes that whole function no-op or not.

Now if you look at the function, all it has (apart from the loop) are assignments to and reads from local variables, and it does not return any value derived from those locals, so it is really a no-op. The question here is why the dead code removal optimizer trips up on such trivial stuff as "true;" and "return;".

Re:Do not attribute to malice ...

[clone53421]

Therefore, optimizing for the benchmark can only benefit the end user.

Just how beneficial is that, if it optimizes for (Step = 0; Step < 12; Step++) but not for (Step = 12; Step > 0; Step--)?

Re:Do not attribute to malice ...

[clone53421]

Viola, you have the exact behavior seen here - no cheating necessary.

When you make the for-loop count backward, it suddenly decides to execute it. Explain that. (http://news.ycombinator.com/item?id=1913315, “Edit (like...#14)”).

Microsoft is irrelevant

[Anonymous Coward]

Oh please. Don't overreact. Microsoft is quickly becoming irrelevant. Google and Facebook are the new borg that we should be worried about.

Re:Microsoft is irrelevant

[Rosco P. Coltrane]

AC is right on the money there. Open-source software has come such a long way that Microsoft products and business are entirely avoidable these days, and therefore are no longer a threat. Google is the true danger of the age because they're fast on the way to make off-line applications obsolete altogether and render the open-source vs. closed source debate moot, as we'll have to swallow their online applications shenanigans without being able to do a thing about it.

Re:Microsoft is irrelevant

[Rik Sweeney]

Google and Facebook are the new borg that we should be worried about.

Considering that these days they spend most their time fighting instead of assimilating, I'd be tempted to say that one of them is the Borg and the other is Species 8472

Re:Microsoft is irrelevant

[Schadrach]

Umm, even in the case that offline applications become obsolete altogether, why would we have to swallow their shenanigans without being able to do anything about it?

It's not like they can lock down the internet or anything, you can always host your own.

Really? You're Going with That?

[eldavojohn]

Next we're going to be shocked that 8th grade history students try to memorize the material they think will be on their test rather than seeking a deep and insightful mastery of the subject and its modern societal implications.

Some things to consider: 1) I'm not doing business with the 8th grader. Nor am I relying on his understanding and memorization of history to run Javascript that I write for clients. 2) You are giving Microsoft a pass by building an analogy between their javascript engine and an 8th grade history student.

Just something to consider when you say we shouldn't be shocked by this.

Real-world usability

[Rosco P. Coltrane]

Benchmarks are very nice and all, but in the end, users using different browsers for real should decide which *feels* faster or better (which isn't the same as being faster or better). If real-world users can't feel the difference, then benchmarks are just there for masturbation value, and quite frankly, on reasonably modern hardware, I've never felt any true difference in rendering speed between the various "big" browsers out there.

I reckon the only thing that truly matters is the speed at which a browser starts up without prefetching (another semi-dishonest technique used by Microsoft for years to make users believe their products start so much faster than the competition incidentally).

Re:Real-world usability

[Anonymous Coward]

I reckon the only thing that truly matters is the speed at which a browser starts up without prefetching (another semi-dishonest technique used by Microsoft for years to make users believe their products start so much faster than the competition incidentally).

Firefox starts up instantly on my Windows 7 PC. That is, of course, because Microsoft has prefetched it into memory. It knows to do this because the SuperFetch service has determined that I use Firefox a lot. IE was prefetched because most of its code was shared with the OS. Not sure if I would call that semi-dishonest or leveraging good design and implementation. Office does some prefetching, but it does it in the same way that Adobe Reader and any other product can do if it wanted to. That can be disabled from start up using the msconfig.exe tool.

Point is, Windows 7 (and Vista) prefetches anything that will fit in memory and is used a lot by the logged on user. Ironically, this is one of the features that most computer-people hate.

Three explanations FTFA

[davev2.0]

There are three possible explanation for this weird result from Internet Explorer:

Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.

Everything in italics is unsupported opinion by the author, yet is treated as fact in the summary and title by CmdrTaco and Slashdot. Perhaps if Slashdot would stick to actual news sites (you know NEWS for nerds and all that), this would be a balanced report with a good amount of information. Instead, it is just another Slashdot supported hit piece against MicroSoft.

Re:Three explanations FTFA

[gl4ss]

it's just speculation on the possible reasons why it happened. should he have waited for the first replies to his post to post the replies to those obvious replies? of course not.

now if you want, you could run the benches yourself - and this is what the blogger wants you to do.

Re:Three explanations FTFA

[Anonymous Coward]

You must be new here.

Re:Three explanations FTFA

[davev2.0]

So, instead the blogger should declare that MS cheated at the benchmarks with nothing more than his results for which he admits that there are at least three plausible explanations?

And, then Taco should treat the author's biased opinion as fact? Remember, the title of this post is "Internet Explorer 9 Caught Cheating in SunSpider."

I don't think so.

And, where is the response from MS? Did anyone ask MS, or did someone find this and go "MS is CHEATING!!11!!one!" without actually investigating or even asking MS? Because, it really looks like the latter, which would make this just more MS bashing blogspam.

Re:Three explanations FTFA

[Omnifarious]

Unfortunately the statements added are basically 'noop' statements that should have no effect whatsoever on anything. It's almost like the original Mozilla engineer (who is not the author of the article BTW) added a comment and had the benchmark result drastically change.

So the author's opinion is fairly well supported by the evidence. In order for me to consider that opinion to be hard fact I would have to see very similar small changes affecting other SunSpider benchmarks in similarly drastic ways.

I'm really quite surprised at the number of people who rush to defend Microsoft on this one. In my opinion the evidence here is, while not damning, certainly very suggestive of benchmark gaming.

Re:Three explanations FTFA

[Arancaytar]

just another Slashdot supported hit piece against MicroSoft.

Isn't that what Slashdot is for?

(Also, you misspelled Micro$oft. You must be new here. :P )

Re:Three explanations FTFA

[Anonymous Coward]

Everything in italics is unsupported opinion by the author, yet is treated as fact in the summary and title by CmdrTaco and Slashdot. Perhaps if Slashdot would stick to actual news sites (you know NEWS for nerds and all that), this would be a balanced report with a good amount of information. Instead, it is just another Slashdot supported hit piece against MicroSoft.

Sorry but "fair" and "balanced" are trademarked by Fox when applied to news. Besides, Slashdot in general has always painted stuff with an anti-MS bias. Slashdot reporting on Microsoft is similar to Fox News reporting on Obama.

No proof?

[1000101]

FTFA:

There are three possible explanation for this weird result from Internet Explorer:

1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.


I'm not saying if what they have done is right or wrong, but this is a sensationalist headline that offers two other "less evil" alternatives to the outcome.

Re:No proof?

[99BottlesOfBeerInMyF]

Headlines are supposed to be succinct summaries and that is enforced by the character limit here. Maybe a better headline would be "Internet Explorer 9 Probably Cheating On Sunspider, But Maybe Just Horribly Written In Ways That Make SunSpider Apply Poorly". Of course that is too long for the title.

The important take away is that a particular SunSpider test is not a valid test for IE 9's performance in that category and that IE 9 will do much, much worse in many real world scenarios. The likelihood is that this is because Microsoft cares more about SunSpider than actual performance (marketing versus functionality) but it is vaguely possible it is the result of really crappy coding that just happens to have the same result.

Re:No proof?

[Anonymous Coward]

There are three possible explanation for this weird result from Internet Explorer:

1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.

I'm not saying if what they have done is right or wrong, but this is a sensationalist headline that offers two other "less evil" alternatives to the outcome.

Frankly, the original article isn't much better. 2 and 3 sit there as weasel words to cover the author's back while he insists that 1 is "the best explanation".

Re:No proof?

[BLKMGK]

Suspect Benchmark Results by IE9 Being Investigated.

There, how hard was that?

Re:No proof?

[PNutts]

Headlines are supposed to be succinct summaries and that is enforced by the character limit here. Maybe a better headline would be "Internet Explorer 9 Probably Cheating On Sunspider, But Maybe Just Horribly Written In Ways That Make SunSpider Apply Poorly". Of course that is too long for the title.

True, but the title of TFA is "Internet Explorer 9 Caught Cheating In SunSpider Benchmark?". That question mark at the end changes the meaning quite a bit. The /. title can fit one more character to accurately reflect the tone of the article.

Re:No proof?

[Anonymous Coward]

Frankly, the original article isn't much better. 2 and 3 sit there as weasel words to cover the author's back while he insists that 1 is "the best explanation".

Also, the third option doesn't exactly rise a serious question about the robustness. I mean, fallback isn't exactly a problem for robustness. Firefox also has some optimizations that have fallbacks, if I remember correctly. (Modded a bit too many posts here, thus anonymous)

Re:No proof?

[99BottlesOfBeerInMyF]

Suspect Benchmark Results by IE9 Being Investigated. There, how hard was that?

Harder than you think since that is misleading. They aren't being investigated as far as I know or the article presents. They were investigated (briefly) and the results are in the blog post.

Cheating allegation too strong

[js3]

Meh I think claiming they are cheating with no evidence seems a little too out there. I've never seen MS brag about how fast their browser is on this particular benchmark, and frankly seems more like a bug than a cheat.

Re:Cheating allegation too strong

[king neckbeard]

They have shown their Sunspider results quite a few times on http://blogs.msdn.com/b/ie/

Re:Cheating allegation too strong

[coofercat]

I don't disagree with you, but I'd love your "bug" idea to become mainstream...

"Critical bug in Internet Explorer boosts performance by 10%"

"Microsoft keen to develop more bugs in the hope they boost performance in other products"

"Mozilla unavailable for comment on how come they persist in reducing bugs when everyone else seems to want more of them"

Re:Cheating allegation too strong

[Anonymous Coward]

If bugs produce this kind of benchmark winning performance, then apparently what Microsoft needs is more bugs. I'm shutting off Windows Update.

Re:Cheating allegation too strong

[BLKMGK]

But did modifying this one test to near impossible speed make that much of a difference? It was obviously anomalous right? What about the other test results? If tweaked do things get screwy and if so what about the other browsers? So far I'm not convinced although certainly it's posisble. Frankly if who they are trying to woo to their browser is Joe Average user then this benchmark, commented on in a blog no Joe Average likely reads, seems silly IMO.

Re:Cheating allegation too strong

[Omnifarious]

And who do you think Joe Average relies on to tell h(im/er) what's good? People like us who read stuff like that, that's who. For years now IE marketshare has been being depressed because people like us keep telling Joe Average to stop using it, even though it's what is pre-installed on Joe Average's computer. So changing our opinions is exactly what Microsoft is after.

Re:Cheating allegation too strong

[Anonymous Coward]

Mostly they seem to complain on that blog about how benchmarks are not representative of real world performance and should not be the focus of a browser design, rather than 'brag' about the results.
E.g.: http://blogs.msdn.com/b/ie/archive/2010/09/14/performance-what-common-benchmarks-measure.aspx

You can argue this has to do with their results being under Opera's, but bragging about Sunspider speed hardly seems the cornerstone of their IE marketing strategy.

Re:Cheating allegation too strong

[The MAZZTer]

They just posted this today.

Re:Cheating allegation too strong

[ihavnoid]

Okay, I think I understand what's going on (at least, what Microsoft is claiming).

1) The sunspider cordic benchmark is actually a huge dead code : runs something, throws the result away, and just measure how long it takes.
2) IE9's javascript engine has a dead code elimination engine. However, since analyzing large amount of code for searching dead code is expensive, they have a parameter specifying how far they will search for dead code.
3) The engineer who was looking for the optimal parameter number simply looked for whatever benchmark he/she could find, and tuned the parameter which yields the highest score,
4) ...which happened to be precisely the size of the cordic benchmark.

The claims on the blog has a point : sunspider is a bad benchmark. Not just because it is stupid to write some code that computes sin() in javascript when you can get super-fast native implementations, but because THE WHOLE BENCHMARK IS A FRIGGING PIECE OF DEAD CODE.

Re:Cheating allegation too strong

[DragonWriter]

Meh I think claiming they are cheating with no evidence seems a little too out there.

There is evidence. Its actually fairly compelling evidence. Its not proof in the sense that there is no other possible explanation, but "unquestionable proof" and "evidence" are two very different things.

Its hard to imagine any plausible non-cheating code that could possibly do what the IE9 code does with the exact SunSpider code but which suddenly performs so much worse with even the slightest variant of the SunSpider code at issue. So the behavior -- not just on SunSpider itself, but the close variants tested -- is evidence for the charge of cheating.

Of course, if IE was open source, we wouldn't have to guess, we'd just look at the source code and see if it was recognizing the benchmark and then special-casing it.

Re:Cheating allegation too strong

[king neckbeard]

If IE is actually detecting that it's dead code, then it's not really a problem. If the fact that it's dead code is more or less a coincidence and it doesn't do the same things with similar dead code, then it would be cheating.

Broken Link

[Anonymous Coward]

Here's the google cache:

http://webcache.googleusercontent.com/search?q=cache:http://digitizor.com/2010/11/17/internet-explorer-9-caught-cheating-in-sunspider-benchmark/

Im shocked!

[miffo.swe]

Microsoft, cheating? Umpossible!

They ALWAYS cheat because its not in their organisational capacity to deliver something good. They deliver mediocre and then warp the surrounding world into their own desired image.

Re:Im shocked!

[BLKMGK]

You sure you aren't talking about Apple with that warp comment? ;-)

Re:Im shocked!

[miffo.swe]

Apple atleast delivers something that works, Microsoft releases pure garbage.

Re:Im shocked!

[BLKMGK]

Yes, that explains their market position. Troll indeed...

3 possible explanations, so why accuse?

[digitaldc]

The article clearly states:
There are three possible explanation for this weird result from Internet Explorer:
1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation.
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.
So, what proof do we have that Microsoft actually cheated?

Re:3 possible explanations, so why accuse?

[jav1231]

They're Microsoft. :)

Re:3 possible explanations, so why accuse?

[YttriumOxide]

So, what proof do we have that Microsoft actually cheated?

As other posters have pointed out, it's either "cheating" or "utter incompetence as developers". Either case doesn't look good for them, but honestly were I in their shoes, I'd PREFER the slack of being called a cheater than the alternative...

Re:3 possible explanations, so why accuse?

[Gulthek]

We have some pretty conclusive test results.

http://news.ycombinator.com/item?id=1913368

Re:3 possible explanations, so why accuse?

[bonch]

So, what proof do we have that Microsoft actually cheated?

http://news.ycombinator.com/item?id=1913315

Re:3 possible explanations, so why accuse?

[BitZtream]

I agree, no proof, so the headline is bunk.

However, I'm inclined to believe MS cheated. History is known to repeat itself, the track record of MS in this case is going to make any intelligent person wonder about this particular incident.

I won't call them cheaters, but I will say that I suspect it. It could be a stupid bug, I totally understand HOW it could happen if I were writing the code. I tend to think of MS as having more resources than I and probably should have caught this sort of thing during testing, but larger things do slip through the cracks.

Re:3 possible explanations, so why accuse?

[deek]

Actually, there is a third option. The benchmark in question contains dead code; it does not return any values, and does not alter any global variables. The Microsoft vm may have just detected this, and skipped the dead code loop.

The real fix in this case is to alter the benchmark code, and make it do something more significant. In fact, if you read the article, that's how the engineer deduces that something is wrong; by adding a "return" to the function, and observing the change in benchmark times.

Of course, it could also be that Microsoft are cheating, or are incompetent, but there are other options than those two.

Re:3 possible explanations, so why accuse?

[YttriumOxide]

Actually, there is a third option.

Not that I can see...

The benchmark in question contains dead code; it does not return any values, and does not alter any global variables. The Microsoft vm may have just detected this, and skipped the dead code loop.

The real fix in this case is to alter the benchmark code, and make it do something more significant. In fact, if you read the article, that's how the engineer deduces that something is wrong; by adding a "return" to the function, and observing the change in benchmark times.

But that's exactly how I come to the conclusion of either "cheating " or "incompetence"... If their optimizer detects that it's dead code and thereby skips it, great. However it should also therefore detect it as dead code with the explicit return - the code is just as dead and should not be doing anything differently. Clearly with the difference in benchmark times, it's not being skipped in this case though. If that was deliberate on the part of the engineers at Microsoft ("When we see this EXACT code, we skip it"), that's cheating (or more precisely "gaming the test", but roughly the same thing). However if it was NOT deliberate on the part of those engineers, then they're completely and utterly incompetent at designing dead code detection.

Re:3 possible explanations, so why accuse?

[Anonymous Coward]

Or the test is on the edge of what the engine considers eligible to be worth checking over for dead code, and adding no-ops as has been done pushes it far enough that it decides that it isn't worth the effort. That could still be considered incompetency, I suppose.

I see everyone going on about how adding no-ops to the code causes it to fail to cheat/fail to be excised for deadness, but has anyone tried changing the signature in other ways? Removing/altering code inside the test instead of adding to it, and seeing whether that has any effect on the time taken?

Re:3 possible explanations, so why accuse?

[YttriumOxide]

Or the test is on the edge of what the engine considers eligible to be worth checking over for dead code, and adding no-ops as has been done pushes it far enough that it decides that it isn't worth the effort. That could still be considered incompetency, I suppose.

From my point of view, yes - I'd consider it incompetency.

Maybe I'm a little harsh in that regard, but as a developer myself that has written optimisation code for virtual platforms (i.e. Translating what looks like an interpreted language in to a compiled form with optimisations), I just can't accept that the engine would handle it that way. The first basic check is to literally strip everything "useless", THEN go ahead and figure out real optimisations. A statement such as "true;" or "return;" (with an implicit return immediately following it such as the end of the block) are both classic examples of things that would be stripped in the first round before any other checks are done.

I see everyone going on about how adding no-ops to the code causes it to fail to cheat/fail to be excised for deadness, but has anyone tried changing the signature in other ways? Removing/altering code inside the test instead of adding to it, and seeing whether that has any effect on the time taken?

That would definitely be an interesting exercise and sorry to say I haven't... currently at work (despite slacking off on Slashdot to write this reply), so I don't really have the time to do so. I might give it a go later though if no-one else replies to you before then to say they have done so.

So I click on the link...

[Kc_spot]

And it leads me to a 500 internal server. Dohohoho, Microsoft.

Re:So I click on the link...

[Anonymous Coward]

They've over-optimized their server to survive a Slashdotting by serving up brief error pages instead of full content. Ingenious!

Internet Explorer?

[TheBilgeRat]

Since when has IE ever been faster? And why bother cheating when you have 51% of the market share?

Browser share from Dec 2009 to Oct 2010

Re:Internet Explorer?

[Anonymous Coward]

> And why bother cheating when you have 51% of the market share?

Because that 51% used to be 91%?

Re:Internet Explorer?

[black6host]

The data you reference shows 59%. However, 51% or 59% it doesn't really matter. What is important is the trend shown. IE is consistently losing market share over time. Were they climbing up the ladder that would be one thing. They're not. This doesn't say anything about whether or not they "cheated", only that your claim that their market share is so high why would they bother to cheat doesn't make much sense, to me.

Re:Internet Explorer?

[commodore64_love]

Mozilla (netscape) used to have 91% share too but I don't see them cheating.

I choose browsers based on features not speed. I like Firefox's addons to enable me to download youtube vids, SeaMonkey's builtin newsgroups/chat/email features, and Opera's Turbo for slow connections (dialup, cellular). As for speed they all seem about the same although FF3.6 does have a memory leak that can be annoying.

Re:Internet Explorer?

[shutdown -p now]

The irony is that IE9 beta is still faster than Firefox 4 in that test even in modified version (which kills IE's 10x lead).

Re:Embarrassing?

[SimonTheSoundMan]

Embarrassing the article got slashdotted. Try the web site on port 8090.

http://digitizor.com.nyud.net:8090/2010/11/17/internet-explorer-9-caught-cheating-in-sunspider-benchmark/

Also embarrassing that you spelt "embarrassing" incorrectly. ;)

Re:Embarrassing?

[SimonTheSoundMan]

Embarrassing, it's CCed at nyud.net.

People miss the point

[Anonymous Coward]

Perhaps Microsoft cheated intentionally. Perhaps it's accidental they over-optimised for the benchmark. Either way, their browser is again proven to be inferior than advertised, to the great disappointment and irritation of their developers and users alike. Seriously, I keep giving them the benefit of the doubt and I'm let down every time...

TDD

[Anonymous Coward]

Why would it seem weird to develop TDD where SunSpider is the test. If that is the current best benchmark test, then normal browsing should be simple.

Re:Really? You're Going with That?

[Sockatume]

2) You are giving Microsoft a pass by building an analogy between their javascript engine and an 8th grade history student.

Indeed. The student would make a better Javascript engine.

MS decision makers ARE 8th grade

[Anonymous Coward]

MS decision makers ARE 8th grade-like.

Win at any cost, even if you have to give all your software away for people to choose it!

free mp3 license

[Anonymous Coward]

http://web.archive.org/web/19981202072738/http://www.mpeg.org/MPEG/mp3-licensing-faq.html

The following letter from Mr. Henri Linde (Thomson) explains the patent and licensing situation for MP3 (aka Layer-3) as of April 1998.

Business Answers MPEG Layer-3 - Version 04.98-3 ...thank you for your interest in our MPEG Layer-3 audio compression technology.

In this document you will find answers to the most frequently asked questions about MPEG Layer-3, the Fraunhofer Gesellschaft, THOMSON multimedia and the licensing of MPEG Layer-3 technology.

The Fraunhofer Gesellschaft.

The Fraunhofer Gesellschaft is the leading organization of applied research in Germany. It operates 47 research centers in Germany with about 9,000 employees, about half of them scientists and engineers. The Fraunhofer Gesellschaft expands to a worldwide Organization, especially in USA and Asia. Home of the Fraunhofer Gesellschaft is Munich.

One of the goals of the Fraunhofer policy is a rapid transfer of innovations into products. The total research expenditure is about US $ 700 million.

The Fraunhofer Institut Integrierte Schaltungen (Fraunhofer IIS), based in Erlangen, is one of the 47 research centers. In the field of high quality low bitrate audio coding, Fraunhofer IIS is the leading international research lab. Fraunhofer IIS has been the main developer of the most advanced audio coding schemes, like MPEG Layer-3 and MPEG-2 AAC (Advanced Audio Coding). Fraunhofer IIS plays a major role in the ongoing work for the MPEG-4 Audio standardization process and contributes to many other standards bodies as well, like ITU-R TG10/4, ITU-R WP10C, AES, MPEG-IPR, and others.

WWW: http://www.iis.fhg.de/audio/

THOMSON multimedia.

The world's fourth-largest consumer electronics group, THOMSON multimedia develops, manufactures and markets: televisions, VCRs, camcorders, audio and communication appliances, satellite decoders, DVD players, as well as color TV tubes and professional television equipment (studio equipment, TV cameras, mixers, etc.).

With its pan-European brands - THOMSON, TELEFUNKEN and SABA - and its American brands - RCA, GE and ProScan - today the group is the leader in the United States, No. 2 in Europe and is strongly increasing its market presence in Asia, Latin America and Eastern Europe.

WWW: http://www.thomson-multimedia.com/ http://www.rca-electronics.com/ or
http://www.nipper.com/

OPTICOM

OPTICOM is a "spin-off" firm from Fraunhofer IIS. OPTICOM was founded in 1995 with the intention to assist you in finding the technical solution best suited for you through consulting and specialized tools.

WWW: http://www.opticom.de/

I) GENERAL

Q. I am interested in source coding technology. What can you do for me?

A. Licensing source code, developing real-time DSP code as well as DSP-based hardware, and developing codec libraries for desktop computers is part of the Fraunhofer IIS activities. The focus is on (mobile) high-quality low bitrate audiovisual communications (e.g. digital broadcasting systems or internet services).

Q. Regarding MPEG Layer-3, do any patent rights exist?

A. As for practically any important technology (and particularly for publicly established standards), you should know that patent rights for MPEG Layer-3 exist. Although others may also hold patents related to the MPEG Layer-3 technology, THOMSON multimedia and Fraunhofer IIS have pooled their interests and grant licenses under their combined patent portfolio.

Q. Who grants licenses under this combined portfolio to use MPEG Layer-3?

A. We have split the licensing activities in three parts:

i) THOMSON multimedia ha

mod parent up

[markdowling]

While MS-IE have disclosed a lot of information lately on their blogs, if they're going to discuss Sunspider results (as they did on 28 October with the IE9PP6 tests) then use of sleight of hand to sex them up is fair game for criticism.

Re:Old news...

[Gadget_Guy]

It is actually a couple of months old. The thing that makes me doubt the claims of cheating is that nobody has been able to find other examples of performance variations in this benchmark in all the time since this came to light. If they were going to cheat, why limit it to the cordic test? Nobody would base their browser choice on this obscure test.

I don't have the beta installed yet, but what I would like to see is the actual calculation changed and then run the tests again. Don't just put in weird code like "true;" but make the javascript plausible. It could be that the addition of these unusual statements are enough to confuse the optimiser so that it resorts back to a completely unoptimised version.

Output?

[ebcdic]

Does this part of the benchmark produce a result or output, and if so is it correct?

And if it doesn't produce any output or a result that's checked, there is plenty of scope for innocent explanations. It could be a bug that doesn't arise when the extra statements are added. Or it could be that part of the code is being optimised away (because the result isn't used) and the analysis isn't clever enough to handle it when the extra statements are present.

Re:Output?

[clone53421]

No. It’s a completely useless busy loop. Optimizing it away completely would be admirable, except that if it is optimizing the code, it strangely fails to recognize that it’s still a useless busy loop if you add return; at the end or true; in the middle.

If Microsoft is cheating...

[VGPowerlord]

If Microsoft is cheating, why wouldn't they cheat a bit better? Of the five browsers, including betas, IE is second from last. Last place is, of course, Firefox, even with the new JS engine. Oh, and that stats image? Taken from the same blog post that originally discovered the Sunspider IE9 issue over a month ago.

Rob Sayre, the Mozilla Engineer who discovered this, filed a bug with Microsoft to get them to look at this issue. However, he didn't file said bug until today, which is likely why this is in the news now rather than a month ago.

Re:If Microsoft is cheating...

[Anonymous Coward]

Oh, and that stats image? Very old. Notice that the Firefox version tested is from September? Huge improvements since then...

http://arewefastyet.com/

Firefox is the purple line. Sunspider test (this one) is on the left.

Re:If Microsoft is cheating...

[Derek Pomery]

That's an oooold image.
Try posting a more up to date one. Firefox is now the fastest at Sunspider on x86 and faster than Chrome on x64.

http://arewefastyet.com/

Then of course there is Kraken...
http://m8y.org/tmp/kraken.xhtml

The problem with Sunspider, and this isn't unique to Firefox, is the shortness of the benchmarks. This does penalise Firefox at bit more which pays up front in tracing a bit for gains on more complex code later. It also encourages optimising for the harness and not the test.

Kraken and v8bench are still good tests. Oh, and just to be clear, IE9pp6 does a darn good job.

Check out its performance on this checkers AI.

http://www.bailopan.net/blog/?p=822

Re:If Microsoft is cheating...

[Rockoon]

x64 Firefox 4.0b8pre
Sunspider = 285.1ms +/- 5.6%

x86 Firefox 4.0b8pre
Sunspider = 249.3ms +/- 4.4%

x86 Opera 11 Alpha
Sunspider = 233.4ms +/- 0.8%

You were saying that Firefox was the fastest at Sunspider. No. Opera is faster (not sure of its fastest.. not installing any IE pre-release)

Re:If Microsoft is cheating...

[Derek Pomery]

Opera 11 alpha is also a slower than Opera 10.63 at other things, such as my Kraken test.

I'm sure it'll be awesome someday, but isn't even close to there yet.

So, yes, maybe they have managed to eke out a bit more speed out of Sunspider, but Sunspider is increasingly meaningless and most gains will be small.

Asa also ran Opera 11 FWIW and got slightly different figures.
http://weblogs.mozillazine.org/asa/archives/2010/10/some_sunspider_numbe.html

As Asa noted:
"Sunspider was designed before any of the browsers had these truly modern JS engines with just in time compilers and because of that and all the progress each browser vendor has made over the last several years Sunspider is no longer particularly useful as a JS benchmark. This is kind of obvious when you see that all of the top scores are pretty much tied. One one hundredth of a second (across 26 tests) separates the slow from the fast and that's just not particularly meaningful. "

Re:If Microsoft is cheating...

[Rockoon]

Have you noticed that Firefox 4 doesnt render anything during the sunspider test? Whats that about? Opera displays the name of the current test and then the time...

It seems to me that Opera is rendering because its supposed to... so why isnt Firefox?

Re:If Microsoft is cheating...

[Derek Pomery]

Rendering has no impact on scores, but...

https://bugzilla.mozilla.org/show_bug.cgi?id=601332#c13

Was basically intended to ensure prettier loading. But in rapid updates like this and their use of innerHTML (STUPID) it had this side effect.

No big deal, you can't say Opera is free of rendering issues, esp the alpha. :-p

Re:If Microsoft is cheating...

[Derek Pomery]

Oh, and BTW, the reason AreWeFastYet.com does not test Opera is for the same reason I'm less attracted to that browser.

They are not open source, and have no standalone javascript engine for automated testing. I've just found them irritating to interact with in general though, even if they are closed source, they could stand to have a much less opaque bug reporting process.

In terms of openness of its development in my experience Mozilla Firefox easily leads the pack, followed by Webkit, Google Chromium, Apple Safari, Opera then Microsoft Internet Explorer.

Mods, please RTFA!

[whoever57]

Fear not, Slashdot, for I have read the fucking article!

Really? Are you sure about that that?

The unnamed "Mozilla Engineer"

Second paragraph of the article:

While Mozilla engineer Rob Sayre was benchmarking Firefox 4 with different browsers, ...

The parent is not insightful, it is merely a troll.

Re:Really? You're Going with That?

[jeffasselin]

1) I'm not doing business with the 8th grader. Nor am I relying on his understanding and memorization of history to run Javascript that I write for clients.

No, but 5 years later you'll let him vote...

Order of Magnitude

[amentajo]

I don't know about those of you suggesting that this isn't cheating. I'm going to go ahead and agree with the author of the article on this one: it's most likely cheating, even following Hanlon's Razor.

To me, an order-of-magnitude difference in an interpreted language by adding non-functional statements is most likely due to using a compiled-in functional equivalent of that particular JavaScript function. The engine probably matches the parse tree to determine whether or not to run the (much faster) machine-code version, and the non-functional statements make the parse tree not match, thus reverting back to executing just-in-time.

Sure, it's possible that the reasoning behind this difference isn't to get ahead on this benchmark. But I'm going to use Occam's Razor to suggest that the order-of-magnitude difference is a result of using pre-compiled versions of specific JavaScript functions, rather than assuming that Microsoft engineers who can optimize that JavaScript function down to 1 ms (with, by the way, a 95% confidence interval of +/- 0.0%!!!!!!!) are incapable of optimizing a functionally equivalent but trivially different version of it below 19.5 ms (with a 95% confidence interval of +/- 1.9%)

Re:Order of Magnitude

[KarmaMB84]

Yeah, it did a lot of good cheating on one test. They're still far behind and nobody has found any other tests they've "cheated" on yet even though this was first posted long ago.

Re:Order of Magnitude

[shutdown -p now]

To me, an order-of-magnitude difference in an interpreted language

There's no such thing as an "interpreted language". A particular implementation of the language may be an interpreter. All modern browser JS implementations, including the one in IE9, are JIT-compiling to native code.

by adding non-functional statements

That sort of thing is actually precisely what a good optimizer should be able to catch.

compiled-in functional equivalent of that particular JavaScript function.

The equivalent of that particular JS function is "void foo() {}". It does a computation, but does not use the result of said computation in any way (doesn't return it as a value, and doesn't update any global state).

Now, why the optimizer is so fragile in IE9, is a good question. But the order-of-magnitude difference itself is not suspicious; rather the fact that making trivial changes to the source trips the optimizer is.

Re:Order of Magnitude

[amentajo]

To me, an order-of-magnitude difference in an interpreted language

There's no such thing as an "interpreted language". A particular implementation of the language may be an interpreter. All modern browser JS implementations, including the one in IE9, are JIT-compiling to native code.

When writing my post, I went by what Wikipedia said about that term, which conveniently accounts for JIT compilation in its description. Turns out there is such a thing!
I don't suggest that there are languages that must be interpreted, only languages that are interpreted (in this case, "interpreted" meaning "not directly executed", which includes JIT compilation).

by adding non-functional statements

That sort of thing is actually precisely what a good optimizer should be able to catch.

That's why I'm suspicious about this: dead code probably should not cause an order of magnitude increase in running time.

compiled-in functional equivalent of that particular JavaScript function.

The equivalent of that particular JS function is "void foo() {}". It does a computation, but does not use the result of said computation in any way (doesn't return it as a value, and doesn't update any global state).

Wow, you're absolutely correct. I just assumed that SunSpider actually checked to make sure that the engine came up with the correct values to catch this kind of thing. So it now seems more likely to me that it just says "if the code matches the cordic function, then sleep for 1 ms, and return." That actually makes more sense with the numbers we see.
So in this case, that code is probably just stubbed out. Why have a stub for a function used in a benchmark test if you're not treating the benchmark specially (which, I believe, is an example of cheating)?

Now, why the optimizer is so fragile in IE9, is a good question. But the order-of-magnitude difference itself is not suspicious; rather the fact that making trivial changes to the source trips the optimizer is.

That's what I was trying to say... it is more likely a symptom of cheating (at least some level of catering to the benchmark rather than to the JS code) than it is a symptom of a botched optimizer that can consistently (95% confidence interval: +/- 0.0%) optimize some code down to exactly 1.0 ms, but "somehow" manage to perform far, far worse than other existing browsers when changing the source in a trivial way, and "all of a sudden" become way more inconsistent (95% confidence interval: +/- 1.9%).

Re:Order of Magnitude

[shutdown -p now]

That's why I'm suspicious about this: dead code probably should not cause an order of magnitude increase in running time.

Actually, that's precisely what a good dead code elimination will cause. Consider this loop in C++:

int main() {
  for (int i = 0; i < 1000000000; ++i) {}
}

A dumb compiler will simply count upwards to 1 billion, which will take a few seconds even on a modern PC. A smart one will remove the loop entirely, since nothing useful is done, and the program will exit immediately as soon as it's run. The difference will be several orders of magnitude!

Haskell had this very problem when it first entered the Language Shootout - most tests were along the same lines, lengthy loops computing something just to throw it away. Turned out that GHC is particularly good at optimizing these sorts of things, and on a number of tests it ended up beating C by a significant margin due to that.

That's what I was trying to say... it is more likely a symptom of cheating (at least some level of catering to the benchmark rather than to the JS code) than it is a symptom of a botched optimizer that can consistently (95% confidence interval: +/- 0.0%) optimize some code down to exactly 1.0 ms, but "somehow" manage to perform far, far worse than other existing browsers when changing the source in a trivial way, and "all of a sudden" become way more inconsistent (95% confidence interval: +/- 1.9%).

Two things of note here.

First, this behavior only shows on one particular function in one specific test of the entire test suite. In fact this is precisely how it was found - the result is so ridiculous (IE 10x faster tan all other browsers!) on that particular test and not on any other - that it immediately stands out, and that immediately prompted an investigation. (If you RTFA, it's actually old news, it just took a while to gather all evidence and officially submit it to MS as a bug report.)

That's not how any sane person would cheat - you'd nib a few milliseconds down here and there, showing advantageous but realistic figures across all tests. Especially with closed source, that is nigh impossible to catch.

Second, it's not "far worse than other existing browsers" when changing the source. It actually still beats FF4 in this test even with the change! Chrome and Opera are faster still, but it's not like the difference there is 2x or even 1.5x.

Ultimately this needs more testing. Best of all would be to try to find some other pattern of dead code that is clearly unrelated to this test (so it couldn't be "wrongly detected" if this is a cheat), but which the optimizer handles in the same way. Finding a few such would definitely prove that this is just optimizer at work, and weird results are likely due to bugs in it (like incorrectly handling "return" as a side effect where it is not). But if no other patterns are found that exhibit this behavior, then this is strong evidence for hardcoding for the test.

Re:Order of Magnitude

[clone53421]

A smart one will remove the loop entirely, since nothing useful is done, and the program will exit immediately as soon as it's run.

And one that’s specifically designed to work well on that benchmark will still sit there churning when you present it with:

int main() {
    for (int i = 1000000000; i > 0; --i) {}
}

Sounds to me like their dead-code detection doesn’t work, so they hard-coded in a few routines that it can detect and nuke instead of actually detecting dead code in general.

Re:Order of Magnitude

[shutdown -p now]

And one that’s specifically designed to work well on that benchmark will still sit there churning when you present it with ...

True, but does it actually do that?

I went ahead to write a simple test, but the results are rather surprising. Here's what I used (not valid HTML, obviously, but IE can handle it):

<script>
function f() {
  var i, n = 1
  for (i = 0; i < 50000000; ++i) {
      n += i;
  }
  g = n; // comment/uncomment as needed
}
 
g = 0;
document.write("[");
start = new Date().getTime();
f();
end = new Date().getTime();
document.write("]");
document.write(g);
document.write("<br>");
document.write(end - start);
</script>

The second number printed out is elapsed time in milliseconds.

Now, when running this code as is, I get ~1600ms on my box. Note that the way the code is written, it cannot skip the computation, because its result is output (it could try to compute the loop in a smarter way, but that's another optimization).

Now try commenting the line where it says "g = n". Once done, the function becomes a no-op, since it no longer changes any visible state. And guess what? Doing that change cuts down execution time more than 3x, to ~500ms, on the same box.

I also wondered if it's the assignment itself that is somehow so expensive, so I stripped the loop out, and ended up with 0ms. So, no, it really does something to the loop.

Now why the results are surprising... if this is honest-to-God DCE, then it should show up as 0ms. That it still spends 500ms in the function is strange - there's no way it should be possible unless the loop still runs in some form. But if it does run, then why does it run 3x faster than the first version?

I've hypothesized that, perhaps, it eliminated the += inside the loop, but not the loop counter. But no - commenting out the += cuts down execution time even further, to ~75ms.

On the other hand, replacing += with any bitwise operator (e.g. ^= or |=) significantly improves the overall numbers, and also makes it so that there's no difference between two versions. With "n |= i", regardless of whether "g = n" is commented out or not, I get 120ms.

Something is clearly going on there in the optimizer, but it's clearly not straightforward. Any suggestions regarding further tests?

Re:Order of Magnitude

[amentajo]

That's why I'm suspicious about this: dead code probably should not cause an order of magnitude increase in running time.

Actually, that's precisely what a good dead code elimination will cause. Consider this loop in C++:

You're right. I should have narrowed the scope of my claim. My apologies.

That's what I was trying to say... it is more likely a symptom of cheating (at least some level of catering to the benchmark rather than to the JS code) than it is a symptom of a botched optimizer that can consistently (95% confidence interval: +/- 0.0%) optimize some code down to exactly 1.0 ms, but "somehow" manage to perform far, far worse than other existing browsers when changing the source in a trivial way, and "all of a sudden" become way more inconsistent (95% confidence interval: +/- 1.9%).

Two things of note here.

First, this behavior only shows on one particular function in one specific test of the entire test suite. In fact this is precisely how it was found - the result is so ridiculous (IE 10x faster tan all other browsers!) on that particular test and not on any other - that it immediately stands out, and that immediately prompted an investigation. (If you RTFA, it's actually old news, it just took a while to gather all evidence and officially submit it to MS as a bug report.)

That's not how any sane person would cheat - you'd nib a few milliseconds down here and there, showing advantageous but realistic figures across all tests. Especially with closed source, that is nigh impossible to catch.

That's how I would do it, too. Microsoft has a lot of smart people -- I'm sure many of them would have thought of that too.
But that didn't happen here. For some reason, it runs too quickly, quickly enough to warrant further scrutiny.

Second, it's not "far worse than other existing browsers" when changing the source. It actually still beats FF4 in this test even with the change! Chrome and Opera are faster still, but it's not like the difference there is 2x or even 1.5x.

Actually, when you ignore the anomalous 1.0 ms, it runs around 2x longer than Chrome and between 2x-3x longer than Opera. I won't install FF4 to try it out, so I'll have to give you the benefit of the doubt on that one.

Ultimately this needs more testing. Best of all would be to try to find some other pattern of dead code that is clearly unrelated to this test (so it couldn't be "wrongly detected" if this is a cheat), but which the optimizer handles in the same way. Finding a few such would definitely prove that this is just optimizer at work, and weird results are likely due to bugs in it (like incorrectly handling "return" as a side effect where it is not). But if no other patterns are found that exhibit this behavior, then this is strong evidence for hardcoding for the test.

Of course. This is circumstantial, so it can't, by itself, prove that they cheated. I also haven't installed the preview to run it myself, yet. But, assuming that the benchmark times are accurate, it'll be hard argue that a benign explanation is more likely than cheating. That's all I'm saying.

Particularly, the 95% confidence interval of +/- 0.0% strikes me as the most suspicious. It's way too convenient that the test ends up taking the same amount of time, to a certain resolution, at least 95% of the time it is run, whereas the alternate spellings of it are much less consistent. Also... the same 1.0ms on different hardware (Sayre's testing machine as well as the author's laptop), all while keeping a 95% confidence interval of +/- 0.0%? That's... dare I say it... inconceivable!

The "roundness" of 1.0ms is convenient, too, but I'm not going to count that as a strike against them. Even though this whole thing is already circumstantial... sometimes, numbers are round.

Re:Order of Magnitude

[shutdown -p now]

Actually, when you ignore the anomalous 1.0 ms, it runs around 2x longer than Chrome and between 2x-3x longer than Opera.

I didn't run the test myself to check it out - I simply looked at the numbers as published on the blog of the person who discovered this issue. Specifically, this graph. Now that I look at it again, I've realized that this is the total SunSpider score, rather than the one for this particular test. My mistake.

It's way too convenient that the test ends up taking the same amount of time, to a certain resolution ... The "roundness" of 1.0ms is convenient, too

I wonder if it's simply the resolution of the timer (possibly artificially rounded up)? So it takes less than 1ms in practice, probably fluctuating there just as much in relative measurements, but the fluctuations are below timer resolution.

By the way, have a look at my post in response to another reply in this thread. I've tried to get the same effect observed on a different (and much simplified) code, but results are rather weird - something is going on, but it's not consistent with full DCE.

Re:Order of Magnitude

[amentajo]

Interesting. After installing the platform preview and confirming the results of the article (1.0 ms, +/- 0%, exactly the same as Sayre and the author, even though the other times were different), I am also getting the same results when looking at your test.

Yeah, bit twiddling makes things faster, but I can think of a logical explanation for that one.
After you run "n += i" 92682 times, n can no longer fit in 32 bits, whereas all the bitwise operations fit comfortably within that range for all values of i, so you can avoid using the "much slower" bigints.

A more far-fetched but powerful explanation is that it could be optimizing certain kinds of iterated bitwise operations on vectors. Iteratively applying |=, ^=, and &= all can be done in O(1) time. Bitwise AND: all zeroes. Bitwise OR: ceil(log-base-2 of 50000000) == 26 binary ones at the end. Bitwise XOR: complicated, but doable. (i XOR n) is the current value. If i is even, then (i XOR n) XOR (i+1) == (n-1 if n is odd, or n+1 if n is even). (i XOR (n-1 if n is odd, or n+1 if n is even)) XOR (i+1) == (n). So, you can get any value of n for any odd value of i using modulo division, as it cycles between n and n +/- 1 on every odd value of i. Do another XOR if you need to figure it out for an even value of i.

I say this is "far-fetched", because it seems like this kind of optimization would rarely be worth the cost.

Next test (also helps to explain what changes when "g = n" is commented out):
"n += 1; n -= 1;" This adds 1 to and subtracts 1 from n each loop, which could hypothetically be optimized away. Observe what it does for commented and uncommented "g = n". I'd recommend increasing the max value of i by a factor of 100 to try and reduce the noise. It should take about a minute and a half or so for each on the machine that you got ~1600 ms for your code.

Hope this helps.

Re:Order of Magnitude

[shutdown -p now]

Yeah, bit twiddling makes things faster, but I can think of a logical explanation for that one.
After you run "n += i" 92682 times, n can no longer fit in 32 bits, whereas all the bitwise operations fit comfortably within that range for all values of i, so you can avoid using the "much slower" bigints.

What really bugs me is not the perf difference between plus and bitwise, but that the version with plus is the only one that exhibits different behavior depending on whether the function lets the result of computation escape or not. That is downright weird.

A more far-fetched but powerful explanation is that it could be optimizing certain kinds of iterated bitwise operations on vectors. Iteratively applying |=, ^=, and &= all can be done in O(1) time. Bitwise AND: all zeroes. Bitwise OR: ceil(log-base-2 of 50000000) == 26 binary ones at the end. Bitwise XOR: complicated, but doable. (i XOR n) is the current value. If i is even, then (i XOR n) XOR (i+1) == (n-1 if n is odd, or n+1 if n is even). (i XOR (n-1 if n is odd, or n+1 if n is even)) XOR (i+1) == (n). So, you can get any value of n for any odd value of i using modulo division, as it cycles between n and n +/- 1 on every odd value of i. Do another XOR if you need to figure it out for an even value of i.

Far-fetched but possible, I guess... but then the timing would be O(1); and it does grow if you increase the iteration coun in practice. And, of course, the same optimization could just as well be applied to all arithmetic operators (in fact, it would make more sense, as it's probably more likely to find addition in a loop than it is to find bitwise AND or XOR!).

Re:Order of Magnitude

[clone53421]

After you run "n += i" 92682 times, n can no longer fit in 32 bits, whereas all the bitwise operations fit comfortably within that range for all values of i

Personally I’d pin that on the fact that if you’re using bitwise operations it drops everything down to a 32-bit integer whereas all mathematical functions will be performed using Javascript’s implementation of binary floating-point.

However that still doesn’t explain all of the oddities about GP’s post.

sayrer's blog entry

[multipartmixed]

http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug-on-a-fragile-analysis/

Confusious say

[masmullin]

... If you no cheat, you a LOOOOOSSSAAAAHHHHH.

Ok, that's just what my old Asian classmate who plays a lot of counterstrike says, whatever, same meaning.

Hmm ...

[lennier1]

Is this like the old Mac OS days where Apple fine tuned the OS to score better at graphics benchmarks, even though the rest still couldn't compete?

Hacker News post with lots of benchmark details

[Gulthek]

An enterprising Hacker News user has come up with some interesting and pretty conclusive results.

http://news.ycombinator.com/item?id=1913368

Not Surprised

[DaMattster]

I am not really surprised but I refuse to single out Microsoft as the only company that behaves in this manner. I am willing to bet most corporations try and fudge tests and benchmarks here there to get ahead. It is done in the name of capitalism and marketing. Look at the methods by which vehicle fuel efficiency is tested. Fuel efficiency tests are suspect at best. I would leave Microsoft alone on this. In my opinion, all browsers suck equally. I have IE, Firefox, Safari and Chrome installed because there are features from all three that I like for different websites and browsing habits. If you could take the best features from all three, you would have the killer browser. In my sole opinion, Chrome sucks less than the others.

Re:Really? You're Going with That?

[Anonymous Coward]

Maybe mozilla should hire 8th graders then :)

Re:Old news...

[SmurfButcher Bob]

Putting a

return;

at the tail of a function is unusual? Are you high?

benchmarking shmencemarking

[Anonymous Coward]

Who cares about SunSpider or whatever, how about benchmarking end user expirience, I'm sure Firefox would end up last on those benchmarks (this post is written in FFX as I'm using it as my primary browser, and have been since the days before 256 name changes).

Re:Old news...

[Anonymous Coward]

Did you submit it to /. at the time, or do you prefer to sit back and let everyone else do the work while you call them out on something that really has no detriment on your quality of life? Even if you didn't want to put in an iota of effort to spread the news, you could have just skipped the story when you saw it here instead of coming to whine. You're like the lazy guard who, when the villagers start screaming that attackers are only a couple hundred metres away, yawns and says "boring, I saw them when they were kilometres away". Well done.

The real bug - closed source vs open source

[jimwelch]

So, what proof do we have that Microsoft actually cheated?

Since this is closed source, we can not verify what is going on, only MS can.

With FF, we can look at the code and pass judgment on the cheat vs bug question.

Original blog posts

[Trelane]

Too bad TFA didn't link to the original blog post (http://blog.mozilla.com/rob-sayre/2010/09/09/js-benchmarks-closing-in/) nor the update (http://blog.mozilla.com/rob-sayre/2010/11/16/reporting-a-bug-on-a-fragile-analysis/) (in which a bug is allegedly filed, though nobody else can apparently see it).

Not embarrassing at all!

[Kaz Kylheku]

I would not hesitate a second to write code to detect a benchmark and supply optimized behavior.

The people who believe in stupid benchmarks and use them to make important decisions (or, at the very least, asses of themselves in forums) should be embarrassed, not benchmark cheaters.

What is the definition of cheating anyway? If the implementation accepts the benchmarking program and delivers a performance, that is a valid result, no matter how it was achieved. If a benchmark calculates the millionth digit of PI, it is a perfectly valid optimization to recognize that code and turn it into a literal constant which spews out the millionth digit of pi.

Yes, that optimization serves the implementors, not the user. So what? Where is the law that says the implementors of a software should not include code which serves them rather than the user? That is the privilege of the developer.

cheap, regular AV cables

[TrogL]

The dollar store (it's not the chain) next to me sells a full line of typical cables (RCA, HDMI, fibre, ethernet) for good prices and I've had no problems. No recognizable brand name - just "made in China".

Re:Old news...

[Gadget_Guy]

Do you seriously see lots of superfluous, empty return statements at the end of functions in javascript? As an example, I just had a quick look at the source to this /. page and didn't find a single example of this. The only time you will ever see a return statement is if there is an early return within a function or a function returns a value. While it is valid to have an empty return statement at the end of a function, nobody ever does it.

In terms of optimising the javascript, the code in question contains a small loop with some straightforward calculations. There were no object handling, memory allocation, string manipulation, or DOM access. There is some array access, but obviously it is able to cope with that. It is a prime candidate for aggressive optimisation, which I suspect is why it performs so well.

Perhaps the mere existence of a return statement in a function is enough of a trigger for the optimiser to go into a conservative mode to ensure that it doesn't inadvertently introduce errors. Remember, the optimisation runs at runtime, and so has to be faster than one from an compiler and therefore must be fairly simple. If the optimisation process is too complicated, then it will end up being slower than just interpreting as you go.

Re:Old news...

[Rockoon]

Not unusual, but it makes the function 1 statement longer..

If you knew anything about JIT compilers, you would know that they have simple heuristics on purpose (compile speed is a strict constraint.) Making something 1 statement longer could remove it as a candidate for quite a few optimizations (inlining, static loop evaluation, loop unrolling, dead code elimination, etc..)

These simple heuristics use quickly evaluated metrics once the source is translated into an abstract syntax tree. The number of nodes in the tree.. the depth of the tree.. the number of conditional nodes..

JIT's are not simply compilers that try to produce the best code possible. JIT's make tradeoff decisions between compile time and the resulting code quality.

precompiled

[atisss]

Actually MS hosts a huge repository of precompiled javascript snippets gathered by bing crawler and precompiled by M$, then IE just looks up correct snippets as bytecode in local snippet database and executes them fast.

So what they stumbled upon was a published javascript snippet that had a bytecode, so here the fast result.

However modified snippets will be downloaded as soon as bing crawls that discussion, and precompiled again, and pushed into next updates.

However the initial test was written poorly by using dead code. What if Mozilla decides to do deadcode optimization? Good test should actually check if calculated values do match (of course this doesn't exclude option of M$ precompiling actual function results)

Re:precompiled

[shutdown -p now]

Actually MS hosts a huge repository of precompiled javascript snippets gathered by bing crawler and precompiled by M$, then IE just looks up correct snippets as bytecode in local snippet database and executes them fast.

Reference?

Re:precompiled

[Anonymous Coward]

No reference, it's just an ironical idea of why only that function works incredibly fast, but no other altered function does :p

Re:Really? You're Going with That?

[Anonymous Coward]

so then, why do we tell the children it matters if they cheat? It matters, but not for about 11 more years.

Re:Old news...

[clone53421]

Somebody ought to try simply turning the if-then-else statement around... i.e. instead of

if (TargetAngle > CurrAngle) {
    NewX = X - (Y >> Step);
    Y = (X >> Step) + Y;
    X = NewX;
    CurrAngle += Angles[Step];
} else {
    NewX = X + (Y >> Step);
    Y = -(X >> Step) + Y;
    X = NewX;
    CurrAngle -= Angles[Step];
}

Use:

if (TargetAngle <= CurrAngle) {
    NewX = X + (Y >> Step);
    Y = -(X >> Step) + Y;
    X = NewX;
    CurrAngle -= Angles[Step];
} else {
    NewX = X - (Y >> Step);
    Y = (X >> Step) + Y;
    X = NewX;
    CurrAngle += Angles[Step];
}

That’s perfectly logical. I’d try it myself, but I don’t have IE 9.

Acid and Compilers

[LBArrettAnderson]

It should first of all be noted that MS is one of the only companies with enough balls not to try to get 100/100 on acid 3 (many of the "requirements" are not part of any official and final specification). I see MS as being the *least* likely to cheat on pointless benchmarks.

Second, adding a return value *does* in fact make a huge difference in run time. There is a very large difference in the way the stack functions, and many optimizations can be made when there is no return value in a function. There are of course some other requirements for this to be possible, but I'm going to go ahead and assume that the JS engine is just being smart about running pointless code. This is not cheating. Perhaps they were encouraged to make this optimization in order to do better on the test, but it could still very well be a valid optimization.

Re:Acid and Compilers

[shutdown -p now]

Second, adding a return value *does* in fact make a huge difference in run time.

There's no return value added. What they basically did was change this:

void foo() {
}

to this:

void foo() {
  return;
}

Note that it's "return;", not "return x;".

Re:Acid and Compilers

[LBArrettAnderson]

The point is that it's entirely plausible that they are parsing "return;" as "return null;" Yes, it's stupid, but it shows that there are real possibilities other than "they are cheating at this test."

And on top of that, maybe the JS spec says that "return;" is interpreted as something other than ending a function without a return statement (like "return null;" for example).

Re:Acid and Compilers

[shutdown -p now]

JS spec says that "return;" is to be interpreted as "return undefined;". However, JS spec also says that reaching the end of function without explicit return should be interpreted as the same exact thing, so inserting an explicit return at the end should have no difference.

Of course it may be that the optimizer reacts to the mere presence of "return" (in which case it's a deficiency - an explicit return by itself, so long as it does not return the value computed by the loop, should not affect DCE for that loop.

Re:Old news...

[shutdown -p now]

Putting a return at the tail of a function is unusual?

Yes, actually, it is quite unusual. Note that we're not talking about "return x;" here but rather plain "return", which does not make any sense (as function will return anyway once its end is reached).

I don't know about JS, but for C# and Java, many IDEs would actually highlight such a statement as dead code, and prompt to remove.

Dead Code Elimination

[TimSneath]

Hi, we've posted an official response and explanation at the bottom of this post:
http://blogs.msdn.com/b/ie/archive/2010/11/17/html5-and-real-world-site-performance-seventh-ie9-platform-preview-available-for-developers.aspx

Bottom line - we built an optimizing compiler into the Chakra JavaScript engine that eliminates dead code where it finds it.

It's a fun conspiracy theory that we'd somehow 'game' SunSpider in this way, but it doesn't make sense that we'd go to all that risk to cheat on just one test out of 26.

Best wishes, Tim Sneath | Microsoft

SunSpider needs to be rewritten

[jejones]

Dhrystone went through the same thing. The first version was written not taking into account dead code elimination, and compilers capable of it pretty well eliminated it all, rendering it useless as a benchmark. Dhrystone 2.0 was revised to prevent dead code elimination, so that to run it you actually had to perform the operations the code was intended to perform.

So... when will we see SunSpider 2.0, and results of running it under various web browsers?

because it improves the average Re:Old news...

[WolfWithoutAClause]

A lot of magazines just run all the benchmark tests and print the average of all the tests. Even a single test that comes back a lot quicker can change the average enough to make the overall results much better than your competitors results.

Incentive

[DrYak]

Honest question: where does it stop?

at the point where there's a financial incentive so people to it on their own.
the free market can regulate itself, but is very short-sighted.

people don't have incentive to split medical costs and spread them among a larger population to dampen the effect of a costly to heal/repare accident.
no incentive because nobody needs usually a doctor, not until they run into a problem and start needing one badly.

on the other hand a farmer, a baker, etc have an incentive : earning money by selling it.

Microsoft bug on SunSpider test

[Anonymous Coward]

All the comments seem to assume M$ simply submitted to the test without doing any testing to see how they would do for slight changes. If so, we are talking about a FAR bigger problem than a simple cheat.
Any manufacturer, IMO, has a duty to test his product, especially when it is going into a benchmark. I would rather have a careful, conscientious manufacturer who (childishly) cheated a little, than a careless one who simply threw something out there.
But then, M$ has done that in the past. I would like to think they have improved, but maybe I am the careless one for hoping that.

More data

[roca]

http://blog.mozilla.com/rob-sayre/2010/11/17/dead-code-elimination-for-beginners/

Summary: The Chakra DCE "optimization" handles ++ but not --, > but not >. In short, it handles exactly the set of operators found in math-cordic, and not others. This makes no sense if it's intended to be a general-purpose optimization.

Furthermore, Sayre also points out that the optimization makes assumptions that aren't true for JS, leading to bugs where it incorrectly eliminates code that is not, in fact, dead.

Re:More data

[roca]

Er, please excuse the quoting errors there.

Note, fixing those bugs requires a fundamentally more powerful analysis framework, or the introduction of dynamic guards, both of which would slow down the JS engine.

precomputed result, eh?? Find it in the program!

[Anonymous Coward]

If the IE result is precomputed, then find it in IE's memory footprint or elsewhere in its files. Find the code that matches the benchmark. Then decompile it. Now you have proof positive that it is cheating. That's what should have been done after the discovery, but before the announcement...