Slashdot Mirror
Intel Caught Cheating In 3DMark Benchmark
EconolineCrush writes "3DMark Vantage developer Futuremark has clear guidelines for what sort of driver optimizations are permitted with its graphics benchmark. Intel's current Windows 7 drivers appear to be in direct violation, offloading the graphics workload onto the CPU to artificially inflate scores for the company's integrated graphics chipsets. The Tech Report lays out the evidence, along with Intel's response, and illustrates that 3DMark scores don't necessarily track with game performance, anyway."
Thanks for telling all of us that the best measure of hardware's performance ingame is... to benchmark it with a game.
A bullet may have your name on it but splash damage is addressed "To whom it may concern."
I thought offloading graphics computations to the CPU was the whole *point* of integrated video.
TODO: Something witty here...
On the one hand, a mechanism that uses the CPU for some aspects of the graphics process seems perfectly reasonable(whether or not it is a good engineering decision is another matter, and would depend on whether it improves performance under desired workloads, what it does to energy consumption, total system cost, etc.), so I wouldn't blame intel for that alone.
On the other hand, though, the old "run 3Dmark, then run it again with the executable's name changed" test looks pretty incriminating. Historically, that has been a sign of dodgy benchmark hacks.
In this case, however, TFA indicates that the driver has a list of programs for which it enables these optimizations, which includes 3Dmark, but also includes a bunch of games and things. Is that just an extension of dodgy benchmark hacking, taking into account the fact that games are often used for benchmarking? Or is this optimization feature risky in some way(either unstable, or degrades performance) and so only enabled for whitelisted applications?
If the former, intel is being scummy. If the latter, I'm not so sure. From a theoretical purist standpoint, the idea that graphics drivers would need per-application manual tweaking kind of grosses me out; but, if in fact that is the way the world works, and intel can make the top N most common applications work better through manual tweaking, I'm can't really say that that is a bad thing(assuming all the others aren't suffering for it).
Just look at the pics. Changing the name of the executable changed the results dramatically. The driver is apparently detecting when it's running a 3DMark (or some other specific apps) and switches to some other mode to boost its scores/FPS markings.
"In a 32-bit world, you're a 2-bit user. You've got your own newsgroup, alt.total.loser." -Weird Al
Is 3DMark the benchmark that will give a higher score to a VIA graphics card if the Vendor ID is changed to Nvidia?
Intel fully admits that the integrated chipset graphics aren't that great. They freely admit that they offload rendering to the CPU in some cases. This isn't a secret.
Oh, ATI was one of the first to cheat on a graphics benchmark quack.exe anyone?
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
In true White Goodman fashion, cheating is something losers come up with to make them feel better about losing.
Is Intel the 500 lb. gorilla in chipsets? Sure, and they got there by 'cheating.' Which is winning.
Aint capitalism grand?
For all you losers who don't know who the great White Goodman is: http://www.imdb.com/title/tt0364725/
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
That was my first thought, too.
Here's the thing, though: They took 3DMarkVantage.exe and renamed it to 3DMarkVintage.exe, and much of that offloading was dropped. So this isn't a general-purpose optimization, which would make sense -- it's a targeted optimization, aimed at and enabled specifically for a benchmark, in order to get higher scores in said benchmark.
It reminds me of the days when Quake3.exe would give you higher benchmarks, but worse video, than Quack3.exe.
Don't thank God, thank a doctor!
"they should be encouraged to release hand coded or special drivers to improve performance in specific games."
Games, sure - but it defeats the point of benchmarks by introducing a new useless variable: how optimized the driver is for that benchmark. I mean, why should 3dMarkVintage.exe be 30% slower than 3dMarkVantage.exe? How does this help anyone except Intel?
Effectively dividing tasks among CPUs is not the issue here. They want to benchmark the GPU and they wanna make sure you don't enable optimizations that are targeted specifically for the benchmark which Intel was doing shamelessly.
Please mod this up; it really is that simple.
It is a miracle that curiosity survives formal education. - Einstein
It's not special drivers for specific games. It's regular drivers with exceptions coded in to make them appear faster on "standardised" tests, which are meant to be an all-purpose benchmark to help consumers identify the sort of card they need (and to compare competing cards). This is cheating to increase sales among the early adopter/benchmarker crowd, impress marketing types and get more units on shelves, and is generally at the cost of the consumer.
I'm not defending Intel at all, but...
ATI's done it: http://www.xbitlabs.com/news/video/display/20030526040035.html
NVIDIA's done it: http://www.theinquirer.net/inquirer/news/1048824/nvidia-cheats-3dmark-177
They've probably done it several times in the past with other benchmarking software as well.
They're all dishonest. Don't trust anyone!
I don't know all the details of when (in relation to AMD buying out ATi) but...
ATi was notorious for cheating on the IQ benchmarks - essentially using a different anisotropic filtering method for the IQ test (the good one), and then the cheating one during the other tests.
The ridiculous part was that Nvidia was caught doing a similar thing, and the outcry (in part driven by ATi calling out Nvidia) forced Nvidia to include admit it and later driver option to select the optimization level used. When ATi was later caught doing the exact same thing, there was no outcry, there was no admission (despite proof), and there was no option in the drivers to turn off the "optimization".
I don't recall the details of why the particular optimization was considered a "cheat" and others weren't (I believe it killed off IQ to the point of ass, and it was something that would never be used in-game).
This was back around the 6800 (Nvidia) vs x800 (ATi) days.
... aaaand ATI has done the same thing at least a couple of times.
The question is probably more easily answered if asked, "Who *doesn't* cheat?". http://www.xbitlabs.com/news/video/display/20030526040035.html
Fact: Everything I say is fiction.
Exactly. If they want to offload GPU processing to the CPUs, then they should do that for ALL programs, not just certain ones in a list.
I'm just guessing here, but maybe because offloading this work to the CPUs decreases CPU performance substantially, they don't want to make these changes generic because it'd make it look like systems with Intel video are slow, especially in any CPU-oriented benchmarks. After all, they pointed out in the article how Intel does this same thing for the "Crysis" game, but even with this offloading working, the game only got a measly 15fps with all extra effects off, which is downright unusable.
The whole thing just looks really shady.
It's not special drivers for specific games. It's regular drivers with exceptions coded in to make them appear faster on "standardised" tests, which are meant to be an all-purpose benchmark to help consumers identify the sort of card they need (and to compare competing cards). This is cheating to increase sales among the early adopter/benchmarker crowd, impress marketing types and get more units on shelves, and is generally at the cost of the consumer.
No need for a car analogy on this one. So it's like what happens when the public schools teach a generation or two in such a way that they are optimized for performance on standardized tests, and when those students eventually enter the working world, they don't know how to make change without a cash register or other calculator of some sort? The way they don't know how to deconstruct an argument? Let alone understand the importance of things like living within your means?
It is a miracle that curiosity survives formal education. - Einstein
linked at the bottom of that article but here (in German but screenshots speak for themselves), the quoted ATI rep basically admits it too by saying that they optimize for the best "visual experience" where that's some mix of visual fidelity, framerate, etc.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Its funny that Intel simply creates an INF file and uses those to detect apps and optimize for performance. I mean, if you are detecting a file name and enabling performance optimizations, why not detect the app behaviour itself and make the optimizations generic ? Clearly you know the app behaviour and you know the performance optimizations work. This seem to me a case where people were asked to ship it out fast and instead of taking the time to plug the optimization into the tool, they just made it a hack. A really bad one too!!!
Sure, but how hard would it actually be for a graphics driver to scan an arbitrary executable and determine a) that it's a game and b) how it will behave when executed? I suppose they could model it after the heuristic and behavioristic features of some antivirus/antispyware applications, but nothing about this problem sounds trivial. There's also the question about how bloated of a graphics driver you are willing to accept.
My guess is that the above concerns explain why this was a poorly-executed hack.
It is a miracle that curiosity survives formal education. - Einstein
You'd think you'd have logic in the GPU that could determine when a certain load was being achieved, certain 3D functionality was being called, etc., and offload some work to a multicore CPU if it was hitting a certain performance threshold (as long as the CPU itself wasn't being pounded...but most games are mainly picking on the GPU and hardly taking full advantage of a quad core CPU or whatever). That makes a degree of sense...using your resources more effectively is a good thing. If that improves your performance scores, well...so what? It measures the fact that your drivers are better than the other card's drivers. That seems like fair play, from a consumer's standpoint. If the competitors can't be bothered to write drivers that work efficiently, that's their problem. Great card + bad drivers = bad investment, as far as I'm concerned. That's the real point of these benchmarking tests, anyway. It's just product marketing.
But trapping a particular binary name to fix the results? That's being dishonest to customers. They're deliberately trying to trick gamers who just look at the 3DMark benchmarks into buying their hardware, but giving them hardware that won't necessarily perform at the expected level of quality. I generally stick up for Intel, having worked there in the past as a contractor and generally liking the company and people...but this is seriously bad form on their behalf. I'm surprised this stuff got through their validation process...I know I'd have probably choked on my coffee laughing if I were on that team and could see this in their driver code.
Hasn't every chipset maker- ever- been busted for fudging benchmark results at some point? Multiple times, usually?
And then they get caught out by the old exe-renaming technique.
Why do they keep trying it? The mind boggles.
I would have thought by now that a standard tool in the benchmarkers repertoire was a tool that copied each benchmark exe to a different name and location and launched that, followed by a launch with the default name; and that the more popular benchmarks had options to tweak the test ordering and methodology slightly to make application profiling difficult.
Well, there is also the interesting tidbit that it doesn't enable those optimizations unless the CPU is an Intel CPU.
Hmm.
---
Marketing execs change all the time. Each one says "Hey! I have an idea...." The programmer who is asked to put in the cheat is not wildly enthusiastic about the idea, knows it won't work and does a quick and dirty hack.
Oh, ATI was one of the first to cheat on a graphics benchmark quack.exe anyone?
Oh this type of thing has been going on for a VERY long time. For example, there was the Chang Modification back in 1988 (It slowed down the system clock that was used as a timing base for the benchmark, resulting in higher benchmark scores).
Don't tell me to get a life. I had one once. It sucked.
Detecting application behavior dynamically is non-trivial. Commonly it is performed by instrumenting the binary, which _degrades_ the performance of the binary. The act of observation destroys the behavior to be observed, so to speak. This is why 3D marks vantage explicitly prohibits "Use of empirical data of application for optimization". _After_ you get the behavior of application, optimization is a lot easier.
That's not interesting. How do you plan to connect a non-Intel CPU to an Intel chipset with integrated graphics?
More data, damnit!
It's really hard to get 15% better performance without doing something underhanded unless your previous drivers were beta quality.
I used to work for a video card manufacturer and game and video developers often did totally retarded things which just happened to work on the cards they developed on but made the software run like crap on ours. We routinely had to implement workarounds for individual games to make them run properly on our cards.
One particular example which springs to mind -- I won't mention the developer or the game -- was an engine which used a feature which we supported in hardware but a certain other card manufacturer whose cards they used performed in software. Rather than configuring said feature once as they should have done, retarded developer repeatedly reconfigured it numerous times in the course of a single video frame, which required us to reconfigure the hardware every time -- slow as heck over an AGP bus -- whereas other card manufacturer just had to execute a few CPU instructions. We had to detect the game and disable our hardware support, so that we would fall back to software and run the retarded code much faster; in that instance there were places in the game where, far from a measly 15%, we'd literally be going from seconds per frame to numerous frames per second.
So it's quite possible to need to detect individual games or applications in order to work around retarded coding which cripples performance on your hardware. The line you shouldn't cross -- and which I don't believe we ever did -- was to render something other than what the developer intended, for example by detecting a shader used by a benchmark and replacing it with one that looked similar but didn't do as much work.
Similarly, the issue here is not Intel punting processing to the CPU when the GPU is overloaded, but the fact that they do so by detecting the name of the benchmark rather than by monitoring the GPU loading and dynamically switching between hardware and software so that it would work on any application. General optimisation is fine, workarounds for retarded developers are fine, but special optimisations for benchmarks which don't affect real applications is getting pretty close to the line.
Oh, ATI was one of the first to cheat on a graphics benchmark quack.exe anyone?
Oh this type of thing has been going on for a VERY long time.
I even remember teapot based hacks (although not the details unfortunately, probably something along the lines of having the teapot hardwired somewhere) back when displaying rotating GL teapots was all the rage to test graphics hardware (ancient history, obviously).
Of course something like Quake was still the stuff of science fiction at the time.
May contain traces of nut.
Made from the freshest electrons.
The behaviour their driver has in the benchmark is also used in several games... ie Crysis Warhead. RTFA.
Just to get things straight bloodhawk said:
Don't remember AMD cheating, But then I have only more recently become a big ATI fan, However Nvidia has a long history of benchmark cheating in drivers in order to make there stuff look better than it is and many times it was far more blatant than what intel is doing here
At the time of quack.exe ATI wasn't owned by AMD, cheating or no cheating we've got to be clear on that one.
I am the lawn!
3DMark Vantage was never a legit benchmark. Heavily tuned for Intel CPU and nVidia GPU architectures it never actually meant a damm thing.
Just compare performance of gf285/295 v. radeon 4870/5870 (any review) in 3DMark and in games. In 3DMark Vantage nVidia cards have close to 50% advantage while in real games radeons sometimes score higher.
The statistical anomaly alone is sufficient to dismiss 3DMark Vantage results as outlier.
All hope abandon ye who enter here.
You cite a 2001 issue as one of the earliest examples? Poor form. ;) :)
Go back another 4 years. nVidia released the Riva 128 which started winning most benchmarks against ATI's Rage Pro (and Rendition Verite etc). Well, a few publications started noticing that the speed advantage was due to the image quality being much worse with no tri-linear filtering, no fog (at least for a few iterations of drivers) and some sort of compressing the textures that made rendered text on some games illegible (a couple of games had the misfortune of having that problem even with their menu system). I remember the comparison images for the nicest benchmark/demo of the day called "Final Reality" were quite telling of the IQ difference. However, most publications of the time just went with fps numbers, so that left ATI with no choice but to "optimize" their new driver set (called "Turbo") especially for 3D benchmarks
Violence is the last refuge of the incompetent. Polar Scope Align for iOS
Both ATI and nVidia have been caught cheating (and by cheating I mean specifically targeting the FutureMark benchmarks to make their products look better than they actually are). The above link is only a single instance. A quick google will net you a good sampling over the last decade or two.
Optimizing a driver for a specific game is not cheating as long as it doesn't affect quality. Optimizing your driver to get inflated scores specifically in a benchmark is cheating.
The behaviour their driver has in the benchmark is also used in several games... ie Crysis Warhead. RTFA.
The issue is that the driver treats different games differently, based on filename. Some get this boost and some don't. Whether you put 3DMark into the boosted or unboosted category, its results will be indicative of some games and not of others.
maybe you don't care, but lots of sales are linked to good benchmark scores. Where the intel product does worse, they're making it look as if it's better. A polished turd is still a turd.
Think of this like comcast's speedboost. It sounds great to be able to be at 50MB/s downstream, except you only get to do it for 30 seconds, thus making real downloads not receive benefit.
wow we're at 50 mb/s! etc al.
I think the point is to benchmark the performance of the gpu. If your fav-game-of-the-month looks fabulous on your friend's hopped up system with xyz graphics card, you expect to get the same graphics performance if you buy the same card, despite having a lower class processor. If the game is already taxing your friend's CPU to play smoothly, imagine the reduced gameplay AND graphics you'll get when you try it on your system, since it's trying to offload GPU work to your already burdened CPU?
There's simply no excuse for changing your behavior when you detect a benchmark app is running. Fraud, fraud, fraud. That's no better than the driver software screwing with the benchmark app as it runs or modifying its output before it's displayed, bugging it into displaying completely made-up numbers of their choosing.
I work for the Department of Redundancy Department.
In any case application specific optimizations are a great tool. They got an extra 18% speed out of the chip with just application specific tweaks. That's a pretty damn significant increase. Ignoring that would be a terrible decision. All graphics drivers should use this and update the drivers every few months as new games come out.
The app-specific optimizations actually made Crysis look like shit, and ate more CPU power (you need an extra core to play Crysis), and the damn thing was still smashed by an equivalent AMD chip that could play Crysis at twice the frame rate (which was 30fps, rather than an unusable 15fps). The benchmark showed that Intel's was about 30% faster than AMD's offering, which in real life use was actually twice as fast as Intel's.
Support my political activism on Patreon.