Slashdot Mirror

← Back to Stories (view on slashdot.org)

FutureMark Confirms nVidia's Benchmark Cheating

Posted by michael on from the red-handed dept.

jlouderb writes "As first reported by ExtremeTech, Futuremark has confirmed that nVidia is cheating on its 3DMark2003 benchmark through eight driver optimizations. The 3D graphics performance war just keeps getting more and more interesting!" See our previous story.

12 of 404 comments (clear)

  1. This is why.. by craigtay · · Score: 5, Insightful

    You don't base your findings on one benchmark. Whenever I go to a site like tomshardware.com they have several different ways to benchmark. Each card has its own strengths, and if a card has cheated it will show up like that.

    --
    Paint.NET, a Free Image Editor, with Source Code Available!
  2. Re:This is why artificial benchmarks don't matter by mskfisher · · Score: 5, Insightful

    Wrong - as they point out in the article, these "optimizations" are usually reductions in quality. They don't just improve performance.

    --
    0x0D 0x0A
  3. Isn't this standard practice? by Nathan+Ramella · · Score: 4, Insightful
    There's no law about fudging benchmarks on a third party application.

    While this isn't a huge suprise, I am happy that there are smart folks out there who spend time to uncover this kind of information. Kudos to you for your efforts!

    Videocard Benchmarks are about as believable as the the 'World's Best Grampa' award.

    -n

    --
    http://www.remix.net/
    1. Re:Isn't this standard practice? by James+Lewis · · Score: 4, Insightful

      It doesn't need to be against the law. Their motive for doing this in the first place was the expectation that their card would gain a better reputation by doing well in that benchmark by cheating. Instead, it has backfired and seriously hurt their reputation. Having a community that can uncover these unsavory practices is deterrent enough.

  4. History repeats itself a thousand times over... by voxel · · Score: 5, Insightful

    This has been done for many years, even the last decade. A good friend of mine works and has worked for almost every major video card company in the buisness for the last decade. What is his job? Make sure THEIR video card gets the best scores on the latest and greatest video cards.

    I am sorry to tell you all, but just because Nvidia was CAUGHT this time, doesn't mean they haven't been "cheating" (by optimizing for a specific benchmark) for the last 6 years.

    I would bet every driver release contains code to help out benchmarks and even specific games. Why do you think Nvidia just said with there latest driver release " *Up to 30% faster frame rates ( *With Unreal Tournament 2002)".

    Its just once in a great while someone notices a performance jump TOO big, or just wants some news worthy-ness and decides to put out a nice PDF file.

    - Jeff

    --
    Modesty is one of life's greatest attributes
    1. Re:History repeats itself a thousand times over... by Cyno · · Score: 4, Insightful

      Maybe this is a good arguement for open source drivers? Afterall I'm paying for the hardware.

  5. Re:This is why artificial benchmarks don't matter by paranode · · Score: 4, Insightful

    This is a problem with Nvidia. The only reason they are competing well with ATI is because they cut so many corners to get their benchmark scores up. It certainly would be nice if Nvidia concentrated on real-world apps and games but it seems like they do not. If you look at the benchmarks historically between ATI and Nvidia's closely competing cards, you'll find that they are closely matched in default runs. However, try turning on 4x anti-aliasing or anisotropic filtering and watch the older, slower, ATI cards beat out the shiniest new Nvidia cards. ATI's image quality has always been superior to Nvidia's. They are all about quantity and need to be focusing more on quality.

  6. Re:Don't don the tinfoil hats prematurely... by rhavyn · · Score: 4, Insightful

    When they did nothing more then change the splash screen, the nvidia card gave out different results. That seems to be detecting the benchmark and cheating.

    Try reading the article.

  7. Wasted Code by JeffRC · · Score: 5, Insightful

    Just think about this the next time you do a 5MB driver download. How much of that code is specifically for detecting and defeating benchmarks? How much of the cheats are part of the instability problems in your system?

  8. Re:lies and statistics. by Surak · · Score: 5, Insightful

    The problem with 'real world testing' when it comes to video cards aimed at the gamers market is basically the difference between a few lousy FPS between the two top-of-the-line cards (and each have similar features, performance-wise) will be virtually indistinguishable in most cases.

    I think people shouldn't get all macho when it comes to this stuff. Honestly, it's like the difference between a 350 hp engine and a 351 hp engine. It doesn't amount to a hill of beans worth of difference except on paper.

    Get over it people.

    --
    My journal has hot /. gossip.
  9. Re:This is why artificial benchmarks don't matter by mskfisher · · Score: 4, Insightful

    In terms of the clipping planes, you're right.

    But the nVidia driver also substituted a shader for one of the water effects, which degraded/modified the image quality.

    And past history has shown that companies are willing to sacrifice quality for performance (see ATI's Radeon 8500 drivers and Quake 3 for an example)...
    It's almost like this is a cold war, of sorts, between the testers/benchmarkers and the card manufacturers.

    --
    0x0D 0x0A
  10. thanks, but... by Shadestalker · · Score: 4, Insightful

    Dear nvidia / ATI / etc.,

    Please optimize your drivers and hardware for the actual applications and games I run, not the synthetic benchmarks designed to simulate workloads. Benchmarks don't use your products, end-users do.