Slashdot Mirror

← Back to Stories (view on slashdot.org)

Futuremark And Gainward Tangle Over Benchmarks

Posted by simoniker on from the graphics-punch-up dept.

An anonymous reader writes "The optimization troubles in GPU land aren't over yet. Yesterday Futuremark released a new anti-optimization patch for 3DMark03. Gainward (who sell graphics cards using NVIDIA hardware) today made a comment that Futuremark had disabled certain features in their ForceWare 52.16 drivers, thus resulting in huge performance drops. A few hours ago Futuremark made an official statement about this : 'The accusation is totally wrong because what it suggests is not even feasible technically. 3DMark03 does not talk to graphics driver, it talks to the DirectX API, which then talks to the driver. Thus, it is impossible for the application to disable GPU compiler...'"

32 comments

  1. Oh.. by OutRigged · · Score: 2, Interesting

    Oh the drama..

    Personally I think benchmarking programs of this kind are outdated. I could really care less when the ATI Radeon 12500XTS gets 100 more 3DMarks then the GeForce TXP xi5000 Extreme. What does that matter?

    What matters to me is real world performance. And by all means, let the companies 'cheat' in that area. If it takes away little/no quality, and makes the games I play faster, more power to 'em.

    --
    RaGe
    We're all just noise on the wires..
    1. Re:Oh.. by Anonymous Coward · · Score: 0

      Benchmark scores generally (or should I say roughly) translate into real-world performance scores. That's why it matters. You could round to the nearest thousand if it makes you feel better.

      The problem with cheating in synthetic benchmarks is that it doesn't necessarily translate into real-world performance. The example of nvidia "optimizing" a synthetic benchmark by not rendering anything outside of the viewable area comes to mind...

      Lastly, synthetic benchmarks report what options the benchmarks were done with. Thus, it would be unfair to optimize with any quality decrease as people would not be able to note that when comparing cards using different drivers.

  2. omg by Apreche · · Score: 3, Interesting

    I wish people would stop fighting over which stupid video card is better than which other stupid video card. All these people who care about benchmarks and all this other shit are the same people who think that audio cable made of gold and platinum will make their music sound better. All that matters is

    a) is your video card good enough to do the things you do.

    b) the price/performance ratio, unless you have tons of money.

    That's all that matters. If your video card can run with vertical sync on and not lose frames in your favorite games, then you're all good. I've noticed Ati does a little better with the DirectX and Nvidia does a little better with the OpenGL. What it comes down to is they are both good cards that both work. Choose the one that suits your needs. Things like VIVO or DVI out are much more important features than 5fps more than the other guy.

    I've got a GeForce FX5900, from gainward in fact. And you know what, it's not the fastest thing there is. It's fast enough for me though. I can play all my games. I will be able to play Doom3 and Half-Life 2. The card will last me 5 years. My TNT2 last me 5 years, there's no reason this one wont. It has VIVO, it has DVI, and I can plug two monitors into it. That's all that matters. If I lose some fps here or get some more over there, it wouldn't bother me a bit. I run vsync anyway so my fps is usually locked at 85.

    So, what does your video card do different than mine? Well there's this one program that makes numbers show up on the screen. Your video card makes bigger numbers show up in this program and mine makes smaller numbers show up. Lucky you!

    --
    The GeekNights podcast is going strong. Listen!
    1. Re:omg by Anonymous Coward · · Score: 0

      And some people want the best card they can get!

      The things they do are:
      >Play with card settings - push it higher and higher.
      >Play games at greatest quality possible (with reasonable performance)

    2. Re:omg by SuuSt · · Score: 4, Insightful

      a) is your video card good enough to do the things you do.

      b) the price/performance ratio, unless you have tons of money.

      And how, preciselly, are we supposed to determine those price/performance ratios if our benchmarking suites are lying to us?

    3. Re:omg by slittle · · Score: 2, Insightful

      Reviews featuring benchmarks taken from real games, preferably the specific games you intend to play.

      Who gives a fig if drivers are 'creatively optimised' for real games, so long as the quality doesn't suffer?

      Benchmark apps are for e-wang waving.

      --
      Opportunity knocks. Karma hunts you down.
    4. Re:omg by RupW · · Score: 1

      I wish people would stop fighting over which stupid video card is better than which other stupid video card.

      The faster and further they push the bleeding edge the better affordable cards for we masses will get. (Assuming they can keep handing the R&D costs off to the early adopters, that is.)

    5. Re:omg by sonamchauhan · · Score: 1

      Who gives a fig if drivers are 'creatively optimised' for real games, so long as the quality doesn't suffer?

      And how would you check that quality hasn't suffered - would you actually buy two cards and run them up side by side on two monitors to check?

      Benchmarking apps provide "the rest of us" a reliable way to compare cards. Any "secret optimizations" or other attempts to silently manipulate benchmarks or game performannce must be seen as the dishonest tactics they are.

    6. Re:omg by karnal · · Score: 1

      That's pretty funny. You rant and rave about how people shouldn't fight about their video card, and then have to make me feel like I'm falling behind with my meager ti4200.

      That just hurts. :)

      --
      Karnal
    7. Re:omg by Elm+Tree · · Score: 1

      Meager ti4200? I'm still running an MX200...

    8. Re:omg by Anonymous Coward · · Score: 0
      Who gives a fig if drivers are 'creatively optimised' for real games, so long as the quality doesn't suffer?
      Because it may not be optimised for the game I am playing, it may only be creatively optimised" for Quake 3 or some other common game.
    9. Re:omg by wolrahnaes · · Score: 1

      And how would you check that quality hasn't suffered - would you actually buy two cards and run them up side by side on two monitors to check?

      Actually, that's basically what the major review sites do. When you see the IQ comparisons at a site like [H]ard|OCP, they put the screenshots output by the game side by side with those from another card at the same spot in the game. Some sites also compare this to the DX reference renderer and the game's software mode.

      Many games also have features to colorize where certain known "optimization" features are being used. I don't remember where I saw this in use, but it was recently and on a major review site.

      Synthetic benchmarks have their place. Just don't base a purchase decision solely on them.

      The video card companies need to learn a few things:

      1. Don't cheat
      You WILL be discovered.

      2. Marketing isn't everything
      The first GFFX chips (a.k.a. "DustBuster"),
      Parhelia,
      Voodoo4/5

      All victims of marketing, but in different ways.

      The GFFX was heavily hyped and seems to have been pushed out the door to meet a release date (and because nVidia was getting creamed in the benchmarks by ATI at the time), so the nV engineers didn't have time to improve the performance some more or at least decrease the heat dissipation. Looking at the current GFFX selection proves that time can heal wounds.

      Parhelia was not marketed heavily to gamers by Matrox (or at least I didn't see that much of it). The problem here was that the gaming mags and websites got wind of the triple-monitor 3D capabilities and started drooling. It was only a matter of time before people were disappointed because they had gotten their hopes up for a product that wasn't designed for what they wanted it to do.

      Voodoo4/5

      The problem here is that the hype pump works a lot faster than engineers. The fact that a quad chip V5-6k couldn't beat a GF2 in some games means that 3dfx was far behind on this one.
      Unfortunately for 3dfx, the hype for the new product killed the last sales of the Voodoo3 (which were only being bought by 3dfx die-hards in those days).

      3. Framerates aren't everything either!
      Anything over 120fps is absolutely useless. Going over the refresh rate won't help you at all. Being a little slower at a better quality is almost always a better choice.

      That last one is bound to get me flamed, so...
      *ducks*

      --
      I used to get high on life, but I developed a tolerance. Now I need something stronger.
  3. Things like these... by Zelet · · Score: 3, Insightful

    ... make me realize how stupid I've been for so long. I used to buy thousands of dollars of computer equipment to be able to play the latest games until a friend of mine asked me, "Why don't you spend $200 and get a console?" You know what, there is no good reason not to. I'm a moron.

    --
    ...And when they came for me, there was no one left to speak out for me." - Martin Niemoeller (1892-1984)
    1. Re:Things like these... by Babbster · · Score: 1, Offtopic
      Insert obilgatory mouse and keyboard comment here...at least that's the kind of thing the die-hard PC gamers always shoot back at me. :)

      I agree with you, especially at the moment. For the price of the latest, greatest video card alone you can get all three consoles brand new plus a game or, if you like the Nintendo line-up, the Gamecube plus eight games (up to double that if you go for the budget titles/used games). Not too shabby for a lot of hours of fun. Right now, the only games I play on PC are Hoyle Casino and Civilization III.

      At the rate I'm going, the next time I upgrade will be when Firaxis comes out with the new version of Pirates! that's supposedly in the works.

    2. Re:Things like these... by SuuSt · · Score: 1

      You don't suggest in your post that one abandon the PC in favor of the console, but that's the general impression it gives. There is no good reason not to get a console, but I'll give you a few good reasons not to give up on the PC:

      Warcraft 3, StarCraft, Half Life 2 (may come to consoles withing a year of release), Massively Multiplayer Game X (minus SWG), BattleField 1942 and it's mods, hell mods in general (they may have had half life for the playstation 2 but you couldn't play counterstrike with it), higher than 640x480 resolution, etc...

      In essence, most of us really have enough room in our wallets for both so theres little point in bickering over it. Admittedly most of the people who do argue are kids trying to defend their decision to get an X-Box for their birthday instead of a game cube (i know i did it when i chose the jaguar as a kid ::shudder::)

    3. Re:Things like these... by Haeleth · · Score: 1

      > For the price of the latest, greatest video card alone you can get all three consoles brand new plus a game...

      And, for the price of a console, you can buy two video cards as powerful as the one inside it - or just one mid-range card, which will already produce better graphics than the console at a higher resolution. The point about the latest, greatest video card is that it has as much power as the next generation of consoles will in a few years time. I'll put it this way, I'm not holding my breath for a PS2 port of Doom 3.

    4. Re:Things like these... by Babbster · · Score: 2, Funny
      I understand. First-person shooters are the light of the PC gamer's life, and if they can't run their FPS at 1600x1200 with full detail then the game sucks. I don't think people who enjoy their PS2s are crying rivers over the fact that Doom 3 isn't going to be on their console. More likely, they're looking forward to playing one of the 20+ potentially good games coming out for their system just in the rest of this month (Battlestar Galactica, Final Fantasy X-2, Fatal Frame 2, Monster Rancher 4, Metal Arms, Secret Weapons Over Normandy to name a few - only ONE of which will be coming out for the PC anytime soon). Then again, maybe at the realization that they can't play any of them on a $500 256MB graphics card they'll just jump off a bridge.

      I think I'll knock this comment down to 1. It has the smell of flamebait but I still feel the need to post it. :)

    5. Re:Things like these... by Haeleth · · Score: 1

      *winces*

      You have a point.

    6. Re:Things like these... by Anonymous+Cow+herd · · Score: 1

      Bah... real gamers have a killer gaming rig AND a console or three :-)

      --
      Ita erat quando hic adveni.
    7. Re:Things like these... by Baikala · · Score: 1
      I think you've made a god point there. All the people I know with a radeon8000/Ti4200 (or any other second-of-the-line-when-it-came-out video card ever since) are serius gamers, price/performance aware, and in general true game lovers that own at least one of the current generation consoles (but not three, I don't know any serius pc gamer who also own all the three consoles).

      I know there sould be lots FPS-only gamers that are always buying the latest state of the art card, overcloakingit like if 280 fps are any different to 272 fps, people who don't conceive digital entretaintment without a mouse, but with each passing year I see fewer and fewer of them.

      --
      16,777,216 comments ought to be enough for any forum!
  4. just returned a gainward 5200 128 by Numeric · · Score: 0, Troll

    i went to a computer show this weekend and i picked up a gainward 5200 128. installed the latest nvidia drivers. installed 3d mark 2001 se (i only play dx8.1 games) so i wanted to test the results. i seriously got a score of 3100. i returned the card and bought an ati 9100 128 pro instead and scored a 7100.

    since i mostly play americas army, i checked my FPS with the 5200 and was scoring around 10 fps on the map SFcsar. i thought AA implemented a new bullet-time feature!

    i think my card may have been screwed up but i didnt feel like troubleshooting it extensively.

    --
    -- ladies and gentlemen we are floating in space!
    1. Re:just returned a gainward 5200 128 by Bobulusman · · Score: 1

      I know you said you didn't want to troubleshoot it extensively, but SOMETHING must have been screwed up. The ATI 9100 came out before the GF4, right? There's no way it could result in double the points of a GF5.

      Now, I've owned both ATI and Nvidia, so I hope no one will call me an nvidia fan-boy, but that just doesn't seem right.

      --
      Cogito ergo sum in Slashdot.
    2. Re:just returned a gainward 5200 128 by Anonymous Coward · · Score: 0

      You've owned both ATI and nVidia? Wow. Those are pretty big companies, you must be mighty rich.

    3. Re:just returned a gainward 5200 128 by damiam · · Score: 1

      The 9100 came out around a year ago, long after the GF4 series (it was on the low-end of ATI's lineup at the time). The FX5200, which came out a few months later at the bottom of NVidia's line, does not perform particuarly well (it falls behind the GF4 MX in several cases). While the numbers (3100 vs. 7100) seem a little extreme to me, the basic scenario is perfectly plausable.

      --
      It's hard to be religious when certain people are never incinerated by bolts of lightning.
    4. Re:just returned a gainward 5200 128 by Anonymous Coward · · Score: 0

      Technology dear William (to quote Rufus):

      From the Riva 128 all the way to the Ti4800, Nvidia relied on its own engineers to create their little miracles. But when Nvidia bought out 3dfx, they took over 3Dfx's last project, which is where the "FX" cores came from. Clearly, if 3Dfx where able to follow up the VooDoo5 with an FX type of product back in 1999-2000, they would dominate the market. Unfortunately, the FX does not live up to the rest of the family.

      I have a Ti4200. And I am not upgrading. First, I need to see some real evidence that DX9 looks any better than DX8. I would rather have 65fps in DX8 with my current card than 25fps in DX9 with a new $150 card. Just my opinion.

  5. Just another pissing contest by Anonymous Coward · · Score: 0

    Since most gamers are male, and males have and embedded need to outperform other males (breeding, etc..) this whole argument basically boils down to who has the bigger schlong. If I buy a product and it doesn't work up to specs I don't purchase another product from that manufacturer. I have owned both Nvidia and ATI, and I prefer the ATI. I tried using the Svideo out on my Nvidia card, but to get it to work I had to run a third-party program. With ATI there was no problem. Consequently I prefer the ATI. Now if they would just make some Linux drivers that work available I would be a happy camper. Just my thoughts!