Slashdot Mirror
More on Futuremark and nVidia
AzrealAO writes "Futuremark and nVidia have released statements regarding the controversy over nVidia driver optimzations and the FutureMark 2003 Benchmark. "Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat."" So nVidia's drivers are optimized specifically to run 3DMark2003... and that's not a cheat.
and i didn't use a cheat sheet, i used a memory priming sheet.
I never said I was smart, I just said I was smarter than you
Though, I do prefer they make application specific optimizations that mean better gameplay.
It's just another piece of information to keep in mind when selecting a new card.
So nVidia's drivers are optimized specifically to run 3DMark2003... and that's not a cheat
Errrr... that seems like a cheat to me!
Davak
But then, I'm Bill Gates.
Fururemark = Nvidia's bitch
Lawyers to the rescue!
Oh, it seems some careless individual has left this big pile of money on the table! Well, we'll just leave for a few moments and maybe when we come back it will have gone away.
Time to update Websters. Cheat just got new semantics.
I think [H]ardOCP stated it best as "Futuremark didn't want to get sued by Nvidia". Nvidia has the legal and financial resources to totally ruin Futuremark and they know it.
And now Futuremark has totally invalidated their own benchmark software by declaring it "open season" for hardware manufacturers to distort the "tests" in any way shape or form they desire to make the numbers higher.
N.
"Nothing strengthens authority so much as silence." - Charles de Gaulle
I don't think this is a cheat, as long as the energy taken to increase the performance in the test is similar to what is used to optimize for particular games.
I'm not going to complain if they can optimize the drivers to get 10% better performance in Battlefield 1942 with no degredation in quality, would you?
The only way I would call this a cheat would be if Nvidia never optimized for any other applications.
Sticks and Stones may break my bones, but copyright will always protect me.
Looks like someone is scared of somebody elses lawyers. Yuck! This is obviously Futuremark trying to appease Nvidia.
If nVidia intends to include driver optimizations for many popular applications, then is it really cheating? Lots of games are built upon the Quake3 engine, and I'm sure the Doom3 engine will be used for some great titles. If nVidia will optimize the drivers for specific games then I'm all for it.
It's not cheating if you don't get caught.
Oh, I did get caught?
No, I didn't. Let's move on, shall we?
Best Windows Freeware
This is politics at its worst, and I'm calling bullshit.
There was no need for this nicey-nice statement other than NVidia threatening lawsuits and Futuremark wanting to protect what assets they have.
Futuremark had every right to call NVidia on their selfish claim and unbelievable hacks. To say that they weren't liable for their own blunder is to say that Futuremark's reputation has been replaced by corporatespeak and a lack of respect almost unparalelled.
What's worse is that I really thought "Yeah, this time the bad guy gets his due" and that NVidia should've known better.
But of course, a few weeks later we've got to put on the nice face again for the public en large.
What a complete waste of time. I know there isn't much respect left in corporate America, but hell, if you can't call a spade a spade, why even bother with the benchmarks when someone can just rewrite an ENTIRE SHADER and only keep a picture clear while the demo is on rails?
This sort of outcome is inevitable as drivers move "up the stack" into the application layer. To get better and better optimizations, the drivers need to know more and more about the application that is requesting the services - thus, we end up violating the strict separation between application and driver.
I think it's all fine and good to have an application specific optimization, but I think it's wrong of nVidia to do it on a application that happens to be a benchmark. The only people benefitting from this optimization is nVidia, not nVidia's customers. Do they plan on making application specific optimizations for all applications? I think we know the answer to that is a resounding no. They just want us to see there product through rose colored goggles, thus missing all it's deficiencies.
On the other hand, if such optimizations do not help other applications, the benchmark was bogus to begin with!
Well, that's excellent... now they can put 'Designed to run 3Dmark2003' on Nvidia product boxes!
I am a filthy pirate.
Cripes already. No one even BOTHERS with #DMark anymore, and after this fiasco no one is ever going to bother with them again. Gamers will use REAL EVERYDAY GAMES to see what runs the fastest again. Looking at some goofy simulation app coming up with scores and people buying into the company and people tricking drivers for particular tests is just crappy and makes 3DMark 100% invalid to any of my concerns in the future. I will only trust reviews that benchmark the latest and greatest games that I will be buying these cards for, whoever can run them fastest at that particular time IS WHAT IM GOING TO BUY. Peroid. Enough of this 3DMark BS.
all you ATI lovers are owned!
I wonder how much it cost NVidia to get Futuremark to release that statement.
NVIDIA works closely with developers to optimize games for GeForceFX. These optimizations (including shader optimizations) are the result of the co-development process. This is the approach NVIDIA would have preferred also for 3DMark03. "
I wonder if ATI would love close cooperation between NVidia and Futuremark...
... FutureMark and NVidia stated that they were proud to announce that former President Bill Clinton had joined their boards and had assumed management responsibilities.
Chr0m0Dr0m!C
Using your analogy, ATI simply building fast video cards is studying for the test. What they are doing is getting a list of the test questions and ensuring they are especially good at answering those. Although they are doing pretty damn good all around studying, they are giving themselves a bad rep by adding a layer of cheating to the mix.
I'm wrong and so are you.
Run Quake3 with the video card and check out the frame rate and image quality. Run it under UT also and every other 3D game you can. Then compare the framerates and image quality. Who cares what it's 3DMark is. Did you a buy the card to specifically run it under 3DMark? Most people buy these cards for playing games so comparing how it runs the actual game to other cards is the only meaningful measurement.
Outdoor digital photography, mostly in New Engl
All your benchmarks are belong to us!
"So on one hand, honey is an amazingly sophisticated and efficient food source. On the other hand it's bee backwash."
Besides, pick any other industry, any other product, and companies are optimizing their products to run fast. J2EE and Databases performance testing comes first to my mind.
So the question in my mind is did Futuremark learn something from the discussions? Is there something it was ignoring in its tests?
I'm trying to not be a cynic and assume a big fat envelope was passed under the table. That what Nvidia did was legitimate.
"Depends on what the definition of is, is."
Those that are following this, should check the pictures on the previous article. The quality of the nvidia "optimized" version sucked (showed big artifacts). That's no optimization, unless there was no image quality loss.
"This card is optimized for quake as long as you follow the left trail, the right trail will just look like crap but nobody follows it anyway".
Why does Futuremark exsist? What do they do?
They make tests for nVidia and ATI! Their entire business is based around these two companies (ok, handful of others , too). Of course they worked out some deal with nVidia. I'm sure that nVidia has ways to lean on them so they changed their tune. FutureMark really can't afford to get on the bad side of the card makers, because if they loose the support of nVidia and ATI, what are they going to do?
It's symbiotic realtionships that make the world go round...
Personally, who gives a crap about a benchmark? I'd much rather see stats of how well cards hold up in specific games than in a damn benchmark. On that line of thinking, I'd also rather see nVidia putting time and energry into making games, not benchmarks, run better on their cards.
If a benchmark doesn't measure performance related to real-world applications, what's the point? If a driver is optimized to run a benchmark faster, that SHOULD mean that the real world apps should run faster, too. If not, the benchmark is useless.
While nVidia has made great cards for some time now (I still use and love my GeForce 3), could it be possible that they're not able to keep up? A lot of the reviews that I've been reading tend to favor the Radeon cards over any anything that nVidia has put out lately. While I doubht that nVidia will become another 3Dfx because they're involved in other markets and I've read about them having US government contracts for this project or that, I would propose that they will not be the huge players that they once were in the vid card market.
If you want to, you can prerender the whole fucking test, stick it in your driver and just play it off instead of actually rendering when Futuremark is running, that would be an "application-specific optimization" too.
The benchmark is ment to reflect performance in the actual game, the reason it takes the same path is merely to make the results comparable. What ATI was something the game *could* have achieved in game, if the operations were properly sequenced. What Nvidia did is to fake a performance it can't actually give if a person had followed the exact same path in the game. That is cheating.
It is pathetic by Nvidia, and it's pathetic by Futuremark to present this press statement. Get some backbone and integrity.
Kjella
Live today, because you never know what tomorrow brings
Now that all the latest games have benchmarking modes, what do we need FutureMark for?
If NVidia wants to do application-specific optimizations that make UT2003 go faster, then that would be great. That's what they should be doing. Those are optimizations that genuinely benefit the user.
Is this truly the only Earth I can live on?
Why did you post that? Frankly, it's offensive and lends nothing to the discussion.
Come on get a life. These are afterall benchmarks. Over the years I've had to run benchmarks for various reasons and you do absolutly everything you can to get the best result you can.
This is a very normal situation. If you do not run them yourself then benchmark data is never better than marketing hype. NVidia is no more guilty than any other company. As a matter of fact they are no more guilty than any person running a benchmark. You always apply you best knowledge and, whether you realize it or not, most of the time prove a predeturmined belief.
Editorial comments like this are wonderful. They make the best shine out in all of us to which we can feel profoundly enlightened. Michael I thank you for taking the time to summarize two press releases by stating the complete obvious, if not for you I might just have been forced to click those god-awful annoying hyper-links.
Since you've obviously missed the "concern" over this whole issue let me help you out in also stating the obvious. Creating a chipset that can run the futuremark settings profoundly instead of processing them like a benchmark is supposed to is a way of, get this, testing out the performance of a piece of hardware. Basically what futuremark has made abundantly clear here is that they are to be viewed not as a benchmark, but as an industry whore who was previously considered a graphics card benchmarking standard that is now seen as a useless tool as it doesn't do what it claims to do.
Benchmarks don't need "helpers" and "shortcuts" they need to test out all the parts of the testing subject to show what it is that subject is truly capable of. So futuremark is in bed with graphics card manufacturers, this is no real surprise, it's just they've sealed their fate as being a worthless benchmark and nothing more than eye candy from now on.
Perhaps you should use the "comment" link like the rest of us common folk and try to acknowledge that you are actually older than 4. Writing your name up on the front of the board for everyone to see really loses its nostalgia around age 6, I hope.
Ignore the "p2p is theft" trolls, they're just uninformed
Sounds like someone threatened to sue for defamation or libel...
No, if you learn for a test using yor textbooks, lecture notes etc.
Yes, in case you actually have an exact copy of test problems.
nVidia could optimize the drivers for typical 3d scenes in games (textbook), or even for a specific game (lecture notes), but benchmarking software is different : it produces a very specific 3d-scene, and if you optimize the drivers to fit perfectly this 3d-scene (copy of tomorrow's test), you're cheating
"...the physical relationship with the President included oral sex, but not sexual intercourse."
Treatment, not tyranny. End the drug war and free our American POWs.
See my user info for links.
I think the money quote is:
However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.
I can sort of see the argument here, but it basically ruins the point of having a standard interface like DirectX. It's also like telling your math teacher, "no, it would be easier for my equations if you made 1+1=3. Now do it because I'm your star student."
It's just an unfortunate choice on nVidia's part.
I don't know about you, but I'd rather have my nVidia card optimized for, say, Quake3 or Battlefield 1942. Someone call ATI and tell them they've got their new marketing campaign.
It hurts when I pee.
ES-SEE-OH VERSUS EYE-BE-EM
1. Get money from ATI
2. Accuse NVIDIA of cheating
3. Get money from NVIDIA
4. make peace with NVIDIA
5. Goto 1
If they optimize the drivers for specific apps, this could be a good thing. However, for the sake of quality, the driver should offer control of what getss optimized.
When you hit puberty, can I pop your cherry?
"Cheating? Our developers, The Franklin Family, would disagree. I think a meeting can be arranged with them, if you wish. Isn't that right Mr. Franklin?"
*shakes hundred dollar bill side to side, speaks in high tone out of side of mouth*
"Sure is, Boss!"
When I read the statement it seemed to indicate that Futuremark was unsure how to write comparitve benchmark now that you can "cheat" at the benchmarks.
I can understand that if there are thing in a GPU that can be optimized for an application then you should go ahead and do it, but of course that means how do you truly, evenly compare the performance of one piece of hardware (now with tons of customizable software) versus another one?
Futuremark has a tough time ahead now getting people to believe they add imperical comparive value.
D.O.U.O.S.V.A.V.V.M.
its been Nvidia Mark for ages.... move along nothing to see here...
Futuremark: "NVidia is cheating! Not as much as ATI, but they're cheating!"
Nvidia: Knock knock
Futuremark: "Who's there?"
Nvidia: "Goons...hired goons."
Futuremark: "Oh...haha...um...Nvidia is actually in the business of application optimisation! Our mistake. Won't happen again."
Seriously folks, this is Nvidia using big bad lawyers to scare Futuremark into capitulating. They might have held their ground, until ATI was proven to be doing the same thing, albeit to a much lesser degree.
Unfortunately, the only person who loses in this scenario is the consumer.
John Maynard Keynes: "When the facts change, I change my mind. What do you do?"
...I'm picturing principal skinner with a laser target aimed at his forehead.
This has been discussed before. Other companies have tried to modify their drivers to produce better results for certain benchmarks. They've always been thrown out as invalid before. I wonder why Futuremark seems to be considering allowing NVidia's enhancement to stand.
There's a line from the story:
"...However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks."
I'm concerned because I feel that allowing video card manufacturers to put code specifically about certain benchmarks in to their product (making their product look better in that benchmark) may not be reflective of real world performance.
However, the benchmark is useless if it doesn't measure real world performance, so I do believe that NVidia could put stuff in their product to make the benchmark run faster that would be beneficial to real applications, so I'm torn.
Some game manufacturers make optimized versions of their code to work with certain video cards, but the normal use is an operating system driver (DirectX...) and I believe using the generic driver is more representative of what you'll get when you use the video card.
It seems that NVidia is arguing that they should be allowed to put optimizations in to their code specifically for the benchmark because they do the same thing with some other populate applications.
how much did that cose them? (....waiting 20 seconds.........waiting..........)
because I have been enjoined by this Holy Office to abandon the false opinion which maintains that the Sun is the centre
Well, after seeing the future mark scores, it looked to me as nvidia fx chips are blowing away ati and its gf4 line.
But I found a really nice german benchmark site 3dcenter.org that had to be the best benchmarks ive ever seen, they actually use the games on each and lists the fps.
Looks like the FX/GF4 5200/4200 4600/5600 (non ultra) are the same. And the ATI 9700PRO/9800 are faster than the 5800 Ultra.
After reading these benchmarks, you can really tell nvidia tweaked the SHIT out its drivers for futuremark...
(blunt)The problem with a lot of the reasoning I see here with people saying they want the card that plays the game they're interested in quickly, is that it's completely stupid. (/blunt)
When you're looking for a video card, you -should- rely on a capable, and untainted/optimized benchmark for comparison simply because you can't predict what the software companies that make the actual games are going to do. Will they support -your- chosen card, or will some other GPU maker offer a better bribe to the developer? You may know that kind of info about games shipping RSN, or already on the shelves, but what about next year's?
Getting the card based simply on one or two games instead of looking at some kind of objective benchmark does no good whatsoever. It's just a way to rope yourself into upgrading the card faster.
My own pointless vanity vintage computing page
Ah, the "benchmark" business!
Imagine a Beowulf cluster of market droids...
I wonder if nVidia just enrolled in the FutureMark beta program...
Total BS, 3DMark2003 is already geared toward specific vendors. I've personally analyzed the data from the driver (since I'm writing one), and they totally favor ATI with the heavy use of PS 1.4 shaders. In fact, the data changes completely if PS 1.4 support isn't claimed. (3x more geometry is sent)
Also, PS 1.4 shaders don't always translate 'up' to PS 2.0 hardware very well, which is why (IMHO) Nvidia started all this hub-bub in the first place.
The only vendor that natively supports PS 1.4 is ATI.
They should have created PS 1.1 shaders for the masses, and then if 2.0 hardware is detected, had 2.0 shaders for everything.
And their "DX9" onyl test is a piece of crap too. They use one or two new instructions in the VS, and PS2.0 is only used for the sky. big whoop. No branching in the VS, two sided stencil, or anything cool.
It's sad the OEMs put alot of stock in 3Dmark, they don't seem to realize that gamers play games all day, not benchmarks.
Futuremark needs to work on random testing, say you would create a "test file" that would be full of random paths and the like. It would make make the test much less visually appealing to watch, but unpredicatable. Then you could run this random test sequencing file on each card. It wouldn't solve everything, but at least it would take away the option to "not render what you don't see."
Futuremark needs to stop apologizing, and start hard coding ways to keep their tests fair. It is going to be the only way to keep afloat after this situation.
From what i could glean, Nvidia changed their software in such away that visuals outside of walls, where you're not supposed to be anyway, were not rendered correctly. I don't see how this is a cheat, if it even marginally improves game rendering. Mind you, it may have made an even bigger impact on 3Dmark, or what have you, than the game. But, that only means that 3Dmark must adjust to the card, and that their software is not able to correctly give a good diagnostic. Thus, they've since talked to Nvidia, and will adjust for the driver changes.
What people aren't realizing is that the first games, 2D games with scrolling screens, did the same thing. The way they could make those games actually perform, is they only rendered a few blocks outside of what is actually shown on the screen. Rendering an entire map in Duke Nukem 1,2,3 (the 2-D version), would have been suicide on a 80386. My god people, it's Engineering, doing the bare minimum to get the job done faster, cheaper.
...and that's not a cheat.
Well, they are right, it's not a cheat. FutureMark, however, has just admitted that as a performance bench mark that might indicate suitability for other tasks their rating sucks.
if program-specific optimizations could be individually disabled by end users. If quality is being force-reduced for certain software then in many cases I would like to undo the reduction.
I already punished the cheaters by buying an ATI Radeon 9500!
ATI isn't going to take out their "optimizations" in the next Catalyst release?
In 2101, this joke will probably still get modded Funny. . . :)
!#@%*)anks for hanging up the phone, dear.
Driver optimization for games is one thing, but driver optimization for benchmarks is indeed cheating. Benchmarks are there to be a good overall unbaised mesurement of performance based on a standard approach. Now if Nvidia is allowed to tweak for a specific application as opposed to tweaking overall, it will bias the results.
When it comes to games, if they want to tweak for a specific game, im all for it. But if you want to tweak a benchmark thats rather unfair.
But then I think ATI does much of the same thing, they just havent been caught.
In America we are imprisoned by our fear of them.
Certain statements in this press release, including any statements relating to the Company's performance expectations for NVIDIA's family of products and expectations of continued revenue growth, are forward-looking statements that are subject to risks and uncertainties that could cause results to be materially different than expectations. Such risks and uncertainties include, but are not limited to, manufacturing and other delays relating to new products, difficulties in the fabrication process and dependence of the Company on third-party manufacturers, general industry trends including cyclical trends in the PC and semiconductor industries, the impact of competitive products and pricing alternatives, market acceptance of the Company's new products, cmdrtaco's and michael's rampant homosexual love affair, and the Company's dependence on third-party developers and publishers. Investors are advised to read the Company's Annual Report on Form 10-K and quarterly reports on Form 10-Q filed with the Securities and Exchange Commission, particularly those sections entitled Certain Business Risks, for a fuller discussion of these and other risks and uncertainties.
You have got to be the only troll to get modded to +5 practically every time you post. Quite a feat, actually.
Do not read this sig.
And I suppose people who cheat at online MMRGPs are just using undocumented program calls and extended functionality. It's all so clear now...
What would be great...
If someone was to reverse engineer the drivers, remove the "Optimisation", recompile and compare results. See what percent the "Optimisations" fudged the results.
Stop the Slashdot effect! Don't read the articles!
NVidia did things that were clearly NOT legitimate, and FutureMark caught them at it. There's a PDF report on FutureMark's Web site (assuming it hasn't met with an "accident" by now) detailing the dirty deeds. Chief among them, IMHO, was a trick where the driver was supposed to draw and update positions of stars in a night sky (involving clearing the background) as one moved along a 3D path; if one stays on the exact preprogrammed track of the demo, it looks OK. BUT... if you turn around (possible in the beta mode of the benchmark) you see that the driver SKIPPED clearing the background; the stars smear like mad. There is NO POSSIBLE WAY their driver was behaving legitimately. (Especially since changing the benchmark's fingerprint oh-so-slightly caused all these quirks to vanish; they were detecting the demo and screwing with things if it was being run...) The rest is just fear-of-pissing-off-the-800-pound-gorilla. A FutureMark developer admitted as much in a newsgroup posting. Sigh...
"My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
I hit puberty a long time ago, and no, you cannot pop my "cherry" as I am a guy, and not even a virgin. I would still like an answer to my original question.
All video card drivers have algorithms to keep from rendering unneccessary portions of the map. What NVidia did, I believe, was to bypass those algorithms and hard-code into the driver the portions that needed to be rendered. That wouldn't work in a real game, where the card must decide what to render at run-time based on user input. Therefor, it's cheating, no different from including an MPEG of the entire 3Dmark demo and showing it in lieu of actually rendering it.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
What a load of crap. This is one of those things that when you think about it too much a bunch of false lines of logic get drawn and you come up with a nonsensical answer. Either that or 3DMark is trying to avoid a lawsuit from nVidia, which no doubt has been threatened.
The point of a benchmark is to test dissimilar systems against common references to get an idea of how they perform against each other in such a way that you have an apples to apples comparison.
If 3DMark writes their program in a way that allows optimization paths for a specific GPU, then it is no longer a benchmark.
You now no longer have an idea of how fast the card REALLY runs as there is no guarantee that game writers will use GPU-specific optimizations. It's the same thing as MMX...Nobody sees the benefits if it's not hardcoded into the program, so what's the point if being uberfast in a benchmark if you won't necessarily see the same results in the real world?
-R
So, out of curiousity, why do you post stuff like this? Why not post to a site thats into psuedoporn? Finally, was the encounter with you GF omitted for the Geeks, or was it omitted because you are incapable of describing that which you do not know?
Yeah, except we are talking about the GeForce FX 5900 here, which if you look at benchmark scores for real games (tom's hardware) is slightly ahead of the Radeon 9800.
...is that nVidia did modify their drivers, and that this is an "optimization" and not a "cheat." They go on to state that optimizations of any kind (including nVidia's "optimizations") are not allowed.
So tell me again how doing something forbidden to increase one's score isn't cheating?
A quote of a quote in their 10/26/98 press release:
Quack, quack.
you cannot pop my "cherry" as I am a guy
That's what you think
No, he was looking at you not your girlfriend.
It's sort of akin to walking around with a backpack full of cinderblocks. That way, when you put down those cinderblocks(ie benchmark), you'll notice how much stronger you got.
Perhaps they should use .NET for their next benchmark. Or Java. That'll be the true test of a video card :-)
we were obliged to update the product to eliminate the effect of optimizations identified in different drivers so that 3DMark03 continued to produce comparable results.
So if a company comes out with new drivers that are more efficient, producing higher performance in their benchmark, they'll just add some nops to even out the playing field? What kind of nonsense is that?
"I would still like an answer to my original question."
Why? Are you a psychoanalyst trying to read my mind through the monitor? Then you will hunt me down and fist me really deep in the poop chute while I'm sleeping?
I think the idea was to test new technologies that haven't been implemented yet in Quake 3 or Unreal Tournament 2003 (like in the upcoming DOOM III).
A quote of a quote in their 10/26/98 press release:
Quack, quack.
The whole point of a benchmark is to help determine real world performance
That's really funny, but I sure hope you're not serious. Never in the history of benchmarks have they ever had more than a passing correlation with the real world, and that will change the day that pigs will begin to fly.
The real issue here is that nVidia's optimizations were very very poor. There is absolutely no excuse for an optimization that doesn't render exactly the same as when it's not there. ATI also optimized for the benchmark, but they did a better job and didn't mess up the rendering, that's all there is to it.
But don't criticize nVidia for messing with their drivers just to improve the score on this benchmark. So did ATI, and rightly so, because on the assumption that a benchmark might occasionally reflect some real app, anything that you do to render the benchmark faster just might help real apps. Good for ATI, good for nVidia. But bad on anyone that does a crap job of it, and that's nVidia here. End of story.
I think the idea was to test new technologies that haven't been implemented yet in Quake 3 or Unreal Tournament 2003 (like in the upcoming DOOM III).
A quote of a quote in their 10/26/98 press release:
Quack, quack.
Futuremark is based in Canada. ATI is based in Canada. nVidia is based in the U.S. Obviously Futuremark is going to do everything they can to help ATI and make nVidia look bad.
Actually, the amazing part is you are probably one of the few that read the article and noticed.
From FutureMarks original report on the Nvidia subject. http://www.futuremark.com/companyinfo/3dmark03_aud it_report.pdf
our findings show that certain NVIDIA drivers seem to detect when 3DMark03 is running and
then replace the 3DMark03's rendering requests with manually implemented alternative rendering
operations.
If the above statement from FutureMark is true, then the report this thread is about is false and I call "Bullshit!".
If the above statement is false, then the story this thread is about MIGHT be true BUT FutureMark should state that there was a mistake and issue an apology to NVidia.
Which is it, FutureMark? No more bullshit! Straight up facts!
Thanks for the clarification! Most of these comments are just yelling 'cheater!' as if they were playing counterstrike!
Really? Everybody on slashdot...which, if everyone who ever belonged to slashdot read it regularly, is about 700,000 people. So I would say there is a large segment of the gamer population who hasn't looked into this thoroughly and believe Futuremark, or who have never even heard of the whole deal and believe 3DMark stats.
Futuremark did the wrong thing here. They should have licensed the test results so they are illegal to publicize if, in Futuremark's estimation, you cheated, which they could concey to you through a C&D. That way they avoid the libel suits, but they keep the integrity of their stuff.
Or they can hold out for a pile of cash, which is what they were obviously doing all along. nVidia was right when they called Futuremark on the extortion angle - nVidia dropped out of the 3dmark beta program, stopped paying Futuremark, so FM go after nVidia until nVidia pay. Sounds a lot like SCO and IBM, except that Futuremark actually had a basis for making a stink. Which makes it all the more sad that they settled. Thanks a lot guys.
-Looking for a job as a materials chemist or multivariat
use this technology with Unreal Tournement, Quake 3 etc?
--
A few months ago, in a dark warehouse in Brooklyn...
*knock* *knock* *knock*
nVidia: Who is it?
*Futuremark goons enter stage left*
Futuremark: You's late wit you's beta program "membership dues". You know what happens wit da peoples dat don't pay they's dues, right?
nVidia: Piss off! We're not paying this year.
Futuremark: Dat's a pretty benchmark score you got there. Be a shame is somethin' BAD happened to it...
*Futuremark pushes nVidia's benchmark trophy over, shattering it*
Futuremark: oops. Looks like ATI is faster dis year. Dat's too bad for you's guys. You sure you don't wants to join da beta?
nVidia: You bastards! Get out!
Futuremark: Think about it. Da boss wants you on board this year, or bad things happen.
*exeunt goons stage left*
nVidia: No more "beta membership dues". We're going to beat those Futuremark goons at their own crooked game!
0 1 - just my two bits
Futuremark Statement:
Paragraph 1: We're making a statement.
Paragraph 2: nVidia didn't really cheat.
Paragraph 3: Most computer games cheat.
Paragraph 4: We don't allow companies to cheat in their code.
Paragraph 5: Therefore, we should cheat in ours.
Nvidia Statement: They should have worked with us for a better cheat.
Joint Statement:We should all cheat together.
Footer 1: Futuremark rocks.
Footer 2: Don't steal our IP.
Footer 3: Nvidia rocks.
Footer 4: Really it does.
Footer 5: Of course, we could be lying.
Footer 6: Don't steal our IP.
The ______ Agenda
I believe the correct interpretation of what FutureMark is saying is that the game writers are building their games differently for the different boards that are out there. That's what they mean by "manufacturer-specific code path optimization is directly in the code source." The source code they are referring to are UT and Q3, as examples.
They are saying that the boards have become different enough that game writers are coding differently for them. Not too surprising, really. That's the way it's always been.
This makes writing a synthetic benchmark extraordinarily difficult, needless to say. I don't know if it's even possible in this case. Perhaps rather than try to come up with one number that specifies how fast a board is, you can come up with a series of metrics for each capability.
While I'm sure that FutureMark has had some pressure applied to them to make this statement, it's not an unreasonable statement on its face. It's just the path they took to get there that is questionable.
I love Mondays. On a Monday, anything is possible.
That seems a bit more appropriate to the story, doesn't it.
There is nothing so silly as other peoples traditions, and nothing so sacred as our own.
course...It is not what you know, it is what the instructor thinks you need to know. If you know what you are going to be tested on and study for just that are you cheating ?? Hardly...
What this says is that the criteria for the test are not real world enough, if optimization for testing ADVERSELSY effects real world performance, I'd have to say the testing criteria are INCORRECT.
errr....umm...*whooosh* *whoosh* Is this thing on ?
That's not in question though... what's in question is NVidia's changing of the shader precision from FP32 to INT12. It's entirely reasonable that this is a tradeoff that would be made to improve speed at a slight cost to render quality -- since the speed improvement can be substantial.
Not clearing the back buffer and other "on rails" cheats are still classified as cheats. It's merely the shader changes that are being considered as possibly OK. Which, btw, is similar to what ATI did. But I don't know that ATI's changes sacrificed visual quality at all. (Not that they haven't done that in the past...)
sweet that just means I can ignore all futuremark benchmarks when I consider a graphics card. Sounds pretty clear to me.
[darn, not strike out tag]
I just want to know when we'll see 3D-DawnMark2k3?
Full screen Anti- {strike}A**ing{/strike} err...Aliasing.
{strike}Pixie Sha{/strike}..ummm.. Pixel Shading.
Volume {strike}Mammery{/strike} Metric Lighting...uhhh...sorry, I faded out for a second.
What were we talking about?
Have you read the moderator guidelines? Well, have you, PUNK? (and I want a Karma: Gnarly option)
Just imagine the mess of the driver source code.
"If 'application' == '3dmark.exe' then enable optimisation."
That's why they will never open source their drivers!
If they could the list of applications which are detected and will use driver specific optimisations and give the possibility to enable them or not (like a 'enable application specific' and a list of applications.)
Then, we could say that it is not cheating but optimisation.
Anyway ATI could reproduce the same 'tweaks' used by nVidia for their benchmark (some fixed plane trick clipping if I remember).
Also it would means that 3dmark'03 could be more optimised. (let's face it, it's slow, could be much faster, using 20% gain with an optimisation, that's mean Futuremark has a problem).
If the benchmark was written properly, then no optimisation would be possible. In this case, Futuremark misses a point.
Same issue has occured with ATI (with the vertex/pixel shader shuffled code), Futuremark could have done this optimisation themself.
NVIDIA, ATI: If you can't open source your drivers, at least, let the user know that you are using specific optimisations on certain applications during the installation, on in a readme file, or better, in the display properties panel, let the user to enable/disable them, instead of hiding the truth !
We know you did it for Quake 3, but this time, it's gone too far !
I need a Sino-Logic 16. Sogo-7 data-gloves, a GPL stealth module...
First off, you have to realize that ATi has been doing this FOR YEARS for its graphic cards... Futuremark and ATi has been in each other's bed for years, and its apparent when you run the games that the 'tests' are based on, and you can see (very easily) that the image quality on the newer ATi (9500 and up) cards is utter crap when compared with the nVidia cards in similar price ranges.
Not only that, but you have ATi's past history of absolute garbage drivers--and the many problems associated with their OpenGL support. Its just plain bogus if you ask me.
Also, nVidia is NOT in Futuremark's developer program, where you get the 'beta' code for the next version of Futuremark so you can 'preview' the technology based 'demos' in the 'benchmark' so that you can see if your drivers will 'work' with the next benchmark... and ATi is.
Huh... How is nVidia supposed to optimize for something that they don't have access to until it is RTM, while ATi has it since day one... and who's making all these accusations again? Hmm... I wonder...
The only thing that nVidia has ever done is optimize for general applications, based on the data being pumped through the card. I believe these were called extensions, and developers could use them if they wanted to... I mean, who would want to speed up their game and make it look better--at the cost of practically nothing? Let me think on this one here...
So in closing, I'd like to say that I love my Geforce4 TI4800 (overclocked to real TI4600 speeds) and that when I was given the chance to have a ATi 9800 in my system--for free--I turned it down because ATi is crap, and I'm all about image quality... And driver stability... And developer support... And OpenGL support... And cost...
p.s. If you think that I'm a Troll, think about this: Anyone who would take the time to look at the actual cards on identical systems, running a time-demo in UT2003--or any other game for that matter--on the same monitors with everything turned up to the max in the driver settings would agree with me. People--it's not about speed, it's about image quality! Everything above 60-80Hz your eyeballs can't see anyway.
Why is it that it's not okay for nVidia to optimise their card for certain applications, but it is okay for certain applications to optimize for nVidia?
Woudl anybody cry foul if Doom III uses an nVidia feature to render a scene in one pass instead of the two it would take an ATI Card (or vice versa)?
no.. if it was counter-strike it would be cheat0r or h4xs or h4x0r or similar
...ATI has my business now. I've been hankering after their screaming monster card anyway (although I don't know how useful it'd be on a 1.4 ghz gaming machine), and this deal with nvidia only confirms my suspicion that they can't be trusted to accurately report the performance of their product.
Nor, it seems, can anyone else.
max
My god carries a hammer. Your god died nailed to a tree. Any questions?
brings back memories of Glide vs D3D vs OPENGL :)
remember competition is good
Don't get me wrong, there is nothing wrong with application-specific optimizations.
But this misses the whole point of 3dmark 2003. Different developers stress the pixel and triangle pipelines in different ways to produce a whole boatload of effects. While major games and engines are often optimized-for, there is no guarantee that ATI or Nvidia will sit down and optimize for the game you just bought.
That said, 3dmark 2003 should be considered a relational tool for generic perfrormance. Consider it a good bet that if two cards perform similarly and acceptably, the two cards should be able to run almost any DX8/DX9 game off the shelf acceptably.
The fact that Nvidia's unopitmized drivers perform significantly behind ATI's unoptimized drivers in 3dmark 2003 raises a significant question:
We all know how well the 5900 does in Quake III, Serious Sam 2, UT2003, etc, but how does it do in ?
I want to know that if I take *insert random DX8 game here* home to play, IT WILL PERFORM WELL. That is the entire point of having a benchmark like 3dmark. To do application-specific optimizations for it is to nullify the entire point of the benchmark.
Man is the animal that laughs.
And occasionally whores for Karma.
Isn't it possible that these graphics card companies can be doing similar operations to other games and their popular benchmarking utilities? For example, I could make my drivers optimized for demo001.dm3 for Quake3, and then some of the reviews that don't use their own demos for testing will be VERY generous to me.
:]
This is probably unrealistic (if you ran the demo with a wider POV, you could probably detect errors easily, at least the way I've heard Nvidia did it with 3dmark), but I'm still going to remain paranoid
So basically Futuremark has said, in not so many words, that our benchmarking software doesn't really mean much in the real world.
Well gee...I could have told them that weeks ago.
"The only vendor that natively supports PS 1.4 is ATI."
Sorry, but that's garbage, pure and simple. Or are you not aware that PS 1.4 support is _required_ for DX9 cards with PS2.0 support. Your complaint may be valid when comparing a GF4 against a Radeon 8500, but is totally bogus when comparing two DX9 cards.
"And their "DX9" onyl test is a piece of crap too. They use one or two new instructions in the VS, and PS2.0 is only used for the sky."
Gee, one minute you're complaining that they use PS1.4 instructions, and now you're complaining that they don't use PS2.0 instructions. PS1.4 instructions _are_ effectively DX9 instructions since other than ATI, no other DX8 chips use them: you need a DX9 chip to run PS1.4 shaders.
And it would appear to be real lucky for nvidia that they don't use many PS2.0 instructions since from the results of their shader test once the nvidia "optimization" of throwing away the shader and running a completely different shader was fixed, shows them running PS2.0 shaders at about half the speed of a Radeon 9800. The low performance of PS2.0 shaders on the FX card seems to be the reason why nvidia hated 3DMark03 so much; there was no way to get a good score without redesigning the chip or "optimizing".
Remind people that Bill Gates was misquoted = instant karma.
Given this blatant dishonesty on the part of Nvidia, (and co-conspiritors, FutureMark) how many here would buy a video card based on an nvidia GPU ? I'll not be buying one any time soon.
complmenting an AC's troll prowess... on something as slight as that... smells fishy to me.
-malakai
-Malakai
A Dragon Lives in my Garage
Problem is, NVIDIA didn't just optimize. Their application specific "optimization" made the images look worse. And when you couldn't notice it, it was because they were clipping outside the camera angle (becuase they knew exactly where the camera was, something they can't do when you're playing UT2003)
Like the original statement by futurmark said, optimization is great. But when you change the image intended by the software designer in order to make it go faster, that's not an optimization. For god's sake, I can turn all the details to low on UT2003 and get it to go faster, but that's besides the point
Reason games specific benchmark don't fly for me (although now that Futuremark has issued this statement, I'm sure ATI will start cheating as well, making the whole thing useless) is that synthetic benchmarks can test features new to the cards that games may not yet have implemented. So I have an idea how it'll perform with future games.
Warning: Opinions known to be heavily biased.
That's right and the new media giants (see yesterdays' FTC Ruling) aren't going to broadcast propaganda, they'll broadcast virtually real news...
There isn't a recession, people are just on a well deserved sabatical.
No really... this stuff really will make your member 3 inches thicker and 8 inches longer.
That's the ticket!!!
I love every bone in her body, especially mine!
Just like my ex-wife considered her lover a "marriage enhancer".
Give me a break...
I have to be honest, the fact that Nvida cheated (and they did cheat) on 3Dmark was funny, but it didn't make me not want to buy an Nvidia card.
But now that they are throwing their weight around (which makes them no better than M$ of Intel - scumbags) I will never ever buy another nvidia card of product that contains anything from nvidia.
I don't care how good the nforce2 is!
buh bye!
What intrigues me most is the inclusion of the period mark. To the original poster, this statement is a complete thought.
Very curious indeed.
OK -
first off, for those of you wondering what the big deal with 3DMark 2003 is - and why you might use it in place of "real games" to benchmark 3D performance - here you go:
3DMark is a test application to benchmark next-generation performance, so that you can get an idea how your video card might handle games that will be out this time _next_ year. Specifically some aspects of 3DMark are geared toward testing DX9 functionality, and it's Pixel and Vertex shaders. No game currently on the market uses these features (at least not that I am aware of).
Secondly, the difference between a cheat and optimization is a fine one. If a given function continually produces the same output for the same inputs, and it takes 1 second to do so, and another function can produce the same results given the same inputs, but only takes 1/2 a second - it can be said to be functionally equivalent. However, it has been optimized. It's entirely possible, even desirable to replace pixel shaders and vertex shaders with routines which are optimized for your hardware. In much the same way that compilers schedule instructions optimally for the underlying CPU architecture, so too can instructions be re-ordered in a pixel shader routine... It's an optimization.
Cheating occurs when people start making approximations (analogies to bringing a cheat-sheet to a test are not valid), or by failing to process (in the case of video cards) the same visual fidelity, and detail that was intended. By example:
A> Reducing texture bit-depth.
B> Reducing geometry detail (merging 2 or more polygons).
This is only cheating if it's not the intent of the original application developer (not driver developer).
A driver developer could make the following optimizations, since they don't affect the intent of the application developer:
A> Pre-calc tables. A classic demo optimization would be to precalc a SIN function table to some level of precision as looking up a value was faster than calculating it on the fly.
B> Replacing various pixel/vertex shader routines with functionally equivalent, but faster ones.
C> Reordering data and textures (keeping detail and fidelity) into more optimal chunks for your hardware architecture.
Those aren't cheats - they are optimizations. Of course, the only way you can tell this is if you have an objective standard to gauge against. 3DMark 2003 doesn't seem to provide this. In order to do so they would need the following:
A> A software renderer for their demo.
B> Timed snapshots of the demo saving uncompressed images from the software renderer to disk.
C> The ability to re-run the demo using a hardware renderer (3D Card and drivers).
D> The ability to take the same snapshots and save them, uncompressed to disk.
E> The ability to do a histogram, per-pixel comparison to the software renders...
This would enable you to arrive at some objective comparison of visual fidelity - instead of the occassionally subjective I think screenshot X looks better than screenshot Y. Without the intent of the 3DMark developers being known, we really can't know how true the hardware vendors and their drivers are to the original vision.
Anything less than 3% difference is highly likely to be indistinguishable from the intent of the developers in this case. 5% to 10% may be visible, but acceptable (i.e. tweaks for speed in place of quality). Over 10% and you're playing with fire.
1) Someone gave someone else some $$$ to shut up and even better yet, spin doctor in their direction.
2) I wish they made more application-wise optimizations. But, i wish they didn't make them on BENCHMARKS. I need Warcraft to go faster, not 3DMark03 (Because frankly, getting 4 FPS and lower really is stupid, this prooves 3DMark's obsolescence, it doesn't mean anything in the real world).
I just went to the store today to get myself a replacement for the 8500 i own and sold to one of my friends. Guess what company got my bucks this time. Yep, ATI isn't pure and virtuous, but at least they apologized. Unite! Speak with your wallets!
Trolls dont like to be Flamebait, because they burn so well. Protect our Troll heritage!
It's fine with me. I have to say, I completely understand NVidia's point of view. Read the press release. It's true what they're saying - almost every commercial 3D game out there has code specific to ATI or NVidia boards, so the game will run as fast as it can. Because FutureMark wasn't taking this into account in their own software, NVidia did it at the driver level.
Is this "cheating"? Well, yes. Why? Because it wasn't done openly, and ATI isn't doing it [as much].
Is it unreasonable? No, not really. FutureMark should be doing two sets of benchmarks - one "raw" benchmark to test raw processing power, and one "optimized" benchmark to test real-world performance. Why do I say real-world performance? Because the performance you'll experience in games will have board-specific optimizations. So to judge performance for those situations, you have to use benchmarks with board-specific optimizations.
But hey, who cares what I think, right?
and what they did might just be completely different.
I think you're pulling this out of your ass. Do you even know anyting about the tests or the optimizations?
PowerVR is cheating!! Tile-based rendering is cheating, cuz it doesn't render whole scene!!!
If you bother reading some of the more intelligent comments, you probably noticed that that's exactly what John Carmack told us last week.
I just think the whole thing is stupid. If nVidia were paying there FutureMark taxes to be part of the beta program, we would have never heard a word about this. FM just wants to make nVidia look bad because they're not helping an IMHO useless program make money.
How many of you play with monitors and at a resolution where you can get much more 85 refresh? Or even 109? Or 120? Your 200 fps card is going to waste.
This same shit's been going on in the compiler world since Whetstone was the hot benchmark. Everybody optimizes code paths for benchmarks, and the only real test is how something works for your application.
Nothing to flame here, move along...
Just junk food for thought...
I think it's fairly safe to say the only people Nvidia cheated were themselves. If you create a card that only looks good in benchmarks and then performs poorly in real world games your customers will take notice. Especially since there are now 4.3*10^26 hardware review sites in every language. All of those sites will run side by side game evals and show the truth. On top of that most people buy these days by word of mouth. If HardOCP says FX is the shit then everyone buys, if they like ATI then ATI sells (Obligitory sheep sound here).
For the record I have one personal expierence with this from ATI as well. My roomate at the time had a Gateway with a GF2 that went bad, Then sent him an early ATI Raedon to replace it. The card was flawless at 100fps in Q3, but if you loaded up Half-Life it ran like crap, never getting over 10fps. Ati coded the original Raedons to look great in Q3 because that was the hot benchmarking game out there at the time. My buddy eventually beat Gateway about the head and neck until they sent him another Nvidia card and then he got his 60+fps in both Q3 and HL.
Just another case of the pot calling the kettle black in my eyes.
Apple free since 1990!
They are now claiming that these are application optimizations. So that means it should make the application (Benchmark) in this case work faster. But wasn't it initially the fact that the Nvidia GPU wasn't rendering the part of benchmark incorrectly (forgot which site found the glitch)that led to finding that Nvidia was 'cheating'? If this optimization doesn't even make the benchmark run correctly, how is it good?
First let me state that I'm thoroughly disgusted with how nVidia and Futuremark have handled this situation. In a vague kind of way, it sort of relates to the government line of "if you don't support this war, you are a traitor". No one is allowed to voice a bad opinion anymore.
Secondly, and this is a new train of thought for me, if nVidia had made the benchmark run faster without sacrificing image quality, I think it should be allowed to detect the benchmark was running and have a code path optimized in the driver for it. This could be used to show exactly how fast the hardware is capable of running optimized to the hilt by the driver developers. It could actually have a benefit of showing game developers how they should code their software, that sequencing instructions in this particular order or using certain architecture specific instructions are THAT much faster. Sort of like how 3DNow enhanced Quake showed off how much faster the K6 could be. Unfortunately this was not what they did, they were caught red-handed, and now they're just throwing their weight around. SHAME ON nVidia.
The benchmarks you link to show the NVidia cards beating the ATI cards on price/performance, and the top cards differ by .8 fps out of 190, or about 0.4%.
The cards you identify as the same are performing close to each other, but are also priced closely (see pricewatch.com), so what's the big deal?
Is that troll I smell?
well not only do they bribe, they spend money on other intersting things too... like hiring a pornstar for their party!
http://www.theinquirer.net/?article=9677
my blog
I know the trendy thing on slashdot now is to bash NVIDIA, but at least RTFA he links to. They benchmark Counterstrike of all games. That has so little to do with new games being released and how pixel sharder performance is etc that its laughable.
So I say that both nVidia and ATI should be ashamed of themselves for such unethical practices.
Comment removed based on user account deletion
Way back when, around 1993 if I recall correctly, Number Nine (remember them?) was guilty of hardcoding the text strings that the Ziff Davis benchmarks used in their WinBench test. Now THAT was cheating.
If you recall a previous story here on /. where algorithmic ineficiencies where used in a DoS attack, the solution to the problem was to use keyed hash functions, so that the attacker, could not find hash collisions. I think the solution to this problem might be similar. For example, if the path that the camera traveling through the benchmark was unknown to the author of the driver, then they could not make "optimisations" based on knowing what does'nt have to be rendered. So if everyone running the benchmark had their own secret camera path, thease type of cheats could not be pulled off. The problem obviously is that one run of the benchmark by one org will not yield exactly the same results as one run by another, but it seems like a smaller problem.
FRA: STFU GTFO
Exactly, as Futuremark put it.
3DMark03 is designed as an un-optimized DirectX test and it provides performance comparisons accordingly.
ATI/Nvidia optimized shaders, not cheating.
Nvidia optimized for rendering in a benchmark, cheating.
You can optimize, as long as you dont know beforehand what is going to be used in the Benchmark.
I want to know the difference between brand-x's efficiency when running game-z compared to brand-y's efficiency. I don't care how the cards rate on Futuremark's standard if they don't allow optimized code. If game code in the real world is optimized, then I want to see tests that measure the cards against each other using optimized code. For all I know, brand-x and brand-y might run exactly the same on Futuremark's tests, but when playing game-z, brand-x could give me three-fold the framerate over brand-y.
This "joint-statement" from both companies seems to be a bit lame as far as I am concerned. I bought an ATI Radeon 7200 last year, and I think I'm going to by a newer ATI Radeon this year... The statement hasn't renewed my confidence in neither Nvidia nor Futuremark. I will not consider Futuremark's benchmark as a valid suite and I certainly won't trust Nvidia to produce quality hardware unless the convince me otherwise.
Looks like Cnet updated their article with the new futuremark.
Big difference in real world tests...
In somewhat (nowhere near) related information, nVidia's drivers greater than 41.72 will not run the DoomIII alpha. It appears they hard coded the card to detect the runtime and display a nice gray screen.
One case where application specifics can really nip you in the bud, unless we weren't talking about leaked software, so I guess....
Oh well.
It's called business.
Can anyone find the link to the original statement FutureMark released when it happened? It was a PDF.
Anyway, it's real easy to make it fair. Like they suggested, put a random element in the test.
"But then they're not consistent".
Run them a few times and take the average. Duh.
"Last one in is a rotten goblin!" - Kepp
They have continued to rule into the 2000's.
Check out their Amiga demo Lapsuus, released under the name "Maturefurk". Awesome display of coding skills. People were accusing them of using code from 3d-mark and Max Payne (which is insane, the Amiga don't run directx :)), that's how good it looked on a 50 mhz 68060.
Released at the Assembly in 2001, it beat all other demos in the combined compo, including demos running on gigahertz plus pc's with 3d accelerated graphic cards.
Ironically, it runs quite well under WinUAE on an XP2400.
How small a thought it takes to fill a whole life
This is like the emperor's new clothes: in my opinion this is probably what happened:
- NVIDIA for some reason decides to pull out of the FutureMark tax program (aka, 'development program' or whatever the heck they call it)
- FM threatens NVIDIA that either they come back onboard or FM will 'leak' a devel version of FM that can be used to portray NVIDIA in a bad light (and probably -only- NVIDIA, the 8% ATI loss was incidental)
- NVIDIA doesn't budge
- FM leaks it and some 'review sites' go all gung-ho over it
- NVIDIA threatens to sue FM out of existance (presumably they think they have a solid case) and ask for damages due to reputation loss etc. etc. etc.
- FM caves and does some creative spin doctoring about the word 'cheating'.
-every- video card manufacturer cheats on benchmarks, come on, if you were the CEO of ATI/NVIDIA wouldn't you do everything in your power to make your hardware look better than your competitor's?
It should be the benchmark sites that should earn their keep and do 'real' benchmarking of the cards instead of spending 5 minutes setting up a test environment with the usual demos and going back to sleep or something.
When a new card comes out, a reputable test site, instead of using demo001 which can be special-cased in drivers for example, should connect online to a crowded q3/ut2k3 server, play there for 15-20 minutes while recording a demo and using -THAT- demo to benchmark the new card =and= re-benchmark the competing card(s). The site should also obviously take screenshots and compare them to make sure that the drivers aren't cheating by lowering visual quality etc. etc. etc.
Obv. one can't go back and rebenchmark all the way to the TNT, but -at least- the current generation cards should be done, for comparisons to older models then, yeah, the older benchmarks should be used (but still not the canned demo001, but the 'online benchmarks' recorded when the older cards came out).
Just my 2c
-- the cake is a lie
Well...of course, that's not even in question. The new card sucked, so they didn't feel the need to pay Futuremark to tell them their card sucked. Sounds reasonable all the way around. We're not questioning whether nVidia are a bunch of whores - they most certainly are. We're questioning whether Futuremark are, and that's a significantly scarier prospect.
nVidia is the one doing the extortion, not Futuremark. They simply threatened Futuremark with a lawsuit.
I'm sure they did, but I'll turn things around a bit - don't you find it suspicious that Futuremark made a much bigger deal about nVidia's "cheating" than ATI's? Seems to me that it was a bit of "punishment" for nVidia not paying them. And I have a decent idea that Futuremark happened upon a pile of cash in the meantime. Looks a bit like extortion to me. Yes, nVidia certainly put the screws on Futuremark - again, I thought that was so damned obvious not to need mentioning, but there you go anyway. So they extorted each other.
Now, to set the record straight, I'm not suggesting that nVidia didn't cheat - they most certainly did. Unquestioned. However, I find it mighty suspicious that Futuremark was willing to do a 180 so quickly on the matter, and I have lost all respect for them in the process. Even if they feared the lawyers, they could have retracted the statement with a bit less enthusiasm. That's why I think they were just waiting for a payout from Day 1.
I guess the ultimate point is this - hardware makers are dogs. They will do anything to make their stuff look faster. That's why we need benchmarks - to keep the cheating from happening. Unfortunately, now we have a situation where the maker of the benchmark, the "referee" as it were, is willing to look the other way for a bit of cash.
So we always knew that nVidia were a bunch of whores - it's sad to discover that Futuremark are as well.
-Looking for a job as a materials chemist or multivariat
" However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks "
Hmmmmm.... Is it just me or did nVidia just buy this new statement from 3dMark/Mad Onion (whomever!!). It's funny how last week they're blowing the word "cheat" all over the place last week. And now this week they're using the word "optimization" AND they're talking about how moved they are that they should include nVidia specific code in future versions!!
Oh and I bet they forgot to mention the nice little $10 million investment nVidia made to their company in show of support. That would be too convenient, now wouldn't it?!?!
On second thought, I am HKP. ;)
I find it funny that people automatically assume an industry giant is at fault. Who cares if they cheated on the benchmark? What really matters is if this benchmark is worth anything as it is. But you'll all say "This is the only DirectX 9.0 benchmark we've got." Bull. I find it funny that Carmack can make a GeForceFX5800 Ultra beat a Radeon 9700 Pro in Doom3, yet everyone thinks of it as garbage. If you don't believe me, finger him and read his plan file, or find it posted on a thousand different websites. Everyone agrees that the FX5900 is much improved over the 5800, yet people mostly find the 9800 being just a speedbump over the 9700. You can do the rest of the logic here.
we all know how geeks love tweaking things to the metal, just look at gentoo's current popularity.
well, having used gentoo for the last year or so, I have to point out that there's really not all that much to tweek.
you can set your compiler options to match your processor, I guess, and you can tweek some video configs in the Xserver, but it's no different than any other distribution.
even redhat gives you different kernels for SMP or other processors.
this whole "tweek to the bare metal" idea of gentoo is distracting to the real nice features of the distribution.
honestly, your machine won't be much faster just because you compiled the kernel with some sort of aggressive gcc options.
If you really think it is, go gets some real benchmarks and time it before you say "well, it just FEELS so much faster."
thank you.
No I'm not.
Actually, we game developers care. We want to make use of new features of graphics cards to increase performance and/or visual quality. But also, we want our games to run on all (relatively-recent) cards, without having to write complex hacks to work around the bugs of each one.
It's no use benchmarking on the latest and greatest games -- because, as developers, we try and avoid releasing games that run horribly (slowly or with obvious bugs) on certain cards. Sometimes we can persuade the manufacturers to fix their bugs, but the timescales can be tight and soemtimes they're quite happy to not fix the bugs, especially if they know their competition runs the codepath you're looking at faster than them, and we get forced to drop the feature. So instead of games pushing up hardware quality, games are held back by shoddy hardware or (more usually) drivers. You're just benchmarking on what the manufacturers already know works. Zzzz. Futuremark's job is to stress-test in advance what's coming up.
And even if Futuremark does things that aren't always what you'd do in games, they are trying to push the cards to the limits to see if they do what the manufacturers claim, or whether they only achieve their claimed performance "in controlled tests".
So some of us talk to the Futuremark guys and say things like, "We're looking into using [technology X] in our next game, but the drivers on cards [A and B] are screwed, works on [C] though. Could you put a section into 3DMark 2004 that uses [X]?". Then, when their card performs miserably at [X] (even though the card's hardware can handle it -- it's just that they've been slack on the drivers) they get shamed into improving their quality at those features. A bit like WHQL for games.
Once the driver bugs for the features are fixed, we can write code that uses them.
Except NVidia decided to stop playing nice when it turned out the latest tests make their cards look quite poor, and noticeably slower than ATI's. So they took their ball and went home (dropped out of Futuremark's beta programme). This is why they didn't know their cheating would be discovered.
And of course, this problem is compounded by people like yourself, Mr Sonic, who see big numbers in Quake benchmarks (you do realise 3D card mfgs "optimise" those too, right?), pop wood, and rush off to buy the latest hovercraft no matter if it's not really "all that".
Incidentally, ATI's optimisations were exactly that: optimisations. Essentially, they were reordering instructions in a shader -- exactly like a compiler optimising instruction order for Intel or AMD processors' particular quirks. The meaning and, more importantly, on-screen output of the code was not altered.
Whereas it's clear to see from the screenshots in the original expose article, that NVidia were not optimising, but actually not running code, causing the onscreen output to look wrong. As developers, we don't want gamers returning our games to the shops "because it goes wrong when you do X". Nor do we want to sweat blood trying to invent ways to avoid their driver bugs.
Oh... someone else who cares about 3DMarks? OEMs. When it comes around to picking what cards to put inside big-name off-the-shelf PCs or, eventually, which chips to surface-mount on the all-in-one motherboard, they're looking for price-performance, and 3DMark is a part of that equation.
I don't buy a graphics card so that I can run benchmarks. I buy a graphics card so that I can run actual applications. Perhaps you enjoy watching 3DMark loop all day long but I like to play games and the games that I play define the process by which I evaluate hardware.
(blunt)To be as blunt as you were, if you buy graphics hardware based on benchmarks it is you that is completely stupid(/blunt)
I just bought an FX 5600 128MB and it SUCKS in UT2003. Slow FPS, and just doesn;t look all that clear. Big waste of money. My buddy's old GF4 Ti 4200 blows it away in FPS and looks in UT2K3. I'm going back to my old 64MB Radeon 9500 which gave faster FPS, and just looks so much clearer. I wish I'd never bought the FX, the only thing it does good is the Dawn pixie demo.
The very fact that nvidia could even find something as simple as clipping planes that needed adjustment and/or z-clears that were unnecessary depending on the scene being viewed (and YES you can perform such optimizations even when an application goes "off the rails") means that 3D Mark IS NOT truly indicative of the best practices of development of graphics-intensive applications. Period.
If 3D Mark were properly developed the _only_ optimizations that nvidia could be left to perform would be related to hardware-specific differences like floating point precision, not common, simple things like clip planes and z-clears.
Futuremark should simply create a test that varies at runtime to defeat specific instruction path optimization. I wouldn't care if it looked horrible as long as the margin of error on the score wasn't more than let's say 2%. This way, the kids who play 3DMark could still have their eye-candy, while the people who use it for real diagnosis can have a standard test.
This is why Emperical testing on 'real world' problems is still valuable when testing competing algorithms, programs, or hypothesises in general. When any standard benchmark is used invariably future developments end up being geared at improving the results against it and not 'real world' performance.
its a crazy idea, but why dont they optimize their drivers to run games and the like? or is that just not realistic?
And Believe it or not I'm going for the piece of shit 5600FX..
Two main reasons.
1) I want to (read : NEED to) start running linux on the desktop and ATI still haven't got the hang of decent linux drivers.
2) Its cheap enough that even though I know it will struggle in half the time that a Radion card will, I can easily justify buying another card when this ones useful life is ended.
Nvidia are TRYING to do the right thing.. they're worth supporting for that reason, its important though they know that we don't want to see them trying to screw us over.
"Consider how lucky you are that life has been good to you so far. Alternatively, if life hasn't been good to you so far
I find the whole thing rather interesting.
First how do you fairly test and compair different graphics cards?
Write your game directly to the standard, IE direct x, open gl or what ever and ignore the exsistance of the cards. Well this isn't really realistic since it doesn't reflect what actually happens. No game writer does this. They all optimize thier software to run well on the hardware they know it's going to encounter. The amount of optimization varries as does the graphics card makers assistance but there is always some level of optimization. This is especially true given todays graphics cards land scape with esentially 2 major manufacturers accounting for the vast majority of 3d cards out there. Doom 3 has 7? different render paths it can use if memory serves correctly to allow for the most performance and best image quality possible on as much different hardware as possible.
Not to mention that the resulting code that just writes to the specs may or may not uses the best method possible for doing something and may just happen to run better on a less capable hardware just because it implements 1 specific thing better in the thier driver/hardware.
Future mark seems to guilty of this type of thing to some extent or another. I've seen comments about rendering things in strange orders forcing graphics cards to do alot more work than nessicary. Heavy use of the dx 8 pixel shaders 1.4's in a so called dx 9 test. PS1.4 being ati's contribution to dx8's pixel shader capabilities.
Ok so if we assume that unoptmized tests aren't really valid and don't reflect real performance how do we allow for optimizations? Do we allow graphics manufacturers to write thier entire own program to produce specified results like some cross platform processor benchmarks try to do? What are we really benchmarking there the graphics card it's self or how well writen the program is? This is also filled with high potential for potential cheating.
Allow graphic card manufacturers to help optimize the code or render path overseen by the benchmark maker(s)? Harder to cheat when a 3rd party has access to and is working with the code also but still open to the quality of the optimizations.
Future mark in particular as a benchmark has always struck me as rather pointless. It's a benchmark trying to guess at how the video cards will perform when they are obsolet essentially. Back when the Geforce 3 came out and the 2001 was all the rage it was the dx 8 test. Well guess what the geforce 3 is getting pretty long in the tooth and dx8 still hasn't really caught on heavily. What are the odds that this benchmark will be anything like the games we're seeing when dx9 becomes popular and what video cards will we actually be running them on? Not to mention what are the odds of dx 10 being out by then?
What about the optimizations that ati/nvidia used? Well the ideas are perfectly valid. It's common place for video cards to ignore or discard information or instructions that don't need to be processed inorder perform as well as possible. Modern cards even have parts specifically dedicaded to this type of thing. Z buffer testing to test if something is going to be blocked before it's even rendered. Clipping the edges of the screen. Anti aliasing only on the edges of objects. Dynamic LoD, AA and Anisotropic filtering. The idea of why do work that isn't needed is common place in graphics cards these days.
What nvidia did is still a cheat because it is based on knowing what the benchmark was going to do ahead of time but not rendering something thats not going to be seen. Not clearing buffers that aren't on screen is perfectly valid as long as it is done inteligently by the driver and not specifically hard coded for a known case.
On that note even actual game benchmarks can be cheated on in the exact same way since they often time uses standardized prerecorded scripts that fall to the same issue. Hardcoded tricks based on knowing the exact route something will follow ahead of time. This is
I would love to see ATI deliver a new driver highly optimized for 3DMark. The driver is to be so optimized you can hardly tell which 3DMark benchmarkinging algorithm is running; producing excellent frame rates at the expense of horrid image quality. Then 3DMark will turn into a driver game for graphics card vendors. In the end no one wins, ATI and Nvidia and playing games, gamers don't have a good benchmark, and Futuremark's future dwindles.
maybe you arent aware of this, but quake 3 is old as hell, IMHO it is pointless to use q3 anymore, you need something to give you SOME idea how a card will compare to another card against, for instance, dx9 cards, otherwise, in 3 months, you might find out you got screwed and have a shitty card.
All major games should be represented, and results should be presented in weighted and unweighted fashion. You're talking about a sample-size of at least 100 here.
social sciences can never use experience to verify their statemen
While not defending anyone in particular.
You guys are reading too much into the use of the word "application" here.
I highly doubt it was used to mean "program" but rather to indicate a set of functions used in a certain way.
If anything this adds justification to Nvidia's old complaint that 3DMark doesn't do things the way an actual game does.
Looks to me they have been trying hard to optimise things a certain way, and along came a special case that broke things (Which is bad, they should allow programmers the option of a non optimised path if they so desire).
I highly doubt that nVidia intended to "cheat" in the 3DMark tests in the way they have been accused. However under pressure from marketing they may have sat on the knowledge once it was discovered. I also find it amusing the implication that not one of the nVidia developers have a copy (personal or otherwise) of the developer version of 3DMark, and yet a reviewer did.
Uh, it wasn't a cheat, I got it. Sorry.
Anyway, Recently Japanese PC News Site "PC Watch" reported that the trident XP4m32LP video chip marks almost 5,000 in 3DMark, but, after renaming benchmark exe, it downs to 2,700. Also they reported the image quality is extremely low while not renamed benchmark is running, compared to renamed benchmark produced. This chip is used in Dynabook SS S7 (Japanese cousin of Toshiba Portege R100), and when it introduced in Japanese market, they touted its superiority in 3D graphics rendering as good as "GeForce Ti4200" and "good for playing FFXI on the road". A lot of Game-OTAKU believed that and bought the machine. Still Toshiba refused to admit they did cheat.
If you want to see what the reporter wrote in "PC Watch", please follow this link to the Babelfish translated article.
To keep these variables to the minimum, hardware and drivers, you have to benchmark. How do people compare Q3 performance then? They run benchmarks.
Synthetic benchmarks have many uses too. A game might not be an accurate performance indicator if it still uses DirectX 8. Or even if it uses DX9 it might not use some features. A synthetic benchmark is a good way to test general performance.
I understand where you're coming from about them (minus you're ignorance about the word benchmark), but if a video card performs well on a synthetic benchmark like 3dMark, then what are the odds that it will run most other games better than the other card?
When you say you want nVidia to spend more time optimising games, well even that is a little silly. Performance isn't free. If they produce drivers that are all around better and correct something that was wrong in the past, then they should do it. But what they did in 3dMark 03 was cheat: in almost all cases, the image quality of the "optimised" frames was worse.
Anyone who's writing a commercial game wants to know what the card is capable of. The benchmark is saying that the card is capable of high performance in certain areas. That's good enough for me. Quit whining and RTFA.
...what nVidia promised Futuremark inorder to get them to validate nVidia's approach.
;))
A serverfarm? (god knows nVidia has alot of them
A boatload of GeforceFX?
Not to sue them?
All the above plus lots of money?
Anyhow, this just proves that anything and everyone has a price. This makes 3dMark a beautiful demo to impress your friends with, but a benchmark tool? Nah...
But am I HKP?
Please read these :
Tom's Review of GF FX5900
Anand's Review of the same card
As you can see from the games figures, the FX5900 is faster than ATI's...
And, in the end, is the performance with applications that matters, not some stupid benchmark.
Doesn't this strike you as somewhat similar to court-TV shows when a lawyer makes an obviously incriminating but possibly unproven remark... gets an immediate response from the opposing lawyer, and withdraws the remark.
That is... once the initial remark is made... the jury (or the users/buyers/slashdotters) know that vVidia cheated. Anyone searching the net will probably find a whole whack of references to this (esp on slashdot). Perhaps it hurts Futuremark's credibility a bit to offer the recant, but by now everyone knows it's just to appease nVidia.
It's like saying "The jury will disregard the previous comment." Once you've heard it, it's pretty hard to disregard. Even an nVidia fan like myself finds it pretty hard to justify getting one of these cards now based on crappy benchmarks. I'll just wait until Doom3 or another new game comes out and then see which is proven to perform better, which is usually a much better than a benchmark anyhow.
Microsoft Products are optimized for 100% stability in idle time operation.
If they can design their hardware to be that "adaptable" to applications by merely providing a software (driver) "enhancement", then more power to them.
Suppose you bought a car becasue it had good solid features/capabilites that are based on today's most popular roads and driving habits. If next year, the most popular roads are paved with a different substance and the speed limits have changed, I'd be FRIGGIN ECSTATIC if my car's performance could be significantly improved (under these new conditions), merely by applying a software update (No "driver" jokes please).
Seem to be the same sort of thing.
My only gripe is the fact that NVidia (and ATI) are making their harware "perform" for a benchmark instead of the most popular games and playing habits. Ant this is nothing short of marketing/sales BS!!!
I'm surprised nobody has brought up the value of open source drivers in this situation. If ATI, NV and all the other GPU board vendors were to provide their drivers in source form, the community could judge the situation based on the actual code instead of making guesses.
Selling hardware without providing source for the interface drivers verges on unethical in the same way that selling (for example) trusses for a building roof without engineering certification and detailed, approved installation documentation is not only unethical but actually illegal in most jurisdictions. In both cases the systems/structural engineer is prevented from knowing essential facts about the capabilities of an important component of the system/structure they are building. The truss manufacturer has to not only provide these things, it has to provide complete engineering data, materials specifications and test data prior to acceptance for use in construction. Then the engineer is required to use only certified trusses, and only according to the manufacturer's approved instructions.
Of course in _most_ situations, the actual impact on safety and health of a video card is unlikely to be large. Nevertheless, proper systems engineering requires complete knowledge of the software as well as hardware structure of any subsystem. I.e., what if I'm building a new version of Prof. Steve Mann's "Mediated Reality" system? Loss of quality could have dire consequences in this case, including loss of essential vision at a critical moment. More seriously, I've seen at least one warning on video-related software regarding the potential for seizures in some people with previously-undiagnosed epileptic tendencies. It is within the realm of possibility that a nonobvious timing interaction between the software drivers and the hardware could cause such a thing in some rare or unlikely circumstance. Where is the engineering data to prove otherwise? It is impossible to determine without full access to the source code.
Certainly in this case, availability to the community of the source code would have allowed complete, early analysis by anyone concerned - peer review - and helped keep the entire process honest and at a higher level of discourse than the "sue me/sue you blues".
As time goes on and computer systems become more and more important to our safety and health, open source must certainly become one of the essential and commonly expected tools to allow systems engineers to properly determine the applicability and safety of subsystems for their applications.
It's easier to be a result of the past, but more fun to be a cause of the future! http://www.spacefinancegroup.com/
> It's not cheating if you intimidate your accuser into recanting the accusation.
The logical conclusion would be to boycott any company who engages in such behaviour.
Unfortunately, even scanning at -1, I could find only 1 such suggestion. Is this Slashdot???
You're missing the point. The benchmark does try to measure real-world performance, and the problem here is that the driver is optimized in such a way that only the benchmark runs faster. So it's not the benchmark that's useless, it's the driver that's cheating.