Slashdot Mirror
More on Futuremark and nVidia
AzrealAO writes "Futuremark and nVidia have released statements regarding the controversy over nVidia driver optimzations and the FutureMark 2003 Benchmark. "Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat."" So nVidia's drivers are optimized specifically to run 3DMark2003... and that's not a cheat.
and i didn't use a cheat sheet, i used a memory priming sheet.
I never said I was smart, I just said I was smarter than you
Though, I do prefer they make application specific optimizations that mean better gameplay.
It's just another piece of information to keep in mind when selecting a new card.
So nVidia's drivers are optimized specifically to run 3DMark2003... and that's not a cheat
Errrr... that seems like a cheat to me!
Davak
But then, I'm Bill Gates.
Oh, it seems some careless individual has left this big pile of money on the table! Well, we'll just leave for a few moments and maybe when we come back it will have gone away.
Time to update Websters. Cheat just got new semantics.
I think [H]ardOCP stated it best as "Futuremark didn't want to get sued by Nvidia". Nvidia has the legal and financial resources to totally ruin Futuremark and they know it.
And now Futuremark has totally invalidated their own benchmark software by declaring it "open season" for hardware manufacturers to distort the "tests" in any way shape or form they desire to make the numbers higher.
N.
"Nothing strengthens authority so much as silence." - Charles de Gaulle
Looks like someone is scared of somebody elses lawyers. Yuck! This is obviously Futuremark trying to appease Nvidia.
It's not cheating if you don't get caught.
Oh, I did get caught?
No, I didn't. Let's move on, shall we?
Best Windows Freeware
This is politics at its worst, and I'm calling bullshit.
There was no need for this nicey-nice statement other than NVidia threatening lawsuits and Futuremark wanting to protect what assets they have.
Futuremark had every right to call NVidia on their selfish claim and unbelievable hacks. To say that they weren't liable for their own blunder is to say that Futuremark's reputation has been replaced by corporatespeak and a lack of respect almost unparalelled.
What's worse is that I really thought "Yeah, this time the bad guy gets his due" and that NVidia should've known better.
But of course, a few weeks later we've got to put on the nice face again for the public en large.
What a complete waste of time. I know there isn't much respect left in corporate America, but hell, if you can't call a spade a spade, why even bother with the benchmarks when someone can just rewrite an ENTIRE SHADER and only keep a picture clear while the demo is on rails?
This sort of outcome is inevitable as drivers move "up the stack" into the application layer. To get better and better optimizations, the drivers need to know more and more about the application that is requesting the services - thus, we end up violating the strict separation between application and driver.
Well, that's excellent... now they can put 'Designed to run 3Dmark2003' on Nvidia product boxes!
I am a filthy pirate.
Cripes already. No one even BOTHERS with #DMark anymore, and after this fiasco no one is ever going to bother with them again. Gamers will use REAL EVERYDAY GAMES to see what runs the fastest again. Looking at some goofy simulation app coming up with scores and people buying into the company and people tricking drivers for particular tests is just crappy and makes 3DMark 100% invalid to any of my concerns in the future. I will only trust reviews that benchmark the latest and greatest games that I will be buying these cards for, whoever can run them fastest at that particular time IS WHAT IM GOING TO BUY. Peroid. Enough of this 3DMark BS.
... FutureMark and NVidia stated that they were proud to announce that former President Bill Clinton had joined their boards and had assumed management responsibilities.
Chr0m0Dr0m!C
Run Quake3 with the video card and check out the frame rate and image quality. Run it under UT also and every other 3D game you can. Then compare the framerates and image quality. Who cares what it's 3DMark is. Did you a buy the card to specifically run it under 3DMark? Most people buy these cards for playing games so comparing how it runs the actual game to other cards is the only meaningful measurement.
Outdoor digital photography, mostly in New Engl
All your benchmarks are belong to us!
"So on one hand, honey is an amazingly sophisticated and efficient food source. On the other hand it's bee backwash."
So the question in my mind is did Futuremark learn something from the discussions? Is there something it was ignoring in its tests?
I'm trying to not be a cynic and assume a big fat envelope was passed under the table. That what Nvidia did was legitimate.
Those that are following this, should check the pictures on the previous article. The quality of the nvidia "optimized" version sucked (showed big artifacts). That's no optimization, unless there was no image quality loss.
"This card is optimized for quake as long as you follow the left trail, the right trail will just look like crap but nobody follows it anyway".
If a benchmark doesn't measure performance related to real-world applications, what's the point? If a driver is optimized to run a benchmark faster, that SHOULD mean that the real world apps should run faster, too. If not, the benchmark is useless.
While nVidia has made great cards for some time now (I still use and love my GeForce 3), could it be possible that they're not able to keep up? A lot of the reviews that I've been reading tend to favor the Radeon cards over any anything that nVidia has put out lately. While I doubht that nVidia will become another 3Dfx because they're involved in other markets and I've read about them having US government contracts for this project or that, I would propose that they will not be the huge players that they once were in the vid card market.
If you want to, you can prerender the whole fucking test, stick it in your driver and just play it off instead of actually rendering when Futuremark is running, that would be an "application-specific optimization" too.
The benchmark is ment to reflect performance in the actual game, the reason it takes the same path is merely to make the results comparable. What ATI was something the game *could* have achieved in game, if the operations were properly sequenced. What Nvidia did is to fake a performance it can't actually give if a person had followed the exact same path in the game. That is cheating.
It is pathetic by Nvidia, and it's pathetic by Futuremark to present this press statement. Get some backbone and integrity.
Kjella
Live today, because you never know what tomorrow brings
Now that all the latest games have benchmarking modes, what do we need FutureMark for?
If NVidia wants to do application-specific optimizations that make UT2003 go faster, then that would be great. That's what they should be doing. Those are optimizations that genuinely benefit the user.
Is this truly the only Earth I can live on?
It's not an optimization if it does not produce the same results! Recall that the shader code that the driver used did not produce the same visual results as the shader code it replaced.
... based purely on application-specific pattern matching, which by its very nature is fragile. As was demonstrated so aptly by the 'off the rails' mode in Futuremark. This isn't an accidental bug: it is obvious that such mechanisms are highly fragile, and are almost certain to cause bad rendering on these applications when they are modified in small ways.
More tellingly, the driver deliberately flaunts the D3D spec by omitting buffer clears, mucking about with clip planes, etc.
As others have said, Futuremark's statement is just covering their legal arse. If someone modifies their code to get better scores in some benchmarks while introducing deliberate bugs (i.e. incorrect rendering), it's a cheat in my book.
I think the money quote is:
However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.
I can sort of see the argument here, but it basically ruins the point of having a standard interface like DirectX. It's also like telling your math teacher, "no, it would be easier for my equations if you made 1+1=3. Now do it because I'm your star student."
Yes, there is an increase in speed, but there was also a degradation in quality in the test. See here.
That's the whole issue...
"I turn away with fright and horror from the lamentable evil of functions which do not have derivatives."
"Cheating? Our developers, The Franklin Family, would disagree. I think a meeting can be arranged with them, if you wish. Isn't that right Mr. Franklin?"
*shakes hundred dollar bill side to side, speaks in high tone out of side of mouth*
"Sure is, Boss!"
Futuremark: "NVidia is cheating! Not as much as ATI, but they're cheating!"
Nvidia: Knock knock
Futuremark: "Who's there?"
Nvidia: "Goons...hired goons."
Futuremark: "Oh...haha...um...Nvidia is actually in the business of application optimisation! Our mistake. Won't happen again."
Seriously folks, this is Nvidia using big bad lawyers to scare Futuremark into capitulating. They might have held their ground, until ATI was proven to be doing the same thing, albeit to a much lesser degree.
Unfortunately, the only person who loses in this scenario is the consumer.
John Maynard Keynes: "When the facts change, I change my mind. What do you do?"
This has been discussed before. Other companies have tried to modify their drivers to produce better results for certain benchmarks. They've always been thrown out as invalid before. I wonder why Futuremark seems to be considering allowing NVidia's enhancement to stand.
There's a line from the story:
"...However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks."
I'm concerned because I feel that allowing video card manufacturers to put code specifically about certain benchmarks in to their product (making their product look better in that benchmark) may not be reflective of real world performance.
However, the benchmark is useless if it doesn't measure real world performance, so I do believe that NVidia could put stuff in their product to make the benchmark run faster that would be beneficial to real applications, so I'm torn.
Some game manufacturers make optimized versions of their code to work with certain video cards, but the normal use is an operating system driver (DirectX...) and I believe using the generic driver is more representative of what you'll get when you use the video card.
It seems that NVidia is arguing that they should be allowed to put optimizations in to their code specifically for the benchmark because they do the same thing with some other populate applications.
have you even read what nvidia's drivers did to "optimize" 3dMark03? hardcoded clipping planes (not applicable in games!), replacing shaders with inferior output, lowering rendering precision to FX12/FP16 (DX9 calls for FP24 minimum!) resulting in visible image quality degradation.
And you say these are not cheats as long as nvidia uses them in games? are you completely lobotomized?
Well, after seeing the future mark scores, it looked to me as nvidia fx chips are blowing away ati and its gf4 line.
But I found a really nice german benchmark site 3dcenter.org that had to be the best benchmarks ive ever seen, they actually use the games on each and lists the fps.
Looks like the FX/GF4 5200/4200 4600/5600 (non ultra) are the same. And the ATI 9700PRO/9800 are faster than the 5800 Ultra.
After reading these benchmarks, you can really tell nvidia tweaked the SHIT out its drivers for futuremark...
(blunt)The problem with a lot of the reasoning I see here with people saying they want the card that plays the game they're interested in quickly, is that it's completely stupid. (/blunt)
When you're looking for a video card, you -should- rely on a capable, and untainted/optimized benchmark for comparison simply because you can't predict what the software companies that make the actual games are going to do. Will they support -your- chosen card, or will some other GPU maker offer a better bribe to the developer? You may know that kind of info about games shipping RSN, or already on the shelves, but what about next year's?
Getting the card based simply on one or two games instead of looking at some kind of objective benchmark does no good whatsoever. It's just a way to rope yourself into upgrading the card faster.
My own pointless vanity vintage computing page
Yes, but, if they'd put the optimization in to only make the Doom 3 *demo* faster, because they knew it was used as a benchmark, you wouldn't support it any more.
There are a lot of "optimizations" you can use if you know in advance what you'll need to draw. You don't even need a 3d engine, you could just pre-render everything and put an mpeg into the driver. I bet that'd be very fast.
It wouldn't make the actual game run any faster, though.
Never underestimate the bandwidth of a 747 filled with CD-ROMs.
Total BS, 3DMark2003 is already geared toward specific vendors. I've personally analyzed the data from the driver (since I'm writing one), and they totally favor ATI with the heavy use of PS 1.4 shaders. In fact, the data changes completely if PS 1.4 support isn't claimed. (3x more geometry is sent)
Also, PS 1.4 shaders don't always translate 'up' to PS 2.0 hardware very well, which is why (IMHO) Nvidia started all this hub-bub in the first place.
The only vendor that natively supports PS 1.4 is ATI.
They should have created PS 1.1 shaders for the masses, and then if 2.0 hardware is detected, had 2.0 shaders for everything.
And their "DX9" onyl test is a piece of crap too. They use one or two new instructions in the VS, and PS2.0 is only used for the sky. big whoop. No branching in the VS, two sided stencil, or anything cool.
It's sad the OEMs put alot of stock in 3Dmark, they don't seem to realize that gamers play games all day, not benchmarks.
I think you don't understand what was done. Nvidia cheated on the benchmark because they new exactly what needed to be rendered. This kind of "optimization" won't help any other application, but it doesn't mean the benchmark is bogus. The same thing could be done by putting optimizations in for time_demo (or something similar) in quake. It will help that specific demo, but it won't help quake.
Jack Valenti and Orrin Hatch will be first up against the wall when the revolution comes.
Driver optimization for games is one thing, but driver optimization for benchmarks is indeed cheating. Benchmarks are there to be a good overall unbaised mesurement of performance based on a standard approach. Now if Nvidia is allowed to tweak for a specific application as opposed to tweaking overall, it will bias the results.
When it comes to games, if they want to tweak for a specific game, im all for it. But if you want to tweak a benchmark thats rather unfair.
But then I think ATI does much of the same thing, they just havent been caught.
In America we are imprisoned by our fear of them.
Certain statements in this press release, including any statements relating to the Company's performance expectations for NVIDIA's family of products and expectations of continued revenue growth, are forward-looking statements that are subject to risks and uncertainties that could cause results to be materially different than expectations. Such risks and uncertainties include, but are not limited to, manufacturing and other delays relating to new products, difficulties in the fabrication process and dependence of the Company on third-party manufacturers, general industry trends including cyclical trends in the PC and semiconductor industries, the impact of competitive products and pricing alternatives, market acceptance of the Company's new products, cmdrtaco's and michael's rampant homosexual love affair, and the Company's dependence on third-party developers and publishers. Investors are advised to read the Company's Annual Report on Form 10-K and quarterly reports on Form 10-Q filed with the Securities and Exchange Commission, particularly those sections entitled Certain Business Risks, for a fuller discussion of these and other risks and uncertainties.
You have got to be the only troll to get modded to +5 practically every time you post. Quite a feat, actually.
Do not read this sig.
And I suppose people who cheat at online MMRGPs are just using undocumented program calls and extended functionality. It's all so clear now...
What would be great...
If someone was to reverse engineer the drivers, remove the "Optimisation", recompile and compare results. See what percent the "Optimisations" fudged the results.
Stop the Slashdot effect! Don't read the articles!
NVidia did things that were clearly NOT legitimate, and FutureMark caught them at it. There's a PDF report on FutureMark's Web site (assuming it hasn't met with an "accident" by now) detailing the dirty deeds. Chief among them, IMHO, was a trick where the driver was supposed to draw and update positions of stars in a night sky (involving clearing the background) as one moved along a 3D path; if one stays on the exact preprogrammed track of the demo, it looks OK. BUT... if you turn around (possible in the beta mode of the benchmark) you see that the driver SKIPPED clearing the background; the stars smear like mad. There is NO POSSIBLE WAY their driver was behaving legitimately. (Especially since changing the benchmark's fingerprint oh-so-slightly caused all these quirks to vanish; they were detecting the demo and screwing with things if it was being run...) The rest is just fear-of-pissing-off-the-800-pound-gorilla. A FutureMark developer admitted as much in a newsgroup posting. Sigh...
"My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
That would be worse than useless. No two test runs could be usefully compared, so the utility of the benchmark itself is called into question. You'd be reduced to aggregation of test results in the hope that they statistically settle out towards some sort of 'true' average for a given card/driver combo.
The inherent value of a benchmark is the notional "apples to apples" comparison, and you've taken even that away. There would now be no reason at all to use 3DMark.
"So on one hand, honey is an amazingly sophisticated and efficient food source. On the other hand it's bee backwash."
All video card drivers have algorithms to keep from rendering unneccessary portions of the map. What NVidia did, I believe, was to bypass those algorithms and hard-code into the driver the portions that needed to be rendered. That wouldn't work in a real game, where the card must decide what to render at run-time based on user input. Therefor, it's cheating, no different from including an MPEG of the entire 3Dmark demo and showing it in lieu of actually rendering it.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
What a load of crap. This is one of those things that when you think about it too much a bunch of false lines of logic get drawn and you come up with a nonsensical answer. Either that or 3DMark is trying to avoid a lawsuit from nVidia, which no doubt has been threatened.
The point of a benchmark is to test dissimilar systems against common references to get an idea of how they perform against each other in such a way that you have an apples to apples comparison.
If 3DMark writes their program in a way that allows optimization paths for a specific GPU, then it is no longer a benchmark.
You now no longer have an idea of how fast the card REALLY runs as there is no guarantee that game writers will use GPU-specific optimizations. It's the same thing as MMX...Nobody sees the benefits if it's not hardcoded into the program, so what's the point if being uberfast in a benchmark if you won't necessarily see the same results in the real world?
-R
It's sort of akin to walking around with a backpack full of cinderblocks. That way, when you put down those cinderblocks(ie benchmark), you'll notice how much stronger you got.
Perhaps they should use .NET for their next benchmark. Or Java. That'll be the true test of a video card :-)
I think the idea was to test new technologies that haven't been implemented yet in Quake 3 or Unreal Tournament 2003 (like in the upcoming DOOM III).
A quote of a quote in their 10/26/98 press release:
Quack, quack.
Actually, the amazing part is you are probably one of the few that read the article and noticed.
Thanks for the clarification! Most of these comments are just yelling 'cheater!' as if they were playing counterstrike!
A few months ago, in a dark warehouse in Brooklyn...
*knock* *knock* *knock*
nVidia: Who is it?
*Futuremark goons enter stage left*
Futuremark: You's late wit you's beta program "membership dues". You know what happens wit da peoples dat don't pay they's dues, right?
nVidia: Piss off! We're not paying this year.
Futuremark: Dat's a pretty benchmark score you got there. Be a shame is somethin' BAD happened to it...
*Futuremark pushes nVidia's benchmark trophy over, shattering it*
Futuremark: oops. Looks like ATI is faster dis year. Dat's too bad for you's guys. You sure you don't wants to join da beta?
nVidia: You bastards! Get out!
Futuremark: Think about it. Da boss wants you on board this year, or bad things happen.
*exeunt goons stage left*
nVidia: No more "beta membership dues". We're going to beat those Futuremark goons at their own crooked game!
0 1 - just my two bits
Futuremark Statement:
Paragraph 1: We're making a statement.
Paragraph 2: nVidia didn't really cheat.
Paragraph 3: Most computer games cheat.
Paragraph 4: We don't allow companies to cheat in their code.
Paragraph 5: Therefore, we should cheat in ours.
Nvidia Statement: They should have worked with us for a better cheat.
Joint Statement:We should all cheat together.
Footer 1: Futuremark rocks.
Footer 2: Don't steal our IP.
Footer 3: Nvidia rocks.
Footer 4: Really it does.
Footer 5: Of course, we could be lying.
Footer 6: Don't steal our IP.
The ______ Agenda
I believe the correct interpretation of what FutureMark is saying is that the game writers are building their games differently for the different boards that are out there. That's what they mean by "manufacturer-specific code path optimization is directly in the code source." The source code they are referring to are UT and Q3, as examples.
They are saying that the boards have become different enough that game writers are coding differently for them. Not too surprising, really. That's the way it's always been.
This makes writing a synthetic benchmark extraordinarily difficult, needless to say. I don't know if it's even possible in this case. Perhaps rather than try to come up with one number that specifies how fast a board is, you can come up with a series of metrics for each capability.
While I'm sure that FutureMark has had some pressure applied to them to make this statement, it's not an unreasonable statement on its face. It's just the path they took to get there that is questionable.
I love Mondays. On a Monday, anything is possible.
That seems a bit more appropriate to the story, doesn't it.
There is nothing so silly as other peoples traditions, and nothing so sacred as our own.
That's not in question though... what's in question is NVidia's changing of the shader precision from FP32 to INT12. It's entirely reasonable that this is a tradeoff that would be made to improve speed at a slight cost to render quality -- since the speed improvement can be substantial.
Not clearing the back buffer and other "on rails" cheats are still classified as cheats. It's merely the shader changes that are being considered as possibly OK. Which, btw, is similar to what ATI did. But I don't know that ATI's changes sacrificed visual quality at all. (Not that they haven't done that in the past...)
Don't get me wrong, there is nothing wrong with application-specific optimizations.
But this misses the whole point of 3dmark 2003. Different developers stress the pixel and triangle pipelines in different ways to produce a whole boatload of effects. While major games and engines are often optimized-for, there is no guarantee that ATI or Nvidia will sit down and optimize for the game you just bought.
That said, 3dmark 2003 should be considered a relational tool for generic perfrormance. Consider it a good bet that if two cards perform similarly and acceptably, the two cards should be able to run almost any DX8/DX9 game off the shelf acceptably.
The fact that Nvidia's unopitmized drivers perform significantly behind ATI's unoptimized drivers in 3dmark 2003 raises a significant question:
We all know how well the 5900 does in Quake III, Serious Sam 2, UT2003, etc, but how does it do in ?
I want to know that if I take *insert random DX8 game here* home to play, IT WILL PERFORM WELL. That is the entire point of having a benchmark like 3dmark. To do application-specific optimizations for it is to nullify the entire point of the benchmark.
Man is the animal that laughs.
And occasionally whores for Karma.
"The only vendor that natively supports PS 1.4 is ATI."
Sorry, but that's garbage, pure and simple. Or are you not aware that PS 1.4 support is _required_ for DX9 cards with PS2.0 support. Your complaint may be valid when comparing a GF4 against a Radeon 8500, but is totally bogus when comparing two DX9 cards.
"And their "DX9" onyl test is a piece of crap too. They use one or two new instructions in the VS, and PS2.0 is only used for the sky."
Gee, one minute you're complaining that they use PS1.4 instructions, and now you're complaining that they don't use PS2.0 instructions. PS1.4 instructions _are_ effectively DX9 instructions since other than ATI, no other DX8 chips use them: you need a DX9 chip to run PS1.4 shaders.
And it would appear to be real lucky for nvidia that they don't use many PS2.0 instructions since from the results of their shader test once the nvidia "optimization" of throwing away the shader and running a completely different shader was fixed, shows them running PS2.0 shaders at about half the speed of a Radeon 9800. The low performance of PS2.0 shaders on the FX card seems to be the reason why nvidia hated 3DMark03 so much; there was no way to get a good score without redesigning the chip or "optimizing".
Problem is, NVIDIA didn't just optimize. Their application specific "optimization" made the images look worse. And when you couldn't notice it, it was because they were clipping outside the camera angle (becuase they knew exactly where the camera was, something they can't do when you're playing UT2003)
Like the original statement by futurmark said, optimization is great. But when you change the image intended by the software designer in order to make it go faster, that's not an optimization. For god's sake, I can turn all the details to low on UT2003 and get it to go faster, but that's besides the point
Reason games specific benchmark don't fly for me (although now that Futuremark has issued this statement, I'm sure ATI will start cheating as well, making the whole thing useless) is that synthetic benchmarks can test features new to the cards that games may not yet have implemented. So I have an idea how it'll perform with future games.
Warning: Opinions known to be heavily biased.
OK -
first off, for those of you wondering what the big deal with 3DMark 2003 is - and why you might use it in place of "real games" to benchmark 3D performance - here you go:
3DMark is a test application to benchmark next-generation performance, so that you can get an idea how your video card might handle games that will be out this time _next_ year. Specifically some aspects of 3DMark are geared toward testing DX9 functionality, and it's Pixel and Vertex shaders. No game currently on the market uses these features (at least not that I am aware of).
Secondly, the difference between a cheat and optimization is a fine one. If a given function continually produces the same output for the same inputs, and it takes 1 second to do so, and another function can produce the same results given the same inputs, but only takes 1/2 a second - it can be said to be functionally equivalent. However, it has been optimized. It's entirely possible, even desirable to replace pixel shaders and vertex shaders with routines which are optimized for your hardware. In much the same way that compilers schedule instructions optimally for the underlying CPU architecture, so too can instructions be re-ordered in a pixel shader routine... It's an optimization.
Cheating occurs when people start making approximations (analogies to bringing a cheat-sheet to a test are not valid), or by failing to process (in the case of video cards) the same visual fidelity, and detail that was intended. By example:
A> Reducing texture bit-depth.
B> Reducing geometry detail (merging 2 or more polygons).
This is only cheating if it's not the intent of the original application developer (not driver developer).
A driver developer could make the following optimizations, since they don't affect the intent of the application developer:
A> Pre-calc tables. A classic demo optimization would be to precalc a SIN function table to some level of precision as looking up a value was faster than calculating it on the fly.
B> Replacing various pixel/vertex shader routines with functionally equivalent, but faster ones.
C> Reordering data and textures (keeping detail and fidelity) into more optimal chunks for your hardware architecture.
Those aren't cheats - they are optimizations. Of course, the only way you can tell this is if you have an objective standard to gauge against. 3DMark 2003 doesn't seem to provide this. In order to do so they would need the following:
A> A software renderer for their demo.
B> Timed snapshots of the demo saving uncompressed images from the software renderer to disk.
C> The ability to re-run the demo using a hardware renderer (3D Card and drivers).
D> The ability to take the same snapshots and save them, uncompressed to disk.
E> The ability to do a histogram, per-pixel comparison to the software renders...
This would enable you to arrive at some objective comparison of visual fidelity - instead of the occassionally subjective I think screenshot X looks better than screenshot Y. Without the intent of the 3DMark developers being known, we really can't know how true the hardware vendors and their drivers are to the original vision.
Anything less than 3% difference is highly likely to be indistinguishable from the intent of the developers in this case. 5% to 10% may be visible, but acceptable (i.e. tweaks for speed in place of quality). Over 10% and you're playing with fire.
While I don't agree with everything you say, I must make one point: if someone offered me a Radeon 9800 Pro to replace my GeForce4MX, I wouldn't -- couldn't. I run Linux. The ATI Linux drivers blow compared to the ATI Windows drivers. For more important, to me, than any cheating NVIDIA might or might not have done, is that NVIDIA has a history of releasing quality products, at decent prices, with good driver support. Further, they were the first company to release OpenGL ICDs for consumer-level cards that were conformant enough to perform well with pro-level apps. They were, and still are, also the first consumer graphics card company whose Linux drivers matched their Windows drivers in performance. ATI's driver situation has gotten better, but until ATI can equal NVIDIA's driver prowess, I'm one loyal NVIDIA customer.
A deep unwavering belief is a sure sign you're missing something...
I think it's fairly safe to say the only people Nvidia cheated were themselves. If you create a card that only looks good in benchmarks and then performs poorly in real world games your customers will take notice. Especially since there are now 4.3*10^26 hardware review sites in every language. All of those sites will run side by side game evals and show the truth. On top of that most people buy these days by word of mouth. If HardOCP says FX is the shit then everyone buys, if they like ATI then ATI sells (Obligitory sheep sound here).
For the record I have one personal expierence with this from ATI as well. My roomate at the time had a Gateway with a GF2 that went bad, Then sent him an early ATI Raedon to replace it. The card was flawless at 100fps in Q3, but if you loaded up Half-Life it ran like crap, never getting over 10fps. Ati coded the original Raedons to look great in Q3 because that was the hot benchmarking game out there at the time. My buddy eventually beat Gateway about the head and neck until they sent him another Nvidia card and then he got his 60+fps in both Q3 and HL.
Just another case of the pot calling the kettle black in my eyes.
Apple free since 1990!
First let me state that I'm thoroughly disgusted with how nVidia and Futuremark have handled this situation. In a vague kind of way, it sort of relates to the government line of "if you don't support this war, you are a traitor". No one is allowed to voice a bad opinion anymore.
Secondly, and this is a new train of thought for me, if nVidia had made the benchmark run faster without sacrificing image quality, I think it should be allowed to detect the benchmark was running and have a code path optimized in the driver for it. This could be used to show exactly how fast the hardware is capable of running optimized to the hilt by the driver developers. It could actually have a benefit of showing game developers how they should code their software, that sequencing instructions in this particular order or using certain architecture specific instructions are THAT much faster. Sort of like how 3DNow enhanced Quake showed off how much faster the K6 could be. Unfortunately this was not what they did, they were caught red-handed, and now they're just throwing their weight around. SHAME ON nVidia.
So I say that both nVidia and ATI should be ashamed of themselves for such unethical practices.
Exactly, as Futuremark put it.
3DMark03 is designed as an un-optimized DirectX test and it provides performance comparisons accordingly.
ATI/Nvidia optimized shaders, not cheating.
Nvidia optimized for rendering in a benchmark, cheating.
You can optimize, as long as you dont know beforehand what is going to be used in the Benchmark.
Actually, we game developers care. We want to make use of new features of graphics cards to increase performance and/or visual quality. But also, we want our games to run on all (relatively-recent) cards, without having to write complex hacks to work around the bugs of each one.
It's no use benchmarking on the latest and greatest games -- because, as developers, we try and avoid releasing games that run horribly (slowly or with obvious bugs) on certain cards. Sometimes we can persuade the manufacturers to fix their bugs, but the timescales can be tight and soemtimes they're quite happy to not fix the bugs, especially if they know their competition runs the codepath you're looking at faster than them, and we get forced to drop the feature. So instead of games pushing up hardware quality, games are held back by shoddy hardware or (more usually) drivers. You're just benchmarking on what the manufacturers already know works. Zzzz. Futuremark's job is to stress-test in advance what's coming up.
And even if Futuremark does things that aren't always what you'd do in games, they are trying to push the cards to the limits to see if they do what the manufacturers claim, or whether they only achieve their claimed performance "in controlled tests".
So some of us talk to the Futuremark guys and say things like, "We're looking into using [technology X] in our next game, but the drivers on cards [A and B] are screwed, works on [C] though. Could you put a section into 3DMark 2004 that uses [X]?". Then, when their card performs miserably at [X] (even though the card's hardware can handle it -- it's just that they've been slack on the drivers) they get shamed into improving their quality at those features. A bit like WHQL for games.
Once the driver bugs for the features are fixed, we can write code that uses them.
Except NVidia decided to stop playing nice when it turned out the latest tests make their cards look quite poor, and noticeably slower than ATI's. So they took their ball and went home (dropped out of Futuremark's beta programme). This is why they didn't know their cheating would be discovered.
And of course, this problem is compounded by people like yourself, Mr Sonic, who see big numbers in Quake benchmarks (you do realise 3D card mfgs "optimise" those too, right?), pop wood, and rush off to buy the latest hovercraft no matter if it's not really "all that".
Incidentally, ATI's optimisations were exactly that: optimisations. Essentially, they were reordering instructions in a shader -- exactly like a compiler optimising instruction order for Intel or AMD processors' particular quirks. The meaning and, more importantly, on-screen output of the code was not altered.
Whereas it's clear to see from the screenshots in the original expose article, that NVidia were not optimising, but actually not running code, causing the onscreen output to look wrong. As developers, we don't want gamers returning our games to the shops "because it goes wrong when you do X". Nor do we want to sweat blood trying to invent ways to avoid their driver bugs.
Oh... someone else who cares about 3DMarks? OEMs. When it comes around to picking what cards to put inside big-name off-the-shelf PCs or, eventually, which chips to surface-mount on the all-in-one motherboard, they're looking for price-performance, and 3DMark is a part of that equation.
Define "convince." Anand's review shows that the FX5900 is indeed a worthy competitor for the latest ATi offering, and the superior Linux drivers put nVidia over the top for Linux users.
If you're concerned about nVidia's ethics, perhaps you should check out ATi's background.
Guess you should read the entire article.
Future Mark build 320 vs. 330. (330 doesnt have the nvidia cheat...)
3,215 - ATI Radeon 9800 Pro 256MB w/ 3DMark03 Build 330
2,821 - Nvidia GeForce FX 5900 Ultra w/ 3DMark03 Build 330
Nvidia using the cheat had - 3,458.
The ATI 9800 Pro is faster.