Slashdot Mirror
More on Futuremark and nVidia
AzrealAO writes "Futuremark and nVidia have released statements regarding the controversy over nVidia driver optimzations and the FutureMark 2003 Benchmark. "Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat."" So nVidia's drivers are optimized specifically to run 3DMark2003... and that's not a cheat.
and i didn't use a cheat sheet, i used a memory priming sheet.
I never said I was smart, I just said I was smarter than you
Though, I do prefer they make application specific optimizations that mean better gameplay.
It's just another piece of information to keep in mind when selecting a new card.
So nVidia's drivers are optimized specifically to run 3DMark2003... and that's not a cheat
Errrr... that seems like a cheat to me!
Davak
But then, I'm Bill Gates.
Oh, it seems some careless individual has left this big pile of money on the table! Well, we'll just leave for a few moments and maybe when we come back it will have gone away.
Time to update Websters. Cheat just got new semantics.
I think [H]ardOCP stated it best as "Futuremark didn't want to get sued by Nvidia". Nvidia has the legal and financial resources to totally ruin Futuremark and they know it.
And now Futuremark has totally invalidated their own benchmark software by declaring it "open season" for hardware manufacturers to distort the "tests" in any way shape or form they desire to make the numbers higher.
N.
"Nothing strengthens authority so much as silence." - Charles de Gaulle
Looks like someone is scared of somebody elses lawyers. Yuck! This is obviously Futuremark trying to appease Nvidia.
It's not cheating if you don't get caught.
Oh, I did get caught?
No, I didn't. Let's move on, shall we?
Best Windows Freeware
This is politics at its worst, and I'm calling bullshit.
There was no need for this nicey-nice statement other than NVidia threatening lawsuits and Futuremark wanting to protect what assets they have.
Futuremark had every right to call NVidia on their selfish claim and unbelievable hacks. To say that they weren't liable for their own blunder is to say that Futuremark's reputation has been replaced by corporatespeak and a lack of respect almost unparalelled.
What's worse is that I really thought "Yeah, this time the bad guy gets his due" and that NVidia should've known better.
But of course, a few weeks later we've got to put on the nice face again for the public en large.
What a complete waste of time. I know there isn't much respect left in corporate America, but hell, if you can't call a spade a spade, why even bother with the benchmarks when someone can just rewrite an ENTIRE SHADER and only keep a picture clear while the demo is on rails?
Well, that's excellent... now they can put 'Designed to run 3Dmark2003' on Nvidia product boxes!
I am a filthy pirate.
Cripes already. No one even BOTHERS with #DMark anymore, and after this fiasco no one is ever going to bother with them again. Gamers will use REAL EVERYDAY GAMES to see what runs the fastest again. Looking at some goofy simulation app coming up with scores and people buying into the company and people tricking drivers for particular tests is just crappy and makes 3DMark 100% invalid to any of my concerns in the future. I will only trust reviews that benchmark the latest and greatest games that I will be buying these cards for, whoever can run them fastest at that particular time IS WHAT IM GOING TO BUY. Peroid. Enough of this 3DMark BS.
So the question in my mind is did Futuremark learn something from the discussions? Is there something it was ignoring in its tests?
I'm trying to not be a cynic and assume a big fat envelope was passed under the table. That what Nvidia did was legitimate.
Those that are following this, should check the pictures on the previous article. The quality of the nvidia "optimized" version sucked (showed big artifacts). That's no optimization, unless there was no image quality loss.
"This card is optimized for quake as long as you follow the left trail, the right trail will just look like crap but nobody follows it anyway".
If a benchmark doesn't measure performance related to real-world applications, what's the point? If a driver is optimized to run a benchmark faster, that SHOULD mean that the real world apps should run faster, too. If not, the benchmark is useless.
If you want to, you can prerender the whole fucking test, stick it in your driver and just play it off instead of actually rendering when Futuremark is running, that would be an "application-specific optimization" too.
The benchmark is ment to reflect performance in the actual game, the reason it takes the same path is merely to make the results comparable. What ATI was something the game *could* have achieved in game, if the operations were properly sequenced. What Nvidia did is to fake a performance it can't actually give if a person had followed the exact same path in the game. That is cheating.
It is pathetic by Nvidia, and it's pathetic by Futuremark to present this press statement. Get some backbone and integrity.
Kjella
Live today, because you never know what tomorrow brings
Now that all the latest games have benchmarking modes, what do we need FutureMark for?
If NVidia wants to do application-specific optimizations that make UT2003 go faster, then that would be great. That's what they should be doing. Those are optimizations that genuinely benefit the user.
Is this truly the only Earth I can live on?
It's not an optimization if it does not produce the same results! Recall that the shader code that the driver used did not produce the same visual results as the shader code it replaced.
... based purely on application-specific pattern matching, which by its very nature is fragile. As was demonstrated so aptly by the 'off the rails' mode in Futuremark. This isn't an accidental bug: it is obvious that such mechanisms are highly fragile, and are almost certain to cause bad rendering on these applications when they are modified in small ways.
More tellingly, the driver deliberately flaunts the D3D spec by omitting buffer clears, mucking about with clip planes, etc.
As others have said, Futuremark's statement is just covering their legal arse. If someone modifies their code to get better scores in some benchmarks while introducing deliberate bugs (i.e. incorrect rendering), it's a cheat in my book.
I think the money quote is:
However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.
I can sort of see the argument here, but it basically ruins the point of having a standard interface like DirectX. It's also like telling your math teacher, "no, it would be easier for my equations if you made 1+1=3. Now do it because I'm your star student."
Yes, there is an increase in speed, but there was also a degradation in quality in the test. See here.
That's the whole issue...
"I turn away with fright and horror from the lamentable evil of functions which do not have derivatives."
"Cheating? Our developers, The Franklin Family, would disagree. I think a meeting can be arranged with them, if you wish. Isn't that right Mr. Franklin?"
*shakes hundred dollar bill side to side, speaks in high tone out of side of mouth*
"Sure is, Boss!"
Well, after seeing the future mark scores, it looked to me as nvidia fx chips are blowing away ati and its gf4 line.
But I found a really nice german benchmark site 3dcenter.org that had to be the best benchmarks ive ever seen, they actually use the games on each and lists the fps.
Looks like the FX/GF4 5200/4200 4600/5600 (non ultra) are the same. And the ATI 9700PRO/9800 are faster than the 5800 Ultra.
After reading these benchmarks, you can really tell nvidia tweaked the SHIT out its drivers for futuremark...
(blunt)The problem with a lot of the reasoning I see here with people saying they want the card that plays the game they're interested in quickly, is that it's completely stupid. (/blunt)
When you're looking for a video card, you -should- rely on a capable, and untainted/optimized benchmark for comparison simply because you can't predict what the software companies that make the actual games are going to do. Will they support -your- chosen card, or will some other GPU maker offer a better bribe to the developer? You may know that kind of info about games shipping RSN, or already on the shelves, but what about next year's?
Getting the card based simply on one or two games instead of looking at some kind of objective benchmark does no good whatsoever. It's just a way to rope yourself into upgrading the card faster.
My own pointless vanity vintage computing page
Total BS, 3DMark2003 is already geared toward specific vendors. I've personally analyzed the data from the driver (since I'm writing one), and they totally favor ATI with the heavy use of PS 1.4 shaders. In fact, the data changes completely if PS 1.4 support isn't claimed. (3x more geometry is sent)
Also, PS 1.4 shaders don't always translate 'up' to PS 2.0 hardware very well, which is why (IMHO) Nvidia started all this hub-bub in the first place.
The only vendor that natively supports PS 1.4 is ATI.
They should have created PS 1.1 shaders for the masses, and then if 2.0 hardware is detected, had 2.0 shaders for everything.
And their "DX9" onyl test is a piece of crap too. They use one or two new instructions in the VS, and PS2.0 is only used for the sky. big whoop. No branching in the VS, two sided stencil, or anything cool.
It's sad the OEMs put alot of stock in 3Dmark, they don't seem to realize that gamers play games all day, not benchmarks.
Certain statements in this press release, including any statements relating to the Company's performance expectations for NVIDIA's family of products and expectations of continued revenue growth, are forward-looking statements that are subject to risks and uncertainties that could cause results to be materially different than expectations. Such risks and uncertainties include, but are not limited to, manufacturing and other delays relating to new products, difficulties in the fabrication process and dependence of the Company on third-party manufacturers, general industry trends including cyclical trends in the PC and semiconductor industries, the impact of competitive products and pricing alternatives, market acceptance of the Company's new products, cmdrtaco's and michael's rampant homosexual love affair, and the Company's dependence on third-party developers and publishers. Investors are advised to read the Company's Annual Report on Form 10-K and quarterly reports on Form 10-Q filed with the Securities and Exchange Commission, particularly those sections entitled Certain Business Risks, for a fuller discussion of these and other risks and uncertainties.
You have got to be the only troll to get modded to +5 practically every time you post. Quite a feat, actually.
Do not read this sig.
And I suppose people who cheat at online MMRGPs are just using undocumented program calls and extended functionality. It's all so clear now...
What would be great...
If someone was to reverse engineer the drivers, remove the "Optimisation", recompile and compare results. See what percent the "Optimisations" fudged the results.
Stop the Slashdot effect! Don't read the articles!
NVidia did things that were clearly NOT legitimate, and FutureMark caught them at it. There's a PDF report on FutureMark's Web site (assuming it hasn't met with an "accident" by now) detailing the dirty deeds. Chief among them, IMHO, was a trick where the driver was supposed to draw and update positions of stars in a night sky (involving clearing the background) as one moved along a 3D path; if one stays on the exact preprogrammed track of the demo, it looks OK. BUT... if you turn around (possible in the beta mode of the benchmark) you see that the driver SKIPPED clearing the background; the stars smear like mad. There is NO POSSIBLE WAY their driver was behaving legitimately. (Especially since changing the benchmark's fingerprint oh-so-slightly caused all these quirks to vanish; they were detecting the demo and screwing with things if it was being run...) The rest is just fear-of-pissing-off-the-800-pound-gorilla. A FutureMark developer admitted as much in a newsgroup posting. Sigh...
"My strength is as the strength of ten men, for I am wired to the eyeballs on espresso."
That would be worse than useless. No two test runs could be usefully compared, so the utility of the benchmark itself is called into question. You'd be reduced to aggregation of test results in the hope that they statistically settle out towards some sort of 'true' average for a given card/driver combo.
The inherent value of a benchmark is the notional "apples to apples" comparison, and you've taken even that away. There would now be no reason at all to use 3DMark.
"So on one hand, honey is an amazingly sophisticated and efficient food source. On the other hand it's bee backwash."
All video card drivers have algorithms to keep from rendering unneccessary portions of the map. What NVidia did, I believe, was to bypass those algorithms and hard-code into the driver the portions that needed to be rendered. That wouldn't work in a real game, where the card must decide what to render at run-time based on user input. Therefor, it's cheating, no different from including an MPEG of the entire 3Dmark demo and showing it in lieu of actually rendering it.
It's hard to be religious when certain people are never incinerated by bolts of lightning.
What a load of crap. This is one of those things that when you think about it too much a bunch of false lines of logic get drawn and you come up with a nonsensical answer. Either that or 3DMark is trying to avoid a lawsuit from nVidia, which no doubt has been threatened.
The point of a benchmark is to test dissimilar systems against common references to get an idea of how they perform against each other in such a way that you have an apples to apples comparison.
If 3DMark writes their program in a way that allows optimization paths for a specific GPU, then it is no longer a benchmark.
You now no longer have an idea of how fast the card REALLY runs as there is no guarantee that game writers will use GPU-specific optimizations. It's the same thing as MMX...Nobody sees the benefits if it's not hardcoded into the program, so what's the point if being uberfast in a benchmark if you won't necessarily see the same results in the real world?
-R
It's sort of akin to walking around with a backpack full of cinderblocks. That way, when you put down those cinderblocks(ie benchmark), you'll notice how much stronger you got.
Perhaps they should use .NET for their next benchmark. Or Java. That'll be the true test of a video card :-)
A few months ago, in a dark warehouse in Brooklyn...
*knock* *knock* *knock*
nVidia: Who is it?
*Futuremark goons enter stage left*
Futuremark: You's late wit you's beta program "membership dues". You know what happens wit da peoples dat don't pay they's dues, right?
nVidia: Piss off! We're not paying this year.
Futuremark: Dat's a pretty benchmark score you got there. Be a shame is somethin' BAD happened to it...
*Futuremark pushes nVidia's benchmark trophy over, shattering it*
Futuremark: oops. Looks like ATI is faster dis year. Dat's too bad for you's guys. You sure you don't wants to join da beta?
nVidia: You bastards! Get out!
Futuremark: Think about it. Da boss wants you on board this year, or bad things happen.
*exeunt goons stage left*
nVidia: No more "beta membership dues". We're going to beat those Futuremark goons at their own crooked game!
0 1 - just my two bits
Futuremark Statement:
Paragraph 1: We're making a statement.
Paragraph 2: nVidia didn't really cheat.
Paragraph 3: Most computer games cheat.
Paragraph 4: We don't allow companies to cheat in their code.
Paragraph 5: Therefore, we should cheat in ours.
Nvidia Statement: They should have worked with us for a better cheat.
Joint Statement:We should all cheat together.
Footer 1: Futuremark rocks.
Footer 2: Don't steal our IP.
Footer 3: Nvidia rocks.
Footer 4: Really it does.
Footer 5: Of course, we could be lying.
Footer 6: Don't steal our IP.
The ______ Agenda
I believe the correct interpretation of what FutureMark is saying is that the game writers are building their games differently for the different boards that are out there. That's what they mean by "manufacturer-specific code path optimization is directly in the code source." The source code they are referring to are UT and Q3, as examples.
They are saying that the boards have become different enough that game writers are coding differently for them. Not too surprising, really. That's the way it's always been.
This makes writing a synthetic benchmark extraordinarily difficult, needless to say. I don't know if it's even possible in this case. Perhaps rather than try to come up with one number that specifies how fast a board is, you can come up with a series of metrics for each capability.
While I'm sure that FutureMark has had some pressure applied to them to make this statement, it's not an unreasonable statement on its face. It's just the path they took to get there that is questionable.
I love Mondays. On a Monday, anything is possible.
That seems a bit more appropriate to the story, doesn't it.
There is nothing so silly as other peoples traditions, and nothing so sacred as our own.
Don't get me wrong, there is nothing wrong with application-specific optimizations.
But this misses the whole point of 3dmark 2003. Different developers stress the pixel and triangle pipelines in different ways to produce a whole boatload of effects. While major games and engines are often optimized-for, there is no guarantee that ATI or Nvidia will sit down and optimize for the game you just bought.
That said, 3dmark 2003 should be considered a relational tool for generic perfrormance. Consider it a good bet that if two cards perform similarly and acceptably, the two cards should be able to run almost any DX8/DX9 game off the shelf acceptably.
The fact that Nvidia's unopitmized drivers perform significantly behind ATI's unoptimized drivers in 3dmark 2003 raises a significant question:
We all know how well the 5900 does in Quake III, Serious Sam 2, UT2003, etc, but how does it do in ?
I want to know that if I take *insert random DX8 game here* home to play, IT WILL PERFORM WELL. That is the entire point of having a benchmark like 3dmark. To do application-specific optimizations for it is to nullify the entire point of the benchmark.
Man is the animal that laughs.
And occasionally whores for Karma.
"The only vendor that natively supports PS 1.4 is ATI."
Sorry, but that's garbage, pure and simple. Or are you not aware that PS 1.4 support is _required_ for DX9 cards with PS2.0 support. Your complaint may be valid when comparing a GF4 against a Radeon 8500, but is totally bogus when comparing two DX9 cards.
"And their "DX9" onyl test is a piece of crap too. They use one or two new instructions in the VS, and PS2.0 is only used for the sky."
Gee, one minute you're complaining that they use PS1.4 instructions, and now you're complaining that they don't use PS2.0 instructions. PS1.4 instructions _are_ effectively DX9 instructions since other than ATI, no other DX8 chips use them: you need a DX9 chip to run PS1.4 shaders.
And it would appear to be real lucky for nvidia that they don't use many PS2.0 instructions since from the results of their shader test once the nvidia "optimization" of throwing away the shader and running a completely different shader was fixed, shows them running PS2.0 shaders at about half the speed of a Radeon 9800. The low performance of PS2.0 shaders on the FX card seems to be the reason why nvidia hated 3DMark03 so much; there was no way to get a good score without redesigning the chip or "optimizing".
Problem is, NVIDIA didn't just optimize. Their application specific "optimization" made the images look worse. And when you couldn't notice it, it was because they were clipping outside the camera angle (becuase they knew exactly where the camera was, something they can't do when you're playing UT2003)
Like the original statement by futurmark said, optimization is great. But when you change the image intended by the software designer in order to make it go faster, that's not an optimization. For god's sake, I can turn all the details to low on UT2003 and get it to go faster, but that's besides the point
Reason games specific benchmark don't fly for me (although now that Futuremark has issued this statement, I'm sure ATI will start cheating as well, making the whole thing useless) is that synthetic benchmarks can test features new to the cards that games may not yet have implemented. So I have an idea how it'll perform with future games.
Warning: Opinions known to be heavily biased.
OK -
first off, for those of you wondering what the big deal with 3DMark 2003 is - and why you might use it in place of "real games" to benchmark 3D performance - here you go:
3DMark is a test application to benchmark next-generation performance, so that you can get an idea how your video card might handle games that will be out this time _next_ year. Specifically some aspects of 3DMark are geared toward testing DX9 functionality, and it's Pixel and Vertex shaders. No game currently on the market uses these features (at least not that I am aware of).
Secondly, the difference between a cheat and optimization is a fine one. If a given function continually produces the same output for the same inputs, and it takes 1 second to do so, and another function can produce the same results given the same inputs, but only takes 1/2 a second - it can be said to be functionally equivalent. However, it has been optimized. It's entirely possible, even desirable to replace pixel shaders and vertex shaders with routines which are optimized for your hardware. In much the same way that compilers schedule instructions optimally for the underlying CPU architecture, so too can instructions be re-ordered in a pixel shader routine... It's an optimization.
Cheating occurs when people start making approximations (analogies to bringing a cheat-sheet to a test are not valid), or by failing to process (in the case of video cards) the same visual fidelity, and detail that was intended. By example:
A> Reducing texture bit-depth.
B> Reducing geometry detail (merging 2 or more polygons).
This is only cheating if it's not the intent of the original application developer (not driver developer).
A driver developer could make the following optimizations, since they don't affect the intent of the application developer:
A> Pre-calc tables. A classic demo optimization would be to precalc a SIN function table to some level of precision as looking up a value was faster than calculating it on the fly.
B> Replacing various pixel/vertex shader routines with functionally equivalent, but faster ones.
C> Reordering data and textures (keeping detail and fidelity) into more optimal chunks for your hardware architecture.
Those aren't cheats - they are optimizations. Of course, the only way you can tell this is if you have an objective standard to gauge against. 3DMark 2003 doesn't seem to provide this. In order to do so they would need the following:
A> A software renderer for their demo.
B> Timed snapshots of the demo saving uncompressed images from the software renderer to disk.
C> The ability to re-run the demo using a hardware renderer (3D Card and drivers).
D> The ability to take the same snapshots and save them, uncompressed to disk.
E> The ability to do a histogram, per-pixel comparison to the software renders...
This would enable you to arrive at some objective comparison of visual fidelity - instead of the occassionally subjective I think screenshot X looks better than screenshot Y. Without the intent of the 3DMark developers being known, we really can't know how true the hardware vendors and their drivers are to the original vision.
Anything less than 3% difference is highly likely to be indistinguishable from the intent of the developers in this case. 5% to 10% may be visible, but acceptable (i.e. tweaks for speed in place of quality). Over 10% and you're playing with fire.