Slashdot Mirror
Futuremark Replies to Nvidia's Claims
Nathan writes "Tero Sarkkinen, Executive Vice President of Sales and Marketing at Futuremark, has commented on the claims by Nvidia that 3DMark2003 intentionally puts the GeforceFX in bad light, after Nvidia had declined becoming a member of Futuremark's beta program. This issue looks like it will get worse before it gets better." ATI also seems to be guilty of tweaking their drivers to recognize 3DMark.
It won't be fast enough next year.
nVidia: "Well, for this program they will never step off the rail, so we can fake it so it looks good from the rail only."
ATI: "Well, this shader program isn't optimally coded - here is a more optimally coded shader that does the exact same thing but more quickly."
nVidia: "Well, you caught us, but we have to cheat because you have it in for us!"
ATI: "Well, you caught us, and although we were doing the exact same thing (only faster), we will remove that code ASAP."
www.eFax.com are spammers
3DMark will look totally sweet because it's *optimised* for both cards.
Its about the OEMs as much or more than the consumer market. They watch the benchmarks closely -- and make decisions based on results.
This is where the money really is, and what is worth fighting for.
I suppose you have to expect some poor practices considering that the top 3DMark card will be considered by many gamers to be the best to buy. It's a massive temptation for such a big industry. I find ATI's decision to remove code which it claims boosts over all performance quite funny.
ATi's tweak yields a 1.9% gain by rearranging the instructions 3dmark issues it's hardware. Anyone familiar with assembly language knows that properly arranging your instructions prevents stalls; the end result, however, is exactly as intended. It sounds to me that this is what ATi did. nVidia, on the other hand...40% gains with very obvious visual errors is..well, wrong.
if nvidia wants to really screw 3dmark, why don't they just make their benchmarker stop working w/ their chips all together...
"3DMark03 was developed strictly according to DirectX9 standard in very close cooperation with Microsoft and other BETA members. If hardware performs well 3DMark03, it performs well in all applications that use DirectX 9. Note that since 3DMark is designed to be an objective evaluation tool, it does _not_ include manufacturer-specific optimizations. This is why it is exceptionally well suitable for objective performance measurement. "
Does this guy work for NVidia?
thank God the internet isn't a human right.
People keep begging that nvidia release their drivers under a open license. Well i guess we now know why they don't. /Esben
"Nobody really checks their email any more. They just delete their spam"
This is why you need many forms of evaluations to properly test something. Just running one program to show you pretty pictures is not going to give any meaningful result. You need to stress test the card in other ways.
And, since one of the main reasons people will buy this is to play flashy and pretty games, ignoring the performance in those games is rediculous.
As long as there are benchmarks, companies will write drivers to get the best score. I would say about 95% of consumers purchase (buying high end cards) based on benchmark scores. Of course you would write a driver that gives the best score to increase your sales. nVidia is just very good, and way better than ATI, at writing drivers that exploit many of the benchmarks now in use.
Is it a bad thing? Not from nVidia's view and ATI is jealous that their code monkeys are falling behind (or understaffed).
Does it make the benchmark invalid? Yes. But it does not matter since 3DMARK has become a "standard." Both card companies will try to exploit the benchies to get the best score and take the "performance crown" and the sales that come with it.
Back when nvidia aquired 3dfx, they began to merge their development teams. The fx is the first card by nvidia to be developed by engineers from both the nvidia and 3dfx groups.
... But who knows! I'm no fortune teller ...
Of course it will work better when you do it their way; It was 3dfx's strength in the beginning, and its downfall in the end.
But I believe that their current development team has yet to hit its stride, and future offerings will see the trophy going back to nvidia...
Online Starcraft RPG? At
Dietary fiber is like asynchronous IO-- Non-blocking!
ATI also is crafty at tweaking their drivers to suck. They should be working on decent drivers instead of cheating on stupid benchmarks to get +1.9%.
I bought a Radeon 9700 Pro. The driver issues almost make it not worth the increased FPS over my Ti4400.
The refresh rate problem in XP is annoying as hell. ATI handles it even worse than NVidia, where you set your "maximum" refresh rate and your "highest" resolution, and it assumes that all lower resolutions can only handle that refresh rate.
There's no way to tell your ATI card, "My monitor can do 1280x1024 @ 85hz, but 1600x1200 @ only 75hz." You either get 75hz in everything if you want 1600x1200, or you get 85hz up to 1280x1024, and have to avoid 1600x1200 use lest ye risk getting "frequency over range".
NV handles it better with the driver, allowing you to set maximum refresh rates for every resolution individually.
These refresh rate tweaking programs don't help either, since half the games out there choke when you use them. PC 3d is in a bit of a sorry state right now, and I'm tired of it.
# Erik
This whole episode has turned into a big mess. NVDA seems to be the bad guy in all of this. Their DX-9 product was delayed and their existing products where only DX 8.0*. The benchmark heavily favours DX-9 parts and NVDA's existing lineup was/is getting smoked in the benchmark by it's main (only) competitor. They decided to go on the offensive and try to kill off this benchmark. The 30 person company that produces 3D Mark have stood their ground against the multi-billion dollar NVDA. NVDA instead of admitting that their Pixel Shader is quite slow when running against 2.0 specs insteads tries to decieve and FUD their way out of it. Looks like they got more than just some patents when they purchased 3DFX...
Now they have painted themselves into a corner and how this will turn out is anyone's guess.
*DX8.1 has PS 1.4 which is actually much closer (AFAIK) to PS 2.0 than PS 1.3 (DX8).
What!? Two giant corporations actually doing something MS-like to make themselves more appealing?! That's unheard of! Why, one might think this is a ploy to increase marketshare! Corporations are our friends, they would never manipulate the people. Damn the man!
IGB: More fun than eating oatmeal!
The problem isn't that benchmarks lie. We all know they do. The problem is we don't know how they lie. Creating open source benchmark applications can show how the driver is excirsed so everyone who wants to know or learn where cards and drivers are strong and weak. Everyone is on the level if everyone can look at the code that came up with numbers. Not to mention there are things to learn from code in benchmarks that excirse the fringe elements of graphics cards and drivers.
The alternative is what we have now: hand waving voodoo. Not only do we have to take the vendor's word they aren't monkeying around with the driver to match execution of the benchmark but now we have to question where the aligence of the benchmark makers.
AFAIK, ATI displays the graphics on screen properly, the drivers are just optimized for the benchmark. One could still consider this cheating. NVIDIA however does not display the graphics properly, it really does cut corners (literally) to get higher scores. ATI got an extra 3% from cheating. NVIDIA got a whopping 24% higher scores from cheating! take a look at the extremetech screenshots:
, a= 41574,00.asp
http://www.extremetech.com/print_article/0,3998
Hardocp
They do a good job of disecting the benchmark, and I'd have to agree that as a DX9 benchmark it fails.
Whatever, it's still just a synthetic mark and nothing more.
-- taking over the world, we are.
I've worked in the PC industry more years than I care to think about. All graphic card vendors tweak their drivers and bios to make their cards look better. If people didn't put so much emphisis on benchmarks for buying decisions then there would not be much reason to tweak things but the reality of the world is they do.
On a side note, me and my team many, many years ago designed, what was at the time, one of the fastest chip sets for the blinding new 12 Mhz 386 PC. We had discovered that the Norton SI program that everyone was using to benchmark PC's based most of it's performance on a small 64 Byte (yes, that is not a typo 64 BYTE) loop. We had considered putting a 64 byte cache in our memory controller chip but our ethos won at the end of the day as cleary what we would have done would have been discovered and the benchmark would have been rewritten. Had we done it however, for our 15 mins of fame our benchmarks would have been something crazy like 10x or 100x better than anything out there.
"TV, a medium as it is neither rare nor well done." Ernie Kovacs
from the what-exactly-are-kid-gloves dept.
Get the answer at Straight Dope
Look there is a clear difference between what NVIDIA and ATI have done here. ATI are not cheating, they look at a sequence of instructions and reorder them if they fit a pattern, but they do exactly the same thing as before. This is central to the kinds of things optimizing compilers and CPUs do. Maybe you thing it's too narrow a path, but it's a minor infraction at best compared to the blatant cheats of NVIDIA, who not only rewrote shaders but did several otehr really heinous things, like disableing screen clear and adding their own hidden clip planes.
It's a real shame that The Register obscured the truth here with an article that attacks ATI for conservatively removing optimizations while giving the real miscreant gets a free pass. ATI should leave their optimizations in IMHO, but maybe you disagree because their mathematically equivalent optimization is not general enough, it's a close call, but they don't deserve what the distorted treatment given in The Register.
I'm confused about what this means. Is the 1.9% difference in ATI performance between Game Test 4 with correct and modified names, or between the current driver and an older version?
Most people here seem to think it's the latter, and I'd agree that they did nothing wrong if that's the case. But it's not obvious to me that they're not accused of the same thing as NVIDIA.
What I'm listening to now on Pandora...
I remember video card companies cheating on benchmarks 10 or more years ago. This was when PCI was the latest thing on the block, and was competing with VESA-local bus. They wrote drivers specifically to detect when they were being called by the PC-Magazine benchmark program, and they'd do some stuff like just returning from every other call, since the prog was just calling the same thing 10000 times.
I lose respect for companies when I hear stuff like this. They should try to reorganize their best-practice protocols and rework their ethics. Then they should read more Scott Adams.
When Quake III runs at 300 FPS on my system under my 9700 Pro with 4x AA, I could care less about 3DMark and what ATI or Nvidia tweak. If the games run smooth and they look good, then go with it. Truth is, the ATI looks better than the Nvidia card under QIII, WCII, JKII, and pretty much everything else I've been playing.
The issue with low FPS is a game problem 9 out of 10 times. The faster the video card, the less the game development houses work to streamline and improve their framerate.
In what way is Ati cheating, really? If you think about it, virtually every modern processor does some minor instruction rescheduling right? Basically, Ati is doing this in the driver and not on-chip, that's the only difference. I'm sure in the next few generations of GPUs we'll see the introduction of hardware features like this. Once the pixel/vertex shaders get ironed out pretty well and a lot of people use them. Right now very few games really make use of them and they spend most of their time emulating hardcoded T&L which is again a part of the driver.
Nvidia is cheating and acting like a child, er, large corporation...but that isn't at all what Ati is doing.
Hey, i want to know the truth just as much as the next guy, but seriously.... does this seem odd that when Nvidia opts out of this hundred-thousand-dollar beta program, this happens? :)
Ive read 5-6 reviews of the FX 5900 and everyone seems to think its great, and rightly gives Nvidia the 3d crown. (Especially concerning Doom ]|[
If you read the interview, its even brought up that the 5900 seems to do just fine in all other benchmarks, only futuremark seems to give it a hard time, and im not buying that crap about Doom 3 benchmarks not being readily available.
If i remember, Toms had a good review of that....
Problem Exists Between Keyboard and Monitor *grin*
My 9700 Pro works without a hitch. Not a single problem.
Is it coincidence or some sort of nVidia inside joke that changing the name of the Dawn executable (fairy.exe iirc) http://www.nvidia.com/view.asp?IO=demo_dawn
to quake3.exe removes those pesky leaves, revealing her suptle nature, and that renaming it to 3dmark2003.exe removes the leaves and her wings? Is the inside joke that they leave "certain things out" of quake3 and 3dmark? Does the government know of the existence of aliens and wormhole portals to other worlds?
put the what in the where?
What upsets me is not that you lied to me, but that from now on I can no longer believe you. -- Nietzsche
Classic.
Think of it this way, when's the last time you saw PC World roast a product that truely deserved it? How many review sites gave WinMe a thumbs up when it's widely viewed in the industry at MS's worst OS to date? We (the public) simply aren't being served if the test companies are cooperating with the companies their testing. Look if a testing company, review site or whatever other lab doesn't occasionaly come out and just say "this sucks" than you know they aren't credible. There's too much out there that sucks, and too few reviewers willing to let the public know before they waste their money.
It's the same reasoning that dictates why consumer's reports will buy their cars anonymously from dealers using third parties instead of getting "special" delivery directly from the manufacture. What we should really see with the behaviour were observing so far is an impetus to develop an open source test benchmark application. By doing this we would assure that the results can't be bought, just like has become common practice in so many other industries.
I decide what games I'm going to play, and I see how they look on each card. The main people that buy highend card generally know others that do also. I just compare their cards with the games I wish to play. I pick the best of the lot, and buy it. Currently I have a GeForce4 4400TI. It does great for the games that I play. I'm sure I will need a new card when Doom3 is released. I will check Doom III out on every card I consider. Then buy the one I think looks the best.
Quake 3 vs Quack 3 Go troll elsewhere. They're both guilty from time to time of doing the same bullshit.
SHOCKED! to find that there is optimization going on here.
(Alphonse enters.)
Your SPECmarks, sir.
Thank you.
Some time ago I had read something to the effect that Nvidia was using "genetic algorithm generation" procedures, to improve its drivers. If so, then it may be possible that the driver cheats just happened as a result of that process, and not as a result of deliberation.
This happens so often in grade school I'm surprised the computer industry hasn't caught on to it yet. If you give students a copy of the exam the night before the exam, the only material they are going to bother to study the question-answer pairs on that exam, and may just remember what the answer to #6 is rather than even try to understand the question.
In order for a driver benchmark to be useful at all, it needs to be kept absolutely secret from the chip manufacturers before the test, and then once it is used and revealed that benchmark needs to be retired, because the next generation of testing should be designed to concentrate on the new features that the graphic card developers are expected to put in their next generation of cards that will be used in the next generation of games.
In short, the best benchmark will always be based on "that sure-to-hit game that's just about to come out."
If it only takes an hour to get a 40% performance increase, why didn't you take that hour before the FX came out, and then you could have actually beaten the radeon 9800.
Or better yet, spend a couple hours and figure out a way to get rid of the locomotive like cooling system built into the FX....
hard core geek-ware
In Doom ]|[, the most advanced graphics game out there currently, NVDA smokes ATI. On test equipment that NVDA only provided the card.
Am I going to "play" 3DMark or am I going to play DOOM 3. For all of you who will be playing 3DMark more than doom, go ahead and get the ATI card. I'll make my decision based upon the stellar DOOM 3 performance.
I never put much stock into benchmarks. One test says one thing another something else.
All this proves is that a benchmark is a highly isolated incident of observable performance.
For example, most charts I see rate the P4 as "faster than the Athlon" at the same clock rate. Yet when I benchmarked my bignum math code I found that the Athlon completely kicked the P4s ass
[K7]
http://iahu.ca:8080/ltm_log/k7/index.html
[P4]
http://iahu.ca:8080/ltm_log/p4/index.html
Does this single test prove the athlon is faster than the P4? Hell no. It proves that using portable ISO C source code todo bignum math is faster on an Athlon. If I used SSE2 the P4 would probably smoke the Athlon, etc...
Can we stop putting stock into this BS?
For the record I have a Ti200. Its decently fast [50fps at 1280x1024 in UT2] and there are no visible artifacts or other "cheats". It works nicely. So if nVIDIA cheated to make their 3dmark score better all the power to them. Screwing around with meaningless benchmarks is a good way to discredit them.
Tom
Someday, I'll have a real sig.
Of course both companies are guilty of optimizing their drivers to get better performance on the benchmark. The fact that ATI is also doing it (though, not to the extreme that nVidia did) should come as no surprise to anyone.
This is business, and nVidia and ATI are different companies, both are doing everything in their power to one-up the other, so I'm sure if anyone looked close enough at various benchmarks, some instances of "foul-play" would be present.
Personally, I don't think the optimizations made by either company is anything to be "ashamed" of, it's not like they're producing fake results, they're simply pushing their own cards to their extent for the benchmark, and that's nothing "unfair."
Trent Polack
www.polycat.net
To what extent does this also cast doubt upon video card benchmarks that use game demos in which the camera moves along a fixed path?
Why has everyone forgotten the http://www.hardocp.com/article.html?art=MTEx">Quac k.exe cheat ATI had with the first radeons? That was (IMO) far worse than what NVidia is doing, since it affected a popular GAME not some lame benchmark that means nothing.
which company are you going to reward for cheating? put it another way, which pile of fertilizer smells better to you?
Error: The first 80386 part (and the slowest) was a 16mhz 80386DX processor. (ditto for the 80386SX, with the 16 bit bus)
You might be referring to a 286 system - 12 mhz 286 boards were actually a hot commodity at one time and the chipset market was crowded with many contenders offering shadow ram features, EMS 4.0, etc.
HBI's Law: Frequency of calling others Nazis is directly correlated with the likelihood of the accuser being Communist.
You mean,
Problem Exists Between Keyboard and Chair.
For the record, my 9700 Pro works great too.
It would be easy enough for ALL of the testing labs to simply rename the benchmark and game application EXE to something random before starting the tests as a matter of course. If they state this fact up front for all to see, it would make special casing like this extinct overnight.
It wouldn't prevent cheating. Data profiling could probably be used also. But it would encourage the driver developers to concentrate on real improvements (just in case they can't identify the test app) rather than concentrate on fooling the test.
There is nothing so silly as other peoples traditions, and nothing so sacred as our own.
First, as we all know, there are lies, damn lies and benchmarks. But a company cheating here can really be damaging to the industry.
In this day and age, the top of the line video cards have more than 100 million transistors, and have become increasingly "intelligent" in sorting and rearranging data, instructions and choosing algorithms to get the best performance out a given engine. Some engines perform better than the others under different circumstances, making benchmarks even more subjective than ever.
Furthermore, there's no established standard on what kind of resolutions, color depths, FSAA, etc. a game should run on, and therefore there's also no standard on how to create a base hardware architecture that can be used as a starting point for comparisons. Sure, we have the same basic features on every card (shaders, T&L, texture units, etc.), but each and every one is implemented in a completely different way and it changes radically every few generations of hardware, making yesterday's optimizations obsolete.
But all of this is actually just fine, because the different architectures and 3D engines combine in different ways in each generation to produce new results, not unlike genes. The 3D graphics industry is truly on a fast evolutionary path, and just comparing back to my old Riva would be enough to make Darwin jealous.
So, a company detecting a specific binary in order to optimize/cheat/whatever how the hardware works in a given situation is an artificial gain that destroys the whole purpose of having better hardware each generation in the first place, and is completely unforgivable for both nVidia and ATI.
They should instead work on improving the "intelligence" of their drivers and hardware, which could arguably provide better results in benchmarks without having to resort to stupid, cheap methods that will damage themselves and the entire industry in the long run.
So, as far as I'm concerned, Futuremark should keep doing what they can to prevent cheating, and the users must support this and every similar effort. The 3D graphics industry will be damned the day it becomes a code war between cheaters and moderators. (not unlike online video games, by the way)
- Otaku no naka no otaku, otaking da!!!
See here for the original /. store describing the Quack / Quake 3 cheat ATI had a while back. MUCH worse than the current NVidia cheat IMO.
Regardless of if you think it is worse, the point is that BOTH companies have cheated in benchmarks so there is NO point in "glorifying" ATI at NVidia's expense. They are just as bad if not worse (and their drivers blow ass ).
Nice to see more unbiased reporting on Slashdot. NVidia gets at least 4 articles ... ATI gets a subtext within an NVidia article. That's balanced....
"Nvidia had declined becoming a member of Futuremark's beta" sound like an invitation, but doesn't cost to be in the Futuremark beta program? Try to be just a bit more accurate, eh?
Onward to the Aether Sphere!
I've been watching benchmarks for VGA cards for years. It seems that ever since I started to read them various companies were accused of (and often found guilty of) cheating. Heck, back when my dad worked for a database company in the 70's they did the same thing.
The only suprise is ATI admitting to tweaking some things and removing the offending (if trivial) code. I suppose they see they can beat nVidia even without cheating, so they win the PR war by saying: Look at how fast our card is WITHOUT any cheating vs. how slow nVidia is even while they DO cheat!
I think not but it has been so long. I was design manager at VLSI Technology. We made a 12Mhz chipset which was our first, I thought for the 386 but it may have been for the 286. Our chip set was used by IBM for their "reentry" back into the ISA bus channel PC's after their big micro-channel effort resulted in them loosing half of their market share by dropping the ISA box for a while.
The cache comment is correct, regardless of the CPU it was going to work with. We reversed engineered the Norton SI benchmark and found out what it was doing. We were tempted but did not go forward as it would have been a useless feature. Point is, any silicon vendor out there hawking their wares knows what the benchmarks are doing and will do "what ever it takes" to either explain away the bad marks and figure out how to make their silicon look better. It becomes a gray area when you start tweaking just to tweak as opposed to adding anything of real value.
"TV, a medium as it is neither rare nor well done." Ernie Kovacs
I haven't forgotten, it should be remembered, I don't think it's worse though. I have mentioned it in other posts and The Register mentioned it in their article. Maybe you should read it before posting?
This "cheat" affected image quality which is how it was first noticed. ATI quickly released new drivers which improved the image quality back to how it should look. And there wasn't any decrease in speed from the "cheating drivers". So in this instance nVidia are worse as they have yet to release a fixed driver or even admit what they did was wrong.
All IHV's provide per game "hints" to the driver of how to work for a give game as well as "hints" of how to work with an unknown game. But this is usually only for making the game work faster on their hardware - without sacrificing IQ - or working around game bugs with their driver.
Maybe everyone should take PowerVR's approach of exposing every driver hint through the registry in english language instead of hiding it in the driver itself.
If the drivers were open-source- they dont have to be "free" software, just open, this could all go away, couldnt it?
-- 'The' Lord and Master Bitman On High, Master Of All
Maybe they'll get more honest results renaming the benchmark executable to "3DLark03" or something. After all, something like this worked once. ;-)
ATI (presumably NVIDIA does as well) has an instruction reordering phase whenever a vertex or pixel shader program is loaded. This attempts to schedule the program so that it executes optimally on the hardware. People at ATI discovered that the optimizer produces a sub-optimal schedule for 3DMark03. This is hardly surprising since in general scheduling is a hard problem. Anyway, they also discovered that they could make it perform 8% better on one of the tests if they ordered the shader programs by hand.
So with the new driver version, ATI detects the 3DMark03 shader program and replaces it with a hand-optimized one that is significantly faster on ATI hardware. The replacement program is exactly equivalent. It doesn't affect the output values at all, mereley the speed at which they arrive. NVIDIA did the same thing but put in a faster program that is not equivalent.
The question you want to know: why is this a bad thing? It's an optimization that only affects one very small set of shader programs. If those shader programs were for a CAD program and were useful in the real world it would be good. Instead they are for a synthetic benchmark that people use to see how fast they can expect the hardware to be. Increasing that score with an optimization not useful anywhere else is deceptive.
It's painfully obvious it's an nvidia fanboy...
Well, that's all well and good, but what does it accomplish? How do we decide who is allowed to work on the standard, because virtually everyone with sufficient skills and clout will have an angle. Do we let hardware developers do it? No, that's like having the fox guard the proverbial henhouse. Game developers? Maybe, although they're tainted too - for one, because they partner with hardware makers, and second, there are instances where they might want to hide a rendering weakness in the game, same as a hardware maker would. Gamers? Generally don't have the expertise or exposure.
So you can see the problem. Futuremark isn't going to opensource their code, and it would be about impossible to build an organization that would be capable of fairly creating a new opensource alternative to 3dmark.
As frightening as it is, the most unbiased player (or at least appropriately biased) in the game would be MS, since they release DirectX. You could argue that they take graft in terms of what features they release in DX, but since that's what we're playing on (don't bring up gaming on linux please), at least it's realistic. And I KNOW they aren't about to opensource anything.
-Looking for a job as a materials chemist or multivariat
Compiler companies have been writing their compilers to recognize particular benchmarks, and generate optimized code for them, for years.
Anonymous Cowards suck.
Open source benchmarks will only give you the opportunity to fiddle with the benchmark to unearth hidden cheats but same thing can also be used tweak the benchmark one way to favor one hardware over another. IOW, it will be even harder to catch cheating/biased reviewers; not a very good idea.
Gentlemen, you can't fight in here, this is the War Room!
Yeah, but Doom ]|[ is just another benchmark. What really matters is how it performs on Halflife 2!
I like nvidia but I'm disappointed that the reply sounds like a justification. From Derek Perez (dperez@nvidia.com):
Since NVIDIA is not part in the FutureMark beta program (a program which costs of hundreds of thousands of dollars to participate in) we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer.
We don't know what they did but it looks like they have intentionally tried to create a scenario that makes our products look bad. This is obvious since our relative performance on games like Unreal Tournament 2003 and Doom3 shows that The GeForce FX 5900 is by far the fastest graphics on the market today.
dp
Who really cares about benchmarks? Personally I don't (that was a rhetorical question). So what if one card can do 10 more mega-flip-flops per microsecond than another one. I don't really care about getting an extra 0.00001fps from Doom3.
I don't believe claims anyway; ATI says their card is faster than NVidia's. NVidia says theirs is faster than ATI's. Bleh....
Is something more along the lines of "live" demos. Not these flash-stuff-by, fixed-POV, fixed-scene demos, but something more akin to a game or whatever. The "Dawn" demo is cool, and would be a neat thing to have in testing as you can see the differences and/or notice any graphical distortions or lag.
A better thing would be to have a demo with a simple multi-room house or whatnot. Various lighting effects, a few characters perhaps, window panes, maybe the outside is blocked by lava or something so that you can pull off a few cool lighting/sprite/particle effects.
How hard would it be for a knowledgable openGL or DirectX programmer to code a small "house" demo, maybe with a small forest, a pond, and a house to walk though. Depending on the details, such a demo would more adequately respresent real-life of the card (say Nvidia shows nicer trees in the forest but slows to xx FPS).
ATI, suddenly finding themselves in a corner, made a very smart decision under pressure.
Point is, they can come out of this wearing the white hat, because they were the first to be such good guys about the issue.
The fact is, even with all Nvidia optimizations in-place, their high-end card will just barely edge out a 9800 Pro without optimizations. Add ot this the fact that ATI, 3dmark and the community will hound them and discount Nvidia's optimizations until they are removed, and you've got an all-out win for ATI.
Remember folks: everyone cheats. Fools take things too far and get caught. ATI has played the fool before, Nvidia plays it now; that is the game.
Man is the animal that laughs.
And occasionally whores for Karma.
Acht. Cut/paste error. Should start out with "What would work well is..."
He's right on the money. PS 2.0 is not always needed, sometimes 1.4 is enough, so there's no point in using 2.0. And in other cases 2.0 is needed because 1.4 just doesn't cut it. 3DMark03 uses 1.4 where it's the smart thing to do, and 2.0 when it's smart thing to do. All DX9 vid-cards support 1.4, since one of the requirements of Direct3D-compatibility is backward-compatibility.
Lesbian Nazi Hookers Abducted by UFOs and Forced Into Weight Loss Programs - -all next week on Town Talk.
We all know ATI did it in the past. We all know Nvidia is doing it now. And even better we know that ATI is up to its old tricks again as well.
x e? Use some kind of encryption to prevent the driver software from snooping in on the rename process and oooh no more cheating....
And how have people figured this out time and time again? Oh, they renamed the executable...
Why does the benchmarking software not rename the executable to some-254-character-long-file-name-random-string.e
I'm sure that there is some way that Nvidia and ATI could get around even this but what are they gonna do make a 75MB driver in retaliation to what the benchmark companies do?
Everyone is blowing this so far out of proportion its crazy. No one cares. The only people that care about these benchmarking applications are the people that write them and the web sites that use them. The general public could care less and probably doesn't even know they exist. All that matters is little Joey gets a new video card and its fast so he tells his friends who then also goes to buy the same card. Anyone with half a brain knows that these benchmarking applications are nothing but one sided. Real life performance never matches up to what the numbers say.
Our initial assessment - as usual - is that we will slaughter them all. The ATI cards are commiting suicide at the gates of our plants as we speak.
Trolls dont like to be Flamebait, because they burn so well. Protect our Troll heritage!
I can't believe this NVIDIA vs. ATI vs. Futuremark crap made the frontpage AGAIN. Is every reader of /. a closet hardware video geek? I'm bored of this story. It is no longer relevant. Please stop posting this, repeatedly. It is nothing new. There's nothing to see here. Move along. They can go about their business.
Don't get me wrong, there is nothing wrong with application-specific optimizations.
But this misses the whole point of 3dmark 2003. Different developers stress the pixel and triangle pipelines in different ways to produce a whole boatload of effects. While major games and engines are often optimized-for, there is no guarantee that ATI or Nvidia will sit down and optimize for the game you just bought.
That said, 3dmark 2003 should be considered a relational tool for generic perfrormance. Consider it a good bet that if two cards perform similarly and acceptably, the two cards should be able to run almost any DX8/DX9 game off the shelf acceptably.
The fact that Nvidia's unopitmized drivers perform significantly behind ATI's unoptimized drivers in 3dmark 2003 raises a significant question:
We all know how well the 5900 does in Quake III, Serious Sam 2, UT2003, etc, but how does it do in ?
Man is the animal that laughs.
And occasionally whores for Karma.
If the government had a clue ,they would sue or FORCE all the major software and hardware companies to open up their source code for compatibility with other Operating systems and other software.
Some of you may think thats impossible but it could be done. Think about it ! All the hardware makers release drivers for ONE damn company - Microsoft. It's not right.
There was a comment that you can do certain operations in different ways and come up with the same result in the end. It sounds like what nVidia might have been doing was re-arranging the operations to run faster, in other words performing a simple optimization of future mark's code. Of there was no VISUAL degedation, then I don't really see much of a cause for complaint.
Obviously any professional game engine is going to have optimization profiles for the major cards, so I don't see this as a big deal.
ReadThe ReflectionEngine, a cyberpunk style n
All computer companies inflate stats. Hell, ALL companies inflate stats (7 out of 10 doctors...95% success rate...etc). It's the game of Capitalism, and the winner takes all....
But yeh, nVidia did cheating and ATi did optimization. That being said, I have 3x more nVidia cards than ATi cards...because they were cheaper for faster. I only get ATi when I build computers for others however. GO CANANDA!
It's AMAZING! I can complete the entire benchmark in under 1 frame!!!!
I optimised the benchmark in a very straight forward way! After starting the benchmark I realised that it performed a huge amount of calculations. ENORMOUS number. However I noticed that after the application had terminated the only difference in the state of my computer was a coupla registry values and I have this text file....
Well the text file is a load of gibberish numbers so I will discard that (don't need to run that part of the program)... The registry seems to change organically anyway. So I'll call that nature.
Sooo... it appears that the net result of the benchmark (discarding gibberish numbers) is to do nothing. So I had an idea (took a millisecond or so). I figure.... don't run the benchmark.... hey presto... problem solved..
Now... DOES MY PROGRAM RUN ON THIS THING???? Hmmm.
"None of this shit works" -W.Shatner
Now that they have "patched" 3DMark so that the NVIDIA cards don't run it as fast, I wonder why I would want to use this program on anything but an ATI card.
Thier comment that the benchmark ran great on the released version but showed errors on the unreleased private devleoper version is like saying "Well it may get 120fps in QuakeIII but if you run it on Super Secret unreleased QuakeIII beta 2 it has errors"
If I have an NVIDIA, S3, Matrox, etc. and they did not pay for the FutureMark beta program then these cards may not be compatible at all. It's almost like a game that only runs on a Playstation and not a Nintendo.
I think this will be the end of 3DMark as a reliable or even relevant benchmark and hopefully we will see more sites use a wider variety of games to showcase new video cards.
BTW how did ExtremeTech (The website that found the "cheat") get thier hands on the developer copy of 3DMark. It costs thousands to be part of the beta program? They must have great ad revenue.
http://www.kubuntu.org/
You look like a total dork writing NVDA.
Nothing will even use the new kit to its fullest for that long.
I'm rapidly coming to the conclusion that the way to go with video cards is to buy one a year old. It's much cheaper, and typically handles all current and near future games perfectly well. The new gizmos, and speed boosts, on these cards rarely provide worthwhile bang for your buck these days.
Use the money you save to buy a faster processor, more RAM, a RAID array or something else that provides a useful improvement in performance outside of the theoretical. Or if you're buying/upgrading card + monitor together, get an extra couple of inches of screen real estate or go for a nice flat panel. The difference in price really is of that order, yet the difference in ability is irrelevant for almost all real applications.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
I don't care about 3dMArk, NVDIA does enough good
hardware and cares about linux, and thats important
to me. ATI doent's care about linux at all, so screw them
Of course, because we all know that no R&D effort goes into those drivers. That's why performance goes up so much with good ones. And releasing the code to those good ones, thus giving away any performance-enhancing algorithms developed during the aforementioned R&D, would in no way competitively disadvantage the company concerned.
It would be beneficial to the user community if the interface specs for these cards were made available by the manufacturers, thus allowing those prepared to put in the effort to write drivers for, say, Linux. But whether to release the actual code for their own drivers, thus probably getting a massive amount of support and quick bug fixes from the geek community but also exposing them to competitive damage, is a commercial decision, and the legal system has no business making commercial decisions.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
If geeks didn't equate 3DMark scores to penis size, no one would give a shit about the stupid benchmarks. Get a freakin life, do you really think OEMs care about how fast their products are??? All OEMs care about is name brand recognition hence the fact that almost all OEM machines use Intel processors and Nvidia graphics cards (usually a Geforce 4 MX). You people really need to get outside more. BTW, I have a Radeon 9500 Pro, and I don't really care about benchmarks as long as the games look good and the porn looks decent ;)
Wait, people use their computers for things other than benchmarking? From the posts I read in the Internet, every new card or driver rev is followed by 100 posts stating how they gained/lost 10 3DMarks and how their e-penis is so much larger than yours because they overclocked their card and got 50 more 3DMarks...what exactly are these "real-world applications" and "games"? How do these increase your e-penis length?
my old employer, acronym generated from Sight Sound Suck, had basically two modes in most drivers: one (WHQL-mode) where the thing was run in the best possible image-quality mode, working around various sundry hardware bugs; and then a set of speed-modes if game-EXEs were found. With benchmarks, it was a bit trickier. One thing they did in particular was to actually attempt to figure out what app was running by doing an analysis of the first N texture loads; which bit us in the ass later on when the same value ended up coincidentally being generated for one of the WHQL tests (where output MUST be exactly correct). NVidia used to be better than this; we had respect for them for at least just implementing the reference rasterizer (fast) in hardware rather than trying to implement new features in hardware without MS's blessing. Those HW features would be implemented well in 3dfx's case, crappy in ours; but the result was identical - if it's not part of the Direct3D API, it's completely useless.
not 286, chip was crippled really, remember dreaming of a 386 with a whopping 2MB of ram though. Stuck to my 8088 (yeah!) and the computers at uni until I could get hold of a monster 486 with 8MB - I was king... Remeber Pentium 60 (or was it 66?) and p90 just had come out then but that was far above my budget... :-)
if (!signature) { throw std::runtime_error("No sig!"); }
I mean, seriously now folks. I think benchmarking these days is nothing compared to what it used to mean. When the Voodoo3's rolled out to tackle the TNT2's, there was a considerable gap between the two. When nVidia introduced the GeForce GPU, the whole game world was changed and the 3dMark score of a GeForce box was 3x that of a non.
;-)
I think benchmarking these days is almost trivial. The scoring difference between a 9800 Pro and a 5900 Ultra, in the end, will only mean about a 15fps difference in your gaming experience. Honestly now, does playing UT2003 at 60fps vs. 45fps really pump your nuts? if so, then you can go ahead and moderate this as flamebait.
And as far as 'optomizing code' for benchmarks, it's industry wide. Intel releases custom compilers just so programs will run faster on P4 chips! Is that cheating? Not really. The programs still run the same, just better on the hardware they chose. Same with nVidia in this situation, the picture still LOOKED the same (unless you enabled the free viewing). So who cares what happens in the background?
My point is, people need to make decisions on their own when it comes to purchasing hardware. It all boils down to personal choice. Some people are hardcore ATI fans no matter what the benchmarks say, others are nVidia fans until the bitter end.
Personally, I choose nVidia because of hardware compatibility issues in the past with several chipsets i used to have, now it's just habitual. People who are on the fence and really don't have their feet in the water when it comes to hardware might be sold by the gold PCB.
In the end, well, it boils down to this. You know what they say about opinions
Rewriting shaders behind an application's back in a way that changes the output under non-controlled circumstances is absolutely, positively wrong and indefensible.
Rewriting a shader so that it does exactly the same thing, but in a more efficient way, is generally acceptable compiler optimization, but there is a range of defensibility from completely generic instruction scheduling that helps almost everyone, to exact shader comparisons that only help one specific application. Full shader comparisons are morally grungy, but not deeply evil.
The significant issue that clouds current ATI / Nvidia comparisons is fragment shader precision. Nvidia can work at 12 bit integer, 16 bit float, and 32 bit float. ATI works only at 24 bit float. There isn't actually a mode where they can be exactly compared. DX9 and ARB_fragment_program assume 32 bit float operation, and ATI just converts everything to 24 bit. For just about any given set of operations, the Nvidia card operating at 16 bit float will be faster than the ATI, while the Nvidia operating at 32 bit float will be slower. When DOOM runs the NV30 specific fragment shader, it is faster than the ATI, while if they both run the ARB2 shader, the ATI is faster.
When the output goes to a normal 32 bit framebuffer, as all current tests do, it is possible for Nvidia to analyze data flow from textures, constants, and attributes, and change many 32 bit operations to 16 or even 12 bit operations with absolutely no loss of quality or functionality. This is completely acceptable, and will benefit all applications, but will almost certainly induce hard to find bugs in the shader compiler. You can really go overboard with this -- if you wanted every last possible precision savings, you would need to examine texture dimensions and track vertex buffer data ranges for each shader binding. That would be a really poor architectural decision, but benchmark pressure pushes vendors to such lengths if they avoid outright cheating. If really aggressive compiler optimizations are implemented, I hope they include a hint or pragma for "debug mode" that skips all the optimizations.
John Carmack
Seriously, it seems like FutureMark pissed nVidia off, so nVidia tried to get back at them by cheating, and now FM's trying to turn everyone against nVidia for cheating, when ATI did the same thing, but ATI pays them money to be in their beta program, so who cares? These things are worthless anyway, the FX5900 still outperformed the 9800 in most of the real world tests, and they were pretty much neck and neck otherwise, 3dMark is supposed to show how well cards perform in real situations, but from the last batch of card reviews I've read, it seems that it doesn't reflect at all on real world performance.
[breathe]
Now, as regrading to releasing the specs for their chips, well... Yank my willy and call me silly, but I can't think of a single good reason.
It's not a real game. People don't "play" 3DMark. I think part of the point of 3DMark was to show how games will perform on a particular card, but there are already in-game benchmarks that will show you more accurately how a game will perform on a particular card. And it is getting clearer that 3DMark doesn't reflect how a card will actually do on games in general since 3DMark scores are not correlated with game performance. Ideally, 3DMark performance should shadow game performance. The fact that it isn't would imply that 3DMark doesn't accurately represent how a game will perform. Sounds like 3DMark is a concept whos time has past.
That hasn't prevented anyone from doing it with binary drivers either.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
Nice explanation, care to explain how to blindside a company & community on an unfair benchmark on the same day nVidia was pimping their fraudulently inflated 3dm2k3 scores?
:mad:
:(
That wasn't the big story that day, the big story was how the 5900 FX "creamed" the 9800 Pro at a secretely released benchmark...secretely released to nVidia.
Was it a karma thing over the alpha leak? That's about the only reason I could understand/forgive.
- "When I say dance, you'd best DANCE motherf*cker!" -Violent Femmes
I seriously considered an ATI card knowing that Nvidia was cheating.
Now that I know they are both cheating, I will stick with my geforce 4 a little while longer.
Stop cheating, Both of you!
less time cheating, means more work innovating!
a frame rate that exceeds the refresh rate of the display. Why on earth would I upgrade? I can't see smaller pixels, faster framerate is irrelevant
:)
A few people have mentioned to me that an FPS higher than the vertical refresh is irrelevant. I've always been confused at that, because I can always tell when something changed that caused my CS/Q3 FPS to drop from 250 to 90, even though my monitor only does 85hz.
I can't see it, I can feel it. And I'll take the pepsi challenge any day and show that I can tell the difference. All true gamers can. It's like when Neo saw everything turn green.
# Erik
... DirectX9 becomes a real standard and available for Linux. Na, thats a lie. Even if this was so, I still would'nt give a shit.
If the writers of the benchmarks actually modeled the real world uses then this would not even be an issue as the "tweaked" drivers would be tweaked for real world applications. Since the most common use of the benchmarks is to make purchasing decisions, the writers of the benchmarks are falling down on the job if they don't closely model the real world applications use. As far as recognizing the sequences of operations to determine if it is the benchmark, the benchmark can render the test screens in such a way that statistically the results are valid even if the sequence varies from run to run. (but please keep a standard sequence to help all of us debug the hardware in the first place). When a driver is being optimized they most likely run the benchmarks as a repeatable means of developing call history, etc. while profiling the code. After all you want to spend 80% of your effort optimizing the most frequently use 20% of the code (or whatever you favorite 90/10 or 80/20 like rule is).
- Tjp
I am in wallow with my inner money grubbing capitalistic pig. ... Oink!
ARB may require 24 bits as a minimum but that has nothing to do with assumed shader precision. What he's getting at Nvidia can't do 24 bit, only better or worse than 24 bit.
Probably there is a very simple reason for that which I just cannot think of right now, but why don't the 3dmark's/[fill in your favourite benchmark] people not have a radomizing function create an arbitrary filename at install time?
For example, they could use only alphanumeric characters, maybe with some rules to make it look more natural and plausible, choose the timerticks as the seed, set just the start menu entry to the right file and there - off you go. It should be very hard for drivers to work out a detection scheme for that.
Of course, if a driver detects this sort of stuff by checking for special sequences, this won't work. But I think it would be very difficult and decrease the actual performance considerably, so it would just not be worth it.
I found it interesting that ATI claimed they simply "optimized" the code by simply shuffling around a few instructions while still getting the same results. That may even be so, but obviously they made this optimisation only for the very specific case of one of 3dmarks benchmarks, else it would have worked as well with the slightly altered drivers as well. Will their engineers look at all and any game out there and optimize that code too or will they come over to my house and write a new driver on the fly when i need one? No? Well then, since i don't benefit from their optimisation in any real-world szenario and it only serves to boost their score a little. In the real world their graphics card will have to deal with suboptimal drivers as well, if they want to improve the situation they should give out a few guidelines to game-developers how to write a fast engine.
This isn't about Nvidia vs. ATI or about defending Nvidia, what NV did by clipping planes was even worse. It's just that there is no justification for cheating on the benchmarks, even if the graphical results are the same. The benchmarks should be an indicator how the card will perform in real-world-szenarios (i.e. games) and any driver-tweaks that are benchmark-specific but don't help performance otherwise are just cheating and make-believe.
"By the way if anyone here is in advertising or marketing... kill yourself." -- Bill Hicks
ATI has played it smart this round. The minimum precision for DX9 is 24 bits and ATI does it.
Hmmm... Pie...
It seems the lesson they learned, wasn't "don't cheat", but "damn we were found, my bad, we'll fix it ASAP".
Buy a console and you don't have to dick around with hardware instead of playing games.
Different precision. How much people is telling everyone about this.
Now the sentence comes from a VIP. I wonder if people will still argue about this.
I feel it like some sort of liberation...
Hoping this is not off-topic.
Perhaps if you spent less time on Slashdot and more time on Duke Nukem Forever, it would be on the shelves. I don't mean to be insulting. After all I'm a huge fan of your work with Half Life and Unreal, but I've been waiting *forever* for DNF and its just vaporware. Do you know - for certain this time - when it will be released?
C - A language that combines the speed of assembly with the ease of use of assembly.
i just heard about disposable dvds. it sounds interesting. does anyone know were i can get some more inforamton...website, trade publication, etc.