Slashdot Mirror
Carmack On ATI's Driver Modifications
CitizenC points out that John Carmack's .plan file has been updated to discuss ATI's driver optimizations. If you weren't paying attention, ATI put code in their drivers to optimize for Quake3, based on the name of the executable - so when running Quake3, you'd get a (good) set of optimizations for the game, but when running the same game after changing the name of the executable, you'd get a default set of optimizations with lesser performance. Some people called this cheating since Quake3 is a typical benchmark application these days.
Basically, here's the gist of what Carmack said:
Uhh, it's bad, but don't quote me on that.
-- Dan
"Making any automatic optimization based on a benchmark name is wrong. It subverts the purpose of benchmarking, which is to gauge how a similar class of applications will perform on a tested configuration, not just how the single application chosen as representative performs.
It is never acceptable to have the driver automatically make a conformance tradeoff, even if they are positive that it won't make any difference. The reason is that applications evolve, and there is no guarantee that a future release won't have different assumptions, causing the upgrade to misbehave. We have seen this in practice with Quake3 and derivatives, where vendors assumed something about what may or may not be enabled during a compiled vertex array call. Most of these are just mistakes, or, occasionally, laziness."
Carmack seems pretty well decided on this one, and not in favor of it. He *does* show a bit of support for having super-ultra-tweak control panels on driver config screens, but that's (almost) an entirely different matter.
It seems pretty cut and dried, at least from his perspective. I for one have got to agree with this viewpoint. Of course, to anyone who has said or will say that you should just rename the executable, I completely agree that this is the only real way of getting an objective test out of it.
Card manufacturers who do this sort of thing *will* get egg on their faces when they start hearing all about the crappy performance of their cards after a new game version comes out with different thinking on what's important visually. Unfortunately, as John points out, a lot of the flack will end up on the developers' doorsteps (misplaced, but a lot of gamers won't know that).
The driver is just software. When Quake3 asks it to do something for the first time, it simply checks the filename attached to the process handle and sets a flag accordingly.
If a job's not worth doing, it's not worth doing right.
It's not surprising to see ATI do something like this. In the business world, this kind of thing may pass off as slightly unethical, but for the target market of graphics card vendors, this is just plain #$*&'ed up. It's quite true that Quake 3 Arena is a very standard benchmarking application nowadays, so any performance gain in the driver for that specific application that does not apply to all programs in general will mislead the consumer in making a purchasing decision. An analogy: no one would appreciate it if a company said their car goes from 0 to 60 in 3.2 seconds and in reality it takes 3.2 seconds just to make it up to 20.
Carmack's right, this whole driver situation in general is a slippery slope. One of the biggest hassles of his job since Quake (#1) came out has been to get all the graphics card companies to play nice and write good drivers. This has not happened for one second. Some companies are better than others (Nvidia is an example of one of the better companies in the field all around), and the situation right now is FAR better than it was. But, between the race for speed, trying to keep up with the most popular applications, and having to support various APIs being pushed by different 500 lb gorillas in the field... it's a mess. ATI was ALWAYS a mess with their drivers, so this doesn't really surprise me. But this isn't right, cause it'll lead to a day when nothing but the lastest Quake game works on new graphics cards. I mean, nothing works AT ALL. And if it's easy enough to get to that situation, I can just see bribes and payoffs... ahem, "strategic partnerships" being made to accomodate game vendors who want to publish a working 3D game at some point in time...
But anyway...
One small comfort is that no one buys ATI cards for performance. Any cards of the GeForce (Nvidia's brand) variety handily whip anything at the same price point with ATI. The big thing ATI has is OEM agreements, and they also sell some really exotic TV/Video Capture/MPEG recording cards that are really snazzy sometimes. I know cause I'm using one of em right now. But their drivers suck in a lot of ways, they were never the fastest, and I'd love to see them stick to a product release schedule EVER. This is the kind of company that gets wiped out when someone new on the scene releases something better/faster/cheaper.
Unfortunately, the last time we saw better/faster/cheaper in the graphics industry was five years ago. Nowadays, you usually get one of those improvements in a new release video card... never all three.
What they did was not "Have Quake run in mode which is faster" but "Have Quake run in mode which has lower quality than it should so it will run faster"
-----------
100% pure freak
So I need to rename all my games, Simcity, Theme Hospital, UT, to quake3.exe to get a optimized version ? :o)
cd \sc2000
ren simcity.exe quake3.exe
cd \hospital
ren theme.exe quake3.exe
cd \ut
ren unreal.exe quake3.exe
...
(already can see it happen!)
--- I am known for the ones who want to find me on the net. Is that a privacy risk or a privilege? One might wonder..
To all you OSS zealots out there, *this* is why NVIDIA's drivers are closed source. You can bet ATI would love to put this whole driver fiasco behind them and just steal the high level OpenGL code (an OpenGL driver has to implement the whole GL API, not just hardware interfacing) from NVIDIA's ICD.
A deep unwavering belief is a sure sign you're missing something...
If they were optimizing for Quake 3 because that is where their largest audience was, and therefore they could satisfy the greatest number of their customers in that way, then yes, I would have a hard time faulting their actions from the standpoint of the free market(although I still wouldn't buy an ATI card, because that is a stupid engineering decision). However, this is not why they optimized for Quake 3.
Confining the dicussion solely to first-person shooters, it should be pointed out that Quake 3 is *not* the most popular game in this genre. Gamespy.com tracks playing statistics -- the numbers they display for today have been consistent for some months(excepting the addition of RtCW). I presume Carmack didn't want to dwell on this at great length in his .plan, and who can blame him? Quake 3's failures are not failures of technology(for which he is responsible) but of gameplay(for which he is not).
The reason that ATI optimized for Quake 3 is that it is used as a benchmarking tool. Their expectation, probably, is that gamers will see relatively high benchmarks for Quake 3 in reviews of ATI cards, and generalize from that to assume that all OpenGL games will enjoy excellent performance on ATI's card. This is likely a false assumption for anything other than Quake 3 or a Quake 3 mod. (Even games based on the Quake 3 engine will not share the performance benefits, unless you rename their executable -- and possibly not even then, depending on how modified the engine is.)
So in other words, the problem is not that ATI has cheated on their drivers to please the massive Quake 3 community. The problem is that they have cheated on their drivers to deceive reviewers, and(they hope) customers who read reviews. This is "self-interest," to be sure, but I do not find it to be "enlightened."
-- Support Ometz le-Serev.
Let me get this straight. ATI takes the time to create optimizations in their drivers that make Quake III run faster, and people are unhappy about it?
You're damn right I am. As someone who uses graphics hardware for scientific visualization, any time hardware manufactures spend on making games faster is a complete waste. With the latest cards and drivers from ATI, we've seen this problem en masse. Games run great, but code that we write (that's general OpenGL and works great on GeForce cards) runs slower than hell on the 8500.
So, if all ATI wants to do is sell cards to gamers (and that's certainly a good market), they can go right on ahead and do that. Nvidia takes more time with their drivers and makes them much more optimized for the general case...so they've got my buisness for now.
Just my thoughts. I'm going to skip the part where I tell everyone else that they're being knee-jerk.
A lot of people still don't realize, what ATI's "optimizations" are about. It's not a optimization specific to Quake 3 which noone could complain about. ATI's drivers forced some features to be disabled massively sacrificing image quality only when Quake 3 is run. It's not clear what they did exactly, but it looks like they were forcing 16-bit textures and/or using lower-resolution mipmaps. See the results here. (The site is in German, but they have some nice details of screenshots, showing the difference onmouseover.)
If you missed the discussion in the first place.
:Our goal for the RADEON 8500 is and always has been to deliver the best possible gaming experience to our customers.
The Register : As we say, if you like the 8500's Quake III frame-rate but aren't willing to put up with the dip in image quality, buy a different card. Or wait for ATI to change its drivers, which, we understand, it's in the process of doing.
HardOCP was the first to publish about this: The Facts As We See Them: It certainly seems to us here at [H]ardOCP that ATi has in fact included application specific instructions in their version 5.13.01.3276 Win2K drivers that make Quake 3 arena benchmarks faster by up to over 15%. Either way, the driver optimisations for Quake III are just one of the (many) factors that differentiate different vendors' products. ®
firingsquad show show some details how the quack.exe is made and concludes:
To some of us, it seems like the evidence points towards intentionally deceptive code designed not only to inflate benchmark scores, but also to keep anyone from finding out. To others, this is nothing more than an overreation to a perfectly legitimate game optimization. In our eyes, anyone who vehemently peddles either of these explanations is either naive or pushing an agenda of their own.
there later in Q&A with ati explains in 2 pages that
yeah right.!
If they would have DOCUMENTED the changes, and put a button in the control pannel (like where all the antialiasing on off, buffer level and all is) it would have been legit, they could have put it in their press release and send them around WITHOUT getting any flames.
There are ways to do things, you can bend the rules, but breaking it will only get fire back. I'm all for specific-engine optimization, and like carmack says, Conformance trade-offs shouldn't be made at the driver level in general, but to this I'll say, if it's documented and available as an OPTION somewhere, it won't apply only to techies, it can be applied to joe schmoe as well "if you click here, game will run faster with a small quality degradation (that you will probably not even notice because at the speed of events in quake 3)". Heck if they can put something like Quancun Aliasing in the Panel options, you can be sure people will understand specific game optimization as well.
Anyways forcing something to the users without their knowledge is just plain bad and lame. Of course people will end up finding out, and of course it will backfire... what is gain from doing that!? there sure is poor judgement at ATI, and there's probably a bunch of people there right now saying "see? told you so!". Management, you should listen to the people in the lower part of your food chain (i.e. QA testers), they might know more about the end-user market and hardware benchmarkarking scene (not the OEM of course) than you do!).
--- Metamoderating abusive downgraders since my 300th post.
Anand did a new look at the 8500 with the new ATI drivers. He claims that the image quality is now identical to the Geforce3 and that the driver optimizations work for all Quake3 engine based games. Anandtech article
1) It is a web site, hence the URL in the story worked.
.plan file they way they are supposed to.
2) I think it's kind of cool that somebody is still using their
While many companies focus upon optimising for benchmarks, most simply optimise the driver paths for the specific cases that those benchmarks use. Any other app that uses similar settings can gain performance from that work. But ATI have made their optimisations dependant upon the name of the app, so no other apps can benefit from their work. While a Quake3 player might not mind, Q3 isn't as widely played today as it once was, and that same player might be less pleased when the card fails to perform to the same standard on any other of their games.
But what makes this particular "optimisation" underhanded is that it's not better or more tuned code, it trades off quality. Have a look here to see the mess that it makes of textures!
Now, if players wanted to see blurry textures in exchange for more performance, they'd simply lower the texture quality slider in the Quake3 game. ATI's drivers do this for them; they're forcing the mipmaps two levels down - a 16x reduction in texture detail - to get the extra speed. And this isn't optional. You can't turn it off, short of using a hex editor.
Particularly, a reviewer running the standard Quake3 High Quality benchmark will never notice the difference (as the frames run by way too fast). There's no extra sliders in the driver, no other indications or switches, and of course no notification in the driver documentation. All the reviewer sees is higher framerates, because the drivers are, quite literally, cheating. They're giving low-texture numbers on the high-texture setting.
I for one applaud ATI's renewed efforts to improve their drivers (and I still plan to buy an 8500DV alongside my existing QuadroDCC), but I feel ATI really have attempted to subvert the benchmark process, and shot themselves in the foot. The strong implication is that, even though their hardware is fine, they don't feel they can compete with nVidia's driver team so they have to resort to methods like these.
Very unwise - they've lost a lot of the support they had as the underdog to nVidia, through these tactics.
Why would anyone engrave "Elbereth"?
Well, actually, while ATI does not actively support Linux themselves, they do release more information about their chips than nVidia does. Because of this, third party driver developers can develop open-source, cross-platform drivers, which with nVidia, you are stuck with Linux on x86. So while nVidia invests more development time, ATI gives more information and therefore has more well-developed drivers on more platforms. Their linux driver implementation may not be as good as nVidias, but will be around even if ATI went out. If nVidia goes, and all we have are the binary drivers compiled for Kernel 2.4.x and XFree 4.1.x, then nVidia owners are in a bit of a fix....
XML is like violence. If it doesn't solve the problem, use more.