Carmack On ATI's Driver Modifications

CitizenC points out that John Carmack's .plan file has been updated to discuss ATI's driver optimizations. If you weren't paying attention, ATI put code in their drivers to optimize for Quake3, based on the name of the executable - so when running Quake3, you'd get a (good) set of optimizations for the game, but when running the same game after changing the name of the executable, you'd get a default set of optimizations with lesser performance. Some people called this cheating since Quake3 is a typical benchmark application these days.

Comments

Carmack's Conclusion

[mESSDan]

Basically, here's the gist of what Carmack said:

Uhh, it's bad, but don't quote me on that.

To boil it all down...

[trilucid]

"Making any automatic optimization based on a benchmark name is wrong. It subverts the purpose of benchmarking, which is to gauge how a similar class of applications will perform on a tested configuration, not just how the single application chosen as representative performs.

It is never acceptable to have the driver automatically make a conformance tradeoff, even if they are positive that it won't make any difference. The reason is that applications evolve, and there is no guarantee that a future release won't have different assumptions, causing the upgrade to misbehave. We have seen this in practice with Quake3 and derivatives, where vendors assumed something about what may or may not be enabled during a compiled vertex array call. Most of these are just mistakes, or, occasionally, laziness."

Carmack seems pretty well decided on this one, and not in favor of it. He *does* show a bit of support for having super-ultra-tweak control panels on driver config screens, but that's (almost) an entirely different matter.

It seems pretty cut and dried, at least from his perspective. I for one have got to agree with this viewpoint. Of course, to anyone who has said or will say that you should just rename the executable, I completely agree that this is the only real way of getting an objective test out of it.

Card manufacturers who do this sort of thing *will* get egg on their faces when they start hearing all about the crappy performance of their cards after a new game version comes out with different thinking on what's important visually. Unfortunately, as John points out, a lot of the flack will end up on the developers' doorsteps (misplaced, but a lot of gamers won't know that).

Poor behavior in a rough industry

[brianvan]

It's not surprising to see ATI do something like this. In the business world, this kind of thing may pass off as slightly unethical, but for the target market of graphics card vendors, this is just plain #$*&'ed up. It's quite true that Quake 3 Arena is a very standard benchmarking application nowadays, so any performance gain in the driver for that specific application that does not apply to all programs in general will mislead the consumer in making a purchasing decision. An analogy: no one would appreciate it if a company said their car goes from 0 to 60 in 3.2 seconds and in reality it takes 3.2 seconds just to make it up to 20.

Carmack's right, this whole driver situation in general is a slippery slope. One of the biggest hassles of his job since Quake (#1) came out has been to get all the graphics card companies to play nice and write good drivers. This has not happened for one second. Some companies are better than others (Nvidia is an example of one of the better companies in the field all around), and the situation right now is FAR better than it was. But, between the race for speed, trying to keep up with the most popular applications, and having to support various APIs being pushed by different 500 lb gorillas in the field... it's a mess. ATI was ALWAYS a mess with their drivers, so this doesn't really surprise me. But this isn't right, cause it'll lead to a day when nothing but the lastest Quake game works on new graphics cards. I mean, nothing works AT ALL. And if it's easy enough to get to that situation, I can just see bribes and payoffs... ahem, "strategic partnerships" being made to accomodate game vendors who want to publish a working 3D game at some point in time...

But anyway...

One small comfort is that no one buys ATI cards for performance. Any cards of the GeForce (Nvidia's brand) variety handily whip anything at the same price point with ATI. The big thing ATI has is OEM agreements, and they also sell some really exotic TV/Video Capture/MPEG recording cards that are really snazzy sometimes. I know cause I'm using one of em right now. But their drivers suck in a lot of ways, they were never the fastest, and I'd love to see them stick to a product release schedule EVER. This is the kind of company that gets wiped out when someone new on the scene releases something better/faster/cheaper.

Unfortunately, the last time we saw better/faster/cheaper in the graphics industry was five years ago. Nowadays, you usually get one of those improvements in a new release video card... never all three.

Re:Poor behavior in a rough industry

[Hollins]

One small comfort is that no one buys ATI cards for performance. Any cards of the GeForce (Nvidia's brand) variety handily whip anything at the same price point with ATI.

This really isn't true any longer. ATI finally released better drivers for the 8500 this week and it keeps up fine with a GeForce3 Ti 500, for $50 less. Here's a review at Tom's Hardware.

Re:Poor behavior in a rough industry

[Namarrgon]

The two problems with ATI's "optimisations" and their quality tradeoffs are:

a) There's no way to turn them off, except by hex-editing the app. They happen automatically, and without the player (or reviewer) even realising, especially in the high-speed benchmark mode.

b) This is not just any old game, not even a particularly heavily played game these days. Its major importance is as the #1 benchmark used by gaming sites.

The conclusion is inescapable. This "optimisation" was not made for players, it was made to subvert benchmarks, pure & simple.

And if you claim to prefer a higher image quality, take a look at what ATI has actually done to the visual quality of the game!

Re:Tell me again: How is this bad?

[Mark J Tilford]

What they did was not "Have Quake run in mode which is faster" but "Have Quake run in mode which has lower quality than it should so it will run faster"

just rename everything ...

[freaker_TuC]

So I need to rename all my games, Simcity, Theme Hospital, UT, to quake3.exe to get a optimized version ? :o)

cd \sc2000
ren simcity.exe quake3.exe
cd \hospital
ren theme.exe quake3.exe
cd \ut
ren unreal.exe quake3.exe
...

(already can see it happen!)

Re:Tell me again: How is this bad?

[Penrif]

Let me get this straight. ATI takes the time to create optimizations in their drivers that make Quake III run faster, and people are unhappy about it?

You're damn right I am. As someone who uses graphics hardware for scientific visualization, any time hardware manufactures spend on making games faster is a complete waste. With the latest cards and drivers from ATI, we've seen this problem en masse. Games run great, but code that we write (that's general OpenGL and works great on GeForce cards) runs slower than hell on the 8500.

So, if all ATI wants to do is sell cards to gamers (and that's certainly a good market), they can go right on ahead and do that. Nvidia takes more time with their drivers and makes them much more optimized for the general case...so they've got my buisness for now.

Just my thoughts. I'm going to skip the part where I tell everyone else that they're being knee-jerk.

LINKS.

[leuk_he]

If you missed the discussion in the first place.

The Register : As we say, if you like the 8500's Quake III frame-rate but aren't willing to put up with the dip in image quality, buy a different card. Or wait for ATI to change its drivers, which, we understand, it's in the process of doing.

HardOCP was the first to publish about this: The Facts As We See Them: It certainly seems to us here at [H]ardOCP that ATi has in fact included application specific instructions in their version 5.13.01.3276 Win2K drivers that make Quake 3 arena benchmarks faster by up to over 15%. Either way, the driver optimisations for Quake III are just one of the (many) factors that differentiate different vendors' products. ®

firingsquad show show some details how the quack.exe is made and concludes:
To some of us, it seems like the evidence points towards intentionally deceptive code designed not only to inflate benchmark scores, but also to keep anyone from finding out. To others, this is nothing more than an overreation to a perfectly legitimate game optimization. In our eyes, anyone who vehemently peddles either of these explanations is either naive or pushing an agenda of their own.

there later in Q&A with ati explains in 2 pages that :Our goal for the RADEON 8500 is and always has been to deliver the best possible gaming experience to our customers.

yeah right.!

ATI had the right to do it but it's HOW that's bad

[tcc]

If they would have DOCUMENTED the changes, and put a button in the control pannel (like where all the antialiasing on off, buffer level and all is) it would have been legit, they could have put it in their press release and send them around WITHOUT getting any flames.

There are ways to do things, you can bend the rules, but breaking it will only get fire back. I'm all for specific-engine optimization, and like carmack says, Conformance trade-offs shouldn't be made at the driver level in general, but to this I'll say, if it's documented and available as an OPTION somewhere, it won't apply only to techies, it can be applied to joe schmoe as well "if you click here, game will run faster with a small quality degradation (that you will probably not even notice because at the speed of events in quake 3)". Heck if they can put something like Quancun Aliasing in the Panel options, you can be sure people will understand specific game optimization as well.

Anyways forcing something to the users without their knowledge is just plain bad and lame. Of course people will end up finding out, and of course it will backfire... what is gain from doing that!? there sure is poor judgement at ATI, and there's probably a bunch of people there right now saying "see? told you so!". Management, you should listen to the people in the lower part of your food chain (i.e. QA testers), they might know more about the end-user market and hardware benchmarkarking scene (not the OEM of course) than you do!).

Re:.plan

[Malc]

1) It is a web site, hence the URL in the story worked.

2) I think it's kind of cool that somebody is still using their .plan file they way they are supposed to.

Quality is lowered to boost benchmarks

[Namarrgon]

If ATI had just optimised for a specific game, I think nearly everyone would be fine with that. Perhaps the only objection you could make is that they might have spent their time optimising for the Quake3 engine instead, covering a wider range of games and benefiting more users. When that game is a benchmark (or rather, THE primary benchmark used to determine real-world performance under a set of well-known and duplicatable conditions), then optimisations specific to that are a little more questionable.

While many companies focus upon optimising for benchmarks, most simply optimise the driver paths for the specific cases that those benchmarks use. Any other app that uses similar settings can gain performance from that work. But ATI have made their optimisations dependant upon the name of the app, so no other apps can benefit from their work. While a Quake3 player might not mind, Q3 isn't as widely played today as it once was, and that same player might be less pleased when the card fails to perform to the same standard on any other of their games.

But what makes this particular "optimisation" underhanded is that it's not better or more tuned code, it trades off quality. Have a look here to see the mess that it makes of textures!

Now, if players wanted to see blurry textures in exchange for more performance, they'd simply lower the texture quality slider in the Quake3 game. ATI's drivers do this for them; they're forcing the mipmaps two levels down - a 16x reduction in texture detail - to get the extra speed. And this isn't optional. You can't turn it off, short of using a hex editor.

Particularly, a reviewer running the standard Quake3 High Quality benchmark will never notice the difference (as the frames run by way too fast). There's no extra sliders in the driver, no other indications or switches, and of course no notification in the driver documentation. All the reviewer sees is higher framerates, because the drivers are, quite literally, cheating. They're giving low-texture numbers on the high-texture setting.

I for one applaud ATI's renewed efforts to improve their drivers (and I still plan to buy an 8500DV alongside my existing QuadroDCC), but I feel ATI really have attempted to subvert the benchmark process, and shot themselves in the foot. The strong implication is that, even though their hardware is fine, they don't feel they can compete with nVidia's driver team so they have to resort to methods like these.

Very unwise - they've lost a lot of the support they had as the underdog to nVidia, through these tactics.

Re:Why NVIDIA's drivers are closed source.

[Animats]

There's another reason. NVidia has two product lines, the "GEForce" for consumers, and the "Quadro" for professionals. The same chips are used for both, there's a jumper on the card that identifies the model, and the driver turns off some features on the consumer model.

Re:I might start supporting ATi...

[Junta]

Well, actually, while ATI does not actively support Linux themselves, they do release more information about their chips than nVidia does. Because of this, third party driver developers can develop open-source, cross-platform drivers, which with nVidia, you are stuck with Linux on x86. So while nVidia invests more development time, ATI gives more information and therefore has more well-developed drivers on more platforms. Their linux driver implementation may not be as good as nVidias, but will be around even if ATI went out. If nVidia goes, and all we have are the binary drivers compiled for Kernel 2.4.x and XFree 4.1.x, then nVidia owners are in a bit of a fix....