Carmack On ATI's Driver Modifications

CitizenC points out that John Carmack's .plan file has been updated to discuss ATI's driver optimizations. If you weren't paying attention, ATI put code in their drivers to optimize for Quake3, based on the name of the executable - so when running Quake3, you'd get a (good) set of optimizations for the game, but when running the same game after changing the name of the executable, you'd get a default set of optimizations with lesser performance. Some people called this cheating since Quake3 is a typical benchmark application these days.

Comments

Poor behavior in a rough industry

[brianvan]

It's not surprising to see ATI do something like this. In the business world, this kind of thing may pass off as slightly unethical, but for the target market of graphics card vendors, this is just plain #$*&'ed up. It's quite true that Quake 3 Arena is a very standard benchmarking application nowadays, so any performance gain in the driver for that specific application that does not apply to all programs in general will mislead the consumer in making a purchasing decision. An analogy: no one would appreciate it if a company said their car goes from 0 to 60 in 3.2 seconds and in reality it takes 3.2 seconds just to make it up to 20.

Carmack's right, this whole driver situation in general is a slippery slope. One of the biggest hassles of his job since Quake (#1) came out has been to get all the graphics card companies to play nice and write good drivers. This has not happened for one second. Some companies are better than others (Nvidia is an example of one of the better companies in the field all around), and the situation right now is FAR better than it was. But, between the race for speed, trying to keep up with the most popular applications, and having to support various APIs being pushed by different 500 lb gorillas in the field... it's a mess. ATI was ALWAYS a mess with their drivers, so this doesn't really surprise me. But this isn't right, cause it'll lead to a day when nothing but the lastest Quake game works on new graphics cards. I mean, nothing works AT ALL. And if it's easy enough to get to that situation, I can just see bribes and payoffs... ahem, "strategic partnerships" being made to accomodate game vendors who want to publish a working 3D game at some point in time...

But anyway...

One small comfort is that no one buys ATI cards for performance. Any cards of the GeForce (Nvidia's brand) variety handily whip anything at the same price point with ATI. The big thing ATI has is OEM agreements, and they also sell some really exotic TV/Video Capture/MPEG recording cards that are really snazzy sometimes. I know cause I'm using one of em right now. But their drivers suck in a lot of ways, they were never the fastest, and I'd love to see them stick to a product release schedule EVER. This is the kind of company that gets wiped out when someone new on the scene releases something better/faster/cheaper.

Unfortunately, the last time we saw better/faster/cheaper in the graphics industry was five years ago. Nowadays, you usually get one of those improvements in a new release video card... never all three.

Quality is lowered to boost benchmarks

[Namarrgon]

If ATI had just optimised for a specific game, I think nearly everyone would be fine with that. Perhaps the only objection you could make is that they might have spent their time optimising for the Quake3 engine instead, covering a wider range of games and benefiting more users. When that game is a benchmark (or rather, THE primary benchmark used to determine real-world performance under a set of well-known and duplicatable conditions), then optimisations specific to that are a little more questionable.

While many companies focus upon optimising for benchmarks, most simply optimise the driver paths for the specific cases that those benchmarks use. Any other app that uses similar settings can gain performance from that work. But ATI have made their optimisations dependant upon the name of the app, so no other apps can benefit from their work. While a Quake3 player might not mind, Q3 isn't as widely played today as it once was, and that same player might be less pleased when the card fails to perform to the same standard on any other of their games.

But what makes this particular "optimisation" underhanded is that it's not better or more tuned code, it trades off quality. Have a look here to see the mess that it makes of textures!

Now, if players wanted to see blurry textures in exchange for more performance, they'd simply lower the texture quality slider in the Quake3 game. ATI's drivers do this for them; they're forcing the mipmaps two levels down - a 16x reduction in texture detail - to get the extra speed. And this isn't optional. You can't turn it off, short of using a hex editor.

Particularly, a reviewer running the standard Quake3 High Quality benchmark will never notice the difference (as the frames run by way too fast). There's no extra sliders in the driver, no other indications or switches, and of course no notification in the driver documentation. All the reviewer sees is higher framerates, because the drivers are, quite literally, cheating. They're giving low-texture numbers on the high-texture setting.

I for one applaud ATI's renewed efforts to improve their drivers (and I still plan to buy an 8500DV alongside my existing QuadroDCC), but I feel ATI really have attempted to subvert the benchmark process, and shot themselves in the foot. The strong implication is that, even though their hardware is fine, they don't feel they can compete with nVidia's driver team so they have to resort to methods like these.

Very unwise - they've lost a lot of the support they had as the underdog to nVidia, through these tactics.