Twilight of the GPU — an Interview With Tim Sweeney

cecom writes to share that Tim Sweeney, co-founder of Epic Games and the main brain behind the Unreal engine, recently sat down at NVIDIA's NVISION con to share his thoughts on the rise and (what he says is) the impending fall of the GPU: "...a fall that he maintains will also sound the death knell for graphics APIs like Microsoft's DirectX and the venerable, SGI-authored OpenGL. Game engine writers will, Sweeney explains, be faced with a C compiler, a blank text editor, and a stifling array of possibilities for bending a new generation of general-purpose, data-parallel hardware toward the task of putting pixels on a screen."

Comments

Software rendering on the GPU

[KalvinB]

Somehow I don't think there's going to be lack of a standardized API much like OpenGL or DirectX even if it's possible to write code for the GPU as easily as the CPU.

The APIs at the most basic level allow Joe and Bob to build their own system, throw whatever graphics card they can find in it and have the same game run on both systems.

As soon as you start coding for a specific GPU you're going to be treating PCs like consoles. I don't care to have to buy multiple graphics cards to play various games.

APIs are for compatibility and general purpose use. The option of flexibility is great for industry use where you're running rendering farms all running identical hardware and custom software.

Re:Software rendering on the GPU

[Anonymous Coward]

Somehow I don't think there's going to be lack of a standardized API much like OpenGL or DirectX even if it's possible to write code for the GPU as easily as the CPU.

Generally, the whole point of standards is that there should be a lack of them.

Re:Software rendering on the GPU

[Hortensia Patel]

You're missing the point entirely. GPUs in the near future will essentially be general-purpose massively-multicore processors. (Not entirely so, but close enough for "GPU" to become a misnomer.)

You don't need separate binaries for Intel versus AMD processors, and you don't have to write to an API to achieve portability between such different CPUs. Given sufficiently general hardware, there's no inherent reason why GPUs should be any different. There will still be a place for libraries, but their role will be more like C libraries than like GL/DX.

you wish

[heroine]

GPUs are going to be driven by the same things that drive game consoles & set top boxes. You can copy pure software, but you can't copy dedicated chips. You can copy video rendered by a CPU but not video rendered on a dedicated chip. Dedicated chips are going to stay cheaper than CPUs for many years. Just because you can decode HDTV in software doesn't mean there still isn't huge momentum behind set top boxes and DTCP enabled GPU's.

abolish standards based programming? Never

[heroine]

Standards based programming isn't going anywhere. That's crazy. We need Direct X, OpenGL, JMF, & MHP if only to outsource large chunks of the programming to cheaper divisions. How great it would be if everyone could base their career on hand optimizing ray tracing algorithms for SSE V, but the economy would never support it. These things have to be outsourced to select groups, firewalled behind a standard & higher paid programming done to the standard.

Re:I hope not!

[Emperor Zombie]

Actually, he's saying the opposite:

TS: I expect that in the next generation we'll write 100 percent of our rendering code in a real programming language-not DirectX, not OpenGL, but a language like C++ or CUDA. A real programming language unconstrained by weird API restrictions. Whether that runs on NVIDIA hardware, Intel hardware or ATI hardware is really an independent question. You could potentially run it on any hardware that's capable of running general-purpose code efficiently.

Um, says who? I don't see it at all

[spoco2]

I just don't get how they can be saying this at all.

Ok... so from the article we have:

Take a 1999 interview with Gamespy, for instance, in which he lays out the future timeline for the development of 3D game rendering that has turned out to be remarkably prescient in hindsight:

        2006-7: CPU's become so fast and powerful that 3D hardware will be only marginally beneficial for rendering, relative to the limits of the human visual system, therefore 3D chips will likely be deemed a waste of silicon (and more expensive bus plumbing), so the world will transition back to software-driven rendering. And, at this point, there will be a new renaissance in non-traditional architectures such as voxel rendering and REYES-style microfacets, enabled by the generality of CPU's driving the rendering process. If this is a case, then the 3D hardware revolution sparked by 3dfx in 1997 will prove to only be a 10-year hiatus from the natural evolution of CPU-driven rendering.

Which they say is remarkably true.

HUH?

So, all these new GPUs really don't speed up your machine? So I can take my nvidia 8800 OUT of my box, leave it to an onboard graphics chipset and I'll be getting the same performance will I?

Yeah.

Right.

I don't see AT ALL how they're getting this right?

Please someone enlighten me, as I'm not seeing it from where I'm sitting.

Re:Um, says who? I don't see it at all

[Antitorgo]

It says it right there in the article.

Current generation GPUs are to the point where you can basically run arbitrary code on the GPU (as shaders) in high vectorizable code. So instead of the monolithic APIs such as OpenGL/DirectX, we'll see custom rendering engines that take advantage of the parallel cores running in GPUs and future CPUs (if Intel has their way).

So in a sense, he was correct. If you take CPUs in his 1999 interview to mean current GPU processors. Which is becoming somewhat of a misnomer nowadays with the CUDA/Firestream SDKs out now.

In the end, his point is that OpenGL/DirectX will probably go away in favor of custom rendering engines based on other techniques that will, in the long run, result in much better/different rendering techniques. (Ex: Voxel based rendering, Real-time ray tracing etc).

Re:Um, says who? I don't see it at all

[GameMaster]

And, I would argue that even that part of his statement is absurd. Most game developers don't even want to have to write their own game engines much less do it from scratch. Even the ones that do write engines (id, Epic, etc.) will, for the most part, never give up their graphics APIs. APIs don't just give programmers access to specialized hardware, they also provide the developer with a raft of low level tools that he/she has no, reasonable, justification or wanting to spend the time re-developing. What will happen, is that APIs like Direct3D and OpenGL will change (as they have been all along) and simplify, as they won't have to handle all the quirky hardware anymore. The idea that engine writers will completely throw out APIs and will go back to writing everything from scratch in C/C++ is almost as absurd as suggesting that they'll go back to writing everything in Assembly language.

Re:Um, says who? I don't see it at all

[Antitorgo]

Actually, I think he says in the article that you'd see APIs like Direct3D and OpenGL staying around and leveraging the CUDA/FireStream APIs instead of relying on hardware specific features (at least, that is what is already happening at the driver level).

I think he envisions that new engines/APIs will come about (such as those that could be Voxel based or based on Ray Shaders, etc.) once the CUDA/FireStream APIs become more mature and the hardware gets faster.

Also, by breaking out of the hardware specific implementations of the SGI-based implementations (OpenGL/Direct3D) it would provide more flexibility for programmers to insert their own customizations into the rendering pipeline. With the hardware designed around one specific rendering design, programmers were limited. With the evolution of GPUs turning into highly parallel GPGPUs and CPUs turning into 80+ core beasts, it opens a whole realm of possibilities.

Re:Um, says who? I don't see it at all

[FuturePastNow]

But integrated graphics aren't about gaming. They're about driving a desktop display with minimal power, heat, and cost. Believe it or not, those three things are actually pluses for non-gamers.

So no shit, a Geforce 6600 (which is what, three years old?) outperforms Intel GMA. No one at Intel cares.

Re:Um, says who? I don't see it at all

[jjohnson]

Part of Sweeney's point that you're missing is that programmers are already writing raw shader code for DX10, and making more general use of CUDA. But more than that, DX and OpenGL are APIs to a particular model of 3D rendering that was, ten years ago, a good match to the benefits of the 3D hardware presented. Now that the GPU is a general purpose processor, the triangles pipeline model for renderers need no longer limit programmers, and they can implement different engine models (like Voxels) because it's all being done in software on a general purpose chip, whether that chip is called a CPU or a GPU.

Re:For once ...

[vux984]

No, that can't be it. Know why? Because...why would you put more processing and thus more heat in one place that already has problems with that?

You mean how floating point units used to in a separate coprocessor?
Or how L2 cache used to be on external chips? (And in some cases was even upgradable.)
Or how modems used to have their own signal processors? But now most use the CPU.
Or how we're moving the memory controller into the CPU right now.

Hell, we've even stuck the majority of complete additional CPUs into the the CPU with our modern dual and quad core chips.

Apparently the author doesn't know much about computers.

Apparently you don't know much about computers either.

The entire history of the personal computer is been one long slide of functionality moving towards the CPU. Sure every now and then something new comes along being done by an add-on processor - like the numeric coprocessor for example.

Sure before the coprocessor you could accomplish the functionality of what a coprocessor does with an 'integer cpu', but a hardware optimized numerica coprocessor was a new feature, one that added tremendous floating point performance in dediated hardware. Within a couple CPU generations the coprocessor had been completely absorbed into the CPU.

The author is speculating that the GPU will see the same fate eventually. And he's probably right.

And why install an overkill graphics processing unit inside the processor if most people won't use it anyway?

Once upon a time people said that about numeric coprocessors. "Only a research scientist would need that!"

Re:For once ...

[ZorbaTHut]

Man, you've got some awful, awful arguments here.

Because...why would you put more processing and thus more heat in one place that already has problems with that?

For the same reason that your CPU isn't spread up among thirty chips distributed throughout your laptop: efficiency and cost. Making one chip is generally cheaper than making two, and the amount of bandwidth inside a single chip is massively higher than what you can do with a northbridge.

And why install an overkill graphics processing unit inside the processor if most people won't use it anyway?

Every latest-generation operating system provides a 3d accelerated desktop. Every latest-generation computer provides the hardware to use it. Programs are going to be taking more and more advantage of that feature.

And why attach it to a part that's waaaaaay harder to upgrade and usually either requires a reactivation or reinstalled of your OS?

See question 1.

And how much harder would graphics hardware driver updates be?

Not at all. For one thing, there wouldn't *be* graphics hardware - it'd be more of a vectorized coprocessor. For another thing, why *would* it be any harder? It's not like people are having horrible trouble updating their USB drivers, even if the USB controller is part of another, larger chip.

And would it overheat laptops a hell of a lot faster by putting more heat in one location? (spoiler alert: yes)

Obviously, if they took existing laptop designs, and slapped a bigger heatsource in the CPU, yes. I'm assuming that computer manufacturers aren't functionally retarded, and they wouldn't do that. (Well, maybe some would, but their computers aren't going to be stable anyway.)

And where would the VGA/DVI output go if there's no graphics card?

The same place it already goes on motherboards that have integrated graphics? It's not like "computers without dedicated graphics cards" is a new concept, unless you've been living in a cave for the last decade.

If you put it somewhere else then why move the graphics processor further away from the outputs?

See question 1. Also, "why not?" - it's not like that extra four inches is going to be a serious problem.

As near as I can tell, your argument comes down to the common logical fallacy:

"They *could* do X. But if they do it the *stupidest way possible*, X is a bad idea. Therefore, X is a bad idea."

When determining whether something is a good idea or not, you have to assume it's going to be done well. If the person in charge of integrating CPUs and GPUs is anything less than a complete unalloyed moron, they'll have come up with solutions to all of those issues of yours.

Re:Er, no

[Narishma]

You missed the point entirely. You don't write different code depending on whether the CPU is from AMD or Intel. In the same way, you won't be writing different code depending on whether the processor (CPU or GPU) is from AMD or Intel or nVidia. You will be writing in a high level language like C++ and it will be compiled to run on the CPU/GPU, not unlike CUDA. So you won't need API's like OpenGL or DirectX anymore.

Re:New market opportunity for render engines?

[mikael]

Before the advent of 3D accelerators, game companies wrote their own low-level renderers. These did the vertex transformation, lighting, clipping and texture-mapped triangle and line rasterization (some companies even explored the use of ellipsoids). Wolfenstein 3D, Quake and Descent as examples.

The low-level graphics rendering part is a very small part of the game engine - rasterizing a textured primitive with some clipping, Z-buffering and alpha blending. But getting this as fast as possible requires a good deal of profiling and analysis to get it as optimized as possible (Brian Hook did a software version of OpenGL for SGI).

3D chip makers gradually took over this area by designing hardware that could do this task far faster than the CPU. First they took over the rasterizing part (3Dfx piggyback boards), then took over the vertex transformation, lighting and clipping through the use of high performance parallel processing hardware (Nvidia TNT). There are other optimisations such as deferred rendering which optimise the order of rendering primitives to save on framebuffer writing.

Initially, all stages of the pipeline were fixed functionality, but this was replaced by programmable vertex transformation (vertex programs in assembler, then vertex shaders in a shading language) needed for matrix-blending in character animation. Fixed functionality for pixel processing was replaced by register combiners (for baked shadows), then by fragment programs and fragment shaders. Geometry shaders were also introduced to handle deformation effects.
There are also feedback features where the output of the rendering can be made to a texture, and thus used as an input texture for the next frame.

All the latest DirectX and OpenGL extensions relate to setting up the geometry/vertex/fragment shaders for this functionality.

That is what Intel and software renderers have to compete against. They would be to implement a set of 3D CPU instructions that allow textured triangles, line and points to be rendered with fully programmable shaders from one block of memory to another (specified by pixel format, width, height, etc...). They could use the memory cache for this purpose, but would have to replace the FPU with hundreds of DSPs to achieve this. Otherwise they would have to provide free access to the framebuffer with hundreds of threads or cores.

Re:Hello world without OpenGL or DirectX?

[Narishma]

Then why aren't commercial games already distributed as C++ source code so that they can be recompiled for AMD or Intel CPUs?

This doesn't make any sense. I didn't say they would release the source code. Only that they would write directly to the CPU/GPU, without using API's like OpenGL, just like they currently do for everything that is not graphics rendering.

Then what will computer graphics programming students use to write a demo of rotating Platonic polyhedra textured with "Hello World"?

The same things they used before OpenGL/DirectX became prevalent. The same things they used during the DOS era.

Re:For once ...

[kestasjk]

The key difference is that L2 cache, memory controllers, FPUs, they all need to interact closely with the CPU.
GPUs generally just take data from the CPU and get rendering, so unless they start sending data back to the CPU much more there's no real reason for them to merge.

Another difference; graphics cards benefit from being updated every now and then. Another; GPUs use a lot of transistors, and because of their parallel nature the more transistors the better, it's not something that will get so small that it can be tacked onto the CPU as a nice extra. There are probably more.

Re:I hope not!

[GleeBot]

I read it differently than you did. He's projecting a world where everything is standardized and faster (less 'bus plumbing' GPUs). In such a world you won't need APIs because you'll have libraries that you can include in the compile process.

No, you won't need APIs because you'll have a standard instruction set. Current graphics APIs are oriented around things like pushing triangles and running shaders on them. Why bother with the complexity of the graphics APIs, if you can create a standardized instruction set for GPUs?

We essentially already have that, but it's wrapped in the paradigm of shader programming. Stop treating the GPU as a graphics processor, and as just this really parallel coprocessor, and you get what Sweeney is talking about.

The basic point is that GPUs aren't really for graphics anymore, and as we move away from a graphics fixation, we'll come to realize that they're just specialized computers. Using graphics APIs to program them would be as stupid as using OpenGL to write a word processor. Sure, with modern capabilities, it's actually possible, but it's a bad conceptual fit.

Re:You are quite wrong

[Anonymous Coward]

The Wheel of Reincarnation has ground decisively to a halt, now that CPUs are going massively-multicore. The idea of special-function processing hardware is as dead as the idea of buggy whips.

Within a generation or two you'll find that multicore CPUs will do a better job of utilizing all of their silicon than dedicated GPUs. At that point there will be no turning back. nVidia and AMD are both in a world of shit, technically speaking.

What about audio?

[serodores]

How long has audio been around? Have you ever seen an audio chip integrated into the CPU? Most of them are done by onboard chips, not on the CPU.

Tim Sweeney has vested interest for this

[jokkebk]

It's also interesting to note that the guy being interviewed is in the business of making 3D engines.

Now ask the question: would 3D engine makers perhaps have something to gain if OpenGL and DirectX would be scrapped, as the interview suggests?

Most game dev labs wouldn't have the resources to build their own engines from the scratch using a C++ compiler, making them to - wait for it - licence a 3D engine like the one this guy is selling.

So in summary, the article paints a picture from the future which would be very beneficial to interviewee, so I'd take it with a grain of salt. Either we'd get some de facto 3D engines replacing OpenGL and DirectX, or the game developers will waste time recreating each new graphics technology advancement into their own engines.

Re:For once ...

[Guspaz]

If the performance degradation 5 years ago was 20-30% because the things were pumping out a ton of interrupts, one would expect the resurgence of multithreaded CPUs to further reduce the impact. They should help with context switches. When you've got eight logical cores, that 30% is suddenly 4%, and much less important.

US-Robotics still sells "hardware" modems, although they're not all that cheap. But back in the day, they did cut 50-100ms of latency off a dialup connection when compared to a winmodem (or controllerless).

Re:For once ...

[Guspaz]

You're exaggerating. Modern Intel integrated graphics fully support DirectX 10, and full onboard video decoding. They're comparable to discrete GPUs from about three years ago.

Do these things scream when it comes to games? No, but they don't need to. For the vast majority of people, serious gaming isn't in the cards. These GPUs need to display a 3D accelerated UI with decent performance, and help out with media decoding. Casual gaming is a plus, although businesses aren't interested in that.

By these accounts, integrated graphics are more than sufficient for the vast majority of users.

Besides, from reading most of the comments above, people are missing the point here; this article isn't talking about merging the GPU into the CPU, it's talking about how discrete graphics are becoming increasingly CPU-like, and how we're going to eventually reach the point where GPUs are so general-purpose, they're actually specialized CPUs. This is, infact, exactly what Larrabee is going to be; a very simple massively multicore CPU sitting on an add-in board. Sweeney's prediction is that AMD and nVidia are both going to take their GPUs to similar places eventually.

Re:For once ...

[Molt]

Since this is the developer who wrote the Unreal 3d engine, which is one of the best in the world at the moment, you'll have to excuse me if I think he knows a little bit about graphics APIs.

Both DirectX and OpenGL are entirely dedicated to the "Here is a triangle-based polygon mesh, some textures, and a few shaders- go draw me a pretty picture won't you?". This is a good approach to a lot of general 3d work but it is not the alpha and omega of 3d rendering.

As a couple of examples it doesn't have good support for voxel-based volumatic rendering which is perfect for cloudy scenes and also could give better support for fractally-defined terrains, it also doesn't support any rendering method which actually uses the equations of curves to render them properly as opposed to turning them into a series of small triangles. Many of these can be shoe-horned into the existing APIs, but when you do so you find you're actually going round the API much of the time and so losing any advantage they give you. There are a number of other examples of rendering methods which won't fit into the existing APIs in the article.

If the article is right and this did come about then there probably would be a series of APIs coming out for the different methods of 3d rendering. Those such as the article's author who're writing the graphic engines for the AAA games almost certainly won't be using them though- they'll be still be hand-crafting their own APIs specific and optimised for the current engine's features and now they'll be building them on top of very low-level graphics APIs with limited features like setting screen resolution and drawing a single pixel as opposed to the polygon-mesh with shaders level of complex APIs today.