Slashdot Mirror
The Future of Real-Time Graphics
Bender writes "This article lays out nicely the current state of real-time computer graphics. It explains how movie-class CG will soon be possible on relatively inexpensive consumer graphics cards, considers the new 'datatype rich' graphics chips (R300 and NV30), and provides a context for the debate shaping up over high-level shading languages in OpenGL 2.0 and DirectX 9. Worth reading if you want to know where real-time graphics is heading."
Ahhh it seems like only yesterday I was sooo impressed by mario, blocky graphics and plain colors with no frills really. Then I was impressed by doom, 3-d, awesome action, multi-player, toystory-animated movie? WOW! Final Fantasy-Good animated movie! double WOW! Now what next? The way things advance today it really is getting hard to tell real world film scenes from CG stuff, I just watched lord of the rings and it really is hard to tell real from CG, now with the hardware getting cheaper and software becoming more advanced what was only a fantastic dream in the 80's, a big movie corp with render farm only dream in the 90's is now becoming possible for the home user. Don't know about you, but I'm still impressed, and I can't wait to see what is just over the horizon.
Silly people.
First, the polygon-based rendering used by the cards is based on fairly special-purpose trickery to get good effects (like shadows - they're not really implemented by the API or the card directly). Second, the other part of really high-quality rendering is high-complexity models - the PC's themselves start to balk at swimming through the massive data sets required in real time. There's always been the speculation that raytracing would catch up to polygon rendering (as CPU's get faster) because the former has sublinear complexity in the number of objects, where the latter is more like linear complexity. _That_ would give you some pretty images!
My own feelings on the little debate is rather simple. Unix machines (Linux, SGI, OS X) are becoming the standard for Hollywood level style movies - they're powerful, you can cluster them with relative ease, and they don't crash that often.
DirectX9 is really about games - render less polygons on the fly, and it only works with one operating system (any guesses on which one?).
As games and movies start to approach each other in graphical abilities, I wonder if OpenGL will become more important as the Unix graphics programmers start getting pulled from their Toy Story 3 seats to help with the guys making Toy Story 3: The Game.
Right now, the #1 reason why OpenGL is still in a good number of Windows machines is John Carmack. Will things change? Maybe, maybe not. But I still wonder about the future.
52 Weeks, 52 Religions with John Hummel
But this is all going to be great fun for gaming, VR, simulation, and so on.
Bruce
Bruce Perens.
A year back or so I did the blender work for a starwars fanflic...Now this was only a fifteen minute film...but the 5 minutes or so of 3d easily took a day to render. As this stuff gets faster, amateur movies will become better and more sophisticated. Low budget films and TV shows will gain access, and the graps of the MPAA will weaken. Anything that makes low-budget films easier is a good thing.
With the internet and a DVD Burner, a low budget film could be distributed DIRECTLY on DVD. Now the films just need to get good enough that people will want them. This would be a good direction for both music and movies.
Cool eye picture. How the heck do you make a model for that?
Brian
> How does the equation change when you consider the resolution used for each animated frame (of Toy Story), and resolutions that are common in the gaming arena?
Not a lot.
Toy Story was rendered at approx 1536 x 922, only 8% more pixels than 1280 x 1024.
As computers get more powerful, our demands on them get greater. The time it takes to render a single frame of a animated feature has stayed pretty much constant over the last few years.
I mean, come on people, it's apples and oranges here. Two similar tools for two VERY different purposes. Rendering 80 FPS at 1600x1200 makes good games, but I doubt there will ever be a day when film frames are rendered in real-time. There's just no reason to!
That's not to say that yesterday's movies couldn't be rendered on today's technology. Absolutely! But tommorow's movies are a different story.
hell, I wish that were true! where do -you- work?
...but as always it's an issue of cost/market/etc. Game development is big business now, it's not a make-something-fun-and-sell-it-in-a-ziplock-bag industry anymore.
I can only speak for consoles, but there have been some interesting developments over the last five years or so...
1. Knowledge prerequisite for engine development has gone up, not down, as was previously thought (hoped?) Some people had thought that with the latest generation of h/w (XBox, PS2, Gamecube) that more programmers would be able to work on the graphics-end (XBox because of DirectX -- PS2 well, because they didn't know any better). But just like on the previous generation of h/w although we don't have to do some of the lower level tasks anymore (s/w render, perspective correction, blah blah) more complex tasks are required for the latest games. I think everyone's hoping (again) that this will change in the -next- generation (e.g. send 3DMax/Maya file to hardware! yeah, right.). maybe. we'll see...
2. The ever-increasing (and always lamented) trend of h/w shy programmers has (maybe?) kept the graphics engine teams small. It still is very common to have one or two man teams building the engine. For example, we have two engine programmers (working on different engines on different platforms) and about 25 on titles. Based on other companies I've been at or seen, this isn't really unusual. If you meant artists (by "graphics department") then yes, there is clearly a trend for having more artists than any other role.
3. Game teams are not oblivious to the severe lack of quality gameplay. Publishers aren't either (really!)
4. Unfortunately, the idea of "game designer" as a profession (outside of a few notable individuals) has been historically ridiculed. It's been ranked with "tester" and "your mom" as far as development teams were/are concerned. However, even though only a small percentage of development houses (still!) recognize "game designer" as a legitimite role, one of the most promising trends has been (perhaps out of necessity?) the steady increase in them. -That- is good news. Basically, more places have someone in charge of "fun."
It'll get better. Probably.
Basically, the explanation is that rasterization takes a time proportional to the number of polygons to render a frame, while raytracing takes time proportional to the log of the number of polygons. That might make you think raytracing should be always faster, which it clearly isn't -- the reason it isn't is that the constant factor in each is very different. So you have a*N vs b*log(N), where b is much bigger than a. As N gets bigger (apparently in the neighborhood of 10 million polygons), the difference between a and b becomes less important than the difference between N and log(N).
The main benefits of raytracing over rasterizing is that it is very easy to get things like reflections, shadows, refraction, and other important effects with raytracing, but with rasterizing, you need to do a lot of complicate and CPU-intensive hacks to get the same effect. Another benefit is that raytracing is parallelizable while rasterization generally isn't. That means that if you have a raytracing accelerator card in your PC, you can nearly double the frame rate or resolution by adding a second raytracing card.
Of course, it's all a chicken-and-egg sort of thing, nobody's going to buy a raytracing card until it's a cheap way to do the rendering they want, and it won't be available unless there is a market. Fortunately, there is research into using the next generation of rasterizing graphics cards to greatly speed up raytracing. This will help bridge the gap, by making raytraced games possible using soon-to-be-existing hardware.
But imagine downloading Toy Story 3 or something to your PC... not as a pre-rendered movie, but as a real-time scripted 3D engine with a soundtrack. Run it in whatever resolution you are able to. Use your own camera angles.
Or play a realtime version of Final Fantasy: The Spirits Within, but walk around the "set" in realtime with the characters or just keep the camera focused on Aki's bizznoobies.
-CausticPuppy "Of all the people I know, you're certainly one of them." -Somebody I don't know
I get sick of hearing this crap. "When will my graphics card be able to do rendering?!", "When will my graphics card be able to display pixar-quality rendering?!", "When will my graphics card be able to put out graphic realism?" ect, ect, ect.
This is a bunch of crap. By the time your playstation 6 or ge-force 7 or whatever the hell it's going to be gets to a point where it can run enough cycles to achieve toy-story quality pictures in real time (which is still years off) the bar will be raised again for cgi.
Just as moores law doubles technology, the technolgy of rendered cgi doubles. Think back when cray supercomputers rendered frames and took about an hour a frame for untextured geometry with little of the properties that are avaliable today. Today, the images still take 1 hour a frame, even though the technology is billions times faster. Why is that? Because cgi artists will continually pump in as much as they can per frame. If it took 20 minutes last year, it's going to take 20 minutes this year because studio x is going to add some new thing that improves quality but still retains their time margin.
Do you honestly think that gpu's are going to be able to achieve real-time radiosity in next couple years? Real time raytracing like renderes have now? Hundreds of thousands of blades of grass with no tricks? Individual hairs? Do you think that will happen anytime soon? Perhaps sometime - but when it does pre rendered images will feature something new that real-time can't match. Face it - real time graphics will never replace the quality of pre-rendered.
While it isn't addressed in the article, there are a LOT of things that need to be handled in the hardware that just isn't there.
- Orders of magnitude more polygons. Artists want more polys. I personally would want at least four polys per pixel at any view, just to make sure that the rendering would be correct.
- Raytracing and Radiosity. Both of these have been proposed in realtime, there is a realtime ratracer (RTRT, google on it) and Realtime Radiosity is a siggraph paper (look it up, too.). BUT they only work with limited numbers of objects. They need to work with the number of polygons given above, and they can't.
- Lights and light maps. Current video cards are limited to just a few lights. The latest generation can only do 8 lights. That works for games, Quake uses just a few and artists hate being limited to them. The video cards would need to handle thousands of light sources, and be able to process light maps (for shadows) in realtime.
- Textures. Currently we use textures to hide the fact that we don't have detail. But as long as that detail is missing, things will look bad up close. The 'ideal image' is not a wireframe with a texture draped over it. The 'ideal image' is based completely on the vertex lists. To build a model at the right detail, each pixel in the texture would need to be replaced with a vertex, including color and other material properties, normals, edges, etc. So each value goes from a 32-bit RGBA color entry to a fairly big (about 1000-bit) structure.
- RAM and BUS speed, and model size. Once we have these massive scenes, we have the bottlenecks of RAM and the system bus. We have always fought these in graphics, which is why the push from ISA to PCI to AGP. Trying to make the graphics better will just compound the problem.
The facts are that these won't go away. We will continue making texture mapped wireframe models for the near future. By the time the graphics card industry can do realtime what movie studios do in months today, the movie studios will be playing with the ideas above.A perfect rendering system is almost near-infinite recursive, requires infinitely detailed models, and takes a long time to render. We can't do the infinite perfect system, but we can tell our artists to let it run for about an hour per frame. That means 'no realtime top of the line movies', no matter what.
frob.
//TODO: Think of witty sig statement
There is an upside to this. Eventually we will reach the point where its impossible for graphics to get better (ie indistinguishble from a photo of the real thing, or maybe vr or something). At that point when there are no more innovations to be made in graphics the game companies will have to, in order to sell games, concentrate on, yep you guessed it, game play.
Why not fork?
You only need all that detail for nearby objects, which is what subdivision surfaces and level of detail processing are for. With procedural shaders and bump mapping, you don't need that much for most surfaces. The detail may be there in the model, but only a small fraction of it needs to go through the graphics pipeline for any given viewpoint. Given that pixel size is finite and human vision has finite resolution, at some point you max out.
For fixed lighting, you can do radiosity in realtime. (Check out Lightscape, now called 3D Viz.) The radiosity map only has to be recomputed when the lights move. Mostly this is used for architectural fly-throughs. Of course, Myst and Riven are basically architectural fly-throughs. (They're rendered with multiple hand-placed lights in Softimage, though; when they were made, the radiosity renderers couldn't handle a scene that big.)
I tend to agree, but at some level of detail, you can render geometry into a texture (maybe with a bump map) and use that until you get really close. Microsoft prototyped a system for doing this automatically a few years back, called Talisman. Talisman was a flop as a graphics card architecture, but as a component of a level of detail system, it has potential.
Moving around in a big virtual world is going to require massive data movement. But we're getting there. This may be the driver that makes 64-bit machines mainstream. Games may need more than 4GB of RAM.
Rendering isn't the problem, anyway. Physics, motion generation, and AI are the next frontiers in realism.
Man, you can tell at a glance who read the article and who didn't.
I'm going to simplify a great deal here to try and boil this down to the essence. John Carmack please feel free to correct any mistakes I make.
Up to this point, the imagery coming out of the gaming graphics cards has been limited by the hardware design of the cards. The feature set implemented by the cards limits how complicated you can get with the details in the final image.
Note that we're not just talking about simple things like pure polygon counts. Film Industry CGI isn't of higher quality just because they throw more polygons at the problem; they have all kinds of highly complex shaders that can generate special textures without changing the number of polygons in the model - if you saw the "special features" on the Shrek DVD, you can see this at work with Donkey's fur.
Rendering all these extra shaders is CPU expensive, which is why the big animation houses have big render farms.
But two things have happened that stand to change that.
The first (and the most ingenious) is that it has been discovered that you can compile any shader into a series of OpenGL language commands. The tradeoff is the number of passes through the pipeline that implementing a given shader may require may well be a large number - but even so, any shader currently in use by a Hollywood Mouse House can, in theory, be compiled into OpenGL and executed on any OpenGL card.
And here's the really cool part - rendering in OpenGL is many times faster than doing it in software on a general-purpose CPU. Many, many times faster.
Secondly, the biggest problem with trying to crank Shrek through your GF2MX400 (assuming you've compiled all the shaders into OpenGL) is that each shader may require 200 passes, but the data structures inside the card lack precision - either not enough bits as a float, or perhaps not even floats at all, but integers.
That means the data is being savaged by rounding errors and lack of precision during each render pass. It's like photocopying photocopies.
BUT, the latest generation of graphics chips have the necessary precision to do 200-pass rendering without falling victim to rounding errors.
Combine these two things together, and you can quite literally take a frame from Shrek, with all the crazy shaders, compile it to OpenGL, and render the frame on your GF6-whatever **faster than the native render platform**
A very good deal faster than the native render platform.
Is this "Shrek in real-time"? No, not by a long shot. But it may well be "Pixar's renderfarm in a box".
Now, as Bruce pointed out, having Pixar's renderfarm in a box doesn't make you Pixar. There is still a requirement for artistic talent. But all that cheap extra horsepower may well mean that the quality of CGI is going to explode for those talented enough to make use of it.
How will this affect games? It makes a bunch of shader techniques that were previously availible only to the movie industry possible within the framework of a game. And it divorces, somewhat, the game visuals from the card's hardware because these shaders are executed as general-purpose OpenGL instructions, not as dedicated hardware on the card. If you, as a game designer, can write a "fur shader" that runs in few enough passes to meet real-time output timings, then you get fur on your model, even if the card doesn't have a built-in "fur shader" or "fur engine".
THIS is why this is all such a big deal. The amount of quality per mSec of render time is about to explode.
Cool stuff!
DG
Want to learn about race cars? Read my Book
... isn't the "rampant piracy" Red Herring they've been feeding the press and their tame politicians in Washington, D.C., it is the possibility that anyone who does have a story to tell will be able to make a quality movie with nothing more than their home PC and a little time.
... a state of affairs the mimicks the current, cartel-controlled situation rather well, actually. Even if only 1 in 100,000 has the story telling talent to put together a good film, that would amount to 2,800 potent competitors to the media cartels.
Suddenly we don't need studios, we don't need actors, and we don't need tens or hundreds of millions to produce a blockbuster movie. And with the internet to distribute the material on, we don't need their distribution network of cinemas either.
The most important talent they rely on is not skill in computer imagery, but skill in telling a compelling story using all of the tools of the visual idiom. This is what most people don't have, and it is an essential element to producing good film.
Like musicians using home-studios to record music, without talent this will go largely unusued, or, more likely, there will be a lot of less-than-good material out there
Musicians really don't need million dollar studios anymore to produce an album, and while this means a lot of junk is pressed onto CD, it also means a lot of musicians are able to produce and market their music outside of the RIAA's cartel, through mp3.com and elsewhere. Hollywood doesn't fear the napstersization of their medium nearly as much as they fear the mp3.com-ization of it, and competition with a few thousand talented people not on their payroll.
This, I think, is why we are experiencing such an onslought of attempts through legislation and back door regulation via the FCC and a little known "standards" body called the BPDG to take both the internet, and general computers, out of the hands of private citizens.
It isn't about 'piracy,' it is about competition, and they don't fear competition from 'everybody' so much as they fear general access to the tools, which means those talented persons not a part of the cartels would be able to compete for viewership and marketshare on a level playing field with the big studios.
And that is something they simply cannot abide.
The Future of Human Evolution: Autonomy
There is already a high-level shading language, even LGPLed, the API is from Apple's QuickDraw3D, it is called Quesa, http://www.quesa.org/. It can sit above any other API, such as OpenGL or Direct3D. It is scene ordinated. It is a pretty cool api, it is a lot easier to use than Direct3D or OpenGL. The file format to save the scene is 3DMF binary or text (XML like); in fact the binary format was appointed to be the binary format for VMRL.
Do you honestly think that gpu's are going to be able to achieve real-time radiosity in next couple years? Real time raytracing like renderes have now? Hundreds of thousands of blades of grass with no tricks? Individual hairs? Do you think that will happen anytime soon?
I used to think as you do. That was before I got a large amount of education while attending Siggraph this year.
At Siggraph, I saw a live demonstration of a real-time raytracer that was also computing a diffuse interreflection solution (radiosity-like, for those who don't understand) on the fly. I also saw a real-time recursive raytracer written by Stanford researchers that was implemented in a GPU's pixel shader. There is currently research on displacement map "textures" that could conceivably let you render thousands of blades of grass or individual hairs without having to send all of that geometry down the AGP bus.
All of these things blow the doors off what people think a graphics chip can do. Your post would have been accurate last year. Not now.
I will agree with one point: software-based rendering will always be able to compute certain effects that will be difficult or cumbersome to do in a GPU. But I'll also claim that the gap is dramatically shrinking.
I'll also say that the two techniques are not really in conflict. You can use them both in conjunction with each other. You can use a hardware-accelerated Z-buffer to help a raytracer determine first-level occlusion. You can use a raytracer to generate textures and maps for a GPU. In the future, we will see both techniques used to compliment each other.