Slashdot Mirror
Ray Tracing for Gaming Explored
Vigile brings us a follow-up to a discussion we had recently about efforts to make ray tracing a reality for video games. Daniel Pohl, a research scientist at Intel, takes us through the nuts and bolts of how ray tracing works, and he talks about how games such as Portal can benefit from this technology. Pohl also touches on the difficulty in mixing ray tracing with current methods of rendering. Quoting:
"How will ray tracing for games hit the market? Many people expect it to be a smooth transition - raster only to raster plus ray tracing combined, transitioning to completely ray traced eventually. They think that in the early stages, most of the image would be still rasterized and ray tracing would be used sparingly, only in some small areas such as on a reflecting sphere. It is a nice thought and reflects what has happened so far in the development of graphics cards. The only problem is: Technically it makes no sense."
Adaptive rendering would seem to be the way forward. Ray tracing has the advantage that you can bail out when it gets complicated, or render areas to the desired resolution. This means a developer can prioritise certain regions of the scene and ignore others: useful during scenes of fast motion, or to bring detail to stillness. The result is similar to a decoded video stream, with detail in the areas that are usefully perceived as detailed. Combining this with eye position sensing (for a single user) would improve the experience.
That completely depends on your point of view.
__ Someday, but not this morning, I'll finally learn to use the preview button.
I guess one has to state the obvious in that by moving to a process which is not implemented in silicon, as with current graphics cards, the work must necessarily be done in software. That means it runs on CPUs and that's something Intel is involved in where as when you look at the computational share of bringing a game to your senses right now, NVIDIA and ATI/AMD are far more likely to be providing the horsepower than Intel.
But really, even if this wasn't a vested interest case (and it may not be, no harm exploring it after all) - the fact remains that we don't actually need this for games. Graphics hardware has gone down an entirely different route whereby you write little shader programs which create surface visual effects on top of the bread and butter polygons and textures. This is a well established system by now and has a naturally compressive effect. It's like making all your visual effects procedural in nature rather than giving objects simple real-world textures and then doing a load of crazy maths to simulate reality. It works very well. Rememeber a lot of the time you want things to look fantastical and not ultra-realistic so lighting is some of the challenge.
Games aren't having a problem looking great. They're having a problem looking great and doing it fast enough and game developers are having a problem creating the content to fill these luscious realistic-looking worlds. That's actually what's more useful, really. Ways to aid game developers create content in parallel rather than throwing out the current rendering strategy adopted world wide by the games industry.
I remember some scenes that I would create with PoV to sometimes take several hours for a single frame to complete. Now we're looking at doing it in real time. Amazing.
... on the subject, from someone that doesn't have a vested interest in seeing real time ray tracing in games becoming a reality.
http://realtimecollisiondetection.net/blog/?p=38
I was a founder of one of the Midwest's first rendering farms back in 1993, a company that has now moved on to product design. Back then we had Pentium 60s (IIRC) with 64MB of RAM. A single frame of non-ray traced 3D Studio animation took an hour or more. We had probably 40 PCs that handled the rendering, and they'd chug along 20 hours a day spitting out literally seconds of video. I remember our first ray trace sample (can't recall the platform for the PC, though) and it took DAYS to render a single frame.
I do remember that someone found some shortcuts for raytracing, and I wonder if that shortcut is applicable to realtime rendering today. From what I recall, the shortcut was to do the raytracing backwards, from the surface to the light sources. The shortcut didn't take into account ALL reflections, but I remember that it worked wonders for transparent surfaces and simple light sources. I know we investigated this for our business, but at the time we also were considering leaving the industry since the competition was starting to ignite. We did leave a few months early, but it was a smart move on our part rather than continue to invest in ever-faster hardware.
Now, 15 years later, it's finally becoming a reality of sorts, or at least considered.
Many will say that raytracing is NOT important for real time gaming, but I disagree completely. I wrote up a theory on it back in the day on how real time raytracing WOULD add a new layer of intrigue, drama and playability to the gaming world.
First of all, real time raytracing means amazingly complex shadows and reflections. Imagine a gay where you could watch for enemies stealthily by monitoring shadows or reflections -- even shadows and reflections through glass, off of water, or other reflective/transparent materials. It definitely adds some playability and excitement, especially if you find locations that provide a target for those reflections and shadows.
In my opinion, raytracing is not just about visual quality but about adding something that is definitely missing. My biggest problem with gaming has been the lack of peripheral vision (even with wide aspect ratios and funky fisheye effects). If you hunt, you know how important peripheral vision is, combined with truly 3D sound and even atmospheric conditions. Raytracing can definitely aid in rendering atmospheric conditions better (imagine which player would be aided by the sun in the soft fog and who would be harmed by it). It can't overcome the peripheral loss, but by producing truer shadows and reflections, you can overcome some of the gaming negatives by watching for the details.
Of course, I also wrote that we'd likely never see true and complete raytracing in our lives. Maybe I'll be wrong, but "true and complete" raytracing is VERY VERY complicated. Even current non-real time raytracing engines don't account for every reflection, every shadow, every atmospheric condition and every change in movement. Sure, a truly infinite raytracer IS impossible, but I know that with more hardware assistance, it will get better.
My experience over the years was ALWAYS with static images that were raytraced. They looked great, but it wasn't until I experienced raytraced animations (high res, many reflective and transparent layers with multiple light sources and a sun-source) that I really saw the benefit and how it would aid in gaming.
The next step: a truly 3D immersive peripheral video system, maybe a curved paper-thin monitor?
I get tired of hearing this talk about real time ray tracing. They might be able to get basic ray tracing at 15 frames per second or more. But it won't matter, the quality won't be as good as some of the high quality images that you see that take hours to render. Sometimes days.
See, the two are incompatible because the purpose is different. With games, the idea is "How realistic can we make something look at a generated rate of 30 frames per second". But with photorealistic rendering the idea is "How realistic can we make something look, regardless of the time it takes to render one frame."
And as time goes on and processors become faster and faster, the status quo for what people want becomes higher. Things like radiosity, fluid simulations and more becomes more expected and less possible to do in real time. So don't ever count on games looking like those still images that take hours to make. Maybe they could make it look like the pictures from 15-20 years ago. But who cares about that? Real time game texturing already looks better than that.
Although I have a hard time arguing in the realm of 3D lighting, I will direct attention to the Beyond3D article, Real-Time Ray Tracing: Holy Grail or Fool's Errand?. Far be it of me to claim that this article applies to all situations of 3D lighting, it may be that Ray Tracing is the best choice for games, but I for one am glad to see an article that atleast looks into the possibility that Ray Tracing is not the best solution; I hate to just assume such things. Indeed, the article concludes that Ray Tracing has its own limitations and that a hybrid with rasterisation techniques would be superior to one or the other.
Demented But Determined.
According to the article, incorrect. Read the 3rd page: http://www.pcper.com/article.php?aid=506&type=expert&pid=3
There, fixed that for you.
Raytracing the shiny first-intersection makes a lot of sense, even if it doesn't sell more CPUs. Sure, some day we will all have stunning holistic scene graphs that fit entirely within the pipeline cache of the processor, but it's not yet time for that.
Every change in developing a game engine requires changes in the entire toolset to deal with how to produce assets, how to fit within render time limit budgets, and how to model the scene graph and the logic graphs so that both are easily traversed and managed.
In the meantime, we have a pretty nice raster system right now, with a development strategy that provides for all those needs. You might not think that fullscale raytracing would upset this curve, but I'm not convinced. What do you do when a frame suddenly is taking more than 1/30sec to render, because the player is near a crystalline object and the ray depth is too high? How do you degrade the scene gracefully if your whole engine is built on raytracing? We've all played games where things like this were not handled well.
I contend that game AI is sometimes more advanced than academic AI because game developers are results-oriented and cut corners ruthlessly to achieve something that works well enough for a niche application. The same goes for game graphics: 33 milliseconds isn't enough to render complex scene graphs in an academically perfect and general way, it will require the same results-oriented corner-cutting to nudge the graphics beyond what anyone thought possible in 33ms. If that means using raytracing for a few key elements and ray-casting/z-buffering/fragment-shading the rest of the frame, game developers will do it.
[
What's the Amiga have to do with raytracing? well, let me explain:
When the Amiga was released, it was a quantum leap in graphics, sound, user interface and operating system design. It could run full screen dual-playfield displays in 60 frames per second with a multitude of sprites, it had 4 hardware channels of sound (and some of the best filters ever put on a sound board), its user interface was intuitive and allowed even different video modes, and its operating system supported preemptive multithreading, registries per executable (.info files), making installation of programs a non-issue, and a scripting language that all programs could use to talk to each other.
20 years later, PCs have adopted quite a few trends from the Amiga (the average multimedia PC is now filled with custom chips), and added lots more in terms of hardware (hardware rendering, hardware transformation and lighting). It seems that the problems we had 20 years ago (how to render 2d and 3d graphics quickly) are solved.
But today's computing has some more challenges for us: concurrency (how to increase the performance of a program through parallelism) and, when it comes to 3d graphics, raytracing! Indicentally, raytracing is a computational problem that is naturally parallelizable.
So, what type of computer shall we have that solves the new challenges?
It's simple: a computer with many many cores!
That's right...the era of custom chips has to be ended here. Amiga started it for personal computers, and a new "Amiga" (be it a new console or new type of computer) should end it.
A machine with many cores (let's say, a few thousand cores), will open the door for many things not possible today, including raytracing, better A.I., natural language processing and many other difficult to solve things.
I just wish there are some new RJ Micals out there that are thinking of how to bring concurrency to the masses...
In a lot of cases in computing, doubling the number of pipelines (read: adding a second core, for example) does not, in fact, double performance unless the problem being worked on is highly parallelizable. For example, this is why one can not accurately describe a machine with two 2.5GHz processors as a "5GHz machine". Most computation that personal computers do on a day to day basis does not scale well, and the average user will reach a point of diminishing returns very quickly if they add many cores to increase performance for these tasks.
So all he's demonstrating here with his "16-core" experiment is that ray-tracing is a highly parallel process, and that throwing lots of cores at it will work effectively to increase performance without reaching that point of diminishing returns (at least, not reaching it very quickly.) Yes, we expect 16 cores to be faster than 4 cores or 1 core, but he's saying that when we're ray-tracing we can expect 16 cores to be almost four times faster than four cores and almost sixteen times faster than one.
Ray tracing looks hyper real on scenes full of plastic and mirrors but it's useless for rendering real-world scenes where radiosity effects dominate.
No sig today...
the fact remains that we don't actually need this for games.
Your post is heavily dependent on availability of suitable hardware. Software can be ported and recompiled to new platforms, but hardware-dependent software has a short lifespan precisely because getting usable hardware doesn't last particularly long. There's a lot of otherwise good enjoyable games which are unplayable now because they depended on the early Voodoo cards or other unrepeated graphics hardware. Now with CPU power ramping back up (relatively speaking), we can pull away from lifespan-limiting dependence on dedicated graphics hardware, and move the full rendering process back to generic CPU hardware.
Torques me off when I want to play something but can't - not because I don't have the horsepower, but because it requires obsolete cards.
BTW: The article notes that RT vastly outperforms raster on very high poly count scenes - that alone is reason to switch.
Can we get a "-1 Wrong" moderation option?
Professer Philipp Slusallek of the University of Saarbruecken demonstrated a dedicated raytracer in 2005, using a 66 MHz Xilinx FPGA with about 6 million gates. The latest ATI and nVidia GPUs have 100 times as many transistors and run at 6-8 times the clock with hundreds of times the memory bandwidth. Raytracing is completely parallelizable, and scales up almost linearly with processors, so it's not at all unlikely that if those kinds of resources were applied to raytracing instead of vectorizing you'd be able to add a raytracer capable of rendering 60+ FPS at the level of detail of the very latest games into the transistor budget of the chips they're designing now without even noticing.
Here's a debate between Professer Slusallek and chief scientist David Kirk of nVidia: http://scarydevil.com/~peter/io/raytracing-vs-rasterization.html .
Here's the SIGGRAPH 2005 paper, on a prototype running at 66 MHz: http://www.cs.utah.edu/classes/cs7940-010-rajeev/sum06/papers/siggraph05.pdf
Here's their hardware page: http://graphics.cs.uni-sb.de/SaarCOR/
I'm sure you thought you were being clever, but you weren't.
If you knew anything about parallel algorithms you'd know that it's fairly common to have things that scale with more like 75% efficiency, and you're still happy. It's all down to how much communication is required, and how often it's required. With raytracing normally (as in, a decade ago when I knew anything about this) you'd parallelise over multiple frames. With real-time rendering you'd need to parallelise within a frame. Depending on your scene, there will be a static cost (bounding box calculations) in addition to the per-pixel costs.
jh
jh
I can't wait to play a game where I'm a shiny silver ball floating above a checkered marble floor.
Read TFA. Ray tracing does NOT happen on the graphics card; it happens on your CPU. And they've got Quake 4 at 1280x running at 90 FPS raytraced already. Since raytracing scales almost linearly, as you add more cores to your CPU (which is likely the future direction of CPU technology improvement), you improve raytracing performance by about the same factor.
It's amusing to read this. This guy apparently works for Intel's "find ways to use more CPU time" department. Back when I was working on physics engines, I encountered that group.
Actually, the Holy Grail isn't real time ray tracing. It's real time radiosity. Ray-tracing works backwards from the viewpoint; radiosity works outward from the light sources. All the high-end 3D packages have radiosity renderers now. Here's a typical radiosity image. of a kitchen. Radiosity images are great for interiors, and architects now routinely use them for rendering buildings. Lighting effects work like they do in the real world. In a radiosity renderer, you don't have to add phony light sources to make up for the lack of diffuse lighting.
There's a subtle effect that appears in radiosity images but not ray-traced images. Look at the kitchen image and look for an inside corner. Notice the dark band at the inside corner. Look at an inside corner in the real world and you'll see that, too. Neither ray-tracing nor traditional rendering produces that effect, and it's a cue the human vision system uses to resolve corners. The dark band appears as the light bounces back and forth between the two corners, with more light absorbed on each bounce. Radiosity rendering is iterative; you render the image with the starting light sources, then re-render with each illuminated surface as a light source. Each rendering cycle improves the image, until, somewhere around 5 to 50 cycles, the bounced light has mostly been absorbed.
There are ways to precompute light maps from radiosity, then render in real time with an ordinary renderer, and those yield better-looking images of diffuse surfaces than ray-tracing would. Some games already do this. There's a demo of true real-time radiosity, but it doesn't have the "dark band in corners" effect, so it's not doing very many light bounces. Geometrics has a commercial real-time game rendering system.
Ray-tracing can get you "ooh, shiny thing", but radiosity can get to "is that real?"
Hybrids do make a lot of sense. The author's argument is the need for a spatial partitioning structure if one mixes ray tracing with rasterization. This is a no-brainer; you'd have such a structure anyway.
In fact, his points actually show why a hybrid is perfect: most surfaces are not shiny, refractive, a portal, etc. Most are opaque - and a rasterizer is much better for this (since no ray intersection tests are necessary). He shows pathological scenes where most surfaces are reflective; however, most shots do show a lot of opaque surfaces (since Quake 4 does not feature levels where one explores a glass labyrinth or something).
Yes, if a reflective surface fills the entire screen, its all pure ray tracing - and guess what, that is exactly what happens in a hybrid. Hybrid does not exclude pure ray tracing for special cases.
Ideally, we'd have a rasterizer with a cast_ray() function in the shaders. The spatial partitioning structure could well reside within the graphics hardware's memory (as an added bonus, it could be used for predicate rendering). This way haze, fog, translucency, refractions, reflections, shadows could be done via ray tracing, and the basic opaque surface + its lighting via rasterization.
Now, I keep hearing the argument that ray tracing is better because it scales better with geometric complexity. This is true, but largely irrelevant for games. Games do NOT feature 350 million triangles per frame - it just isn't necessary. Unless its a huge scene, most of these triangles would be used for fine details, and we already have normal/parallax mapping for these. (Note though that relief mapping usually doesn't pay off; either the details are too tiny for relief mapping to make a difference, or they are large, and in this case, traditional geometry displacement mapping is usually better.) So, coarse features are preserved in the geometry, and fine ridges and bumps reside in the normal map. This way, triangle count rarely exceeds 2 million triangles per frame (special cases where this does not apply include complex grass rendering and very wide and fine terrains). The difference is not visible, and in addition the mipmap chain takes care of any flickering, which would appear if all these details were geometry (and AA is more expensive than mipmaps, especially with ray tracing).
This leaves us with no pros for pure raytracing. Take the best of both worlds, and go hybrid, just like the major CGI studios did.
This sig does not contain any SCO code.
Disclaimer: I work for NVIDIA. I speak not for them.
People keep saying this, that raytracing scales up better than rasterization. It's simply not true. Both of them have aspects that scale linearly and logarithmically. They do scale differently, but in a related sort of wy.
Raytracing is O(resolution), and O(ln(triangles)), assuming you already have your acceleration structures built. But guess what? It takes significant time to built your acceleration structures in the first place. And they change from frame to frame.
Rasterization is O(ln(resolution)), and O(triangles). Basically, in a rasterizer, we only draw places that we have triangles. Places that don't have triangles have no work done. But the thing is, we've highly pipelined our ability to handle triangles. When people talk about impacting the framerate, I want to be clear what we're talking about here: adding hundreds, thousands, or even a million triangles is not going to tank the processing power of a modern GPU. The 8800 Ultra can process in the neighborhood of 300M triangles per second. At 100 FPS, that'd be (not suprisingly) 3M triangles per frame.
Modern scenes typically run in the 100-500K triangles per frame, so we've still got some headroom in this regard.
Cheers.
I currently have no clever signature witicism to add here.
The obvious problem with layering hacks on top of hacks to make your games look better is that it takes more time and money to pay programmers to develop them.
However, one of the clear trends in gaming is spending more on game art and artists than on programming and programmers, and maybe a less obvious problem is that working with hacks is a lot harder for the artists, as well. As an example: setting up shadows in rasterization is not too difficult, codewise (it's fairly expensive, timewise, but clearly workable if you don't have too many lights).
However, from an artist's point of view, rasterized shadows are wonky, counterintuitive, and fiddly to use. You have to specify which lights cast shadows of which objects on which other objects. The specific angles of lights, cameras, and the scene in question can cause ugly artifacts, which may or may not be possible to eradicate. This is particularly a problem in interactive play, where it may not be practical to preview all possible combinations of the above.
So, an overlooked advantage of ray tracing may be to provide a better artistic *medium* for game developers -- if you can afford it....
This is a rather good point; at some point, adding more polygons doesn't do anything to make an unrealistic scene look more realistic. This is true for raytracing and polygon rendering alike. Ray tracing has some advantages here, for what it's worth; it scales better with huge scenes, and it can represent non-triangular primitives natively (though all the fast ray-tracers I've seen only deal with triangles). I wouldn't call reflection a non-issue; currently, no one cares because current implementations aren't very good, and there aren't any better alternatives to compare with. Same with refraction. Ray-tracing can do soft shadows, but they're computationally more expensive (at least, all the approaches I know are).
Ray tracing is just the next step towards realism. Once we start doing ray tracing, we can move on to global illumination. Photon mapping is a GI algorithm that works like ray tracing in reverse; it casts rays out from the light source, and then they can bounce of off or be absorbed by the objects they hit (depending on surface properties), and their final location is stored as a point in a data structure called a photon map. Then, when you ray trace, you use the local density of the photon map to approximate the amount of light. Photon mapping can be used to simulate ambient light, caustics (patches of light reflected off of or refracted by shiny things), and subsurface scattering in a way that is physically correct and unbiased. See "The Light of Mies van der Rohe" animation on this page for an example of ambient light, or here for some images of caustics.
Ray tracers can do all the things that polygon renderers can do, plus a bit more. Once the hardware gets fast enough (and it looks like that will happen within a few years), there's no real reason not to use ray tracing. Photon mapping is more expensive (there's an nlogn sort involved in constructing the photon map), so it will probably be quite a while before we start to see real-time global illumination updates, but that's the next big step, and we can't go from here to there with the polygon rendering algorithms we're using now.