Ray Tracing for Gaming Explored

Vigile brings us a follow-up to a discussion we had recently about efforts to make ray tracing a reality for video games. Daniel Pohl, a research scientist at Intel, takes us through the nuts and bolts of how ray tracing works, and he talks about how games such as Portal can benefit from this technology. Pohl also touches on the difficulty in mixing ray tracing with current methods of rendering. Quoting: "How will ray tracing for games hit the market? Many people expect it to be a smooth transition - raster only to raster plus ray tracing combined, transitioning to completely ray traced eventually. They think that in the early stages, most of the image would be still rasterized and ray tracing would be used sparingly, only in some small areas such as on a reflecting sphere. It is a nice thought and reflects what has happened so far in the development of graphics cards. The only problem is: Technically it makes no sense."

Comments

Adaptive techniques: make or break

[MessyBlob]

Adaptive rendering would seem to be the way forward. Ray tracing has the advantage that you can bail out when it gets complicated, or render areas to the desired resolution. This means a developer can prioritise certain regions of the scene and ignore others: useful during scenes of fast motion, or to bring detail to stillness. The result is similar to a decoded video stream, with detail in the areas that are usefully perceived as detailed. Combining this with eye position sensing (for a single user) would improve the experience.

Re:Adaptive techniques: make or break

[morgan_greywolf]

Linux hackers are far better coders then most people who use Visual Studio Um, those two groups aren't mutually exclusive. Many of us *nix hackers also have day jobs that require us to use tools like Visual Studio. You make assumptions that aren't true.

"How will ray tracing for games hit the market?"

[bobdotorg]

That completely depends on your point of view.

This isn't what we need in games

[Lurks]

I guess one has to state the obvious in that by moving to a process which is not implemented in silicon, as with current graphics cards, the work must necessarily be done in software. That means it runs on CPUs and that's something Intel is involved in where as when you look at the computational share of bringing a game to your senses right now, NVIDIA and ATI/AMD are far more likely to be providing the horsepower than Intel.

But really, even if this wasn't a vested interest case (and it may not be, no harm exploring it after all) - the fact remains that we don't actually need this for games. Graphics hardware has gone down an entirely different route whereby you write little shader programs which create surface visual effects on top of the bread and butter polygons and textures. This is a well established system by now and has a naturally compressive effect. It's like making all your visual effects procedural in nature rather than giving objects simple real-world textures and then doing a load of crazy maths to simulate reality. It works very well. Rememeber a lot of the time you want things to look fantastical and not ultra-realistic so lighting is some of the challenge.

Games aren't having a problem looking great. They're having a problem looking great and doing it fast enough and game developers are having a problem creating the content to fill these luscious realistic-looking worlds. That's actually what's more useful, really. Ways to aid game developers create content in parallel rather than throwing out the current rendering strategy adopted world wide by the games industry.

Further Reading

[moongha]

... on the subject, from someone that doesn't have a vested interest in seeing real time ray tracing in games becoming a reality.

http://realtimecollisiondetection.net/blog/?p=38

How far we've come in just 15 years

[dada21]

I was a founder of one of the Midwest's first rendering farms back in 1993, a company that has now moved on to product design. Back then we had Pentium 60s (IIRC) with 64MB of RAM. A single frame of non-ray traced 3D Studio animation took an hour or more. We had probably 40 PCs that handled the rendering, and they'd chug along 20 hours a day spitting out literally seconds of video. I remember our first ray trace sample (can't recall the platform for the PC, though) and it took DAYS to render a single frame.

I do remember that someone found some shortcuts for raytracing, and I wonder if that shortcut is applicable to realtime rendering today. From what I recall, the shortcut was to do the raytracing backwards, from the surface to the light sources. The shortcut didn't take into account ALL reflections, but I remember that it worked wonders for transparent surfaces and simple light sources. I know we investigated this for our business, but at the time we also were considering leaving the industry since the competition was starting to ignite. We did leave a few months early, but it was a smart move on our part rather than continue to invest in ever-faster hardware.

Now, 15 years later, it's finally becoming a reality of sorts, or at least considered.

Many will say that raytracing is NOT important for real time gaming, but I disagree completely. I wrote up a theory on it back in the day on how real time raytracing WOULD add a new layer of intrigue, drama and playability to the gaming world.

First of all, real time raytracing means amazingly complex shadows and reflections. Imagine a gay where you could watch for enemies stealthily by monitoring shadows or reflections -- even shadows and reflections through glass, off of water, or other reflective/transparent materials. It definitely adds some playability and excitement, especially if you find locations that provide a target for those reflections and shadows.

In my opinion, raytracing is not just about visual quality but about adding something that is definitely missing. My biggest problem with gaming has been the lack of peripheral vision (even with wide aspect ratios and funky fisheye effects). If you hunt, you know how important peripheral vision is, combined with truly 3D sound and even atmospheric conditions. Raytracing can definitely aid in rendering atmospheric conditions better (imagine which player would be aided by the sun in the soft fog and who would be harmed by it). It can't overcome the peripheral loss, but by producing truer shadows and reflections, you can overcome some of the gaming negatives by watching for the details.

Of course, I also wrote that we'd likely never see true and complete raytracing in our lives. Maybe I'll be wrong, but "true and complete" raytracing is VERY VERY complicated. Even current non-real time raytracing engines don't account for every reflection, every shadow, every atmospheric condition and every change in movement. Sure, a truly infinite raytracer IS impossible, but I know that with more hardware assistance, it will get better.

My experience over the years was ALWAYS with static images that were raytraced. They looked great, but it wasn't until I experienced raytraced animations (high res, many reflective and transparent layers with multiple light sources and a sun-source) that I really saw the benefit and how it would aid in gaming.

The next step: a truly 3D immersive peripheral video system, maybe a curved paper-thin monitor?

Re:How far we've come in just 15 years

[Twisted64]

Imagine a gay where you could watch for enemies stealthily...

How's that voice recognition software working out for you?

Now hear this

[suso]

I get tired of hearing this talk about real time ray tracing. They might be able to get basic ray tracing at 15 frames per second or more. But it won't matter, the quality won't be as good as some of the high quality images that you see that take hours to render. Sometimes days.

See, the two are incompatible because the purpose is different. With games, the idea is "How realistic can we make something look at a generated rate of 30 frames per second". But with photorealistic rendering the idea is "How realistic can we make something look, regardless of the time it takes to render one frame."

And as time goes on and processors become faster and faster, the status quo for what people want becomes higher. Things like radiosity, fluid simulations and more becomes more expected and less possible to do in real time. So don't ever count on games looking like those still images that take hours to make. Maybe they could make it look like the pictures from 15-20 years ago. But who cares about that? Real time game texturing already looks better than that.

Re:Now hear this

[IceCreamGuy]

from TFA:

At HD resolution we were able to achieve a frame rate of about 90 frames per second on a Dual-X5365 machine, utilizing all 8 cores of that system for rendering. The quote is referring to Quake 4. So they already can raytrace a semi-modern game at 90 FPS, and they have a graph that very clearly shows raytracing at a performance advantage as complexity increases. Just look at the damn graph (page three), the point where raster performance and raytracing performance intersect can't be more than a couple years off, and it's apparent that we may even have crossed that point already. Continue becoming tired of hearing about raytracing, the rest of us will sit patiently as the technology comes of age. Personally, I'm tired of hearing about this HD stuff, I mean, it's not like HD TVs will ever be mainstream, with their huge pricetags and short lifespans. Oh wait...

Re:Now hear this

[suso]

The quote is referring to Quake 4. So they already can raytrace a semi-modern game at 90 FPS, and they have a graph that very clearly shows raytracing at a performance advantage as complexity increases. Just look at the damn graph (page three),

I don't have to look at the damn graph to tell you that what people are going to want is this

And what they are going to get is this

And, they should just be happy with this (which, is pretty awesome)

My point is that real time photorealistic rendering will never catch up with what people expect from their games. It will always be behind. If all you want is mirrors, then find a faster way to implement them at the expense of a bit of quality.

Re:Now hear this

[The_Wilschon]

I think you still should look at (and understand) the damn graph. The point of the article was that if you want a given complexity of scene (which translates into quality of image), you only have to get a little bit more complex than current game scenes before current ray tracing techniques become faster than current raster techniques. Thus, ray tracing at 30 fps will look better than raster at 30 fps in the near future, perhaps already. Ray tracing is the quickest known route to better graphics quality at the same frame rate in games.

Yes, what can be produced will still be behind what people want or expect. But ray tracing will be less far behind than rasters in the near future.

All of this is according to TFA; I don't know much about this from a technical standpoint.

Re:Now hear this

[tolan-b]

I think you're missing the point. The reason Quake 4 looks crap raytraced was because it wasn't written to be raytraced, no shaders are being applied (because they were all written for a raster engine) so of course it looks bad. This stuff is just research.

One of the biggest hurdle in game graphics is geometry detail. Normal mapping is just a hack to make things appear more detailed, it breaks down in some situations. Raytracing will allow *much* higher geometry detail than rasterisation. Better reflection, refraction, caustics and so on are just gravy.

Re:Now hear this

[Cornelius the Great]

You completely missed the parent's point. Traditional rasterization chugs when a scene gets complex enough (I think the complexity is O(n)). Ray tracing scales very nicely (O(Log n)) and you can throw in stuff like TRUE reflection/refraction with minimal decreases in performance, with millions more polygons. Yes, rasterization is faster in current games, but throw in hundreds of millions of polygons into a scene and see what happens.

Furthermore, rasterization requires tricks (many would call them "hacks") to make the scene approach realism. In games today, shadows are textures (or stencil volumes) created by rendering more passes. While they look "good enough", they still have artifacts and limitations falling short of realistic. Shadows in raytracing come naturally. So do reflections, and refractions. Add some global illumination and the scene looks "real".

Rasterization requires hacks like occlusion culling, depth culling, sorting, portals, levels of detail, etc to make 3D engines run realtime, and some of these algorithms are insanely hard to implement for best case scenarios, and even then you're doing unnecessary work and wasting unnecessary ram rendering things you never see. Raytracing only renders what's on the screen.

That being said, I don't think raytracing will completely replace rasterization, at least not right away. Eventually, some games may incorporate a hybrid approach like most commercial renderers do today (scanline rendering for geometry, add raytracing for reflections and shadows). Eventually, 3D hardware will better support raytracing, and maybe in another decade we'll begin to see fast 3D engines that use ray tracing exclusively.

Re:Now hear this

[roystgnr]

they have a graph that very clearly shows raytracing at a performance advantage as complexity increases.

No, they have a graph that very clearly shows that raytracing while using a binary tree to cull non-visible surfaces has a performance advantage over rasterizing while using nothing to cull non-visible surfaces. Perhaps someday a raster engine will regain that advantage by using these "BSP Trees" as well.

in the player's best interests, natch

[Speare]

Daniel Pohl, a marketer at Intel

There, fixed that for you.

Raytracing the shiny first-intersection makes a lot of sense, even if it doesn't sell more CPUs. Sure, some day we will all have stunning holistic scene graphs that fit entirely within the pipeline cache of the processor, but it's not yet time for that.

Every change in developing a game engine requires changes in the entire toolset to deal with how to produce assets, how to fit within render time limit budgets, and how to model the scene graph and the logic graphs so that both are easily traversed and managed.

In the meantime, we have a pretty nice raster system right now, with a development strategy that provides for all those needs. You might not think that fullscale raytracing would upset this curve, but I'm not convinced. What do you do when a frame suddenly is taking more than 1/30sec to render, because the player is near a crystalline object and the ray depth is too high? How do you degrade the scene gracefully if your whole engine is built on raytracing? We've all played games where things like this were not handled well.

I contend that game AI is sometimes more advanced than academic AI because game developers are results-oriented and cut corners ruthlessly to achieve something that works well enough for a niche application. The same goes for game graphics: 33 milliseconds isn't enough to render complex scene graphs in an academically perfect and general way, it will require the same results-oriented corner-cutting to nudge the graphics beyond what anyone thought possible in 33ms. If that means using raytracing for a few key elements and ray-casting/z-buffering/fragment-shading the rest of the frame, game developers will do it.

General purpose CPUs: a REALLY bad way to do this.

[argent]

Professer Philipp Slusallek of the University of Saarbruecken demonstrated a dedicated raytracer in 2005, using a 66 MHz Xilinx FPGA with about 6 million gates. The latest ATI and nVidia GPUs have 100 times as many transistors and run at 6-8 times the clock with hundreds of times the memory bandwidth. Raytracing is completely parallelizable, and scales up almost linearly with processors, so it's not at all unlikely that if those kinds of resources were applied to raytracing instead of vectorizing you'd be able to add a raytracer capable of rendering 60+ FPS at the level of detail of the very latest games into the transistor budget of the chips they're designing now without even noticing.

Here's a debate between Professer Slusallek and chief scientist David Kirk of nVidia: http://scarydevil.com/~peter/io/raytracing-vs-rasterization.html .

Here's the SIGGRAPH 2005 paper, on a prototype running at 66 MHz: http://www.cs.utah.edu/classes/cs7940-010-rajeev/sum06/papers/siggraph05.pdf

Here's their hardware page: http://graphics.cs.uni-sb.de/SaarCOR/

Not ray tracing, radiosity

[Animats]

It's amusing to read this. This guy apparently works for Intel's "find ways to use more CPU time" department. Back when I was working on physics engines, I encountered that group.

Actually, the Holy Grail isn't real time ray tracing. It's real time radiosity. Ray-tracing works backwards from the viewpoint; radiosity works outward from the light sources. All the high-end 3D packages have radiosity renderers now. Here's a typical radiosity image. of a kitchen. Radiosity images are great for interiors, and architects now routinely use them for rendering buildings. Lighting effects work like they do in the real world. In a radiosity renderer, you don't have to add phony light sources to make up for the lack of diffuse lighting.

There's a subtle effect that appears in radiosity images but not ray-traced images. Look at the kitchen image and look for an inside corner. Notice the dark band at the inside corner. Look at an inside corner in the real world and you'll see that, too. Neither ray-tracing nor traditional rendering produces that effect, and it's a cue the human vision system uses to resolve corners. The dark band appears as the light bounces back and forth between the two corners, with more light absorbed on each bounce. Radiosity rendering is iterative; you render the image with the starting light sources, then re-render with each illuminated surface as a light source. Each rendering cycle improves the image, until, somewhere around 5 to 50 cycles, the bounced light has mostly been absorbed.

There are ways to precompute light maps from radiosity, then render in real time with an ordinary renderer, and those yield better-looking images of diffuse surfaces than ray-tracing would. Some games already do this. There's a demo of true real-time radiosity, but it doesn't have the "dark band in corners" effect, so it's not doing very many light bounces. Geometrics has a commercial real-time game rendering system.

Ray-tracing can get you "ooh, shiny thing", but radiosity can get to "is that real?"