Ask Slashdot: Why Don't Graphics Cards For VR Use Real-Time Motion Compensation?

dryriver writes: Graphics cards manufacturers like Nvidia and AMD have gone to great pains recently to point out that in order to experience virtual reality with a VR headset properly, you need a GPU capable of pushing at least a steady 90 FPS per eye, or a total of at least 180 FPS for both eyes, and at high resolutions to boot. This of course requires the purchase of the latest, greatest high-end GPUs made by these manufacturers, alongside the money you are already plonking down for your new VR headset, and a good, fast gaming-class PC. This raises an interesting question: virtually every LCD/LED TV manufactured in the last 5 or 6 years has a 'Real-Time Motion Compensation' feature built in. This is the not-so-new-at-all technique of taking, say, a football match broadcast live at 30 FPS or Hz, and algorithmically generating extra in-between frames in real time, thus giving you a hyper-smooth 200-400 FPS/Hz image on the TV set with no visible stutter or strobing whatsoever. This technology is not new. It is cheap enough to include in virtually every TV set at every price level (thus the hardware that performs the real-time motion compensating cannot cost more than a few dollars total). And the technique should, in theory, work just fine with the output of a GPU trying to drive a VR headset. Now suppose you have an entry level or mid-range GPU capable of pushing only 40-60 FPS in a VR application (or a measly 20-30 FPS per eye, making for a truly terrible VR experience). You could, in theory, add some cheap motion compensation circuitry to that GPU and get 100-200 FPS or more per eye. Heck, you might even be able to program a few GPU cores to run the motion compensation as a real-time GPU shader as the rest of the GPU is rendering a game or VR experience.

So my question: Why don't GPUs for VR use real-time motion compensation techniques to increase the FPS pushed into the VR headset? Would this not make far more financial sense for the average VR user than having to buy a monstrously powerful GPU to experience VR at all?

Comments

Sickening

[bobbutts]

I find the effect sickening on a flat TV. I'd gather it's worse in goggles.

Re:Sickening

[Immerman]

I believe something related is already being done on the Oculus and probably the Vive as well - in that each frame, after being rendered, is geometrically manipulated to approximate the changes that should be seen due to head motion since rendering began.

You can't use the same technique as used for televison, interpolating between frames, because that introduces far more lag, which I believe is the single largest contributor to nausea. After all you can't interpolate between frames until the second frame is ready, so even if interpolation were instantaneous you couldn't render frame N+0.5 until frame N+1 was finished, so you'd be inserting a minimum extra half-frame worth of additional lag in exchange for doubling the higher frame rate. Pretty sure that would be a lousy deal, especially considering the ugly artifacts such interpolation introduces.

Re:Sickening

[Martin Blank]

It reminds me of the Hobbit movies, in particular of the battle on the river. I was taken out of the movie by the splashes. They looked fake, but I knew that this was more because the movie was shot at 48FPS and so captured the motion better.

So does it look fake because it is fake, or does it look fake because it's different from what we expect to see?

That doesn't work because...

[Anonymous Coward]

You can't change the angle at which the scene is rendered by interpolating between frames.

It's not the raw framerate. It's that the scene your viewing has to match where you're looking that quickly or you get motion sick.

Latency

[Kjella]

Broadcast TV with this stuff on has considerable latency. It looks at the next frame and then interpolates its way there, might work reasonably well to watch a recorded show but would be pretty horrible for real time motion from VR. Next question?

Re:Latency

[ndnet]

This!

Even with a stable framerate, this technique intentionally delays the next frame to add compensation frames.

As an example, let's have a magic VR helmet running at 120Hz and instant processing (ie, 0ms GTG time, which doesn't exist) and a video card capped at a perfectly stable 30 FPS (aka 30Hz).

We will split a second into ticks - we'll use the VR helmet's frequency of 120 Hz, so we have 120 ticks, numbered 1 to 120. (Just to annoy my fellow programmers!)

We therefore get a new actual frame every 4th tick - 1st, 5th, 9th, etc.

Without motion compensation, we would display a new frame every 4th tick - 1st, 5th, 9th, etc.
With ideal (instant) motion compensation, we can't compute a transition frame until we have the new frame. So we could, theoretically, go real frame #1 on 1st tick, computed frame based on #1 and #2 on 5th tick, real frame #2 on 6th tick, computed frame based on #2 and #3 at 9th tick, etc.

This would also be jerky - 2 motion frames then 3 at rest? We could push the frames back a tick and fill the interval with three compensation frames, but then we increase the delay, which is always higher than this example (and is multiplicative). So we'd have frame #1 at 5th tick, computed frames at 6th/7th/8th, frame #2 at 9th tick, etc. You've now introduced a minimum 4 tick delay, which at 120Hz is 1/30 of a second, or 33ms! To an otherwise impossibly-perfect system!

What about historical frames instead to PREDICT the next blur? Well, then, when something in-game (or, really, on screen) changes velocity, there would be mis-compensation. (Overcompensating if the object slows, undercompensating if it speeds up, and miscompensation if direction changes).

There's more problems, too:
- This doesn't help when the video card frameskips/dips.
- Instant GTG and instant motion frame computation do not exist. At best, they're sub-tick, but you'd still operate on the tick.
- Input delay already exists for game processing, etc.
- Increased input delay perception would be exponential to the actual length of the delay. For example, 1-2ms between keypress and onscreen action? Hardly notable. 50ms delay just to start registering a motion on screen and course correct? Maybe OK, maybe annoying. 150-200ms? Brutal.

Because it works by delaying frames

[Qzukk]

In order to add a frame between 1 and 2, you have to have received both frame 1 and frame 2. People are already getting sick because what they see and do don't match, you're going to make it worse by making what they see lag further behind what little the headset picks up.

Re:They do.

[Lord Crc]

Look up "asynchronous time-warp". /thread

Pretty much this. Here's a video explaining time warping, it also has some links for more details in the video description.

https://www.youtube.com/watch?v=WvtEXMlQQtI/