Ask Slashdot: Why Don't Graphics Cards For VR Use Real-Time Motion Compensation?

dryriver writes: Graphics cards manufacturers like Nvidia and AMD have gone to great pains recently to point out that in order to experience virtual reality with a VR headset properly, you need a GPU capable of pushing at least a steady 90 FPS per eye, or a total of at least 180 FPS for both eyes, and at high resolutions to boot. This of course requires the purchase of the latest, greatest high-end GPUs made by these manufacturers, alongside the money you are already plonking down for your new VR headset, and a good, fast gaming-class PC. This raises an interesting question: virtually every LCD/LED TV manufactured in the last 5 or 6 years has a 'Real-Time Motion Compensation' feature built in. This is the not-so-new-at-all technique of taking, say, a football match broadcast live at 30 FPS or Hz, and algorithmically generating extra in-between frames in real time, thus giving you a hyper-smooth 200-400 FPS/Hz image on the TV set with no visible stutter or strobing whatsoever. This technology is not new. It is cheap enough to include in virtually every TV set at every price level (thus the hardware that performs the real-time motion compensating cannot cost more than a few dollars total). And the technique should, in theory, work just fine with the output of a GPU trying to drive a VR headset. Now suppose you have an entry level or mid-range GPU capable of pushing only 40-60 FPS in a VR application (or a measly 20-30 FPS per eye, making for a truly terrible VR experience). You could, in theory, add some cheap motion compensation circuitry to that GPU and get 100-200 FPS or more per eye. Heck, you might even be able to program a few GPU cores to run the motion compensation as a real-time GPU shader as the rest of the GPU is rendering a game or VR experience.

So my question: Why don't GPUs for VR use real-time motion compensation techniques to increase the FPS pushed into the VR headset? Would this not make far more financial sense for the average VR user than having to buy a monstrously powerful GPU to experience VR at all?

Comments

Sickening

[bobbutts]

I find the effect sickening on a flat TV. I'd gather it's worse in goggles.

Re:Sickening

[Anonymous Coward]

There are, however, things that make VR implementations easier and less error prone.

If the VR helmet itself, which has motion sensors, detects motion, it could apply motion compensation while it awaits for a new frame.

The approach could be limited to head movements, be truly helpful and nigh flawless if done correctly.

So really, the poster does make a valid point.

Re:Sickening

[Immerman]

I believe something related is already being done on the Oculus and probably the Vive as well - in that each frame, after being rendered, is geometrically manipulated to approximate the changes that should be seen due to head motion since rendering began.

You can't use the same technique as used for televison, interpolating between frames, because that introduces far more lag, which I believe is the single largest contributor to nausea. After all you can't interpolate between frames until the second frame is ready, so even if interpolation were instantaneous you couldn't render frame N+0.5 until frame N+1 was finished, so you'd be inserting a minimum extra half-frame worth of additional lag in exchange for doubling the higher frame rate. Pretty sure that would be a lousy deal, especially considering the ugly artifacts such interpolation introduces.

Re:Sickening

[Martin Blank]

It reminds me of the Hobbit movies, in particular of the battle on the river. I was taken out of the movie by the splashes. They looked fake, but I knew that this was more because the movie was shot at 48FPS and so captured the motion better.

So does it look fake because it is fake, or does it look fake because it's different from what we expect to see?

Re: Sickening

[D.McG.]

Just that much lag is enough to induce motion sickness; which is why 90fps is preferable. The latency, measured from head movement to photons, needs to be under 20ms to keep one for vomitting.

That doesn't work because...

[Anonymous Coward]

You can't change the angle at which the scene is rendered by interpolating between frames.

It's not the raw framerate. It's that the scene your viewing has to match where you're looking that quickly or you get motion sick.

They do.

[Anonymous Coward]

Look up "asynchronous time-warp". /thread

Re:They do.

[Lord Crc]

Look up "asynchronous time-warp". /thread

Pretty much this. Here's a video explaining time warping, it also has some links for more details in the video description.

https://www.youtube.com/watch?v=WvtEXMlQQtI/

Re:They do.

[allcoolnameswheretak]

The HTC Vive uses a different technology, I don't remember what it was called, but it seems to be more effective at preventing nausea, as I read in many reviews that people tend to experience less nausea using the Vive for long periods than with Oculus.

I own a Vive, and when playing Elite I usually get between 45 and 55 frames per seconds, yet the experience seems entirely smooth to me and I don't get any nausea at all. I find no perceivable difference to playing a game running at 90 FPS.

Re:They do.

[MarbleMunkey]

The Vive uses Interleaved Reprojection, which drops the framerate to 45 fps, and then doubles (and warps the second copy of) each frame. This is enabled when the fps drops below 90. More info 'here.

Latency

[Kjella]

Broadcast TV with this stuff on has considerable latency. It looks at the next frame and then interpolates its way there, might work reasonably well to watch a recorded show but would be pretty horrible for real time motion from VR. Next question?

Re:Latency

[ndnet]

This!

Even with a stable framerate, this technique intentionally delays the next frame to add compensation frames.

As an example, let's have a magic VR helmet running at 120Hz and instant processing (ie, 0ms GTG time, which doesn't exist) and a video card capped at a perfectly stable 30 FPS (aka 30Hz).

We will split a second into ticks - we'll use the VR helmet's frequency of 120 Hz, so we have 120 ticks, numbered 1 to 120. (Just to annoy my fellow programmers!)

We therefore get a new actual frame every 4th tick - 1st, 5th, 9th, etc.

Without motion compensation, we would display a new frame every 4th tick - 1st, 5th, 9th, etc.
With ideal (instant) motion compensation, we can't compute a transition frame until we have the new frame. So we could, theoretically, go real frame #1 on 1st tick, computed frame based on #1 and #2 on 5th tick, real frame #2 on 6th tick, computed frame based on #2 and #3 at 9th tick, etc.

This would also be jerky - 2 motion frames then 3 at rest? We could push the frames back a tick and fill the interval with three compensation frames, but then we increase the delay, which is always higher than this example (and is multiplicative). So we'd have frame #1 at 5th tick, computed frames at 6th/7th/8th, frame #2 at 9th tick, etc. You've now introduced a minimum 4 tick delay, which at 120Hz is 1/30 of a second, or 33ms! To an otherwise impossibly-perfect system!

What about historical frames instead to PREDICT the next blur? Well, then, when something in-game (or, really, on screen) changes velocity, there would be mis-compensation. (Overcompensating if the object slows, undercompensating if it speeds up, and miscompensation if direction changes).

There's more problems, too:
- This doesn't help when the video card frameskips/dips.
- Instant GTG and instant motion frame computation do not exist. At best, they're sub-tick, but you'd still operate on the tick.
- Input delay already exists for game processing, etc.
- Increased input delay perception would be exponential to the actual length of the delay. For example, 1-2ms between keypress and onscreen action? Hardly notable. 50ms delay just to start registering a motion on screen and course correct? Maybe OK, maybe annoying. 150-200ms? Brutal.

It's not throughput, it's latency.

[interiot]

It's not throughput that matters, it's latency. If there's more than a tiny delay between turning your head and your eyes seeing the viewport move, then many people get bad motion sickness.

Because it works by delaying frames

[Qzukk]

In order to add a frame between 1 and 2, you have to have received both frame 1 and frame 2. People are already getting sick because what they see and do don't match, you're going to make it worse by making what they see lag further behind what little the headset picks up.

Latency

[PhrostyMcByte]

High frame rate is important for motion data, but it's also (and perhaps primarily) about latency. It is *much* more jarring in my Rift to experience latency -- akin to rapid sea-sickness -- than to have a lower frame rate.

Getting more motion information would be great but we can't sacrifice latency for it, and those TVs tend to have a very noticeable amount of latency. Not that this is an unsolvable problem -- I just haven't seen it yet.

Latency

[maorb]

The biggest reason is latency. People start getting nauseous when they turn their head in a VR headset and their view doesn't change quickly to match the head movement. Motion Compensation on TVs relies on having at least two frames (or 22ms worth of frames at 90fps if this were a current-gen VR headset) already at the TV in order to do the calculations for a frame in between, and in practice they could be buffering 2-3 seconds worth of frames for their calculations and you'd never notice that your TV is displaying your channel 2-3 seconds behind everyone else because it's not interactive.

Unfortunately even 22ms of desynchronization between our heads and our eyes is uncomfortable for most people.

I imagine that at a high enough refresh-rate (180hz?), you could get away with the added delay (11ms at 180hz, 6ms at 360hz), but at that point the image would already be very smooth so you wouldn't be gaining anything of value.

Huh?

[Nemyst]

you need a GPU capable of pushing at least a steady 90 FPS per eye, or a total of at least 180 FPS for both eyes

Um, what? That's entirely wrong. You need a steady 90 fps, that's it. There's no doubling because of eyes, this isn't 3D TVs where you need to alternate frames. The only other concern is that the resolution is higher than 1080p.

Re: Huh?

[Miamicanes]

Actually, it's worse than that. 90fps is the rock-bottom MINIMUM. If you want a frame rate that satisfies the Nyquist minimum for high-contrast peripheral vision, even 400fps is on the low side.

The human eye is really hard to fully satisfy with immersive video, because trying to define the eye's "resolution" or "frame rate" is kind of like trying to define the resolution of an Atari 2600 or an Apple IIe. The fovea has relatively high color resolution, but comparatively poor luminance resolution. Peripheral vision has poor color perception, but is extremely sensitive to motion.

That's why 60hz CRT multisync monitors looked OK when you looked directly at them, but visibly flickered if you saw the screen in peripheral vision while focusing on something else, like a book in your lap.

ATW FTW

[WaffleMonster]

Oculus 1.3 runtime for the Rift was released with async timewarp. When it was released DK2 users used to earlier runtimes without it were all over the boards with phrases like "holy shit" and "DK3" to explain how ATW changed everything for them. Jitter issues magically disappeared overnight with only a simple software update.

More generally there is one and only one "trick" for improving VR quality going forward and that is foveated rendering. This technology is absolutely critical to any serious vision of future HMDs.

To provide some context cones of our eyes cover a massive (cough cough) 15 degrees of arc. That's it. You can't even lean back and read 1/4 of what is on your monitor without moving your eyeballs around to do it. 4k is overkill.. 1080 is overkill... The future in VR is entirely locked up in sensing eye orientation and optical and or electronic steering of relatively low resolution displays in response.

Re:Because there is no such thing as magic

[Jeremi]

You feel motion but you don't see it and your brain is drawing two different opposing conjectures.

... which is actually kind of amusing when I think about it. There's a watchdog circuit somewhere in your brain dedicated specifically to checking whether or not your sensory inputs match up, and when it detects that they don't, it assumes that you are drunk or high (or otherwise somehow poisoned) and initiates the upchuck routine. How many generations of questionable-quality-alcohol drinkers did it take to evolve that?