Slashdot Mirror
Ask Slashdot: Why Don't Graphics Cards For VR Use Real-Time Motion Compensation?
dryriver writes: Graphics cards manufacturers like Nvidia and AMD have gone to great pains recently to point out that in order to experience virtual reality with a VR headset properly, you need a GPU capable of pushing at least a steady 90 FPS per eye, or a total of at least 180 FPS for both eyes, and at high resolutions to boot. This of course requires the purchase of the latest, greatest high-end GPUs made by these manufacturers, alongside the money you are already plonking down for your new VR headset, and a good, fast gaming-class PC. This raises an interesting question: virtually every LCD/LED TV manufactured in the last 5 or 6 years has a 'Real-Time Motion Compensation' feature built in. This is the not-so-new-at-all technique of taking, say, a football match broadcast live at 30 FPS or Hz, and algorithmically generating extra in-between frames in real time, thus giving you a hyper-smooth 200-400 FPS/Hz image on the TV set with no visible stutter or strobing whatsoever. This technology is not new. It is cheap enough to include in virtually every TV set at every price level (thus the hardware that performs the real-time motion compensating cannot cost more than a few dollars total). And the technique should, in theory, work just fine with the output of a GPU trying to drive a VR headset. Now suppose you have an entry level or mid-range GPU capable of pushing only 40-60 FPS in a VR application (or a measly 20-30 FPS per eye, making for a truly terrible VR experience). You could, in theory, add some cheap motion compensation circuitry to that GPU and get 100-200 FPS or more per eye. Heck, you might even be able to program a few GPU cores to run the motion compensation as a real-time GPU shader as the rest of the GPU is rendering a game or VR experience.
So my question: Why don't GPUs for VR use real-time motion compensation techniques to increase the FPS pushed into the VR headset? Would this not make far more financial sense for the average VR user than having to buy a monstrously powerful GPU to experience VR at all?
So my question: Why don't GPUs for VR use real-time motion compensation techniques to increase the FPS pushed into the VR headset? Would this not make far more financial sense for the average VR user than having to buy a monstrously powerful GPU to experience VR at all?
I find the effect sickening on a flat TV. I'd gather it's worse in goggles.
You can't change the angle at which the scene is rendered by interpolating between frames.
It's not the raw framerate. It's that the scene your viewing has to match where you're looking that quickly or you get motion sick.
Look up "asynchronous time-warp". /thread
Broadcast TV with this stuff on has considerable latency. It looks at the next frame and then interpolates its way there, might work reasonably well to watch a recorded show but would be pretty horrible for real time motion from VR. Next question?
Live today, because you never know what tomorrow brings
It's not throughput that matters, it's latency. If there's more than a tiny delay between turning your head and your eyes seeing the viewport move, then many people get bad motion sickness.
In order to add a frame between 1 and 2, you have to have received both frame 1 and frame 2. People are already getting sick because what they see and do don't match, you're going to make it worse by making what they see lag further behind what little the headset picks up.
If I have been able to see further than others, it is because I bought a pair of binoculars.
High frame rate is important for motion data, but it's also (and perhaps primarily) about latency. It is *much* more jarring in my Rift to experience latency -- akin to rapid sea-sickness -- than to have a lower frame rate.
Getting more motion information would be great but we can't sacrifice latency for it, and those TVs tend to have a very noticeable amount of latency. Not that this is an unsolvable problem -- I just haven't seen it yet.
The biggest reason is latency. People start getting nauseous when they turn their head in a VR headset and their view doesn't change quickly to match the head movement. Motion Compensation on TVs relies on having at least two frames (or 22ms worth of frames at 90fps if this were a current-gen VR headset) already at the TV in order to do the calculations for a frame in between, and in practice they could be buffering 2-3 seconds worth of frames for their calculations and you'd never notice that your TV is displaying your channel 2-3 seconds behind everyone else because it's not interactive.
Unfortunately even 22ms of desynchronization between our heads and our eyes is uncomfortable for most people.
I imagine that at a high enough refresh-rate (180hz?), you could get away with the added delay (11ms at 180hz, 6ms at 360hz), but at that point the image would already be very smooth so you wouldn't be gaining anything of value.
It doesn't help that the technique is also *very* prone to artifacts, and those artifacts would differ per eye, making them even more noticeable in 3D.
you need a GPU capable of pushing at least a steady 90 FPS per eye, or a total of at least 180 FPS for both eyes
Um, what? That's entirely wrong. You need a steady 90 fps, that's it. There's no doubling because of eyes, this isn't 3D TVs where you need to alternate frames. The only other concern is that the resolution is higher than 1080p.
It works in a TV broadcast because it's a stream. It will work in a 3d streaming video to a headset but that's without head tracking.
It can't work for interactive head tracking. It's a thing called latency and it's the reason people get headaches from VR. Your brain does it's own head tracking and when what you see doesn't match you get vertigo and or a headache.
You also get vertigo from confusing your brain by spending too much time in zero G. A ride in an elevator can make you lose your cookies. That's because it remembers both what it expects with gravity and without. You feel motion but you don't see it and your brain is drawing two different opposing conjectures.
Actually, motion compensation requires both the current frame and one or more future frames to be able to compute intermediate frames. With a media player where the full video already exists it is simple enough to access "future" frames. With a TV showing video inbound from a broadcaster, cable company or media device, you can delay the output by 1/30th or 1/24th of a second, delay sound similarly, and no one is the wiser. P>
But in video gaming, delaying the video by 1/30 or 1/24 or even 1/20 of a second has serious effects on game play. You really don't want a gaming device that delays the video just so that it can insert extra frames, you need to see the most recent frame as soon as you can.
I'm an American. I love this country and the freedoms that we used to have.
NVIDIA does exactly that : warps the image according to most recent (or even predicted) head position.
Works on their recent high end cards
http://www.slashgear.com/nvidi...
Oculus 1.3 runtime for the Rift was released with async timewarp. When it was released DK2 users used to earlier runtimes without it were all over the boards with phrases like "holy shit" and "DK3" to explain how ATW changed everything for them. Jitter issues magically disappeared overnight with only a simple software update.
More generally there is one and only one "trick" for improving VR quality going forward and that is foveated rendering. This technology is absolutely critical to any serious vision of future HMDs.
To provide some context cones of our eyes cover a massive (cough cough) 15 degrees of arc. That's it. You can't even lean back and read 1/4 of what is on your monitor without moving your eyeballs around to do it. 4k is overkill.. 1080 is overkill... The future in VR is entirely locked up in sensing eye orientation and optical and or electronic steering of relatively low resolution displays in response.
It wouldn't do the job unless increased motion-sickness is the intended goal.
a) What you're describing is frame doubling or similar tricks (every manufacturer has it's own name for it). It only makes the system APPEAR better and brighter. This is often done by simply doubling the frames or turning the backlights on and off between changing frames. This makes it appear "crisper", brighter and smoother, it's only a trick though and most displays already do it, it doesn't actually improve the actual experience, only tricks your brain into thinking it is at first glance and often when you do have a real 60 or 120fps source, it interferes with it (and thus needs to be disabled, either manually or automatically). It does increase fatigue.
b) VR needs to be as close as realtime as possible. When you move your head, your brain expects the world to move right away (most current research pegs the human eye's temporal resolution to be at about 10ms or ~100Hz in bright environments). So when you make your 60Hz game (~16ms latency) into a VR game, you WILL get some issues because you are basically lagging an entire 5-10 frames behind the whole time (your input latency on very high end systems is typically ~10ms, processing latency and video card latency add another 20-50ms and then LCD latency (the time it takes for an LCD to go from black-to-white, not gray-to-gray) is typically another 10ms).
c) Most frame doubling that uses a 'smoothing engine' requires the next frame to be available. You're now adding 32ms of latency (on a 60Hz system or ~3 "frames" for your eyes) just to get a smoother image so that is entirely useless.
A 'good' VR system would require end-to-end control over the individual pixel arrays of the LCD displays as well as direct access to any sensor inputs (which all have inherent latency as well). The second you say "USB" you've already lost 10-100ms and when you say HDMI/VGA/DVI, you've lost another 8-16ms (because those are built for streaming images to "TV's". Hell, most video cards these days also use some form of double or even triple buffering (meaning your image comes out 1-2 frames too late already) in order to do all sorts of post-processing tricks (aliasing etc).
Custom electronics and digital signage for your business: www.evcircuits.com
These are not the same problems at all. Look at gaming monitors and Ping times to better understand the problem. (These aren't the same thing, but at least the underlying reasoning is closer). The VR problem is one of synchronicity. If your sitting looking at a screen of a movie or something, you don't need to change the image very often, and your brain will work it out. If there is something you are carefully paying attention too, like a ball in a sporting event. 30 FPS is not smooth enough (you will see gaps.), this also is worse with Tv's without phosphor display. And the ball will noticeably skip. So this trick gives you more frames with less time between them, so it won't skip. It's not perfect and some people get weirded out by the effect. BUT this has NOTHING to do with the VR problem. In this case it's synchronicity. Think about it, real life is relatively continues, there is no 30 fps per second, and there is reasonable belief that synchronicity has a much much faster ruler. You have a near perfect understanding between doing something and watching what happens when you do it. And if it seems real, but it's not. Turns out a large part of the population will get weirded out. And in the ol venn diagram way of thinking about it, that population and influential gamers is pretty damn high. So no it has to be fast ass cards. (there is a little bit of similarity, but one of the other problems has to do with speed of action. If your fast enough you solve by far the largest of synchronism issues. (remember when the rift first came out. Everyone was excited because they largely solved the synchronicity issue, but on low-res screens. The low res screen is problematic with moving input that is trying to reflect reality.If you move at the right speed you will get stuttering. While the timing will be right,but resolution stuttering will weird some people out. So it has to be fast enough to allow synchronicity enough, it has to be pixel dense enough that it doesn't skip unexpectedly differently from real life. None of this can be cheated, because we are not watching movies in real life. We don't ADAPT like when watching a movie.
I have a nexus player which I like, but if I cast anything to it, or play local content with MX player I get the soap opera effect making a 24 fps movie look like it's at 60. Is it the Nexus player doing this or my TV? I've posted elsewhere and some said it's my TV but if so, wouldn't it do it with everything? Like Netflix YouTube etc.
Drives me nuts.
Many posters have pointed out that latency in the graphics card is an inherent problem.
There's also latency in the monitor, and latency in the computer which creates each frame. Merging the functions of the monitor and the graphics card should allow a latency reduction of 1 frame (at the display rate). (This brings up new problems, of course. Interfaces change, and the monitor has to handle the weight and heat of the graphics card function - a problem if the monitor is a VR headset.)
Contribute to civilization: ari.aynrand.org/donate
... does exactly this, and it works a treat. Only difference to TV's is, it is only used when your framerate goes below 90 fps.
It does look weird when it goes on for longer periods, but the alternative would be juddering which is insta-puke-city.
A good VR experience (and preventing motion sickness) requires fast response time. This requires low latency of the entire chain from the motion sensing device, through the USB connection, OS process scheduling, scene calculation and rendering and any buffering in the video card and display.
A system that is able to respond quickly can obviously produce more frames per second. But just creating more frames per second without reducing latency will not help the experience feel more convincing (or prevent you from feeling queezy). It will just look a bit smoother.
In playback of canned video latency does not matter much. In fact, generating these in-between images actually increases overall latency as the system has to delay the next image while calculating and displaying the in-between image. As long as an equivalent delay is inserted into the audio nobody notices this. But it won't work for VR.
There is, however, a method to produce faster response without calculating more images per second. The most critical movements are those of the head and the change in the scene from such motions can be approximated by simple panning. It's not perfect, but does work to reduce motion sickness.
Stop worrying about the risks of nuclear power and start worrying about the risks of not using nuclear power.
There are already GPUs that can do 90FPS without interpolation techniques which would not provide as good of a latency or eye tracking. In a couple of years the same level of performance will be cheap and widespread. Why would manufacturers focus on short term tricks and not the real thing?
what good would extra 60 frames _from the past_ do to your VR experience? other than make you puke
BTW Microsoft once championed the idea of generating 3D from a bunch of 2D tiles, this would enable tricks like motion compensation for tile reusing. It went pretty well for everyone involved (read billions invested/wasted by all MS 'partners').
https://en.wikipedia.org/wiki/...
Who logs in to gdm? Not I, said the duck.
And even more important once we see single pass widely implemented it will truly be 90 fps with no confusion.
You ask why we don't generate a video frame that's halfway between the current video frame and ONE THAT WILL HAPPEN IN THE FUTURE. That is quite probably the stupidest question I've heard a nerd make.
I find it sickening when it's disabled, as motion is much more jittery and stutters while panning on today's blur-free LCD panels.
On earlier TVs that would mask this with blurring or phosphor fade between frames, this was not an issue. Today's sets have high refresh rates with little artifacts and blur, so pans and motion appear full of stutter, especially on a large set where an object/person in motion travels larger physical distances between frames
Twinstiq, game news
FPS isn't cumulative like that. 90 FPS "per eye" is just 90 FPS overall. It's the draw area (number of pixels) that increases. And, with what we currently have, it doesn't actually increase. Both Oculus and Vive are 2160 x 1200, 25% more than the usual 1920 x 1080.
You need to know what the next frame will be before you can interpolate towards it. This means delaying everything, That's a big no-no in VR.
Also, motion compensation is not perfect. That's why all these "60fps!" trailers people like to put on YouTube look quite underwhelming.
Real 48/60fps video looks a lot nicer.
systemd is Roko's Basilisk.
Time-warp is not motion compensation, not in the sense used in the question.
It does compensate for motion, yes, but "motion compensation" is a different thing in video terms (it should probably be called "motion compensated interpolation.")
systemd is Roko's Basilisk.