Measuring Input Latency In Console Games

The Digital Foundry blog has an article about measuring an important but often nebulous aspect of console gameplay: input lag. Using a video camera and a custom input monitor made by console modder Ben Heck, and after calibrating for display lag, they tested a variety of games to an accuracy of one video frame in order to determine the latency between pressing a button and seeing its effect on the screen. Quoting: "If a proven methodology can be put into place, games reviewers can better inform their readers, but more importantly developers can benefit in helping to eliminate unwanted lag from their code. ... It's fair to say that players today have become conditioned to what the truly hardcore PC gamers would consider to be almost unacceptably high levels of latency to the point where cloud gaming services such as OnLive and Gaikai rely heavily upon it. The average videogame runs at 30fps, and appears to have an average lag in the region of 133ms. On top of that is additional delay from the display itself, bringing the overall latency to around 166ms. Assuming that the most ultra-PC gaming set-up has a latency less than one third of that, this is good news for cloud gaming in that there's a good 80ms or so window for game video to be transmitted from client to server."

Comments

Transfers to PC Game Ports too...

[TheSambassador]

Only in the ports that the PC gets from the consoles (or even ones that happen to be released on both systems) do I notice the horrible latency. It's awful in Oblivion, Fallout 3, Bioshock, and plenty of others. Part of it has to do with V-Sync, but turning that off doesn't eliminate all of it. I can't believe that 133ms is the norm. I've grown up a PC gamer, and that's definitely one of the top reasons I *hate* console FPS games.

Re:Transfers to PC Game Ports too...

[Kral_Blbec]

Side note about Oblivion and Fallout 3. I think it is delayed intentionally to make it feel like someone actually is moving and make it more RPG-like.They aren't supposed to be twitchfests. Many FPS have the char move so fast it isn't humanly possible, turning, running, switching weapons etc.

Re:Transfers to PC Game Ports too...

[KulSeran]

1) Input is often sampled only once per frame. That is why quake at 120fps feels more responsive, the time between you pressing a button and the game noticing you pressed the button is reduced.

2) Input and actions are often determined on a per-frame basis. Meaning the fastest delay you can get is a single frame. Consoles tend to have games that run at a target frame rate (30, 24, 60) that determines how much visual flavor the game can have (60hz leaves less time to draw and update stuff than 30hz). So, at 30Hz, the fastest you can hope to see an action is 1 frame after the game detects it. That amounts to 33ms-66ms depending on your timing of the press in relation to the frame processing (we are asynchronous to the technology after all).

3) After a game renders a frame, it is usually buffered, so it has to swap and display the buffer. With V-Sync (consoles tend to v-sync automatically) that means it has to wait the 16ms for a 60Hz screen to refresh. But it only attempts to swap once a frame, introducing an aditional frames worth of delay in the display.

This is where the 99ms minimum response for a 30FPS game came from. 48ms for a 60FPS game.

Then you have to take into account that because of threading and networking, there are often more buffers in the game that only swap once a frame. This can introduce additional frames worth of delays as developers attempt to use the hardware to its limits.

Reality check

[girlintraining]

...average lag in the region of 133ms. On top of that is additional delay from the display itself, bringing the overall latency to around 166ms.

Considering that until very recently all displays had an inherent lag of about 70ms -- and that new [LCD] technology has pushed that higher. But we're only considering half the equation: The average human response time for auditory or visual input is 160--220ms. This increases as we age. We are also part of this system and we're a helluva lot more lagged than our technology is.

I want an upgrade.

Re:Reality check

[Mprx]

The only inherent display latency of a CRT is the time taken for the beam to arrive at any particular part of the screen. In the worst case this is one frame, which at a reasonable refresh rate (100Hz+) will be only 10ms or less. A good LCD (there's only one on the market, the ViewSonic VX2268wm) updates in the same line by line fashion as a CRT, and will add only a few more milliseconds switching time latency.

Of course you still have the latency in the input/processing/rendering stages, but this doesn't have to be very high (increase input sampling rate, avoid any interpolation, disable graphics buffering, etc). The only reason most modern console games are unplayable is because reviewers all ignore latency, and low latency can be traded for higher graphics detail which the reviewers pay attention to.

Perceived latency has nothing to do with reaction time.

Re:Reality check

[Hurricane78]

The average human response time for auditory or visual input is 160-220ms.

You know exactly that you're talking bullshit. The statement is true, but is irrelevant, because this is the response time when the pipelining of predicted actions does not work. How else would we be able to do any high-speed actions?

The brain *expects* a bang and a flash when we press the pistol trigger. If it's too late, this will show later, when the predictions and reality are compared again.

You see the monster, and pipeline a shot, some ms later, your hands press the trigger. Now you get the signal of higher pressure form your fingers which goes into the input pipeline. But the bang and flash arrive much too late at that same pipeline. So when they later come out of it again, the discrepancy still is there. Which messes with your ability to predict things.

Try playing a keyboard with 100 ms of lag in-between. At 200 ms it is next to impossible.
Try it with a online shooter with the additional 200 ms ping. Good luck winning that match!

Re:Reality check

[sahonen]

Actually, raw reaction time, which doesn't even change too much between 20 and 30, is not the primary element of skill at first person shooters. I've looked at the raw reaction time (i.e. click your mouse when you see a light turn on) of many gamers, some who absolutely dominate me and some at or below my level, and there was no real correlation between that reaction time and skill. From what I've gathered, I've determined that skill at FPS games is more a function of experience and training rather than raw reaction time.

The basic categories that set an elite gamer apart from an average or newbie gamer go something like this:

Predicting your opponent and being unpredictable yourself: Knowing where your opponent is going to be, and acting in a manner that your opponent can't predict. If you can put your crosshair where you know your enemy is going to be, and he can't do the same, you're going to win even if he has better raw reaction time than you. This is a function of experience with the game.

Decision making: Evaluating the importance of the various high-level goals in the game, deciding which ones to prioritize, and acting on that decision. Making better decisions, making them faster. Again, a function of experience with the game.

Aiming skill: If an enemy appears on your screen away from your crosshair, how quickly and accurately you can move your mouse to put the crosshair over him. This is a function of training, learning exactly how much mouse movement corresponds to how much movement on screen, and being able to precisely produce that movement with your hand. This is often confused for reaction time when watching people play, but really, the reaction time component is only in seeing the enemy and deciding to shoot him. The rest is muscle memory.

This is where input lag really hurts, it's very very important that your field of view appears to correspond to your mouse movements with absolutely no lag. Console games don't suffer from this because aiming with console controllers is far less precise than using a mouse, so the input lag "hides" behind the imprecision of the joystick. When the game meets the PC where people are using mice, the lag between moving your mouse and your on screen view changing becomes perceptible.

Movement skill: The ability to manipulate your controls to allow you to travel faster. Not just finding the most efficient routes, but being able to use quirks in the game's movement code to give yourself more velocity. Another function of training, getting the control inputs just right can be difficult to master.

Teamwork: In team-based games, communication, chemistry, planning, and effective group decision making.

DDR?

[koinu]

Anyone can make a comment how the lags affect gameplay on DDR? I still hesitate to buy an LCD TV and stay with my CRT, because I am not sure about it. When playing DDR, I usually listen to the music and the rhythm, so I really don't know exactly what would happen with a LCD TV.

I've seen people playing DDR with Samsung LCD TVs on Youtube. It seems it's working well.

Re:DDR?

[bjorniac]

DDR or any rhythm/timing based game will be perfectly fine with a fair amount of lag so long as the lag is consistent. The game isn't based much on reaction times, more hitting the pads at the right intervals. Once you get accustomed to the lag (which should happen naturally as you dance) the actual amount won't matter so much - you just have to move 160ms before the arrow hits the circle or whatever, something you will have been doing already, moving to land on the beat, rather than waiting for the beat and then moving. This differs from, say, a shooter like counter-strike, where you have to react as fast as possible to what is a non-rhythmic, supposedly non-predictable event (unless the opposing team comes out in synchronized swimming formation).

Inconsistency in lag would be a killer here, as it is everywhere, as it would be essentially adding a random component to your timing that you have no control over. But any time you do rhythmic work you're doing predictable lag compensation already - eg clapping on the beat requires you to start the motion before the beat happens rather than react to it.

Re:DDR?

[Anpheus]

One thing Rock Band has done, and presumably this came from somewhere else or has propagated to Guitar Hero and other rhythm games, is that you can set the video latency and audio latencies separately and finely tune the system so that it looks and sounds like you want it to be.

Rock Band 2's guitar controller actually has a tiny light sensitive component and a cheap microphone, so that you can auto-set your game. It's really very handy, and took only fifteen seconds or so. The result was that when a note crosses the "active line" of the game is when I should both strum it / hit it / sing it and hear the result.

Are you certain there is no way to do the same thing with DDR?

Re:DDR?

[mwvdlee]

I'm sorry, perhaps I'm misunderstanding you, but in the world of music 500 BPM is far from "low". Most "danceable" music generally is somewhere between 120 and 130 BPM, Drum-and-bass (which most people would consider quite fast) is about 170-180 BPM. Finding anything over 200 BPM is uncommon and usually for novelty sake. Perhaps the measurement you're talking about is something else than Beats Per Minute?

Re:DDR?

[CompassIIDX]

He shouldn't have referenced "BPM" because it's not really accurate, but by "500 BPM," he's talking about the rate at which the notes are falling down the screen. Many people take advantage of Hi-speed settings which allow you to increase this rate, thus decreasing the total number of notes your brain has to process at any given time. So a 125BPM song at Hi-speed 4 scrolls the notes at a rate of "500BPM." The actual beats per minute remain the same, though, obviously.

Re:DDR?

[Judinous]

Yes, by BPM I was referring to the DDR setting used to control the speed at which the notes flow past the screen. The reason that it must be turned up for higher difficulty songs has less to do with the number of notes on the screen at once, and more to do with the amount of separation between them. At low speeds, there are not enough vertical pixels separating the notes to distinguish the order that they are actually coming in, and whether they are simultaneous (jumps) or not. When played at "normal" speeds, the notes will even overlap each other making a solid "wall" that is nearly impossible to work out, even if you were to pause the game and dissect the screen at your leisure.

Re:DDR?

[nbates]

I used to have a monitor connected to my wii. Then I bought a Samsung LCD TV and I noticed the lag. Not directly, but indirectly. Both my partner and I noticed that we got worst in playing. We seemed to miss the markers every time.

I went through the manual and didn't find any lag data, but I found a "game mode" option. Turning the option on improved the experience and our scores. So I guess that you should read the manual before you buy an LCD TV to check if it has a "game mode". I read that this mode reduces the post processing in the TV so the signal is presented on screen faster.

Another place where I found lag was an issue is on sound. I thought about replacing two of my Home Theater's speakers by my TV's frontal speakers, sending those two channels from my PC to my TV and the rest of the channels to the HT. However, the sound on the TV was delayed by a small fraction of a second. Enough to be audible. You can hear an "echo". This could be solved right away if the TV had latency data, so I could force a delay on the rest of the sound channels. Too bad my TV doesn't have this information.

By the way, my TV is a Samsung 40' Full HD. Can't remember the exact model right now.

Musicians can detect very small amounts of latency

[silverspell]

It may be that console gamers have learned to expect around 100-150ms of input latency, perhaps thanks to visual cues that help to justify the latency on some level. (If I decide to jump, it takes a certain amount of time to react to my thought and make that happen; if I tell Mario to jump, maybe he takes about the same amount of time to react to the stimulus. It makes a certain kind of sense.)

But I assure you that musicians find that level of latency unacceptable. When you're playing a software synth live, performing with other musicians, even 75ms of latency is very noticeable and makes you feel like you're playing through molasses. Same thing with recording -- if it takes longer than 25-30ms to hear my own sound coming back at me, I definitely notice it. Virtuosic music regularly exceeds an input density of 50ms per event!

Atari 2600 has less latency

[Visoblast]

On the old Atari 2600, the game has to be written around rendering fields (half frames) of video. On NTSC, that is 59.94 fields per second, or a little under 16.7ms. Input is usually read during vertical blanking between fields. That makes for not much more than 33.3ms latency in the worst case of input change just after vertical blanking.

Maybe new isn't really better.

Re:Shooting Pause...

[Jeek Elemental]

I think what you see is simply hitting the max number of inflight bullets? Software limited yes but probably based on what the hardware can handle.
If the game uses hardware sprites (quite possible) it may be limited by the total number of sprites on screen.

So when you hit this max number you wont be able to fire any "new" bullets until an old one hits something or goes offscreen.

How can they miss this ?

[billcopc]

OK, I'll be the first to concede that I am more sensitive (or attentive) to lag issues, being an audio/video hack myself, but how can 4+ frames of lag be ignored or even tolerated in any action game ?

I already consider the 3-frame LCD lag inacceptable and utterly shameful.. I mean the data is there, put it up already! If the de-crapifying filters need that much lookahead to function, they need to be refactored to use look-behind, and if the copycat engineers can't fix it, at least give an option to disable it per-port so we can play our games.

Now on the development side, as a so-so game dev myself, I can't think of any valid excuse for Killzone's 12 frames of lag. What the hell are they doing in the loop ? Here's what a game loop is supposed to look like :

for (;;)
{
    if(button_pushed(1) && ga_hasammo(ga_PEW_PEW))
    {
        ga_plWeapon::spawn_bullet(); /* MOTHERFUCKING PEW PEW!!!1!! */
    }

    render_scene();
}

Notice the lack of "sleep(9000)" statements ? So that's what, 20 usec worth of code ? Take input, spawn bullet, play sound and draw the goddamned frame already! If that takes you 200 msec to process, then your game is really running at 5 fps with a shit ton of interpolated frames in-between, and you should probably go back to writing Joomla plugins.

Ten years ago, this shit would not have flown. We used to tweak the everloving crap out of our loops, and VSYNC was the norm, which made late frames painfully obvious. To deal with it, we used hard-timed loops and every single piece of code had to obey the almighty strobe. You had 16 or 33ms to render your frame, and if that wasn't enough well, you had to tweak your code. Today, now that even game consoles have gone multicore, there is no excuse. You could even have one thread acting as a clock watcher, monitoring the other tasks and telling them to hustle (e.g. degrade) if they're falling behind.

To prioritize anything else is to betray the game's purpose: to entertain via interactivity. If a game is going to sacrifice interactivity, I might as well go watch Mythbusters instead :P

The latency issue with the Wii.

[bezenek]

At the Hot Chips symposium last month, Rich Hilleman, Creative Director for Electronic Arts, commented on the 100ms delay inherent in the Wii remote (Wiimote). I assumed there was an issue in the delay involved in sensing the accelerometers, but this article shows 100ms is not any different from other consoles.

I wonder what Rich Hilleman was really getting at? Maybe people are more sensitive to delays when they are a result of a full-body-type movements rather than a button-press.

This is interesting stuff, and it would be a good thing if some graduate student did a thesis on it. (Free Ph.D. here--no thinking required!)

-Todd

Multiple Mismtatched Stimuli

[Plekto]

Often the real problem players have isn't the latency itself, because our brains will accommodate almost any lag as long as it's uniform(witness the lack of "frames" for most movies, despite being (usually) at a mere 24fps). What causes the problem is actually when you have more than one set of stimuli that are going at different rates. This is most noticeable with audio and video not being in sync.

With an LCD display, this is magnified greatly unless you are going directly from the computer or machine to speakers with their own amplifier built in(or headphones).

Case in point - I like to play Rockband with my son. On a CRT, it was fine. On a LCD I had to set the audio lag to 0ms. THEN set the video to sync with that. Adding delay to the audio as well as the video made it impossible to get a decent result - one has to be set to zero.

eg: audio lag is tested at 20ms. Video lag is 35ms. The correct settings are 0 and 15, because the audio will always have that delay in it no matter what you do. Putting both at the recommended 20 and 35 yields a combined 55ms between your finger and the result. Though they are in sync, they feel "laggy" as our brains are used to video running at about 60fps continuous/30fps interlaced if we grew up in the era of CRT displays. So 55ms feels like we just dropped 3 frames versus one in this situation. And that's just about at the threshold of delay between what we hear and what we see where it starts to become annoying.

Note - it's also why you need a $20 dedicated sound card. Often games hammer the CPU and on-board audio chip set when a big group of sounds come in and that also causes lag in Direct-X games for a moment(which most all console ports are, exacerbating the issue).

Re:you are full of shit

[mabinogi]

You're wrong.
20 milliseconds is 22 feet - that's quite a distance, and yes, it would be difficult to play with someone that far away.