The End of Video Coding? (medium.com)

An anonymous reader writes: Netflix's engineering team has an insightful post today that looks at how the industry is handling video coding; the differences in their methodologies; and the challenges new comers face. An excerpt, which sums up where we are:

"MPEG-2, VC1, H.263, H.264/AVC, H.265/HEVC, VP9, AV1 -- all of these standards were built on the block-based hybrid video coding structure. Attempts to veer away from this traditional model have been unsuccessful. In some cases (say, distributed video coding), it was because the technology was impractical for the prevalent use case. In most other cases, however, it is likely that not enough resources were invested in the new technology to allow for maturity.

"Unfortunately, new techniques are evaluated against the state-of-the-art codec, for which the coding tools have been refined from decades of investment. It is then easy to drop the new technology as "not at-par." Are we missing on better, more effective techniques by not allowing new tools to mature? How many redundant bits can we squeeze out if we simply stay on the paved path and iterate on the same set of encoding tools?"

Comments

What else would one do?

[MBGMorden]

Should they just adopt new and inferior solutions and hope for the best?

To me this is the "science" part of Computer Science. Do research into new algorithms and methods of video encoding, but it would be stupid to start adopting any of that into actual products or live usage until and unless it tops the more traditional methods in performance.

Article is much more interesting than summary

[SuperKendall]

This is one case where the actual article is well worth reading, with a ton of links off to other areas to explore, and more interesting detail than the summary presents... well worth taking a look if you are at all interested in video compression and where the state of the art is going.

Re:Article is much more interesting than summary

[alvinrod]

If humans truly are incapable of discerning the difference in a controlled study, doesn't that suggest that the test is flawed because it is being too strict for some arbitrary reason?

To better illustrate what I mean,say I want to buy hosting for a service and want 99% uptime. However, the person considering providers throws out any without guarantees of 99.999% uptime. They're not actually doing what I want and I may end up paying more than I would otherwise need to for no good reason. Or suppose I have a machine that judges produce and will remove anything that it thinks shoppers won't purchase (as a result of appearance, bruising, etc.) so that I don't waste resources shipping it to a store that will eventually have to throw it out as unsold. I want that machine to be as exact as possible because if it's being more picky than the shoppers, that's wasted produce I could otherwise be selling.

Huh?

[Anonymous Coward]

Unfortunately, new techniques are evaluated against the state-of-the-art codec, for which the coding tools have been refined from decades of investment. It is then easy to drop the new technology as "not at-par." Are we missing on better, more effective techniques by not allowing new tools to mature?

What a stupid statement.

Is the expectation we adopt crappy replacements to "allow them to mature?"

They can mature until they're as good as what we have, not replace it with something which doesn't work to give it room to grow into something which doesn't suck.

Either you have a working replacement, or you have a good idea and a demo.

"Not-at-par" means the latter -- you don't have a mature product, and nobody is going to adopt it if it can't do what they can do now. Saying "ti will eventually be awesome" tells me that eventually we'll give a damn, but certainly not now.

It's bad enough I have to fight my vendors that I'm not accepting a beta-rewrite and suffering through their growing pains to get to the mature product they're trying to replace. I'm not your fucking beta tester, so please don't suggest I grab your steaming turd and live with it until you make it not suck.

Boo hoo, immature technologies which don't cover what the technology they're trying to replace aren't being allowed to blossom into something useful. Make it useful, and then come to us.

Misses the real problem.

[Anonymous Coward]

Let's say for argumentation that a new and much more efficient video codec was just invented.

The trouble is that it will immediately be locked up behind patents, free implementations will be sued, and it'll be packed with DRM and require per-play online-permission.

Our main problem isn't technology, it's the legal clusterfuck that has glommed onto the technology landscape.

I'd say we're moving at a pretty healthy pace.

[PhrostyMcByte]

H.264 was king. Now we've got H.265 and AV1 which have not entirely replaced H.264 due to compatibility purposes, but have still gained significant traction.

On the audio side, AAC replaced MP3, and Opus is set to replace AAC. Opus can generally reach the same quality as MP3 in less than half the bits!

So I don't see this stagnation they talk about. These algorithms are generally straightforward and codec devs, even if they don't have a hyper-efficient implementation yet, will be able to see the benefit -- it's just a matter of investing in their time to develop high quality code and hardware for it.

New vs old

[DrYak]

but it would be stupid to start adopting any of that into actual products or live usage until and unless it tops the more traditional methods in performance.

The logic behind the article is that the new techniques will never top more traditional (or at least could not have a way to achieved in the current state of affair), because most of the resources (dev time, budget, etc.) are spent optimizing the "status-quo" codecs, and not enough is spent on the new comer.
By the time something interesting comes up, the latest descendant of the "status-quo" would have been much more optimized.
It doesn't matter that the PhD thesis "Using Fractal Wavelets in non-Euclidian spaces to compress video" shows some promising advantages over MPEG-5 : it will not get funded, because by then "MPEG-6 is out" and is even better just by minor tweaking every where.
Thus new idea like a PhD thesis never get funded and explored further, and only further tweaking of what already exist gets funded.

I personally don't agree.

The most blatant argument is the list it self.
With the exception of AV-1, the list is exclusively only the actual list of block based algorithm : MPEG-1 and it's evolutions (up to HEVC) and things that attempts to do something similar while avoiding the patents (the VPx serie by On2, Google).

It completely ignores stuff like Dirac and Schroedinger :
completely different approach to video compression (based on wavelets) that got funded, developed and are actually in production (by no less than the BBC).

It completely ignores the background behind AV-1 and how it relates to Daala.

AV-1 was designed from the ground up not as an incremental evolution (or patent circumvention) over HEVC, it was designed to go along a different direction (if nothing else, at least for the reason to avoid the patented techniques of MPEG, as avoiding patent madness was the main target behind AV-1 to begin with).
It was done by AOMedia, where lots of group poured resources (including Netflix themselves).

Yes, on one side of the AV-1 saga, you have entities like Google that donates their work on VP10 to serve as a basis - so were's again at the "I can't believe it's not MPEG(tm)!" clones.

But among other code and techniques contributions (beside Cisco's Thor which I'm not considering for the purpose of my post), there's also Xiph who provided their work on Daala.
There's some crazy stuff that Xiph has been doing there : stuff like replacing the usual "block"-based compression with slightly different "lapped blocks", more radical stuff like throwing away the whole idea of "coding residuals after prediction" and replacing it with what "Perceptual Vector Quantization", etc.
Some of these weren't kept for the AV-1, but other crazies actually made it into the final product (the classic binary arithmetic coding used by the MPEG family was thrown away for integer range-encoding, though they didn't go as far as use the proposed alternative ANS - Asymmetrical Number System)

Overall, incrementally improving on MPEG (MPEG 1 -> MPEG 2 -> MPEG 4 ASP -> MPEG 4 AVC/H264 -> MPEG 4 HEVC/H265) get hit hard by the law of diminishing returns. There's only so far that you can reach be incremental improvement.

Time to get some new approaches.

Even if AOMedia's AV-1 isn't that much revolutionnary, that's more out of practical considerations (we need a patent-free codec available as fast as possible, including available quickly in hardware, better end up selecting thing that are known to work well) than for not having tried new stuff.
And even if some of the more out of the box experiment didn't end up in AV-1, they might end up in some future AV-2 (Xiph is keeping experimenting with Daala).