Netflix Uses AI in Its New Codec To Compress Video Scene By Scene (qz.com)

An anonymous reader shares a Quartz report: Annoying pauses in your streaming movies are going to become less common, thanks to a new trick Netflix is rolling out. It's using artificial intelligence techniques to analyze each shot in a video and compress it without affecting the image quality, thus reducing the amount of data it uses. The new encoding method is aimed at the growing contingent of viewers in emerging economies who watch video on phones and tablets. "We're allergic to rebuffering," said Todd Yellin, a vice president of innovation at Netflix. "No one wants to be interrupted in the middle of Bojack Horseman or Stranger Things." Yellin hopes the new system, called Dynamic Optimizer, will keep those Netflix binges free of interruption when it's introduced sometime in the next "couple of months." He was demonstrating the system's results at "Netflix House," a mansion in the hills overlooking Barcelona that the company has outfitted for the Mobile World Congress trade show. In one case, the image quality from a 555 kilobits per second (kbps) stream looked identical to one on a data link with half the bandwidth.

Comments

What exactly is Netflix doing?

[Anonymous Coward]

So, after reading the summary and RTFA, it is still not clear to me:
1. Are they sending different streams depending on user's bandwidth? That would not be news.
2. Are they preventing rebuffering in the middle of the scene by pausing at its end? Probably not...
3. Are they using different encoding parameters and bandwidths for different shots in a movie? That seems interesting... but apart from breaking down the movie by scenes, don't we already have variable bandwidth encoding?

Re:What exactly is Netflix doing?

[GuB-42]

What I think is that they devised an algorithm, probably based on neural networks, that is particularly good as estimating the perceived quality of the picture.
This data is then used to adjust the level of compression of each part of the picture, so that the least important parts of the picture get compressed more aggressively to save bitrate for the more important parts.
This is nothing new really, the idea of using AI techniques and perceived quality to help with compression is decades old. The interesting part here is that it is done on a commercial scale.

Re:What exactly is Netflix doing?

[Anonymous Coward]

My guess is it's the codec equivalent of "See that tree in the background? Yeah, that's going to be there for a while so just draw it once and leave it there until we tell you otherwise and we'll only send you the data about the stuff that's actually moving."

Re:What exactly is Netflix doing?

[squiggleslash]

Bear in mind that while they may be using a new codec - that is, a piece of code designed to convert a video from one format to another - they're not using a new format, if they did, then they could no longer use the hardware acceleration they pretty much rely upon to ensure devices can play their content in real time, and without destroying the battery.

So insofar as they're doing the above, they're not doing it any more than any other codec does, because only compressing changes is what every current codec does well, and pretty much a fundamental assumption behind each mainstream format, from MPEG-1 and H.261 onwards.

Re:What exactly is Netflix doing?

[Guspaz]

It sounds like they've basically just written a better psychovisual engine for driving variable bitrate encoding. Since their work was done in concert with some universities, it's possible that we could see it make its way into x264 and x265 in the future, if they publish their work.

Re:What exactly is Netflix doing?

[Anonymous Coward]

This is nothing new really, the idea of using AI techniques and perceived quality to help with compression is decades old. The interesting part here is that it is done on a commercial scale.

Sound like it is about the right timespan.
Next time there is an article about new improved batteries or some innovative way to create more storage space on a chip: Expect a couple of decades until it will be common enough for you to run into it without being on the bleeding edge.

Re:What exactly is Netflix doing?

[rainmouse]

I hate these things, they never work as stated. All it means is they can save money by buffering ahead even less. Days past if you had a poor connection you could load a video and pause it and it would buffer through. Now if you pause it buffers what it seems to think is all you actually need. It never is.

Re: What exactly is Netflix doing?

[Anonymous Coward]

Could be, but when AAC came around for audio compression, the interesting concept was that a pschyoacoustic model was applied to identify which of (at the time 7) multiple compression model paths would provide an the most optimal perceived result of the compression for the given audio samples. Better encoders would choose better paths for encoding. So, the best encoders would identify which compression method to use for each audio segment and how long said audio segment should be.

H.264 and H.265 offer a massive amount of opportunities to tweak encoding of macro blocks. There are spatial considerations (block size), temporal considerations, motion consideration, etc... each individual block can be a differ type (I, P, B, etc...). Each block can be mapped compressed relative to another block considering time and space. Each block can select a different set of coefficients for frequency identification (DCT for example) as well as gradients (quantization). Each block can be stored for optimal management of loss related to congestion (NAL).

To be fair, if I covered every possible case in H.265, I can be here a long time.

I used to write encoders for these standards and I would often target optimal allocation of bitrate relative to PSnR and SSIM. These are great metrics for attempting to model optimal quality following decompression. Unfortunately, I was too early to also optimize bitrate allocation relative to improved perceived quality relative to specific areas of interest which is something we can do today by applying computer vision modules that can simulate what is likely to be most interesting to humans and draw their attention. For example, consider that in "Back to the Future" when watching the scene where Doc types the date into the car computer, a computer can now identify that a human would most likely be drawn to watch the LED digits most closely. So, allocating a greater bitrate there would be better than to Doc's fingers and head movement.

Modern machine learning methods such as those used by Google to recognize a red dress in a photo and catalog it appropriately can easily be used for this type of encoding process and as Audun Mattias Ãygard has been publishing on his blog, these algorithms are public and well known today.

I experimented with this technology for identifying optimal image compression some time back. The tech wasn't ready yet

Bitrate allocation relative to areas of interest could easily allow for 50% bitrate savings. As the tech improves, I could see a great deal more especially in the area of quantization.

Now, if we used more AI for CABAC and CAVLC to precondition the dictionaries, we may have a great model for H.266

Re:What exactly is Netflix doing?

[Anonymous Coward]

That's what modern video compression already does. What is new here is they have an AI that makes a determination on what constitutes a noticeable loss in video quality and then seems to use that for more intelligently chosen keyframes (ie, when the "this bit is going to stay here for a while" is declared) and perhaps providing metadata on how far you need to buffer to get a "scene" so that they know when it may be safe to stutter for a second if needed without being noticed.

Re:What exactly is Netflix doing?

[Wootery]

This may well be DRM, not just over-engineering.

I wonder if Netflix worry about a situation where you can dump a movie to disk by doing a save-state on your VM. (See also: their offline-viewing feature is only available on locked-down devices.)

I've seen Chromecast cache over 3 minutes of Netflix, but I'm fairly sure that's about as high as it gets.

Re: What exactly is Netflix doing?

[Anonymous Coward]

That's what they do already. I was watching a gardening show on satellite TV and the signal was disrupted for a second. That was enough to fry the compression blocks of a gardeners face so that the next 30 seconds were of a large red Dahlia with eyes providing gardening tips with a Devonshite accent. I can't unsee that.

Re:What exactly is Netflix doing?

[AHuxley]

Creating a vast overview of different many nations telco conditions.
Optical, POTS to optical, coax. Is the national internet shared a lot and slow most nights?
Once that regional data is sorted movies can be made ready for local conditions. What is the actual connection from an ISP to the user when movies are been requested.
Is the connection good but the amount of bandwidth shared with a lot of other providers due to cost or other per nation issues?
Once local conditions are finally understood a movie can be compressed as needed per user.
Thats much better than testing for say "100" down once at mid day or late night when a user set up their system.
Night time movie watching might present very different network conditions as everyone else is using all the bandwidth in that part of a city or nation.
So that users movie is then very responsive to such issues rather than a one time network test.
So every part of an action movie, cartoon or drama gets just the compression it needs on very different networks.

AI?

[jgotts]

Why are they calling it AI? That's silly.

It's just an improved encoding scheme with better algorithms.

Nothing new to see here. We've been improving video encoding schemes since we started encoding video.

Re:AI?

[Anonymous Coward]

Because it's the trendy buzzword for software right now. Just like 3D printing was the trendy hardware buzzword a few years ago, and just as meaningless.

Re:AI?

[Cornwallis]

Because AI is this year's great new marketing term, like 3d printing was a few minutes ago, like cloud computing was a few hours ago...

All it means is someone can charge more for something, somewhere while us poor suckers get used to [my opinion] what most certainly will be dumbed-down lower-rez imaging.

Watch. In a few years they'll be pushing what, to my eyes, looks like 8-bit pixellated graphics as high quality and it will be accepted. But I'm a cynic.

Re:AI?

[TheRaven64]

Hi, welcome to 2017. AI is now defined by the media to mean 'thing using algorithms'. In related news, algorithm is now defined to mean 'scary thing the reader probably doesn't understand'.

Re:AI?

[grumpy-cowboy]

Let's call it : Algorithms Intelligence. :)

Re:AI?

[Anonymous Coward]

It also leverages blockchain microservices in the IoT.

Re:AI?

[Anonymous Coward]

We need to leverage artificial intelligence to implement a cloud based app to support user's Web 2.0 3-D printing requirements. By creating new synergies and shifting paradigms, we can maximize our efficiencies and grow revenue.

Re:AI?

[Anonymous Coward]

But we are in the 3D printed future already. Why just this morning I saw a 3D printed car parked outside a 3D printed house. It's not just baubles and D&D figurines.

Re:AI?

[Anonymous Coward]

To be really charitable, perhaps it is an intent to go beyond content neutral analysis of waveforms and errors, to an understanding of meaning and importance that a human viewer would likely notice in specific parts of a scene. For example: "here is a face, faces are important, let's keep the quality up here" vs. "here is a rock, this rock is just as good as a million others, we don't have to be exactly right here", perhaps someday up to something so high level and crazy as 'that's the New York skyline, as long as it keeps looking like the New York skyline I can play around with it tons and the viewer will still be happy'. An algorithm that could process an existing scene and make these qualitative decisions could be considered AI in the range of the current definitions of AI. And yes I say 'current definitions' of AI because it's such a moving target. And before anybody says 'universities have been researching that for years', well that's the point: getting from R&D to a commercial implementation would be a major advance and one worth crowing out. If Netflix has a system that can analyze thousands of hours of video and make these bandwidth reductions cost effective and reliable, that is a big deal.

Captcha: depends (as in, "Is it AI?" "Depends.")

Re:AI?

[wagnerrp]

What happened? I blacked out.

Re: AI?

[Anonymous Coward]

As well as machine learning, deep neural networks and cloud computing.

Re:AI?

[neutrino38]

Well no that is not silly at all. Modern video codecs are in fact a toolbox containing different techniques that are more suited for such r such type of scenes.
- action movies would have better movment over definition
- still sequences
- sequences where only part of the scene is moving.

Using deep neural network to
1- identify which type of scene and adjust codec settings on the fly
2- compare the rendering to the original uncompressed version
3- readjust if necessary and learn from the situation

would be a breakthrough IMHO.

Re:AI?

[MrL0G1C]

Most of which is already done and none of which requires AI.

Re:AI?

[Namarrgon]

The improved algorithms are being driven by machine learning. They trained it to recognise when a scene needs a higher bitrate to look good, so that humans don't have to guide the encoder.

I don't necessarily agree that all machine learning or neural networks merit the term "AI", but nor do I class software that must be trained on a dataset to function as an "algorithm", except in the broadest sense.

Re:AI?

[LinuxIsGarbage]

In addition to AI being trendy, there are robots, specifically the idea that robots are going to replace jobs.

Machines, and automation have been replacing jobs since the dawn of the Industrial Revolution, and has been more aggressive with the invention of Microprocessors.

For some reason people seem to think that there's a new phenomenon where Lt. Commander Data and Bender are going to be replacing jobs.

It's just

[jpellino]

middle-out.

Sigh.

[Anonymous Coward]

It's an algorithm. It isn't AI. We really should have never started using that term. Your 'AI' photo tools are still filters, this is still an algorithm, none of it is 'AI'. I have never seen such an epic failure on the part of supposedly knowledgable professionals (Zuckerberg is one of the worst, though it's likely partly marketing spin in his case) to grasp concepts and utterly miss the forest for the trees. The media is the dumbest of all. We really are more stupid today, to an alarming extent.

Re:Sigh.

[Anonymous Coward]

Maybe thats a lower case L not an upper case i... AL... for ALgorithm.

Re: Sigh.

[Anonymous Coward]

Algorithms are hand coded by a human programmer. Machine learning involves sets of images used to "train" a generic self-modifying algorithm to make a decision. It is fed millions of pictures of various scenes until it provides the correct decision.

You sa "AI"?

[bobbied]

I don't think that word means what you think it means...

Re:You sa "AI"?

[Anonymous Coward]

Or maybe it doesn't mean what you think it means?

GPU Transcoders

[caferace]

VBR isn't rocket science, and not new. Great that they're using it. GPU transcoding is really helping these days.

Re:GPU Transcoders

[kriston]

Exactly. DISH Network and DirecTV have been using this since the early 1990s and big-dish C-Band has used it even longer.

I fail to see what is new here, and, from what I understand, I'm even more surprised that Netflix wasn't aware of this technology from the very beginning.

Re:GPU Transcoders

[omnichad]

VBR within a frame is relatively new or not talked about much if it isn't. This is talking about which macroblocks to give more bandwidth to based on their content and relative importance within the frame.

Re:GPU Transcoders

[Anonymous Coward]

That's a super long way of phrasing "I don't get it"

For all the people saying this isn't AI

[Solandri]

Netflix does use AI in developing the video compression algorithm. The problem with encoding videos with lossy algorithms is that video quality is a subjective thing. You need a person to watch it and tell you how good the video quality looks. This makes it rather slow and difficult to do A/B testing, not to mention how boring it is watching the same clips over and over with different encoding.

Netflix got around the problem by using machine learning to teach a computer when video quality looked good. They had a bunch of people watch videos with different compression and rate the quality, then told the AI that their ratings were gospel. It then analyzed the different videos and decided for itself which features were associated with good quality. Once the computer was generating the video ratings as people, they had a rapid way to do A/B testing. That allowed them to optimize their compression algorithm in much less time than with using humans to rate video quality.

I'm not sure why Summary links to some popular news article which talks in general about Netflix using AI, instead of linking to the actual Netflix page describing exactly what they did. This used to be the sort of technical detail you'd expect from slashdot submissions.

Re:For all the people saying this isn't AI

[Anonymous Coward]

I agree this is the sort of technical detail we used to expect from slashdot submissions. Not anymore. Things were definitely better back in the days before we lost CmdrTaco to autoerotic asphyxiation. We warned him many times; but he just couldn't break the habit.

Re:For all the people saying this isn't AI

[DontBeAMoran]

http://s2.quickmeme.com/img/e8...

Re:For all the people saying this isn't AI

[dinfinity]

This is why /. should allow some comments to become 'top comment'.

Yours is more valuable than the submission, TFA and all other comments combined.

Re:For all the people saying this isn't AI

[zlives]

i wonder if they took into account viewing medium. i have a hard time comparing video details on an smaller screen vs a larger screen.
what looks good on a monitor/tv looks bad on a 160" 4k projector... to me
netflix does a decent job mostly though.

Call it what you want...

[cetan]

Call it anything you want: "Netflix uses bagels to compress video" I don't really care. I just wish they would take a closer look at the darkest parts of a scene and stop compressing the hell out of it. Visible gradients ruin every single scene always.

Re:Call it what you want...

[DontBeAMoran]

Maybe your display brightness/contrast settings are wrong?

Re:Call it what you want...

[zlives]

they do a better job on their 4K stuff for sure, still no comparison to 4k media. i wonder if they compress based on BW availability.

Re:Call it what you want...

[cetan]

I view streaming content on a variety of devices off of a perfectly acceptable cable internet connection and I still see the compression, but the worst of it is seen on the "main" family TV. Netflix offers the best experience (followed by Amazon Video, followed by the truly horrific Google Play), but it's still there.

I fully admit that I am not a hardcore video guy and not obsessed with tweaking a bunch of TV settings so there is indeed room to make adjustments. That said, I'm very happy with up-scaled DVDs of the same movies on the same TV. Adjusting contrast/brightness would only force the shadows even deeper for disk-based video and that's not an acceptable trade-off.

I should clarify my previous statement above. When I wrote "Visible gradients ruin every single scene always" I didn't meant to imply I'm seeing gradients all the time. I'm only seeing them in scenes containing large percentages of darkness/black.

Re:Call it what you want...

[DontBeAMoran]

Maybe their new CODEC guided by human votes will use more bandwidth for dark scenes from now on.

i beg to differ...

[Anonymous Coward]

"No one wants to be interrupted in the middle of Bojack Horseman or Stranger Things."
Actually, if I am every watching Bojack Horseman... interrupt me any way possible. Use bullets if necessary.
 

Netflix Could Have Been Good

[BrendaEM]

Netflix is disappointment growing by the days. Soon Nextflix may have only a single video in 1000 categories with compression showing a single frame as all.

Re:Netflix Could Have Been Good

[BrendaEM]

Sorry, trackpad got the best of me.

Re:Netflix Could Have Been Good

[Anonymous Coward]

Why the fuck do you post in monospace?

Re:Netflix Could Have Been Good

[Anonymous Coward]

Why the fuck do you post in monospace?

It certainly didn’t make the post any more interesting or worthwhile, which I think was the intention. That, in and of itself, was quite a feat, because the post wasn’t actually worth reading in the first place.

The Furture of Netflix

[BrendaEM]

Soon Nextflix may have only a single video in 1000 categories with compression showing a single frame as all.

Re:The Furture of Netflix

[Anonymous Coward]

Markov generated comment?

What did they use before?

[Anonymous Coward]

Natural Intelligence, or what?

(some press releases are so stupid -- they trigger a kind of denial-of-service attack in me).

Compression won't solve buffering.

[shess]

Except in edge cases, videos don't stutter because they take slightly more bandwidth than you have available. They stutter because the buffers aren't deep enough to overcome network jank, and my understanding is that streaming providers use shallow buffers for content-protection reasons (it's not like you're going to suddenly switch streams 45 minutes into a movie).

Put another way, the difference between a 500 kbps stream and a 250 kpbs stream isn't going to improve your rebuffering experience on a link with 25mbps of bandwidth available, because the problem is an artificial barrier between you and Netflix.

Re:Compression won't solve buffering.

[by (1706743)]

Tangentially related, it's rather infuriating that (at least on a Chromecast) going back 30 seconds requires re-buffering. Perhaps, as you say, this is due to content protection reasons.

In any event, it makes missing a bit of dialog a frustrating experience -- I'd love a "skip back ten seconds and turn on subtitles temporarily" button, with all the content already buffered...

It's a kind of censoring

[yes-but-no]

Take a scene of a pretty mother breast feeding. What a male considers as interesting blocks/parts of the image is totally different from that of a female. The AI may choose to drop details from one block than another based on its training set (or based on what it thinks the viewer cares). Essentially now the viewer is served only stuff that the server thinks what may be liked. That is it's producer doing the choosing; rather than the consumer. Not sure if it's a good thing or bad..but at times we want to see the original -- not the altered/watered-down version.

More marketingspeak.

[JustNiz]

>> compress it without affecting the image quality,

If the compression used is in any way lossy, affecting image quality is by definition inevitable.

Please stop calling simple algorithms AI

[marcgvky]

Please stop calling simple algorithms AI. Algorithm != Artificial Intelligence. Stop losing our language to the losers in the marketing department. Respectfully, Huge Dilbert Fan

Re:Please stop calling simple algorithms AI

[Anonymous Coward]

Please stop calling simple algorithms AI.

If you actually knew anything at all about what they were doing, then you’d know that it’s not just a set of simple algorithms. Incidentally, thanks for level-setting our expectations for you.

Buffer Interruptus

[fahrbot-bot]

"We're allergic to rebuffering," said Todd Yellin, a vice president of innovation at Netflix. "No one wants to be interrupted in the middle of Bojack Horseman or Stranger Things."

Or porn. "Yes, yes, yes..." (buffering ...) [ Nooooooooooooooo.... ]

Psychovisuals

[DrYak]

Yup, it's a psychovisual model.
Like there has been used in video compression for quite some time.

There is a primary source link mentionned elsewhere in this thread.

The novelty is that these one use machine learning (SVM according to the source).

(As opposed to older psychoauditive models used in compression of MP3, Vorbis, etc. which were based on clear rule, such as "a loud beat from a drum will mask whatever was playing the main melody".
This one learns automatically based on a crowd-sourced quality evaluation)

Re:Psychovisuals

[Anonymous Coward]

This links are also relevant:
http://techblog.netflix.com/20...
http://techblog.netflix.com/20...

"It's not in the box....

[Anonymous Coward]

....it's in the band"