ExtremeTech Wages War of the Codecs

prostoalex writes "ExtremeTech tested Windows Media, DivX, QuickTime/Sorenson and QuickTime/MPEG4 codecs. They encoded clips from Matrix Reloaded, Monsters, Inc., X2 and Spider-Man. QuickTime/Sorenson won the encoding speed contest, for the quality tests read the entire review, as each movie sample was encoded with 500KB and 1MB bitrates. Video samples provided on the site as well, so see for yourself."

Comments

stills vs. motion...

[garcia]

I took a look at there sample images and found very little difference (other than MPEG4 obviously) in their quality.

It is very difficult for ME to decide between them. I have never actually seen any QT movies up for download as far as real movies go. Most movies are encoded with divx and seem to work just fine.

Do people really care about minor differences in quality when the file sizes are down to 710mb? I know I don't. Blurred motion is just something I deal with when I download something.

Encoding time is important only if you do this regularly. For those of us just watching a movie it doesn't matter. Whatever gives me the smallest file size with a decent picture is what I want to go w/.

Re:stills vs. motion...

[badasscat]

One big factor which is neglected in this article is compatibility.

I don't think this was neglected at all; it clearly influenced the choice in what codecs to test in the first place. The four codecs in this article are the four most popular - wmv comes standard on PC's, QuickTime is Apple's standard and works across all major platforms, Divx is still the non-Linux geek's codec of choice (and it works with WMP) and MPEG-4 is now supported in the latest QuickTime.

The codecs chosen for review, then, are the ones that work with the players used by the greatest number of people. A lot of earlier posters complained about this or that codec not being included here, but they obviously missed this pretty critical point of the article. It doesn't matter to me, as someone working for a commercial enterprise that has to encode videos for our customers, whether Xvid or whatever offers slightly fewer artifacts. Because the fact is my customers probably don't have that codec and aren't going to bother downloading it just for me. Even Divx is probably barely at the saturation point where it's worth covering in an article like this, but for certain purposes and for a certain audience (PC gamers, for example), it's worth considering.

As others have pointed out, there are articles out there dealing with the lesser-used codecs if you just want to know who the absolute quality winner is. But in the real world and unless you're encoding video only for yourself, whatever codec wins in absolute quality is basically irrelevant. What matters is which codec offers the best quality among those in widest general use, and I thought this was a decent article on that basis (though in all honesty simply seeing the examples is probably good enough - I don't need an explanation of how blocky MPEG-4 is in an image, I can see it myself).

And it seems to me that what this article is saying is that if you want to use a cross-platform codec that everybody probably has (even on Linux), use plain old QuickTime. If you want to encode for the geek crowd, use Divx. If you want the best quality overall and you don't care about excluding a small percentage of the audience, use WM9. Whatever you do, avoid MPEG-4. Simple, and helpful to any professional whose job includes either encoding or contracting out encoding of videos for customers.

I don't care.

[MooKore 2004]

Thanks to the hard work of the Mplayer team, I can play any video format I want. If you havent tried it, you should.

Get mplayer

Quality versus Speed

[jamshid42]

OK, the article blurb claims that QT/Sorenson had the fastest encoding times, but also had the third-worst quality (only QT/MPEG-4 was worse). DivX seems to have the best quality, which, in my opinion, should be the end goal.

Think about it, how many times are you going to encode a movie? How many times are you going to watch it? Typically, you are going to encode once and probably watch it multiple times. Therefore, I would happily accept a little longer processing time in the beginning if that means I will end up with a better quality production.

Re:Quality versus Speed

[cft_128]

Don't forget that speed is also important for bragging rights and computer platform holy wars.

Not the best evidence.

[markv242]

Posting still images isn't the best way to point out video artifacts due to compression. Post five seconds of compressed material (all of this qualifies under fair use) and let the users see the artifacts themselves. The human eye is much more likely to spot the artifacts in a movie because of our perception of motion.

Re:Not the best evidence.

[pla]

Posting still images isn't the best way to point out video artifacts due to compression.

If I considered that as their worst methodological flaw, I'd tend to agree with you completely.

However, for those who haven't bothered to read the article, two points (well, more than two, but as examples...) stuck out as completely invalidating their results:

"In fact, even the original high bitrate MPEG2 on the DVD struggles in places."

and...

"We therefore took the uncompressed clips and created new "master clips" by encoding them to very high bitrate (around 8 megabit) files using Indeo 5.1 compression, as all our test applications could easily read this format"

Anyone else spot the problem, here?

First of all, starting with lossy source material automatically injects artifacts into the video. A codec that looks for similar ways to trim bits as the original (MPEG2), ie, MPEG4, will natually have a distinct advantage in having fewer artifacts in the final result. Not that I can think of any means by which they could have obtained high-quality "raw" video, but any valid test of an encoder's capabilities would require them to do so.

And, as if that didn't introduce enough bias, they then reencoded in Indeo 5 format, before using each "real" codec under consideration. Again, that injects its own artifacts, and favors codecs that look for similar ways to trim bits. But, all four of the codecs tested can deal with MPEG2 as source material, so even the "to make it fair" argument falls flat here.


Overall, this so-called "comparison" has zero external validity, in the strict experimental sense. They managed to waste a few hours of CPU time, nothing more.

At the very least (if they couldn't get ahold of raw HQ video), going straight from MPEG2 would have given a meaningful comparison of "how it will look ripped from a DVD". But by the Indeo pass, they removed even that as a possible claim.

QTPro doesn't have the best encoders

[SideshowBob]

Sorenson Pro (which has 2-pass and VBR encoding) isn't available in the $30 QTPro package. Use Sorenson Squeeze or MediaCleaner.

Also, QuickTime's MPEG-4 encoder is not the best MPEG-4 encoder out there. But there are better ones available, and of course MPEG-4 being a standard, the output of those other tools will be playable in QT Player.

So to make the comparison valid, both in terms of encoding speed an quality, some other tool should've been used.

Thanks, I think?

[MalaclypseTheYounger]

What a wishy-washy article. To sum up and save you the 2 minutes of your life to read that article, all 4 techs are good, and they are all good for something, bad at others. I wonder if the author could have sat on the fence any MORE when comparing the codecs.

I for one, will continue to obey my DivX Masters, they have always been good to me. It seems that the author had a hard-on for QuickTime and M$, both of which annoy the ever living crap out of me... QuickTime, with it's little icon in the toolbar that just won't go away, and Microsoft because I just can't trust them to not tell MPAA which movies I have on my HDD.

$0.02 Flamebaiting, Trolling response concluded.
(and my Karma just got back to Neutral, dang it)

Stupid test

[stratjakt]

Test of some scenes from copyrighted movies? What're the results for? What format is best downloaded off of Kazaa?

And the samples are all live action.. Test encoding some hand drawn animation (ie; an old bugs bunny), a computer generated animation, a anime style animation, a dialogue type scene, a live action scene with a lot of action, black and white vs color, etc, etc.

The types of images on screen greatly affect the performance of different algorithms.

Plus, each codec has about a million tweaks and optimizations for different types of footage.

I doubt highly that there's one clear "winner". It's really not that simple.

Which is why I hate sites like ExtremeTech that always have to boil it down to "this product is the best, the rest suck!".

Like the ATI vs nVidia flamewars. ATI may benchmark faster, yet nVidia has effects in games ATI lacks. There is no clear "this one is the best". Or Intel vs AMD or Linux vs Windows, etc, etc..

Nothing in the realm of computer science is that simple.

Fair and yet unfair comparison

[ebrandsberg]

Unfortunately, due to the fact that the source was itself the result of mpeg encoding, it could unfairly impact the ability of the various codecs to handle the content. On the flip side, much of the content people are encoding is actually decoded content, i.e. from a digital camcorder, etc.

What would be interesting is taking the original raw film footage (that hasn't been digitally compressed with a lossy method) and encoding, then comparing the results.

On2 VP4

[Anonymous Coward]

As sad as it sounds, I'm not surprised that XviD was left out. After all, alot of these reviewers pay attentions to what is being *marketted*. But I am surprised that On2 failed to get their VP4 Personal Codec noticed by this reviewer. I guess On2's marketting group dropped the ball when it came to make ExtremeTech aware of it's offerings.

What were they thinking?

[NaugaHunter]

Aren't there any 'Stock-footage' type DVD's out there that include a license to redistribute they could have used? Or couldn't they at least have tried to get permission for the clips they used?

While they might not have want to try to argue fair use through education or reviewing, they could have found at least one clip they could distribute. Hell, rent a high end digital camera and make one. Tape traffic on a highway, both daytime and nighttime, and you've got a motion video test, or a fountain, or anything.

Sorry

[pclminion]

Sorry, but any "test" where there's no scientific definition of what an "artifact" is, nor any mathematical definition of "image quality," is total bull. Yes, it's important to include subjective experience in the criteria, but we also need hard numbers. Where are the hard numbers on luminance distortion? Chrominance distortion? How many bits per pixel do you pay for each decibel of noise reduction? What's the worst case performance (no correlation between frames)? Best case performance (no difference between frames)?

I'm sorry, but some hand-waving, subjective "Hey, this thing kinda looks better than that thing" is not a test. Calling it a "War of the Codecs" is even more ridiculous.

Of course people care

[poptones]

Not all of us use this stuff for DOWNLOADING MOVIES. I haven't downloaded a movie in ages - I don't have the bandwidth. However I have several on my hdd and use xvid not only for archiving music videos and shows I enjoy, but also to get around the general DVD suckiness (movies that degrade over time, stutter, and require me to go back to the disc every time I want to watch one).

I like having all my movies and music and shows just a mouse click away. No fondling media, no DVD drives whooshing and movies stuttering halfway throgh because some tiny piece of schmutz got on the precious disc. In order to do this, I don't care at all what 500kbs or 1mbps files look like - The Twins effect alone occupies about 2GB on one of my drives, and I still haven't been able to produce a rip of Natural City that satisfies me even when the last one I tried was nearly 4GB (lots of film grain in that one and I don't care to lose it).

Yes... many of us care about quality. In fact, this is the very reason I rip DVDs - so the programs I enjoy play (more smoothly) from my hard drive.

Becauses it's a Mac of course!

[Sycraft-fu]

Don't you know, EVERYTHING is better on a Mac. Heck, put an Apple logo on your glass, your drink will taste better :).

In all seriousness it comes form the fact that many Mac users toss around apple marketing terms (like Velocity Engine) without understanding what they mean (it's a floating point vector math unit, like 3dnow or SSE2). They just assume it makes things better since that's what the hype claims.

Re:But no Xvid?

[rbegga]

No kidding. Taking into account that they bash the Apple/Quicktime MPEG-4 quality, and that the article is published by Ziff-Davis (Who I consider to be the least credible source for information in the industry), you might as well be reading an article published by Microsoft about how great WMV is.

The opposite is true

[RoLi]

Whether they hide the first pass or not, you'll need two if you really want quality. It's the only way for the codec to know for sure where it can spare bits and where it can't.

If you always encode to 650MB or 1300MB/Movie, then yes.

But if you want to use disk space efficiently, you get too big file sizes for easily encoded movies and too bad quality for hard encoded movies.

IMO, nothing beats quality based encoding, ie. you specify a quality setting and the movie will have whatever size is needed for that quality.

As soon as the CD dies as the major storage for movies (being replaced either by hard-disk or DVD), we will hopefully see more focus on quality-based encoding and less on bitrate-based, because it's pointless.

not realistic

[halfelven]

XviD is not "just as good as DVD but at 700 megs". Be reasonable. XviD, DivX and QT/MPEG4 are actually close relatives, they all "speak" MPEG4 "dialects".

I played with a lot of different codecs, including MPEG4-like mutants such as DivX, XviD, ffmpeg, etc. If i limit myself just at comparing DivX and Xvid, then:
- XviD is slightly faster than DivX, all else being equal
- XviD has slightly better quality than DivX, all else being equal, but it's an extremely close call (and sometimes the opposite is true)

So, in the Extremetech benchmark, if you replace DivX with XviD, it would fare slightly better overall. But definitely nothing as ridiculous as "owning the competition".

Facts please, not emotional knee-jerk reactions. Thank you.

Re:But no Xvid?

[contradyction]

Taking into account that they bash the Apple/Quicktime MPEG-4 quality, and that the article is published by Ziff-Davis (Who I consider to be the least credible source for information in the industry), you might as well be reading an article published by Microsoft about how great WMV is.

OH NO!!! The article didn't say that Apple is 100% awesome and they didn't replace the 's' in Microsoft with a dollar sign. That must mean that the article was written by Bill 'Son of Satan' Gates himself!!! Or it could just mean that the videos generated by the Quicktime encoder didn't look as good as the ones generated by the WMV9 and the DivX encoder. Look at the comparisons for yourself and realise that in this case the Apple tool was a distant third.