Concrete Comparisons of Theora Vs. Mpeg-4

icknay writes "With the upcoming Firefox 3.5 and HTML5 video, there's natural interest in Theora vs. Mpeg-4, but without much evidence either way. Here's clips encoded at various rates to provide concrete comparison between Theora and Mpeg-4. Theora performs decently, but requires more bandwidth than Mpeg-4 (although this is a 1.1alpha release of Theora and Theora has a much better license than Mpeg-4). The quality comparisons are very subjective, but you can try the clips yourself and see how it breaks down. There was an earlier discussion about this, but it lacked much concrete evidence. (Disclosure: it's my page.)"

Comments

Surprisingly different

[spud603]

I sort of knew Theora was a bit behind than Mpeg-4, but I didn't realize by how much. The Theora clip that has a 60% higher bitrate than the Mpeg-4 still looks fuzzier to my eyes (especially the moving grass).

Re:Surprisingly different

[stdarg]

I noticed in the last discussion that Theora does better when you take a single frame and look. It seems to have a lot more details. However, it's apparent in the clips that the detail comes at the expense of smoothness between frames. If you watch the background it's jumping around a lot, making it look fuzzy, presumably as Theora tries to preserve various details.

Re:Surprisingly different

[TheRaven64]

Theora is based on VP3, which is a generation older than MPEG-4. It's been improved a lot, but it's still old technology. Tarkin had a lot more potential, but a few years ago Theora was doing something and Tarkin was still mostly theoretical so the developers focussed on Theora. In hindsight, this may have been a mistake. Theora competes well with MPEG-2, but no one is using MPEG-2 for web distribution.

Longer term, Dirac looks more promising. It's comparable quality to H.264, is royalty-free, and has two open source implementations. Schroedinger, the newer one, is MIT licensed, and so can be use anywhere. Currently, the CPU load is too high for everyday use, however.

Subjective Measures

[Anonymous Coward]

Subjective measures are really the best way to evaluate video quality. There are (objective) quantitative measures such as PSNR, but they don't really tell you what the impact of video compression does for the eye. Video quality evaluations mostly involve showing clips (like these) to a large amount of people and asking them which they liked better. There is a lot to consider in terms of how the video responds to packet loss, jitter, etc.

60% more bitrate for same quality

[hattig]

The important line from the article: "Theora uses 1600kbps, or about 60% more bandwidth than Mpeg-4 to reach about the same quality."

Also useful to get some scale: "The uncompressed clip is 349 megabytes, while the 1600kbps Theora clip is 2 megabytes -- Theora may lag Mpeg-4 at this time, but it still yields great compression."

and "Theora is significantly better than Mpeg-2. Mpeg-2 required about 2400 kbps to hit the subjective quality level above, 50% higher than Theora's bandwidth."

Some things I would have liked to have seen: 250kbps, 500kbps, 2mbps, 8mbps videos, with subjective quality difference (rather than same subjective quality at different bitrates). Theora is apparently very good at lower bitrates, and not everybody has an awesome broadband connection, so they may be forced to watch lower-bitrate streams. Does the HTML5 video tag support selecting streams based upon available bandwidth?

License

[XanC]

The license is the single most important thing. It determines whether or not you can use the software at all, or for your specific purpose, whatever that is.

When we're talking about establishing a standard for the Web, which everybody is expected to be a) able and b) allowed to use, there is nothing more important than the license.

Re:I thought Theora was GPL-ed?

[Draek]

The reference implementation of Theora, like that of Vorbis is under a BSD-style license to help it gain wider adoption, so your point is valid even for propietary browser makers such as Opera.

Three "errors" in this test

[YA_Python_dev]

There are three things that this test doesn't consider:

1. for the same bitrate (1000 kbit/s) the Mpeg-4 file is 5.2% bigger than the Ogg one;
2. nobody uses video alone like in this test, there's always audio and the audio codec associated with Theora (Vorbis) rocks: same quality as MP3 for half the bitrate. Bits saved on the sound can be used to improve the video; and, yes, it is apples-to-apples comparing the overall bitrate of Ogg/Theora+Vorbis against an all-Mpeg-4 solution.
3. but the most important detail is that they used a constant average bitrate encoding with Theora, which is known to give inferior results for the same bitrate to simply setting the quality to match the desired bitrate.

For real life examples, that also include sound see "YouTube / Ogg/Theora comparison" and "Another online-video comparison".

Re:Three "errors" in this test

[clone53421]

If the codec is included with their browser, that won't be a problem.

Re:Three "errors" in this test

[Machtyn]

I believe the issue is that, even though Ogg Theora has a better license, the codec was really bad compared to other, similar codecs. At least, that has been the going concern. Given a choice between a better user experience or a better license *most* users will choose experience.

The need to prove Ogg Theora is better is to attempt to counter this concern.

Re:Help me out, please

[TheRaven64]

The license means that every product that includes an encoder or decoder for MPEG-4 (including AVC / H.264) needs to pay the MPEG-LA a small free for every version they sell (or give away). This is incompatible with Free Software. Imagine that FireFox included an MPEG-4 implementation. The Mozilla Corporation makes enough money that they could afford to pay the maximum annual fee for this license, but what happens after you download it? If you give a copy of FireFox to someone else, then you need to pay the license fee (except you can't, because the MPEG-LA doesn't offer licenses except in large quantities). Maybe Moz. Corp. could pay that license too, but what happens in a few years time when they decide to stop? Suddenly, no one can redistribute any copies or derived works of FireFox. The root problem is that it is not possible to get a license for MPEG-4 that permits the kind of arbitrary redistribution that Free Software entails. Although the license fees are capped, they are capped annually, so each year you need to pay again or you no longer have a license to distribute code implementing the patents.

This is why Theora is better as a standard format. Anyone can implement it, at no cost and with no restrictions. H.264 is better quality, and so makes sense as an optional format for HTML 5 to support, but requiring it would mean that it would be impossible for the second-most-popular web browser to be HTML compliant. Of course, in an ideal world, the W3C, Mozilla Corporation, Google, or some other interested party would just buy the H.264 patents outright and let them lapse, but somehow I don't think that's very likely.

Re:License

[Quantumstate]

People providing decoders have to pay the license fee.

"Royalties to be paid by end product manufacturers for an encoder, a decoder or both (âoeunitâ) begin at US $0.20 per unit after the first 100,000 units each year. There are no royalties on the first 100,000 units each year. Above 5 million units per year, the royalty is US $0.10 per unit."

This causes issues for free software especially with the gpl because there is a clause which says you cannot restrict the distribution of the code but by having to pay a license fee this is a restriction.