H.264 vs. Theora — Fightin' Words About Patentability

An anonymous reader writes "Thom Holwerda from OS News has penned a rebuttal to claims from Daring Fireball's John Gruber that Theora is a greater patent risk than H.264. Holwerda writes, 'And so the H264/Theora debate concerning HTML5 video continues. The most recent entry into the discussion comes from John Gruber, who argues that Theora is more in danger of patent litigation than H264. He's wrong, and here's why.'"

Comments

Patent risks

[Elektroschock]

If you want to get rid off patent risks abolish software patenting of technical standards, embrace open standards.

Re:Patent risks

[Sir_Lewk]

You keep on bring this up. The answer is, and always will be, because software is math. Under US patent law math is not supposed to be patentable.

You might not agree with this, but that is in fact why most of us argue that software should not be patentable. I suspect you confuse comprehension with agreement.

Re:Patent risks

[im_thatoneguy]

So just because it's on a computer and not manufactured it shouldn't be patented. Poppy-cock.

The virtual world should get the same protection as meatspace. If you invent a new non-obvious widget you should be able to patent it regardless if a CNC machine or injection molder was required to manufacture it.

Re:Patent risks

[EvanED]

You do the realize that at the heart of the RSA algorithm (and most if not all cyptography algorthims for that matter) is a series of mathematical equations...

You could say the same thing about physical inventions, and it would only be a bit more far-fetched IMO. "At the heart of an internal combustion engine is just the physical principle that a fuel-air mixture burning will create an increase in volume." The presumably patentable thing in the case of the internal combustion engine is the clever idea that you can harness that increase in volume to turn linear motion into rotational motion, and use that to, for instance, move a car forward. Similarly, there may "just" be some number theory behind the RSA algorithm, but the thing I think should be patentable in its case is the clever idea that you can harness those equations in order to do public-key cryptography.

(I don't actually know whether the internal combustion engine was patented, or what parts of it were, etc.; but you get the idea.)

Therefore under US law it should not be patentable, yet for some reason because it relates to computing it is was granted a patent.

I can't speak to what should or shouldn't be patentable under current law; I'm just saying that I think a system that allows at least some aspects of software to be patented makes perfect sense.

Re:Patent risks

[b4dc0d3r]

Nope it's dishonest. The same thing applies to software and hardware patents. There's no distinction in that explanation that cannot be applied to mechanical inventions. You've made the case for voiding all patents, not just software.

A far more coherent argument would include the rapid evolution of software, with examples such as GIF. Of course GIF is a special case which allowed a better candidate, PNG, to become more common (with a side journey into IE's market dominance holding back PNG acceptance while IE's PNG support sucked). So GIF encouraged invention, the "legitimate purpose" of patents. So a good argument is difficult to make.

Very simply, math is abstract and remains free. The formula to calculate mortgage interest, or the location of a thrown object given the initial vector, gravity, atmospheric pressure (density), and external force such as wind, are not patentable. You can discover them and write papers and be influential in the field, but never patent it.

PKZIP had a patented compression method. Zlib did the same thing, just using a different method, and created compatible files. MP3 encoders bypassed Fraunhofer patents. Maybe the output wasn't byte-for-byte the same, nor the compression levels equal, but there is a serious hole in the argument for software patents when you can just do the same thing a different way and get around the patent.

The answer to Microsoft's Linux patent FUD is: Show us the patent, we'll work around it. The only place that fails is specific implementations like H.264 or FAT LFN or MPEG or SMB which are multi-platform. You can't program around a patented file format. Did you know that reverse engineering is valid for compatibility purposes? Why would that exist if not for a purpose? Then when you successfully reverse engineer something for compatibility, you can't use it because of patents. Why even have that exception if patents make it irrelevant?

That's my main argument these days - if I can break the DMCA for interoperability, I can ignore patents for the same reason.

The perfect mouse trap is invented, or the perfect lawn mower. I can choose to buy the patented solution, or go with a competitor. I buy in, paying extra for the patented hardware, and break a part. Is it allowed to fabricate my own parts to replace a broken one? Not if it violates the patent. So why can I reverse engineer for software interoperability, but I can't for hardware? Law may say one thing, here's my argument.

Hardware which is validly patentable does not have an interoperability requirement. Your saw does not have to work with other saws, so the parts may be patented. A company like Black and Decker could patent a battery which powers their tools, and they would own interoperability among products but I'm free to choose a competitor or build my own, which need no interoperability.

Software patents are increasingly used to protect a desirable object, so that content creators or consumers or both have to pay to create and/or access the content. Here's a new video codec, use it for a while while we hammer out the standards, then I pull the trigger and require payments. If you are the sole distributor of content and content will be consumed on your device (such as Pez and the Pez dispenser), patent away. But a method of packing up video to be shared with your devices, other companies, competitors and those in unrelated fields - interoperability is a requirement, and patents simply don't make sense.

I have a mathematical algorithm which uses psychoacoustic modeling to reduce the audio data which must be encoded, resulting in smaller files. Brilliant idea, which is unpatentable by itself. The algorithm is unpatentable by itself. I can develop a separate model to accomplish the same thing, better, using the exact same algorithm (with a different model underneath). Psychoacoustics happens to be a fundamental part of the universe, which should be unpatentable like (natural) DNA is, or a description of gravity.

So I u

Re:Patent risks

[pem]

There is a difference between "describe-able with math" and "is math".

Perhaps, but that difference is small, and shrinking everyday. I think that is why several people, including myself, feel that it is splitting hairs to let the whole math thing get in the way of the debate. It's a distraction. I believe that most, if not all software patents are bogus, and many hardware patents are bogus, but that does not *necessarily* mean that there should be no software patents, just that we're going about the whole patenting thing the wrong way.

Or to put it another way....

Simulations of CMOS computer chips are so good that (in most cases, if you are not doing a shrink to a new node where there are new physical problems to find), if you design a circuit, you can simulate it (using software), and have well above 90% confidence that it will work just like your sim after you spend a million bucks to build a mask set to do the photolithography do do the chip. (If you think about it, that must be true, else no one would bother building any really dense silicon chips at all because it would be too risky.)

SO...

The design, as simulated by the computer, just using software and math, is not patentable, because it's "just math." But OTOH, the design, as put into the real hardware, should then not be patentable, because its operation is utterly predictable (and predicted) by the non-patentable prior-art act of simulating it!

Now, you can use all the weasel words you want to distinguish the hardware from the software, and in fact, this is probably, kind of, sort of, how we got to the weasel words that allow software patenting "in conjunction with a machine." BUT, there is (to my mind) a fundamental problem that a chip designer using math and software tools to build hardware is not doing a job that is FUNDAMENTALLY more difficult or technical, or more important to the economy, or which requires more invention, or greater math skills, or even skills which are really that different than a lot of highly skilled software designers.

So, what distinguishes the results to make one patentable and the other one not. Well, we've already got "one is realized in real hardware." OK, where does that leave a logic design that could be stuffed into an FPGA or into a real chip? In one case, it's quasi-software -- easily reprogrammable, arguably as non-patentable as software should be. In the other case, it's an invention realized in hardware.

For a living, I write software, I write hardware that gets synthesized into CMOS, and I do emulation of the hardware in FPGAs. To me, these are all fundamentally the same, and it would be intellectually dishonest for me to argue otherwise. OTOH, I am the inventor or co-inventor of 16 patents, mostly hardware, and mostly junk -- the kind of patents that big companies put in their arsenals for the whole "mutually assured destruction" thing that fell apart with the arrival of non-practicing patent trolls.

So, my opinion is that the system is broken, but that in trying to fix the system, it is a counterproductive distraction to try to split hairs between hardware and software.

Before someone posts only the xiph link

[discord5]

So before someone starts the whole "which codec is better" flamewar again: someone at xiph thinks theora is better, ars thinks h264 is better, and this guy has a do it yourself kit in the form of a shell script.

Have fun arguing, as the past few articles have been quite fruitful in that area. Sadly few have realized (despite it being the main focus of most of those articles, but hey, who reads those) that quality will not be the merit to win this battle.

Re:Before someone posts only the xiph link

[TheRaven64]

That Xiph.org thing keeps being posted. I personally detest it because it compares H.264 and H.263 in the same page, so people who skim the article look at the smaller images and say 'look how much better Theora is!' And they're right, Theora is better than H.263. It's not especially clear which is better in the cherry-picked frame but, as you say, it is obvious which is better when you watch the videos. It's even more noticeable when you grab the lossless DIRAC version of the source movie how much both of them lose.

these don't seem like strong arguments

[YesIAmAScript]

In the first part, he takes Gruber's working (submarine patents) too literally. Gruber didn't mean literally patents that are applied for earlier but not granted yet. Gruber misspoke himself. Instead, he means companies who have patents that are already granted and they later will decide applies to new situation and then sue. If Theora becomes successful, it will meet with plenty of these, just as any other software success does now.

In the second part, oddly, given that he rails against strawmen, the argument creates a strawman.

The quoted response veers rapidly from addressing facts (whether Theora is within patent guidelines) to making a prediction 'I predict that MPEG LA may counter that they know groups have been pressured into licensing patents in order to use Theora.' Then it shoots down the prediction and thus claims to counter the argument. But that prediction is just a prediction, it isn't the issue at hand. And countering prediction you made up yourself doesn't necessary counter the actual argument which is that H.264 has a patent defense pool and Theora doesn't.

Thom is a jackass.

[carlhaagen]

There. I said it. Why? Because he counters Gruber's arguments with identical retorts, completely failing to see beyond his own nose, failing to realize and admit that all he is doing is just pulling his end of the rope in this tug of war, instead of coming up with anything worthwhile to consider in the choice of h.264 v theora.

It's been said, but it's important

[Endymion]

I love Free Software. I really do. I normally piss-off people with my fairly hard-line GNU/RMS attitude towards software. In most cases, I will drop features so I can run the Free version of something, and all of my code is GPL3.

But in this case, the so-called Free solution is the wrong choice to make. H.264 has won, and it won years ago. Now, an argument can be made that making a stand is important. But in this case, there is a pragmatic and strategic reason not to: taking a moral stand with Theora will damage other things, namely HTML5 and potentially Firefox itself.

PNG won out in the end over GIF, mostly, because it had better features. But what enabled that win was that they could both be used at the same time. If early Mozilla branches simply removed GIF support, the browser would have been dead in the water. Nobody would use it, because the images people already have were in GIF format. Only because both formats were supported could Mozilla be even considered by most people.

Today, people have data in H.264 format. A lot of data. A long list of hardware devices are made that support it directly. This data is not going to vanish, and people will want to play it. Firefox can choose to support that, or they can choose to become less relevant over time. Chrome is getting surprisingly strong uptake, and IE (ack) is getting much less offensive as time goes on. (aside: this competition is pretty awesome - browsers were starting to stagnate for a few years, and the rush for new features has been revived)

Playing people's data and being compatible with most modern and future hardware is the pragmatic reason to support H.264; the strategic reason is that the moral stand is not about video codecs! It's about removing Flash and related proprietary solutions. Playing the SAME video stream (a .mp4 in H.264 format) in flash or the <video> tag is neutral as far as codecs go, but it opens up the idea of a Free player.

The battle over codes needs to be left for another day.

As for how to actually implement it, Mozilla et al needs to take a cue from how distros handle MP3 and other patented codecs - foreign "non-free" repositories. The details on how you do that are highly flexible. Mozilla seems to like over-engineering things, so I'm sure they can come up with a Clever Codec Plugin Scheme to automate this, as long as the actual codec is 1) a separate project, and 2) developed outside the org.

Please - I love firefox, and this video issue is the one issue that could break them in the long-run. People like their YouTube.

addendum

[Endymion]

None of what I just said should be taken as a reason to not use Theora in addition to H.264. Push the Free solution, of course, but in parallel like what happened with PNG.

Firefox could actually be blind-sided by this

[MikeRT]

One of my fears here is that Firefox will be hit as hard by IE 9 as Netscape was with IE 4. Mozilla seems largely oblivious to how ambitious IE 9 is. A hardware-accelerated, multi-process, significantly more standards-compliant browser that supports H.264 out of the box would be just the thing for Microsoft to potentially stop Mozilla dead in its tracks on Firefox adoption.

A moral win?

[KingSkippus]

And what Mozilla and the Theora-or-nothing crowd are missing is that even staying with Evil H.264, the video-tag/HTML5 is still a huge moral win over Evil Proprietary Flash.

I'm sorry, but I just plain don't see it that way. It is simply substituting one proprietary format with another. In fact, using the "technically better is better, period" argument of the poster above you, because Flash includes more features than simple video, we should be striving against having a video tag and just continue using Flash.

The GIF argument just isn't applicable. When everyone standardized on GIF, there really wasn't a viable alternative that worked nearly as well. There is a viable alternative to H.264. Also, keep in mind that when GIF became a de facto standard, the legal environment surrounding patents was much different. It was a time when there was question over whether a compression algorithm could even be patented, and the chance that anyone would actually sue over it was virtually nil. Now, the sue 'em all strategy is actually a lucrative business model.

Come to think of it, didn't we go through many of these same arguments around 10 years after GIF became the de facto standard? Wasn't the questionability of the patent-encumbering of it a primary driver behind the development of the free PNG format? Didn't it take around like two friggin' decades for PNG to be as widely supported because we didn't really know better in making GIF the de facto format?

Don't you agree it's pretty damned stupid to repeat that exact mistake yet again under the whole "fool me once, fool me twice" tenet?

Re:A moral win?

[Endymion]

You act like H.264 and Theora are both new, and therefore one equal footing, and so there is a choice.

There isn't. MPEG video is already entrenched. It won so long ago, that hardware manufacturers are now assuming H.264 in most every device. Your "choice" is that we should somehow make the entire hardware and software industry magically switch away from the last few years of work they did, all the current and upcoming products they are releasing, etc.

Yes, I wish this wasn't the case, and I wish that a patent-free format was used instead. But wishing for things that fly in the face of reality is the attitude of religious nuts, not engineers.

My argument is that any patents in any of these formats, and all technical features, are 100% irrelevant. Normal people don't care. What they do care about is if they go to the local electronics Big-Box retailer and buy a camera, that they can post the video on the net. And that video will be in H.264 format. They care about watching youtube/etc. Which is H.26{3,4} format.

If a moral stand is desired, which it should be, it should be done by:
    1) Promoting the proper solution, patent-free, as an alternative
    2) Dodging the problem so you don't drive people away from your cause. ("make the codec separate from the browser")
    3) Use H.264 anyway, and accept the patent lawsuits as a proper form of Civil Disobedience, and get patent law changed.

The path Mozilla is taking is to going to cause normal users to say one thing and only one thing:
    "Hmm. I browse to $cool_new_video_site and it doesn't work. It does work in IE and Chrome. Firefox must be broken, so I'll use IE instead."

How is driving people away a win? The scope here is greater than a video codec.

Re:First Post

[MrNaz]

Why on Earth does HTML5 need to even specify the codec? I mean the tag doesn't specify an image format, why should not just have a src= attribute and any video supported by the system will play in it. That way it'd be the same as the change from GIF to PNG all those years ago, where those who want to use GIF could, and those who needed / wanted the free option (which was also superior) could use it without killing support for the other.

I don't see why this is an either / or issue.

Re:When h.264 isn't h.264.

[Endymion]

Yes, I am well familiar with the mess that makes up the technical features of "H.264", or more precisely, "MPEG 4, Part 10 AVC", the Part 2 variants ("XVID").

None of these technical features matter, as most people won't have any idea what you mean. What does matter is that people are currently buying cameras that capture video in Baseline profile, that magically works on a surprisingly number of devices. What matters is that many current devices, and most future devices support High Profile in hardware.

At no point does Theora enter into it. No devices make it, and no* devices play it (in hardware).

[*] Almost none. Exception are minimal and not significant enough to matter.

Mr. Horn, you're mucking FUD & I'm calling you

[Qubit]

According to the article here, MPEG LA CEO Larry Horn said this (emphasis mine):

In addition, no one in the market should be under the misimpression that other codecs such as Theora are patent-free. Virtually all codecs are based on patented technology, and many of the essential patents may be the same as those that are essential to AVC/H.264. Therefore, users should be aware that a license and payment of applicable royalties is likely required to use these technologies developed by others, too.

When asked directly about the MPEG patent holders:

Ozer: It sounds like you are saying that some of your patent holders own patents that are used in Ogg. Is that correct?

Horn: We believe that there are patent holders who do.

Okay, Horn: Who are the patent holders and what the patent numbers?

Ozer: It sounds like you’ll be coming out and basically saying that to use Ogg, you need to license it from MPEG LA. Is that correct?

Horn: That is not what we said. We said no one in the market should be under the misimpression that other codecs such as Theora are patent-free.

Ummmm... You're just spreading FUD and trying to be coy about it. But you just look like a smarmy used-car salesman. I call bullshit.

I have a good deal of respect for people like Monty who get this kind of shit thrown at them day-in and day-out from whatever weak-willed, money-over-morals, cardboard-cutout figurehead the MPEG-LA props up today to go and do their dirty work.

Mr. Horn, your arguments are hollow and your acts of fear-mongering are unbecoming of any man. I'm not sure I'd go so far as to call your actions reprehensible had you not graduated from Yale and then gone on to get a J.D. from Columbia. I mean, honestly, is the quantity of cash they're throwing at you so large that you can pile it on top of your morals like steel weights in a flower press, keeping your inner sense of honor pressed down so it doesn't jump up and kick your ass for being a manipulative and deceitful businessman?

Show us the patents or shut up.

Compression? Who needs it?

[itsybitsy]

Soon lossy compression will be irrelevant.

I demand full resolution video without lossy compression. Screw the compressors.

Bandwidth and storage capacity will soon make lossy irrelevant even at HD * N scales.

I want the FULL resolution that the cameras recorded.

If you don't then enjoy your pixalated movies.

Re:First Post

[Anonymous Coward]

No, he's not suggesting that all devices support all codecs. He's quite rightly suggesting that HTML5 should support all codecs. Why on earth should it be stuck with one? That makes no sense whatsoever. It would make HTML5 very limited and that is not something that W3C or any other standards body should ever even think of doing. The 'img' tag doesn't support one type of image format, nor does the 'object' or 'embed' tag support only one form of content for their usage.

By the way, Flash is NOT a video codec, nor a video standard. Flash video *IS* H.264 (MPEG-4) video.

Re:First Post

[gig]

> I don't see why this is an either / or issue.

In one word: standardization.

For the same reason we standardize markup around W3C HTML5, we standardize video around MPEG-4 H.264. We need to be able to make just one Web app and have it work on any browser from any manufacturer. We need to be able to make just one video and have it work in any video player from any manufacturer. If we don't have that, then consumers cannot choose their own preferred browser, or preferred media player. They get stuck using IE6 or Windows Media Player solely to decode nonstandard Web apps and video. It's not acceptable.

Modern consumer electronics devices have one video codec burned into hardware, and that is MPEG-4 H.264. This is almost 10 years old now. Same as DVD players all had MPEG-2. That's the reason the H.264 codec exists. If you want to publish a video that will play on iPod and other media players, iPhone and other smartphones, various set-top boxes, both FlashPlayer and QuickTime Player, both YouTube and iTunes, that is H.264. If you want to play video that was made with Flip camcorders, or Kodak camcorders, or Canon cameras, or Nikon cameras, or Panasonic cameras, or iPod/iPhone, that is H.264.

A key thing to understand is that MPEG-4 is not owned by any one company. The patents are not held by any one company. They are put into a pool and licensed equally to all comers. This puts all the consumer electronics manufacturers on equal footing. Flip is not going to cease to exist one day because a submarine patent takes all their devices off the market. The entire MPEG-4 group would address the submarine patent, all the manufacturers are protected from litigation in this way. That's just not true with Ogg.

On a Mac/PC, if you are somewhat technical, you can load all kinds of software codecs, most of which are made for authoring or some other special purpose, not made for consumer playback. Same as you can happily make Web apps for IE6 if your company uses IE6. But if you want to share Web apps with the world, you use HTML5. If you want to publish video for the world, you use H.264.

Also, you have to understand that video authoring tools all work with H.264 for many years now, and not with nonstandard formats. Where you see Ogg video, or Windows Media, or Real Media, or any other nonstandard media, they were very likely created from H.264.

This all has nothing to do with HTML5. As I said, H.264 is almost 10 years old and both YouTube and iTunes and both FlashPlayer and QuickTime Player play it. That *is* Web video. H.264 plays in Firefox today, and will play there tomorrow. HTML5 standardizes *markup* not video. So browsers now have to become video players. If Firefox doesn't want to do that, then they will see their users make an exodus for Chrome or Safari. There isn't any way to turn back time to when Ogg was current technology and rewrite history and re-encode the incredible amount of video that is stored in H.264.

If you imagine that Mozilla was saying "we can't support UTF-8" that is the same as them not supporting H.264. The UTF-8 text is already out there, and there's no other technology to replace it, and that is the same with H.264 video. A Web browser that can't play YouTube is not a Web browser.

Re:First Post

[nog_lorp]

If we don't have that, then consumers cannot choose their own preferred browser, or preferred media player.

Actually, if we DO have that, I cannot choose my preferred browser: an open source one. Yeah, big problem there for slashdotters - FOSS cannot implement H.264 because of the patent encumbrance.

Flip is not going to cease to exist one day because a submarine patent takes all their devices off the market. The entire MPEG-4 group would address the submarine patent, all the manufacturers are protected from litigation in this way. That's just not true with Ogg.

Licensing the H.264 pool does not protect you from submarine patents, but it doesn't matter as submarine patents are no longer viable.

It is true that it is incredibly important that codec management no longer be a complication for the user. HTML 5 handles this well:
<video>
  <source src='video.mp4'>
  <source src='video.ogg'>
</video>

There: it plays the "more advanced", more established, license-fee-encumbered codec if available. Otherwise, it falls back on the FOSS codec.
There should be a single format that all browsers support, but this codec should not require an exorbitant license fee without excluding a huge segment of the browser market (FOSS). As for protection from patent trolls, it is reasonable to presume that Google will side with any other company that is targeted for use of Theora.

All browsers should support Theora, and publishers can host H.264 (or any other codec) seamlessly on top of that if they want.