Microsoft Patents GPU-Accelerated Video Encoding

An anonymous reader writes "Microsoft has just received a patent that grants the company the rights to GPU-accelerated video encoding, which may be the primary technology that takes advantage of the horsepower of the GPU in today's consumer applications. The broad patent covers tasks to perform motion estimation in videos, the use of the depth buffer of the GPU, to determine comprising, collocating video frames, mapping pixels to texels, frame processing using the GPU, and output of data to the CPU."

Comments

Extra Extra!

[WrongSizeGlass]

A Graphics Processing Unit has been used to accelerate video!

If this doesn't qualify as 'obvious' then we are all doomed.

Re:Extra Extra!

[autocracy]

It seems that the idea was apparent long before the patent came about. I think the underlying reason that we haven't seen it yet is that the tradeoff value wasn't present yet. The GPU must beat out the CPU by a sufficiently wide price and performance margin for the workload before anybody bothers with specialized code for it.

In fact, this is a math coprocessor revisited. Remember those?

Re:Extra Extra!

[onionman]

Why would it be obvious that hardware designed to accelerate 3d rendering - transformation, lighting and rasterisation - can accelerate the compression of video frames?

It seems that you are 'obviously' wrong.

It's seems incredibly obvious to me. Of course, I've worked on FFT code for Cray vector units which were around a long time before 2004. If you can't see the relationship between vector processing, FFTs, and any form of video compression/display, then perhaps you shouldn't be in charge of determining what is "obvious" regarding this particular patent.

I have long felt that our patent system is ridiculous because it allows such silly patents. If something is obvious to an expert in the field, then it shouldn't be patentable.

Re:Extra Extra!

[Jeremiah Cornelius]

We saw this at SIGGRAPH for YEARS before 2000.

I think that Be Computer had colliding patents - not to mention SGI and nVidia.

Re:Extra Extra!

[dgatwood]

You're being sarcastic, right? Why would a glorified vector engine be useful for doing video compression, which is basically lots and lots of vector math? It's so obvious that anybody with even basic knowledge of video compression would immediately understand how the two problem spaces map onto one another with no instruction whatsoever.

It's so obvious that ATI released software to do it within a year of when that patent was first filed, which means they were working on it at least a year before that, which means that multiple people independently came up with the idea at the same time, which means it is obvious.

Heck, other companies had already been doing this, and even held patents on it five years earlier. Okay, so texture compression and video compression aren't quite the same thing. One deals with a single image, one deals with compressing a series of images.... Yeah, that's not obvious to anyone who has never seen someone make a flipbook during class in elementary school.

Re:Extra Extra!

[Cornelius the Great]

In fact, this is a math coprocessor revisited. Remember those?

Yes, those chips that handled floating-point operations so well that they eventually were integrated directly onto the CPU die itself; ie- 80386 CPU + 387 co-processor evolved into a single 486DX with integrated FPU.

Still, I don't see the why you're comparing them to GPUs... FPUs were small in comparison, even compared to early fixed-function rasterizers from the 90s; today GPUs are multi-billion-transistor chips with hundreds of programmable stream processors (with faster/higher bandwidth memory) that not only cover all of the rendering pipeline, but can do general-purpose computation as well. While small GPUs are getting integrated into future CPUs (AMD Fusion, Intel Sandy Bridge, etc), I'm doubtful discrete graphics will disappear in the way x86 math-coprocessors did, at least for the foreseeable future.

Re:Extra Extra!

[dgatwood]

Except that conceptually, it's a trivial extension of texture compression, which video cards have supported natively since at least the late 1990s. The only reason we weren't doing video compression is that the video cards weren't fast enough and/or were too power hungry to offer an advantage over CPUs. The patent office should not be awarding patents for discoveries, and that's all GPU-based video decompression really is---discovering that suddenly GPUs are faster than CPUs and things that were impractical (but widely discussed) years before are now practical.

I've only skimmed patent, so it's possible that the summary sucks and that there's something novel and unobvious here, but at a glance, this patent really does look like an explanation for a straightforward mapping of video compression onto a GPU which with the possible exception of the motion estimation would have been obvious a decade ago. For that matter, if you had asked somebody "how would you do motion estimation on a GPU" a decade ago, they probably would have come up with a similar solution.

Then again, it's a software patent, and the design process for nearly all software is obvious to someone with suitable skills in the field, which is why these patents are almost universally crap anyway.

Re:Extra Extra!

[IICV]

That's why I'm an advocate of an adversarial patent system, something kinda like this:

You (as a private individual or a company) can sign up to be a "patent examiner", for a minor fee. You specify which areas in which you have expertise (and you (or your employees) may need certain certifications as specified by professional groups in that area, depending). When a patent is submitted, it is required that the patent clearly and specifically state what problem the covered art solves - e.g, in this case, it would be something like "efficiently encoding video using components found in a commodity computer".

Then, a few examiners are picked at random. They're given a day or so in isolation, with whatever reference materials they want to bring (no networked devices, though) to figure out how they would solve the problem. At the end of their isolation, they just need to produce a couple of sketches of how they would go about solving the problem.

If a majority of the examiners (who should be experts in the field of the patent) produce any solution sketches that are largely similar to the patent, the patent is rejected - because clearly, if when experts in the field set their minds to solving that problem they come up with the to-be-patented invention, then it's not novel; it's just an obvious evolution no one else has gotten around to doing yet.

Re:Extra Extra!

[onionman]

No, but providing someone with a unit to calculate (for example) a transformation doesn't mean that you give them an efficient way of computing FFTs. Of course if you give them a general purpose matrix-vector multiplier then it does. In 2004 a graphics card used a fixed function unit, and today it uses a general purpose one. Assumptions about how obvious it is that other applications can be performed don't carry back to previous generations of the hardware.

I do believe that you are arguing in good faith and that you are not a troll, and I understand your argument. Here are the things which I believe were obvious (to general practitioners in the field) in the 1990s:

1. Video display and video (de)compression are essentially large linear algebra problems which lend themselves to vector processing.

2. Video display hardware will continue to improve.

3. Eventually video display hardware will become sufficiently powerful to perform video (de)compression.

The reason I believe that those were all obvious to practitioners in the field in the 1990s is because they were obvious to me in the 1990s and I wasn't even a specialist in the field. (I was, however, working on large number-crunchy stuff). I would suspect that the true experts in those fields were well aware of all three of the above points even in the 1980s.

I'll have to end my participation in this thread, now, and get back to real work...

Re:Extra Extra!

[jedidiah]

> It may be obvious now, but Microsoft applied for this patent in 2004, well before stuff like OpenCL and CUDA came around.

Nevermind OpenCL. How about things like PureVideo, VDPAU and xvmc?

Accelerating video with the video card is about as obvious as it gets.

This sounds like YOU patenting something: Visualize something that's obvious but hasn't quite gotten here to do hardware limitations and then rush off to the patent office.

The only difference between you and Microsoft is that Microsoft has lawyers on retainer.

Re:Extra Extra!

[MobileTatsu-NJG]

That's because they are almost all of the type 'use a [general-purpose device] to do [specific thing].'

No, they're not. That's what the eyeball-grabbing-summary says. The actual patents say "use these 10 steps to achieve this one result". For example: Slashdot ran a story that Microsoft patented page-turning on an e-book reader. There was a big dog-pile as people leapt in to claim prior art, citing a million Flash apps that allowed page turning. Nobody bothered to read the bit that described, in detail, how it was a gesture input that included things like how many pages to turn, what multiple fingers would do, and how the UI would respond graphically to let the user know how all that was going to go down.

This is not an occasional occurance. It happens every time. You lot are so busy looking for reasons to say software patents are bad you don't check on any of the things you're arguing about. You're shooting your own credibility in the foot, arguing with me isn't going to fix that.

Patenting the mere use of a product?

[EmagGeek]

Are they serious? This is virtually the same thing as someone inventing a car and me winning the patent on "driving cars."

Re:Patenting the mere use of a product?

[BatGnat]

It specifically says GPU, dont sell them as GPU's anymore. Lets call them MFPU's, multi-function processing units (or some other use of the letters MF).

Nothing to see here, move along.

[Anonymous Coward]

Oh look, another patent that shouldn't have been granted. The only thing the modern patent system is good for is buying new boats for patent lawyers. Does this still surprise anyone?

Re:Nothing to see here, move along.

[thePowerOfGrayskull]

I'm sort of afraid of the mayhem that will occur a bit before the current system will fall apart

Non-indemnified hardware running rampant! Patents running naked in the streets!

MS making their own thicket?

[ciaran_o_riordan]

This patent mightn't change much, but it's the weight of the hundreds of patents that's spoiling the AV field.

Microsoft is a member of MPEG-LA, but they pay more royalties than they make from the organisation, so they're probably eager to make their own AV thicket.

* http://en.swpat.org/wiki/Microsoft
* http://en.swpat.org/wiki/MPEG_LA
* http://en.swpat.org/wiki/Audio-video_patents

2004

[iamhassi]

Before we get a million "Adobe does this!" comments RTFA: "Microsoft applied for the patent titled “Accelerated video encoding using a graphics processing unit” in October 2004"

Far as I know no one was doing this in 2004

Re:2004

[samkass]

Before we get a million "Adobe does this!" comments RTFA: "Microsoft applied for the patent titled "Accelerated video encoding using a graphics processing unit" in October 2004"

Far as I know no one was doing this in 2004

Still not enough information. Patent claims can change between the original filing and the version that gets granted by amending the patent application. It's done by trolls... get a patent application going and keep it in limbo. Someone else comes to market with something cool, and you add it to your patent application. If it gets approved, viola! You have a patent that is "back-dated" years before the invention hit the market.

I'd be curious to see exactly what the patent said in 2004 and compare that to the then state-of-the-art.

Re:2004

[Anonymous Coward]

Try again with "GPU motion compensation". Plenty.

Enforceable?

[airfoobar]

The patent was filed in 2004, and there must be loads of prior art. Companies such as Nvidia and ATI have had GPU-accelerated video encoders for years now.

Regardless, this patent should never have been granted. It's all because of the patent office's massive backlog, and their decision to accept every random patent to reduce it.

Re:Badaboom?

[wagnerrp]

This patent is specifically for running motion compensation calculations on the GPU, and everything else on the CPU. Badaboom runs everything on the GPU, so the patent does not apply.

Prior art is EASILY provable -- this patent is BS.

[Anonymous Coward]

I can PERSONALLY prove prior art on this patent.

I have custom code from a project back in 1997-1998 for a Chromatic Research MPACT video card that used it to offload either MPEG-1 or H.263 video encoding process to the card.

I also have code from the same era that offloads both H.263 and/or MPEG-1 encoding to a video card that is based around a combination of a Trident 9xx series video chip and an 8x8 VCP.

So, I can PROVE I have WORKING code that does what this patent is for that was written in 1998 or earlier.

Re:Badaboom?

[DrgnDancer]

What is confusing? Microsoft does something. Microsoft applies for a patent on that thing. A patent lawyer who knows very little about the tech in question, has about 600 applications he's currently supposed to be processing, has been instructed that he can't work overtime this week by his boss, but also that he is too far behind on his portfolio and needs to catch up, and who doesn't make near as much as his buddies from law school do to begin with, looks at it. He thinks "I don't even know what half of these words *mean*", then notices that Microsoft filed the patent. Through his haze of pain and frustration he dimly remembers that Microsoft is an "Innovative and economy driving company" and says "fuck it." He hits the "Approve" button.

His boss is happy because his numbers are better this week, and there is no real penalty for approving patents that later get overturned. Even assuming that Microsoft ever attempts to defend the patent rather than just threatening small companies with it in hopes that they'll cave without a court battle.

The things currently wrong with the patent system which this story demonstrates:

1) Patent attorneys often don't understand the tech they are expected to review. This is less of a problem with "real" patents, since the device being patented is just that. A device. If it does what it says it does, in the way it says it does, understanding why isn't all that important. Software is essentially algorithms. If you don't understand them, then judging their uniqueness is difficult.

2) The reviewers in the patent office are phenomenally overworked right now. There are literally tens of thousands of applications backed up. I saw some patent official guy at the end of the Bush administration say that if all applications stopped, right then, he could maybe catch up in a year or two. I don't imagine it's gotten better. Both Bush and Obama have authorized more reviewers, but it seems to be like filling the ocean with a teaspoon.

3) Patent reviewers make a fraction of what patent attorneys in private practice make. This means that they're always looking to get out and get into private firms. Probably not all of them, but like any rational human, most want to make more money and get more respect.

4) There is no real penalty for screwing up. Most patents never get defended in court, because the companies that own them mainly used them as bargaining chips, or to threaten smaller, defenseless, companies. Even if the patent does go to court, it'll take years to invalidate, and no repercussions fall on the approver.

Eliminating software patents would, in one stroke, alleviate or eliminate two of these four problems. Probably the most serious two. It'd be awful nice if it happened. The alternative is probably the whole system collapsing under its own weight eventually.