Slashdot Mirror
Microsoft Patents GPU-Accelerated Video Encoding
An anonymous reader writes "Microsoft has just received a patent that grants the company the rights to GPU-accelerated video encoding, which may be the primary technology that takes advantage of the horsepower of the GPU in today's consumer applications. The broad patent covers tasks to perform motion estimation in videos, the use of the depth buffer of the GPU, to determine comprising, collocating video frames, mapping pixels to texels, frame processing using the GPU, and output of data to the CPU."
A Graphics Processing Unit has been used to accelerate video!
If this doesn't qualify as 'obvious' then we are all doomed.
Are they serious? This is virtually the same thing as someone inventing a car and me winning the patent on "driving cars."
Oh look, another patent that shouldn't have been granted. The only thing the modern patent system is good for is buying new boats for patent lawyers. Does this still surprise anyone?
This patent mightn't change much, but it's the weight of the hundreds of patents that's spoiling the AV field.
Microsoft is a member of MPEG-LA, but they pay more royalties than they make from the organisation, so they're probably eager to make their own AV thicket.
* http://en.swpat.org/wiki/Microsoft
* http://en.swpat.org/wiki/MPEG_LA
* http://en.swpat.org/wiki/Audio-video_patents
Expert in software patents or patent law? Contribute to the ESP wiki!
... and most companies use dedicated hardware encoders like those produced by envivo and others. Besides that they still have to rigorously defend such a patent since this sounds like one of those that might get thrown out in court.
But what about programs like Badaboom that already use GPU acceleration in their encoding? Patents confuse me to no end.
Isn't one requirement of a patent for it to be non-obvious?!!!
That a well informed person in the relevant field probably wouldn't think of the invention in the natural course of events?!!!
No way in heck should this have been granted as this use is beyond obvious -- it is sitting on your face wriggling!!!
Before we get a million "Adobe does this!" comments RTFA: "Microsoft applied for the patent titled “Accelerated video encoding using a graphics processing unit” in October 2004"
Far as I know no one was doing this in 2004
my karma will be here long after I'm gone
This will make it almost impossible for smaller companies to make fast video encoding applications. They will have to start paying royalties if they want to encode video using the GPU in applications such as FRAPS or any video converter. Their products will either have to become more expensive or remain inferior to products made by larger companies.
The patent was filed in 2004, and there must be loads of prior art. Companies such as Nvidia and ATI have had GPU-accelerated video encoders for years now.
Regardless, this patent should never have been granted. It's all because of the patent office's massive backlog, and their decision to accept every random patent to reduce it.
Four of the top executives nVidia were found dead in a hotel room after apparently participating in a group suicide...
But what about programs like Badaboom that already use GPU acceleration in their encoding? Patents confuse me to no end.
The patent application was received in October of 2004 according to the article. So I assume Badaboom would have to precede that or produce some form of prior art preceding that date to defend themselves should Microsoft resort to litigation after failing to agree to a licensing deal with Badaboom's creators. Regardless, a cursory glance proves that Microsoft could out lawyer them whether they are right or not so I believe with a 98% confidence that BadaboomIt is facing some serious liabilities.
My work here is dung.
The problem with this patent might be that attempts to patent the problem, i.e. all solutions to a problem, not a specific solution. This type of abuse should be stopped. It isn't stopped by giving companies a slap on the wrist, that is invalidating a few claims and letting the others stand.
Of course, since I'm a slashdotter, I didn't actually read the patent ;-)
Hey don't blame me, IANAB
brb patenting using the internet for gathering information
Looks like it's time to patent how clever you can use MS Windows.
They make 31 specific claims. They specify offsets and macro block sizes. And using SAD and MSE in motion.
Could someone who knows look over the actual patent. Are they patenting some specific type of GPU acceleration or something everyone has been using?
I've never understood how anyone could patent using something for its intended purpose.
I can PERSONALLY prove prior art on this patent.
I have custom code from a project back in 1997-1998 for a Chromatic Research MPACT video card that used it to offload either MPEG-1 or H.263 video encoding process to the card.
I also have code from the same era that offloads both H.263 and/or MPEG-1 encoding to a video card that is based around a combination of a Trident 9xx series video chip and an 8x8 VCP.
So, I can PROVE I have WORKING code that does what this patent is for that was written in 1998 or earlier.
Pretty sure the prior art goes back waaaaaay beyond 2004. 3Dfx was out of buisness by the time this patent was filed. In other news, the fastest counterclaim lawsuit has been filed by any/all video card manufacturers in business before 2004.
A computing device used to compute?
Holy schmoley, say it ain't so!
Crowd sourced obliteration of this patent.
Let's list the prior art. If you know of a patent that is prior art please list it here. If you know of a program or computer science paper or article that is prior art please list it here. Provide links if possible. If you review the patent and find a flaw please list it here with your explanation of what the flaw is. If you find any part "obvious" please indicate why. Good hunting.
This patent is nothing more than a description on how to use a general purpose processor to perform specific tasks. Adding to that, it describes a way to use computers to handle video. And using GPUs to do work is fundamentally very old technology, as they basically are glorified vector processors. So, how can such an obvious and overreaching patent pass regarding such fundamental technology? Is this not a obvious application of this particular technology?
Slashdot, fix your code or at least hire someone who is competent at it to do it for you.
Please provide a link to a web page with further details. Thanks.
Weren't "Toaster" boards in Amigas doing video encoding on GPUs in the early 1990's?
thanks for sharring..
Serçuk
A programmable GPU just consists of a bunch of processors. How is encoding videos on them anything new compared to implementing these algorithms on any other specific hardware/architecture ?
I'd be pissed (and I don't mean drunk) if I was AMD (nee ATI) or Nvidia. Microsoft doesn't even make GPUs, yet now they've patented a way to limit one of the major uses of them and by extension, control the manufacturers of them. How long before the graphics houses need to license (i.e. pay extortion) to MS in order for their hardware to be fully utilized by end users without fear of a lawsuit? It's just another Microsoft tax by another name.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
all they describe there is how to express the motion estimation process in terms of the DX9 graphics api. the way paralell processors are programmed today (with OpenCL or CUDA) is very different and does not require any of the steps described in the patent. In fact the OpenCL programming model is equaly suited for programming todays multicore CPUs as well as GPGPUs.
This might not matter too much. Using the GPU to assist in video encoding might be less of a good idea than many people think. Many complex procedures during encoding are not all that suited for parallelization. Take entropy coding for example. You probably have most of a chance for doing anything useful with motion estimation, but that's still quite hard. A bunch of people have worked on adding GPU acceleration to x264, as part of their thesis. There wasn't any real success. Most of them failed to make it actually useful, since cache considerations and the like prevented them from using nicer algorithms than exhaustive search.
As for existing encoders, like Badaboom, they mostly aren't all that fast or good. You can probably beat them with x264 on fast settings and still get similar or even better quality.
foldl1' (\ a f -> (f =<<) . a) fs
A) If you haven't read the patent, read it. ...
B) If you don't understand how to read a patent (and odds are you don't if you've never written one or aren't a lawyer, even if you think you do), recognize that your knee-jerk reaction to it may not be accurate
Slashdot has a long and glorious record of flamefests because 99% of its readers don't understand patents or how to read them and think a well-written fairly-narrow patent is covering some broad obvious area.
Now I'm sure the flamefest will happen, anyway, but I suppose there's always a chance ...
If you actually read the claims of the patent, it only covers doing motion estimation on the GPU and everything else on the CPU. This seems like a rather obvious part to offload to 2004's marginally-programmable GPUs.
Why would a glorified vector engine be useful for doing video compression, which is basically lots and lots of vector math?
Because in October 2004, mainstream video cards weren't necessarily "a glorified vector engine". Many were still very much fixed-function.
One deals with a single image, one deals with compressing a series of images.... Yeah, that's not obvious to anyone who has never seen someone make a flipbook during class in elementary school.
Flipbooks are like motion JPEG: every frame is a keyframe, and no motion compensation is used. The claims in the present patent relate specifically relate to the use of motion estimation data.
'nuff said. I think there's another Anonymous Coward around here who claims he has a working cold fusion reactor in his garage, and another one who invented time travel and could use it, but he just doesn't want to.
Graphics cards are used to RENDER video all the time. But, the actual encoding of video has always been done with the CPU, at least up until a few years ago. It's because the encoding itself is a complex algorithm, and GPUs had not really been used to implement arbitrary algorithms before, only the simplistic, high-volume calculations needed to render 2D/3D scenes onto your monitor. It's not until recently that GPUs have become popular for highly-parallelized, general-purpose programming (GP-GPU) that these kinds of patents come about.
Still doesn't mean it should be patented. It's just like all those patents that are for "Patent for doing X... over the internet". It is silly, but not obvious, since GP-GPU is a relatively new field (in the last few years).
My submerged patent on using a space engine to SLOW DOWN a space ship will make me BILLIONS!
Check out part U19.
Someone invents the knife and Microsoft patents cutting meat with it. Society is not well served by patents like this.
Look, we already went through the era of patenting obvious, well-known process X "with a computer". Then we went through patenting X "on the internet". We look back on that now, and we say, "Duh, putting it on a computer or on the internet didn't make it novel."
Is putting it on a GPU any better? No.
Can I get a patent to use a CPU to Centralize all or most of the Processing that a computer does into a single Unit?
So, you mean that if I take 11 cylinders instead of 10 or 12, I could get a patent on a new car trunk, just because of that? Or, if I would get patent on 3 CPU word processing, or 4 CPU word processing or double butt *%&#+/@, whatever...
If I could, I would. Unfortunately, very little about Chromatic Research is available on the web. However, I can verify that, although the project ended prior to any commercial products SHIPPING, we had in lab demos of working H.263 encode/decode being done on a Chromatic MPACT board. Actually, I think I still have one of these old boards around here somewhere. I also saw working MPEG1 accelerated encode being done on an MPACT2 board, though, if memory serves, it could only accomplish I-frame only in realtime, but the code was there for full IBP delayed encoded. I also actually WROTE and worked with code that did similar operations with both H.263 and MPEG1 on an 8x8 VCP processor. However, the card that used the VCP part was a combo board with a separate Trident 9xx series VGA chip, so it might or might not qualify. Additionally, I have seen a very similar in house demo of the old Philips TriMedia part doing much the same in terms of video encode/decode. The biggest point to recognize is this however: 1) Chromatic Research at the least had in-house H.263 and/or MPEG1 encoding on a VGA GPU in 1997. 2) In 1998 the company laid off half its employees and shortly thereafter was acquired by ATI Technologies. Which means *ATI* has PRIOR ART.
I remember in the old-days. Then there was products like the Video Blaster. It was useful for encoding and decoding video. It was actually a video-sort of card. You would connect the graphics card, then you would connect the screen-output to the Video Blaster. Then the cable went from the Video Blaster to the monitor. And then you got an overlay on some area of your screen where you played your video. This card was not for high-end stuff, but really catered the consumer market. But, originally the first video was through dedicated video-processors. This was how it worked. You connected the camera to the special card, then this card would help out with the compression of the video.
Microsoft have not been granted the rights to GPU accelerated video encoding at all. They have been granted the rights to a method of doing so.
The claims seem to revolve around handling certain parts of video encoding in a GPU vs certain parts in the CPU but the site is slashdotted so I can't review it at the moment.
All that said, if I were looking for prior art, I would look at SGI patents for SGI's Indigo IMPACT and/or IMPACT Compression board hardware (e.g. see http://www.wordiq.com/definition/SGI_Indigo2) and even better, the slightly later "O2" workstation graphics they implemented in 1997 (see http://www.wordiq.com/definition/SGI_O2 ). The IMPACT graphics video handling was done all in hardware off the CPU as far as I know, but the O2 had a unified memory architecture and integrated graphics in such a way that some video texture operations were handled on the graphics chipset (the MJPEG compression?) and some in the CPU (texture storage in general purpose RAM). Whether this split of CPU/GPU operations matches the claims MS is patenting, I don't know and would welcome informed comment.
(More broadly, I would add that I thought PCs were doing video decoding on the GPU as far back as Nvidia's Riva TNT if not the slightly earlier Riva 128 (1998). Don't know any implementation specifics tho.)
--LP
...how well patents are benefiting small businesses. I mean, that's one of the biggest reasons to support patents, right? To protect the little guy.
So, when Microsoft gets a patent, it's big news. But if a tiny little company got the same patent, no one would know about it or hear about it. Then when Microsoft discovers the idea "independently" and finds out about the patent later on, no one would be the wiser. They could negotiate a negligible license fee and act like nothing happened, until a commercial linux distribution tries to do the same thing. Then they'd sue the distribution.
Until the public interest is served by patents, I'd be happy to see patents go away.
The diversity and expression of human opinion is essential to human survival.
Nvidia and ATI have had this technology out for quite awhile. How on earth did Microsoft nab a patent on this. Oh wait that's right, they use blind moles to read and approve patents.
Well, then you'd be the perfect person to look at the claims and tell us if they're doing the same thing you did! Here's a hint: if you offloaded *all* your encoding to the video card, the patent does it differently.
Since the patent only covers encoding on a graphics processing units, just invent a special video processing unit for that. Since the VPU can also do graphics work, it can replace the GPU on graphics cards.
The Tao of math: The numbers you can count are not the real numbers.
I'm confused, how can Microsoft get the patent for that? They don't even design GPUs? Beyond that, I've been using a GPU encoding app for over a year badaboom (or something like that) which allows any Nvidia CUDA enabled card to be used for video encoding instead of the cpu, actually quite fast for most conversions. This stinks to me, was this oversight on Nvidia and others parts for not patenting this, or being specific enough in the patent? Maybe I'm confused still as I don't see how they could award a patent to microsoft in 2010 for something I've been using, unrelated to microsoft, since 2008 (i think)
Apple demoed Core Video on July 1, 2004
http://macdevcenter.com/pub/a/mac/2004/07/01/wwdc.html
As for the APIs, Core Image and Core Video will be the graphic equivalents of Core Audio. For performance the key is that image processing will not be done in the CPU but in the GPU. Phil Schiller demonstrated the use of some of the more than 100 filters and transitions that will be supplied. He applied filters to live video using a simple application built on top of these APIs.
It shipped as a working OS X API in April of 2005.
I've had it, I'm writing my elected officials & USPTO to suggest a faster, cheaper, and reliable review system.
I am just as passionate about the patent trolling b******t that comes out of the patent office, but please allow me to explain a few things to you:
1) Many patent attorneys actually have degrees in engineering or science, and do in fact understand what the patent is for. The problem lies in the fact that most patents are granted for minor differences in how technical concepts are implemented.
2) Yes, they are asked to do a lot, but if you had a cozy government job with a guaranteed pension, do you really care? Since when is it the patent lawyer's fault when the Feds are the fox watching the chicken coup?
3) Please refer to number 2.
4) The penalty from screwing up comes in the form of courtroom drama when the patent is declared invalid. No repercussions for the lawyer? Almost never, no matter what field of the law. Should the lawyer go to jail with their client if there is a conviction in a criminal defense case? Most countries say no.
Software is already protected by copyright, which in my view is sufficient. Software patents are just another form of corporate taxation. When people finally care about this, maybe something will change.
The intent of the "ordinary skill in the art" phrase is that the person doing the judgement must be actually skilled in the art, but only to an ordinary level. Thus, it's not a layperson, but an ordinary person from the field in question.
So software patents must not be obvious to an average developer in that area of programming, genetic patents must not be obvious to a geneticist, and machines must not be obvious to a mechanical engineer.
Seems to me MSFT is creating a utility patent on a process where they knew the software patent would fail. That is in addition not only to this patents subject matter being obvious to those familiar with the state of the art (hence not most patent examiners, which is why the system is very very broken), but it is old hat to those in the industry having been done for almost 2 decades at this point. Just because the GPU is bigger badder faster down't mean things running on the GPU become re-novelized.
... or ... Well you get the idea. I am very much opposed to patenting the same concept doing the exact same thing because of the environment. No novel interface, no novel usage. This is just citing one possible application using a GPU for computes, when there were already many many apps doing this.
Time to patent running applications on CPUs that have more than 2000 pins on the package. or have more than 81 cores on the CPU. Or have an aggregate flops of better than 1 teraflop on a chip. Or use direct optical connections to the chip. Or running applications on a mobile device in a watertight shell underwater
- Tjp
I am in wallow with my inner money grubbing capitalistic pig. ... Oink!
Yet Another Science Fiction Patent
There are lots like it, patents on ideas that were only theoretically possible at the time (and not a groundbreaking idea to anyone familiar with the technology), but with predictable advances in technology would definitely become possible. Look for patents relating to electric vehicles (and I mean ANYTHING with electric motors and wheels or props) and "fly-by-wire" type systems (in applications where they wouldn't have been remotely possible until the 2000s or so) dating back as far as the 70s.
I wonder if, as a more tangible example, you could get away with patenting space elevators and launch loops. The principle is exactly the same.
"When information is power, privacy is freedom" - Jah-Wren Ryel
I thought you said Microsoft. Egad... you did say Microsoft. What's the world coming to when MS is stealing a patent that obviously belongs to Nvidia or AMD? Hell they can't even build an OS that doesn't suck. What would they know about GPU's?
"We are just a war away from Amerikastan. When god vs god the undoing of man." Dave Mustaine
I find it odd that the summary claims video encoding "may be the primary technology that takes advantage of the horsepower of the GPU in today's consumer applications." Haven't they ever heard of video games?
TMPGEnc (and I'm sure others) have supported offloading video encoding operations to NVIDIA GPUs via CUDA for several years at least. I'm sure there are other applications that support CUDA. How is this patent different from that?
I really shouldn't have used someone else's email address for this account.
Unless that code was published in one of the official publication it does not count as prior art.
You may want to have a read of the patent. As usual the slashdot submission is HIGHLY misleading, this is not a broad patent for video encoding at all, rather a very specific method of hardware accelerated encoding.
Microsoft would not win a patent fight with Texas Instruments. They've been doing this - using "GPUs" in embedded processors to accelerate video compression and decompression for at least a decade now, if not longer.
The society for a thought-free internet welcomes you.
"All Amiga systems can display full-screen animated graphics with 32, 64 (EHB Mode) or 4096 colors (HAM Mode). Models with the AGA chipset (A1200 and A4000) also have 128, 256 and 262144 (HAM Mode) color modes and a palette expanded from 4096 to 16.8 million colors. The Amiga chipset can genlock -- adjust its own screen refresh timing to match an NTSC or PAL video signal. When combined with setting transparency, this allows an Amiga to overlay an external video source with graphics. This ability made the Amiga popular for many applications, and provides the ability to do character generation and CGI effects far more cheaply than earlier systems. Some frequent users of this ability included wedding videographers, TV stations and their weather forecasting divisions (for weather graphics and radar), advertising channels, music video production, and 'desktop video'. The NewTek Video Toaster was made possible by the genlock ability of the Amiga."
http://en.wikipedia.org/wiki/Amiga#Graphics
Video Toaster 1990
http://www.youtube.com/watch?v=zyGCYoZ5Nlk
http://www.youtube.com/watch?v=ErU-cs1S0Kw
....WHOOOOSH!
Well, only Russian words from well known vocabulary may describe this situation. Thanks God, we in Europe do not have to obey stupid American patent law and practice! Pestets!
And how do you RENDER a picture (S3TX)? Oh, look. you ENCODE/DECODE the textures. Wow.
And no, the actual encoding has NOT always been done with the CPU. Earlier, it was done with a floating point co-processor. Whose descendants are the GPUs.
Man, it is tiring to have to explain this to the slashdot drones every time one of these stupid patent stories comes up.
THE PATENT ONLY PATENTS WHAT IS WRITTEN IN THE _CLAIMS_ IN THE PATENT. NOT THE TITLE OF THE PATENT. NOT THE ABSTRACT. NOT THE DESCRIPTION. NOT THE STUPID SLASHDOT STORY. ONLY THE CLAIMS.
For this patent that means if you implement this ideas EXACTLY AS IN THE 39 CLAIMS except for, for example '7. The method as recited in claim 6, wherein the pixel offset includes a range of 1 to 16 pixels.' and your pixel offset is 17, you are cool. You are fucking golden. Gedit? So quit your whining about previous art until you find something that follows THE CLAIMS. OK? THE FUCKING CLAIMS!!!!!!!
http://www.theonion.com/articles/microsoft-patents-ones-zeroes%2C599/