Notes On The Future of Video on Linux

Dina's Dream points out two interesting articles currently running on LinuxPower, and linked from Gnotices (GNOME news site) as well. "The first article is a really good summary of the current state of affairs of video under Linux and the direction we should take. Questions are bounced back between a few very knowledgeable people, including GStreamer developers, SGI people and Alan Cox. The second article is a set of lessons learned by Chris Pirazzi while working at SGI. Chris was involved in a lot of Video API programming at Silicon Graphics, and raises a few very good points based on his experience. All people even remotely working on video drivers or software should read these points and take them to heart."

Comments

The Future Is Here for me already

[anewsome]

My pvr production is running slick as can be, now that I have ditched all the capture card madness (WinTV, Buz, etc). All I need now is a $30 firewire card and my trusty Canopus ADVC-100 and I capture full frame 720x480 video in better than DVD quality. If you are considering doing a lot of capture/encoding of anything, I highly recommend this setup. dvgrab running under linux seems to be very stable and the captured video is virtually identical to the source.

SSSCA

[Anonymous Coward]

Well, if SSSCA passes, there won't be any video on linux.

Complete MultiMedia Architecture

[Anonymous Coward]

There is one group working on a complete multimedia network architecture. More information can be found unter www.networkmultimedia.org.

Quote from the web-page:

"The goal of this project is to design and implement a network-integrated multimedia infrastructure for Linux as well as other operating systems. Since many low-level parts of such a system already exist (like en-/decoders, de-/compression, multimedia networking APIs, CORBA, and others), our primary goal is the development of an architecture, which integrates these usually isolated modules. This unified architecture will offer a simple and easy to use interface for applications to integrate multimedia functionality. Therefore, it can be used as enabling technology for traditional multimedia applications, but also for ubiquitous computing and mobile computing.The result of our work will be made available as OpenSource."

- Jarman

Re:My experiences with Linux

[Steveftoth]

Personally, I've never had problems with RedHat installs (YMMV), the only time I ever have problems with *nix system is when I don't know where the files are that allow you to configure the system. Since every distribution of linux, BSD and Solaris all seem to be a little different in their config.

Re:DivX ;-)

[Alan Cox]

DivX is waste of energy and programmer time. Its so patent trapped that one yawn from the movie industry to the patent holders and it'll be gone except in very limited form

Something like VP3.2 holds more promise (and hey I got the lib to actually -compile- on a non windows/mac box two days ago)

Now combine VP3.2 video with Ogg audio and you have a credible media format. Add xhtml navigation and you have something really cool

video focus

[paulbd]

the slashdot headline is more accurate than the article's actual title. the author's approach comes almost entirely from a consideration of video. if he was starting from a primary interest in audio, he would have talked about many different issues, and mentioned different kinds of solutions. gstreamer is a cool system, but it needs to be stressed over and over that gstreamer is an architecture for building applications. it does not offer any mechanisms for inter-application communication or synchronization. since most people want to do a lot more than write a particular plugin for gstreamer, gstreamer doesn't help us when the challenge is not providing an architecture for a single program, but one for multiple applications on the same (or even networked) system(s). when you want to run a cool video processor along with a really nice FX rack for audio, gstreamer can't help you unless the author of each component had decided to implement his stuff as a gstreamer plugin. since this cramps the GUI style rather considerably, its unlikely that many people will choose to do this. finally, i would note that although it has become customary to sync audio to video, this actually makes very little sense when the temporal resolution of audio (22000-96000 frames/sec) is vastly greater than video (20-30 frames/sec). its really just an artifact of the way technological development has happened, of who has the most power in the entertainment "content" business, and of the fact that we generally consider visual data more significant than acoustic data. we'd be in much better shape is the conventional approach was to sync a video stream to audio, since we could easily and uniformly take advantage of the much better clock resolution that audio devices provide.

Re:video focus

[vektor_sigma]

Paul: You mention that it makes little sense to sync audio to video and then state that this is what most applications do.

On the contrary, syncing video output to the audio is much easier, since audio resampling is a big pain. It's much easier to just watch the soundcard buffer and decide approximately when to show the frame as you imply. You only run into big difficulties when you want to sync audio to the video (or more accurately, sync video blits and audio output to the vertical refresh rate of your output device).

There are times when this is a very important. You imply that video framerates are between 20-30fps, which is quite small. For playback of video sources we need to handle framerates of 50fps and 59.94fps. At such a high framerate, the monitor refresh must track the video input in order to achieve smooth video. See the link to Dave Marsh's faq on judder I mention in the article.

In these cases, you need to set the monitor to the correct refresh rate and then watch the error between the refresh and the sound card clock, and resample the audio when necessary: playing with the refresh rate on the fly would cause a monitor resync and disturb playback!

If video cards let applications drive the refresh of the monitor (software genlock), we could run it based on the audio clock and get the advantages you describe. However given the current state of the hardware it's better to do these small resamplings. That said, actually doing this in linux is still infeasible without some updates to the APIs and drivers, which we are working on.

-Billy Biggs

Re:video focus

[Omega Hacker]

First of all, the focus of the article was indeed multimedia, from the point of view of the average person who wants to use Linux to watch DVDs and videos off the net. These people really couldn't care less about perfect sample accuracy of their $10 sound card. Trying to force pro-audio qualities onto every machine is impossible, and highly counterproductive in the face of other more visible issues.

GStreamer is indeed an infrastructure for building multimedia apps. Absolutely everything it does is under the auspices of the elements that it loads and connects together. Claiming that GStreamer has no means of inter-application communication is false, as there are already numerous elements both for communication with sound servers and other applications (network source/sinks, etc.). A set of Jack elements are being written, as well.

GStreamer also has nothing to do with the GUI whatsoever. Elements are GObjects that have no GUI in them at all, they focus on doing what they're supposed to. Any GUI exists as a seperate entity on top of the processing pipeline as managed by GStreamer.

If your worry is that LADSPA won't be used, don't. LADSPA plugins are fully supported with a shim under GStreamer.

As for whether to sync to audio or video, you actually have it quite backwards. First of all, the most difficult situation is not with progressive video, but with field-based video (which both NTSC and PAL are, BTW), where the vertical rates are 50 or 59.95Hz. Compare this to a CRT, and you have significant problems finding a decent match between them.

As for the "content" business somehow making video more important than audio, you're ignoring the fact that video *is* more important than audio when both are present together in the same stream. There are several order of magnitude more bits in the video than than the audio, anyway.

Now, to the technical reasons it makes vastly more sense to absorb changes in the audio clock:

1) 90+% of computers have sound cards with clocks that can be best described as "kinda sorta correct". They vary wildly around the real 44.1k baseline. If anyone notices this, there's not much they can do about it. More fundamentally, how are you going to maintain any kind of stream clock when the audio rate is changing during the presentation (as it will when temperature and other things change in your computer, since the clock is susceptable to this)?

2) The quanta of video playout is much greater than that of audio, on the order of 500+ times larger. The goal of any good video playback is to synchronize the playout of a video frame (theoretical time unit) to a vertical refresh (physical time unit). If at any point those become desynchronized, you will have a discontinuity of at least one vertical retrace, or on average about one 75th of a second. Specifically, you'll have a video frame that is suddently displayed for between 50% and 100% longer than it's supposed to be, in the middle of a bunch of correct frames. This is *blatantly* noticable by anyone with a normal visual cortex.

Audio, and the human ear, is much more forgiving. If the program simply drops or duplicates a sample every once in a while to maintain minimal drift between the two (video and audio) clocks, it will be altering 1/44,100th of the samples in that second. If done wrong, it will cause a click. Doing it "less wrong" to avoid clicks is trivial.

So the decision is between locking to a highly variable audio clock (think 3.6 seconds per hour per 0.1% off) and having video that jerks and sputters whenever the theoretical and physical frame times disagree, or doing some resampling to the audio where necessary, with the possibility of some loss of quality that is undetectable by 99% of people watching videos on their computer with tinny speakers.

Me, I don't like video-induced headaches, I'll resample the audio.

Re:Whatever happened to Broadcast 2000?

[qqtortqq]

Its interesting you mention broadcast2000. I used to use it, but gave up due to crashes and things, but it was a neat program if they ever got the bugs out of it.

From: http://heroinewarrior.com/bcast2000.php3

After a long period of deliberation on the matter, Broadcast 2000 has been removed from public access due to excessive liability.

We've already seen several organizations win lawsuits against GPL/ warranty free software writers because of damage that software caused to the organization. Several involved the RIAA vs mp3/p2p software writers. Several involved the MPAA vs media player authors. You might say that warranty exemption has become quite meaningless in today's economy.

Fucking dmca....

Re:DivX ;-)

[LinuxGeek8]

You should check out the PLF distro (not PLD).
It contains non-free rpms for mandrake, installable with urpmi/rpmdrake.
Only mplayer is not available in binary rpm, just source.rpm.

PLF is actually built on a current cooker distro (cooker-mandrake-devel), but there is someone recompiling them against mdk 8.1.
Just try them.

Re:DivX ;-)

[MartinG]

Interesting. I hadn't seen vp3.2 before.
I notice their license is derived from the Mozilla Public License 1.1
Can anyone comment on whether that is good or bad news? eg. will the gstreamer folks be able to include vp3.2 codec support and distribute it as a plugin?

My experience with Video on Linux

[lkaos]

Video on linux has always been one of my favorite things.

The biggest advance in v4l was essentially the XFree86 4.x release since it encorporated an Xvideo extension allowing for really nice video play back in Linux.

There are a couple cards that are extremely well supported. I personally use an ATI AIW and the MPEG playback in incredible. In fact, I prefer MPEGs in Linux verses Windows simply because I think Linux does a better job at using the RGB conversion stuff at the hardware level than Windows does.

Of course, the biggest foe to video in Linux has been the fact that many of the best accelerations (iDCT) are simply not supported because card makers fear releases 'techinical secrets.'

Another problem is the split in video display APIs. Prior to the Xv extension being released, the Linux kernel had a video4linux API. The second version of that API is incredible as it has so many features that would allow for truely pluggable components.

Unfortunately, all X stuff is implemented in user space so cards that have integrated display and video stuff end up supporting everything in user space and then providing a loopback mechanism to work with the kernel API. It's a little messy but hopefully everything will get worked out as things progress.

Otherwise, hats off to all the hard work of the gatos project, the v4l2 project, and linuxvideo. If you haven't tried all the really cool stuff available for video on linux, you really should.

Re:DivX ;-)

[volsung]

Or we could do what they do in Windows land, which is bundle every dependent library with the program and install them if they aren't already there. This is easy in a world of shrink-wrapped CD installs (where you have space to burn), but is a good deal more cumbersome with downloaded applications.

urpmi (in Mandrake) already does exactly what you request from rpm. It resolves all the dependencies and will either download the RPMS from an online source or ask you to insert your installation CDs. Hopefully Redhat will snarf it into their next distro. (That would actually be quite appropriate given how much Mandrake has snarfed from RedHat. Both distros would benefit from some cross-pollenation.)

a refreshing perspective

[FrostyWheaton]

This has to be the only article I have read in a long time that stresses the importance of doing things the right way, instead of the wrong way or, heaven forbid, the "Max Power" way.

There are many core issues with video on any Unix that need to be hammered out now to ensure that things will go well both now and in the future.

As the author mentions several times, adapting refresh rates to video frame rates and working with the monitor's vertical sync as well as audio sync etc, are all very important things that need to be implimented before Video for (insert favorite unix here) will become anything more than a glorified hack.

The first logical step is to impliment what is needed to do things right, and to impliment them in the right(proper) way. the X-protocol should be fully implimented in Xfree, and the kernel should be extended to enable applications to be written which can make full use of the hardware, with minimal kludge-work.

Then the focus moves to making the "killer-app" type media production tools and players. The power of Open Source is the ability to build on the work of others. However, stealing someone's hack to adapt refresh rates, and jamming it into your own code is not an optimal solution. Focus on doing things right the first time, anything less (especially when dealing with core issues) is just asking for untold headaches and frustration in x years, when we are kicking ourselves for not doing the right thing the first time

Most video utilities apt-gettable from Freshrpms

[Nailer]

The #1 problem is ease of software installation and configuration. Sure rpms are great, until you get some that depend on something else.

Or you could just get apt-get and let it sort out the dependencies in your rpms for you automatically. The Freshrpms apt archive (for Red Hat 7.2) has packages for

• Xine
  
• Ogle
  
• MPlayer (source packages only as Mplayer isn't Open Source)
  
• Drip (a DivX encoder)
  
• Transcode (another DivX encoder)
  

And all you need to install them is type apt-get installl (whatever) to fetch the rpms, whatever dependencies they need to install, and install the packages.