Notes On The Future of Video on Linux

Dina's Dream points out two interesting articles currently running on LinuxPower, and linked from Gnotices (GNOME news site) as well. "The first article is a really good summary of the current state of affairs of video under Linux and the direction we should take. Questions are bounced back between a few very knowledgeable people, including GStreamer developers, SGI people and Alan Cox. The second article is a set of lessons learned by Chris Pirazzi while working at SGI. Chris was involved in a lot of Video API programming at Silicon Graphics, and raises a few very good points based on his experience. All people even remotely working on video drivers or software should read these points and take them to heart."

Comments

SSSCA

[Anonymous Coward]

Well, if SSSCA passes, there won't be any video on linux.

video focus

[paulbd]

the slashdot headline is more accurate than the article's actual title. the author's approach comes almost entirely from a consideration of video. if he was starting from a primary interest in audio, he would have talked about many different issues, and mentioned different kinds of solutions. gstreamer is a cool system, but it needs to be stressed over and over that gstreamer is an architecture for building applications. it does not offer any mechanisms for inter-application communication or synchronization. since most people want to do a lot more than write a particular plugin for gstreamer, gstreamer doesn't help us when the challenge is not providing an architecture for a single program, but one for multiple applications on the same (or even networked) system(s). when you want to run a cool video processor along with a really nice FX rack for audio, gstreamer can't help you unless the author of each component had decided to implement his stuff as a gstreamer plugin. since this cramps the GUI style rather considerably, its unlikely that many people will choose to do this. finally, i would note that although it has become customary to sync audio to video, this actually makes very little sense when the temporal resolution of audio (22000-96000 frames/sec) is vastly greater than video (20-30 frames/sec). its really just an artifact of the way technological development has happened, of who has the most power in the entertainment "content" business, and of the fact that we generally consider visual data more significant than acoustic data. we'd be in much better shape is the conventional approach was to sync a video stream to audio, since we could easily and uniformly take advantage of the much better clock resolution that audio devices provide.

Re:video focus

[vektor_sigma]

Paul: You mention that it makes little sense to sync audio to video and then state that this is what most applications do.

On the contrary, syncing video output to the audio is much easier, since audio resampling is a big pain. It's much easier to just watch the soundcard buffer and decide approximately when to show the frame as you imply. You only run into big difficulties when you want to sync audio to the video (or more accurately, sync video blits and audio output to the vertical refresh rate of your output device).

There are times when this is a very important. You imply that video framerates are between 20-30fps, which is quite small. For playback of video sources we need to handle framerates of 50fps and 59.94fps. At such a high framerate, the monitor refresh must track the video input in order to achieve smooth video. See the link to Dave Marsh's faq on judder I mention in the article.

In these cases, you need to set the monitor to the correct refresh rate and then watch the error between the refresh and the sound card clock, and resample the audio when necessary: playing with the refresh rate on the fly would cause a monitor resync and disturb playback!

If video cards let applications drive the refresh of the monitor (software genlock), we could run it based on the audio clock and get the advantages you describe. However given the current state of the hardware it's better to do these small resamplings. That said, actually doing this in linux is still infeasible without some updates to the APIs and drivers, which we are working on.

-Billy Biggs

a refreshing perspective

[FrostyWheaton]

This has to be the only article I have read in a long time that stresses the importance of doing things the right way, instead of the wrong way or, heaven forbid, the "Max Power" way.

There are many core issues with video on any Unix that need to be hammered out now to ensure that things will go well both now and in the future.

As the author mentions several times, adapting refresh rates to video frame rates and working with the monitor's vertical sync as well as audio sync etc, are all very important things that need to be implimented before Video for (insert favorite unix here) will become anything more than a glorified hack.

The first logical step is to impliment what is needed to do things right, and to impliment them in the right(proper) way. the X-protocol should be fully implimented in Xfree, and the kernel should be extended to enable applications to be written which can make full use of the hardware, with minimal kludge-work.

Then the focus moves to making the "killer-app" type media production tools and players. The power of Open Source is the ability to build on the work of others. However, stealing someone's hack to adapt refresh rates, and jamming it into your own code is not an optimal solution. Focus on doing things right the first time, anything less (especially when dealing with core issues) is just asking for untold headaches and frustration in x years, when we are kicking ourselves for not doing the right thing the first time