Which Open Source Video Apps Use SMP Effectively?

ydrol writes "After building my new Core 2 Quad Q6600 PC, I was ready to unleash video conversion activity the likes of which I had not seen before. However, I was disappointed to discover that a lot of the conversion tools either don't use SMP at all, or don't balance the workload evenly across processors, or require ugly hacks to use SMP (e.g. invoking distributed encoding options). I get the impression that open source projects are a bit slow on the uptake here? Which open source video conversion apps take full native advantage of SMP? (And before you ask, no, I don't want to pick up the code and add SMP support myself, thanks.)"

Comments

ffmpeg

[bconway]

Use the -threads switch.

Re:ffmpeg

[morgan_greywolf]

Similarly, mencoder supports threads=# where # is something between 1 and 8.

Re:ffmpeg

[mweather]

Apple computers ARE PCs. They coined the damn term.

Re:ffmpeg

[m0rph3us0]

No it doesn't the only time you want to use multi-threading in a single CPU environment is because asynchronous methods for IO are unavailable or the code would be too difficult to re-architect to use asynchronous IO. If the application is seriously IO bound threads can even make the situation worse by causing random IO patterns.

Ideally, the number of threads a program uses should be no more than the number of processors available. Otherwise, you are wasting time context switching instead of processing.

Re:ffmpeg

[ydrol]

Darn, I forgot a minor detail in my question. I was really asking about the various front-end apps (dvd::rip, k9copy, acidrip etc), I got the impression that none seem to notice they are running on an SMP platform and pass the necessary switches by default to the backend.

Some may argue this is a good thing, but for the time being SMP is the way forward for faster processing as MHz has maxed out, in consumer PCS. So when they start buying octo-core CPUs they dont expect it to run at 1/8th speed by default.

I was also being a bit lazy. I could have checked up on each app in turn, but I asked /. instead.

Re:ffmpeg

[VGPowerlord]

True, but in most contexts, "PC" is the shortened form of IBM-compatible PC (which is really outdated), and is usually just stands for Windows these days.

Re:ffmpeg

[MadnessASAP]

If I may offer a suggestion, I'm not too sure on what your setup is but on mine I have 2 DVD drives each on separate IDE buses and 2 SATA drives (also on separate buses) rip from the DVD to drive 1 and encode from drive 1 to 2. OF course it all depends on a variety of factors but using that certainly helped that.

Re:ffmpeg

[Nikker]

Running multiple cores with an ide interface is going to kill you regardless because you are only encoding in memory not really storing much there. Basically you have a cap of about 40MB/s for anything larger than about 40MB.

Re:ffmpeg

[ksheff]

That's the point. If the xvid encoder is single threaded, then to keep all the cores busy, one must run multiple instances of ffmpeg with each one encoding a different file. For the given Makefile, that is what make will do when the -j switch is used.

transcode, of course!

[morgan_greywolf]

transocde uses separate processes for everything.

Beat me to it!

[BLKMGK]

x264 via meGUI from Doom9 is what I use to compress HD-DVD and BD movies - also on a quad core. I have some tutorials posted out and about on how I'm doing it. Near as I can tell you cannot dupe the process on Linux due to the crypto - Slysoft's AnyDVD-HD is needed.

Playback - I use XBMC for Linux. It is also SMP enabled using the ffmpeg cabac patch. the developers of this project have been VERY aggressive at taking cutting edge improvements to the likes of ffmpeg and incorporating them into the code. Since Linux has no video acceleration of H.264 SMP really helps on high bitrate video!

Handbrake

[vfs]

Handbrake has always used both of the cores on my system for transcoding.

Re:Handbrake

[catmistake]

that's because Handbrake uses ffmpeg

Re:Which part of Open Source didn't you get?

[pushing-robot]

OP is asking for open source tools. You cited a commercial one that doesn't provide source.

VisualHub (the front-end app) may be closed, but ffmpeg is LGPL.

And the GP was suggesting using ffmpeg, not VisualHub.

Re:Simple...

[j00r0m4nc3r]

Running multiple instances of the same code concurrently in multiple threads is simple. Even running mutually exclusive parts of the same code concurrently in separate threads is easy. Converting complex serial algorithms to effectively utilize multiple cores is generally not simple. And writing code that can scale and balance across n number of cores/threads is extremely hard. There are all sorts of synchronization issues to deal with, scheduling issues, data transport issues, etc.. and it becomes increasingly hard to debug code the more cores/threads you throw in. I think the stigma is justified.

keyframes

[Anonymous Coward]

Actually, the MPEG stream resets itself every n frames or so (n is often a number like 8, but can vary depending on the video content). These are called keyframes (K) and the delta frames (called P and I frames) are generated against them. Because of this, it is really easy to apply parallel processing to video encoding.

Re:keyframes

[DigitAl56K]

Actually, the MPEG stream resets itself every n frames or so (n is often a number like 8, but can vary depending on the video content).

That is not true for MPEG-4 unless you have specifically constrained the I/IDR interval to an extremely short interval, and doing so severely impacts the efficiency of the encoder because I-frames are extremely expensive compared to other types.

Keyframes are usually inserted when temporal prediction fails for some percentage of blocks, or using some RD evaluation based on the cost of encoding the frame. Therefore unless the encoder has reached the maximum key interval the I frame position requires that motion estimation is performed, and thus you can't know in advance where to start a new GOP.

In H.264 due to multiple references you would certainly have issues to contend with since long references might cross I-frame boundaries, which is why there is the distinction of "IDR" frames, and this would certainly not be possible threading at keyframe level.

Granted, for MPEG1&2 encoders threading at keyframes is a possibility, although still not one I'd personally favor.

avidemux

[Unit3]

I've noticed a lot of talk about commandline options, but not the nice guis that use them. Avidemux is open source, cross-platform, gives you a decent interface, and uses multithreaded libraries like ffmpeg and x264 on the backend to do the encoding, so it generally makes optimal use of your multicore system.