Audio Format Listening Tests Concluded

Pointing to the conclusions of this listening study, nullity writes: "The results are interesting, and show a high variation in the performance of the various codecs on different musical styles. Ogg seems to work well on dance music, WMA8 on chamber music, etc."

Comments

WMA8

[af_robot]

Ogg seems to work well on dance music, WMA8 on chamber music, etc.

Like requiem...

Who cares about 64 kbps tests?

[splorf]

These tests are all at 64 kbps and most people use much higher bitrates for real music. I'd like to see comparisons at 128k bits minimum, and preferably 160k or 192k, which is what most quality mp3's are at, for direct comparison.

Re:Who cares about 64 kbps tests?

[keller]

People who wnat to stream audio of course! .K

Re:Who cares about 64 kbps tests?

[ProtoCat]

64Kbps is where the flaws of a codec are truely exposed. It's a great median between being too high to produce much results and too low where everything completely falls apart. You may not think any of this has any relevance to you as you're encoding above 128Kbps, but it actually does make a difference when you stress your encoder with a difficult piece of music.

However, if the difference between sounding 'good' and sounding 'accurate' mean little to you, as someone who'd make an argument of 64Kbps tests being worthless would, then you really aren't the intended audiance of such tests. You can merrily use any of those encoders at 128-192Kbps without ever really noticing or caring much.

I, personally, would like to see OGG1.0, MP3 Pro and WMA8 take on some real tough to beat codecs such as Dolby's AAC High-Complexity Mode (which no AAC freely available encoder supports, including QuickTime) and Sony's ATRAC3. But, that'd be kinda moot, because most people out there do not have access to those toys.

For now, I'm content to just watch people hop around and proclaim whatever they want as king of audio formats while sticking to 256Kbps Fraunhoffer MP3 (archival purposes) and 192Kbps LAME HQ MP3 (general usage) as something both widely supported and pratically indistinguishable from the source. Even if AAC-HC and ATRAC3 were freely available, it'd take an awful large effort to wean people off of MP3 so far as support base and to migrate them to a new format. New P2P programs, new players/plug-ins (in some cases) and new hardware players. Not gonna happen for a while.

Re:Who cares about 64 kbps tests?

[flipflapflopflup]

> Are you pondering what I'm pondering?

I thought your sig was part of your comment, and was going to agree entirely. 64 kbps tests are pointless because no-ones uses those rates, and at higher rates the differences become negligable.

At the kind of bit rates that real people actually use (160, 192, and up), it takes a real pro/audiophile/picky git to tell the difference. Which makes the whole thing seem a bit pointless really.

Chances are then, it's not going to be audio quality that makes or breaks these standards. Look at betamax...

Re:Who cares about 64 kbps tests?

[nagora]

64Kbps is where the flaws of a codec are truely exposed.

Running your car over a cliff is where the flaws in its safety system are truely exposed but I don't tend to drive over cliffs much.

However, if the difference between sounding 'good' and sounding 'accurate' mean little to you, as someone who'd make an argument of 64Kbps tests being worthless would, then you really aren't the intended audiance of such tests.

What do you mean by this? 64Kbits is worthless for listening to any music I own while 128 is good enough to not actually annoy me much of the time so why should I be interested in these tests? Are you saying that the intended audience for these tests are people that are not interested in the quality of the music they're listening to?

TWW

Re:Who cares about 64 kbps tests?

[gleam]

You don't often intentionally hurl your car at 45 miles an hour into a steel box, either, but insurance companies do it all the time to see how well a particular car stands up to the abuse.

Even if you don't knowingly take the insurance institute's results (or federal crash-test ratings) into account, the company selling you insurance does, and your premiums will be higher.

To say "just because i'll never do something this way it has no merit" is silly. Performance in a 45-mile-per-hour offset crash will tell a car company how well it would stand to you accidentally bumping into the corner of your garage, or into the bumper of another car.

Tests like this are important because they're indicative of performance at all bitrates. If you want to know WHICH codec will sound the best at 128kbit, you should look at which codec sounds the best at 64kbit--the two are likely to be the same.

There are two intended audiences for this test: 1) people trying to decide which audio format to use for a stream (which are very often in the 32-64kbit range)

and

2) people who realize these tests can tell us much more than simply which codec performs best at 64kbit, and want to know how to maximize the quality-to-diskspace ratio on their own encodings.

Hope this clears something up for you.

-gleam

Re:Didnt read the article yet but....

[mccalli]

it seems that music you listen closely too sounds better with WMP, and fast, not listened to music sounds ok w/ ogg.

Well....not quite. There's a different frequency distribution between electronic, pop acoustic and classical music.

Specifically, electronic music, which most dance stuff is, has a very flat frequency distribution. See this for yourself - load your favourite media player, siwtch on the graphic equaliser graph and watch how basically nothing happens except in the mid-range.

Now try again with an orchestral piece. There will be much more variation, though in most it will tend towards the top end.

Now try again with rock. Tends towards the bottom and top, with middle frequencies missing.

Keep going with any format you feel like mentioning...you'll get the same.

Actually, this is a striking example of how recording techniques can ruin sound as well. Take a look at the Apollo 440 album - Gettin' High on Your Own Supply. A good mixture of guitars and electronics, right? Well, look at the frequency graph again. See how virtually every guitar frequency variation has been cut out: this music was recorded digitally, mostly using samples by the looks of it. The normal variations you'd associate with having guitars play live are all filtered out, and the graph goes back to the flat digital sound again.

Cheers,
Ian

So I guess its time for a new frontend

[CableModemSniper]

I guess grip will have to use Genre info from CDDB to decide what to encode the the files as now. I wonder if you coudl set up something to optimize individual tracks. Like scan a wav and pick the best codec for the frequencies used in the audio.

MP3PRO not MP3

[Anonymous Coward]

I noticed a number of confused posters here... The tested codecs were AAC/MP3PRO/OGG/WMA, not MP3. Had mp3 been tested, it would have lost every round as all of the tested codecs are vastly superior to plain MP3 at this bitrate.

It also should be noted that the only two samples that WMA beat OGG at (indeed the only ones that it didn't totally flop on) were two very simple samples that are demonstrations of two differnt weaknesses in the current revision of vorbis. Orignally the results page had some very interesting commentary from Monty on this, but it looks like it got pulled.

With the exception of those two samples, OGG clearly won. Even including those, it was only beat out by MP3PRO by a small margin. When you factor in that MP3PRO isn't available at anything but such low bitrates and that it's substantially more propritary then MP3, it seems like pretty much a no-contest.

Multi-Codec Codec anyone?

[BlackGriffen]

Considering that different codecs do better at different music w/ different frequency spreads, who else thinks that the next generation of audio codecs will be multi-modal; in effect, be several codecs in one. Then have each codec specialize on certain types of music. Perhaps even have them run in an advanced mode where they do a frequency analysis of whole songs, rather than just using genre, to automatically select the best codec for the job. Perhaps even use different codecs for different sections of the song. That would definitely help songs like Bohemian Rhapsody and orchestas with movements, etc.

Would this be too time consuming to implement or what?

BlackGriffen

Outiers skewing the results?

[Outland Traveller]

Looking at the data, it looks the two samples where Ogg performed poorly ended up being encoded at a significantly smaller average bitrate than any of the other encoders.

The table at the end lists LiszBMinor with an average ogg bitrate of 45 and BachS1007 with an average bitrate of 47. Since the other codecs encoded those samples at a bitrate 64 or higher, this may explain the results.

The results may point to a flaw in Ogg's VBR login rather than in the lossy compression scheme.

Re:Outiers skewing the results?

[jonathan_ingram]

If you read the thread on HydrogenAudio (which is the message board where most of these tests / codecs are discussed), you'll find the following information from Monty, the lead developer of Vorbis:

Ogg had a very low bitrate (in the forties) on all the classical samples, which is the way it should have been (Classical solos with their deep noise floors and simple harmonics are relatively easy). But the real reason Ogg scored so low in both (and Beauty Slept as well) was a) the tuning behind noise normalization is still not perfect. This is the very first release of that feature, and the test found flaws b) also the first release of new, more aggressive stereo modes and I think that they too need more analysis infrastructure driving them.

I expect Ogg's performance on Liszt and Bach to be very subpar NN performance. The poor performance on BeautySlept and Waiting was most likely insufficient stereo analysis. Ogg had the infrastructure to win those four samples, but the encoder didn't know how to do it yet (because I didn't know it would be necessary).

Re:No Control

[Anonymous Coward]

Here, there is no control - crappy experiment.

The participants could have just been scoring on "this is different to the unencoded track, therefore it must be worse".

So put a copy of the unencoded track as a test track and see if it gets marked down (and also, of course do NOT tell the participants that it is there).

Umm.. did you bother to read about the testing methodology before coming to your grand conclusion here?

ABC/HR.. as in ABC/Hidden Reference... as in, there is a copy of the original track included as a hidden reference on every single trial.

The users are given 2 sliders per sample laid out on a panel. The samples are loaded in random order. On the sliders for each sample, one slider is for the original sample, and one is for the encoded. These are also randomized per sample. The user does not know which is which. If they happen to rate the original sample less than 5.0 (highest rating, meaning it should be transparent), then their results are disregarded entirely for that sample.

Re:Yes, look at the SCORES

[micromoog]

mp3PRO has no hope of gaining enough market share to become a worthy competitor. It's a very proprietary extention to MP3.

mp3PRO has one very specific advantage over all the other formats on the market-share front. It has the characters m, p, and 3 in it. Everyone has heard of mp3, and people who don't care about the open source cause (read: the vast majority of people) will buy an mp3PRO device way before considering an Ogg Vorbis device.

As I've said before, name is really important when marketing comes into play. And Ogg Vorbis' name simply blows.

Test compares codecs, not formats

[benwaggoner]

This is an interesting and relatively well done test (although it appears that the listeners knew which format they were listening to, so it wasn't truly double-blind, and a anti-MS and pro-Ogg bias can't be ruled out).

However, some discussions seem to be focusing on this saying AAC is bad or WMA is bad, when really it refers to the particular implementations in codecs of those formats.

For example, the Apple MPEG-4 AAC-LC encoder was used for AAC. This is a Low Complexity version of the format. Also, the Apple encoder has a strange limitation where it automatically converts 44.1 stereo to 32 stereo at that data rate. This isn't required by the AAC format. Other AAC encoders yield MUCH better results, and beat MP3 Pro in double-blind testing. I haven't seen any double-blind comparisons between AAC and Ogg.

Also, the WMA8 encoder is due to be replaced by the backwards-compatible WMA9 in early September. Of course, there may well be improved versions of the other encoders by then as well.