Audio Format Listening Tests Concluded

Pointing to the conclusions of this listening study, nullity writes: "The results are interesting, and show a high variation in the performance of the various codecs on different musical styles. Ogg seems to work well on dance music, WMA8 on chamber music, etc."

Comments

Who cares about 64 kbps tests?

[splorf]

These tests are all at 64 kbps and most people use much higher bitrates for real music. I'd like to see comparisons at 128k bits minimum, and preferably 160k or 192k, which is what most quality mp3's are at, for direct comparison.

Re:Who cares about 64 kbps tests?

[keller]

People who wnat to stream audio of course! .K

Re:Who cares about 64 kbps tests?

[flipflapflopflup]

> Are you pondering what I'm pondering?

I thought your sig was part of your comment, and was going to agree entirely. 64 kbps tests are pointless because no-ones uses those rates, and at higher rates the differences become negligable.

At the kind of bit rates that real people actually use (160, 192, and up), it takes a real pro/audiophile/picky git to tell the difference. Which makes the whole thing seem a bit pointless really.

Chances are then, it's not going to be audio quality that makes or breaks these standards. Look at betamax...

Re:Who cares about 64 kbps tests?

[nagora]

64Kbps is where the flaws of a codec are truely exposed.

Running your car over a cliff is where the flaws in its safety system are truely exposed but I don't tend to drive over cliffs much.

However, if the difference between sounding 'good' and sounding 'accurate' mean little to you, as someone who'd make an argument of 64Kbps tests being worthless would, then you really aren't the intended audiance of such tests.

What do you mean by this? 64Kbits is worthless for listening to any music I own while 128 is good enough to not actually annoy me much of the time so why should I be interested in these tests? Are you saying that the intended audience for these tests are people that are not interested in the quality of the music they're listening to?

TWW

Re:Who cares about 64 kbps tests?

[TummyX]

64 kbps tests are pointless because no-ones uses those rates, and at higher rates the differences become negligable.


With MP3 maybe, but OGG, WMA, MP3Pro and ACC all aim to get MP3@128Kbs quality at 64Kbs. That's what this test is FOR! Try comparing a 128KBps MP3 with a 64KBps OGG file. You'll find the quality to be quite comparable. It's a pity the test didn't include MP3 (at 128KBps) so we could see how good these new codecs really are.

And BTW, you might as well say it is pointless to test 128, 160 or 192kbps and that we should all be testing uncompressed 1500Kbps audio.

Re:Who cares about 64 kbps tests?

[gleam]

You don't often intentionally hurl your car at 45 miles an hour into a steel box, either, but insurance companies do it all the time to see how well a particular car stands up to the abuse.

Even if you don't knowingly take the insurance institute's results (or federal crash-test ratings) into account, the company selling you insurance does, and your premiums will be higher.

To say "just because i'll never do something this way it has no merit" is silly. Performance in a 45-mile-per-hour offset crash will tell a car company how well it would stand to you accidentally bumping into the corner of your garage, or into the bumper of another car.

Tests like this are important because they're indicative of performance at all bitrates. If you want to know WHICH codec will sound the best at 128kbit, you should look at which codec sounds the best at 64kbit--the two are likely to be the same.

There are two intended audiences for this test: 1) people trying to decide which audio format to use for a stream (which are very often in the 32-64kbit range)

and

2) people who realize these tests can tell us much more than simply which codec performs best at 64kbit, and want to know how to maximize the quality-to-diskspace ratio on their own encodings.

Hope this clears something up for you.

-gleam

Re:Who cares about 64 kbps tests?

[jorleif]

What do you mean by this? 64Kbits is worthless for listening to any music I own while 128 is good enough to not actually annoy me much of the time so why should I be interested in these tests?

I suppose he meant that since at 128Kbps all codecs perform quite well. Therefore personal preferences affect the results too much, i.e. some people like bass boosts and lots of treble, although this is not accurate in the sense that it differs from the original recorded signal.

If 128kbps is 'good enough' for you then I too suppose you would fall into the sounding 'good' instead of sounding 'accurate' category.

Re:Who cares about 64 kbps tests?

[Anonvmous Coward]

Oh but it is CD quality.

"We took 1,000 people and had them listen to an NSync song off a 64-kbit WMA file, and then off the CD. When asked 'Do you like the song any better?', 999 out of 1,000 people said "No. Can I go now?"

So yes, there is no quality difference between WMA and CD's, it's just a matter of asking the right question.

Should compare Ogg as a single entry

[Russell Coker]

Let's assume that anyone who likes Ogg and is seriously into music will compress their music with both Ogg variants and use the best variant for each file.
Therefore we should also consider taking the best of the two results and comparing it to mp3.
From a quick look at the results it appears that Ogg will still be edged out by mp3 when analysed in such a fashion, but it's much closer.
Also a test on several bit rates would be useful.

Re:Sound Artifacts

[Anonymous Coward]

I always equate WMA audio to the sounds of someone whistling in a tin bathtub. Every peice of WMA I ever heard, from static files to streamed audio, has that tinny, whistling sound to it. I don't know if its me (Not being subjective and all!), or if its just poor encoding options (By everyone? Well, its possible I guess) or just WMA.

Either way, I stick to Ogg these days. MP3s @ 192kbps are fine, but take up more space. Ogg sounds better than MP3 (IMHO, to my ears, etc. Your mileage may vary, yada yada), keeps my disk usage down, and all my audio is playable across all three OS's that I use (WindowsNT 4 in work, Linux & Syllable at home)

Re:Sound Artifacts

[funky womble]

I have problems with some 192Kbps CBR MP3s, I find --r3mix usually sounds a lot better (and takes up less space), though sometimes I neeed to bump it up to ABR (usually around 220Kbps if it's a difficult piece).

What about VBR?

[xA40D]

Okay, so everybody has pointed out that a 64k bitrate is useless for their needs. Some claim 128K is the best, others 192K. Personally I prefer a variable bitrate - indeed OGG encodes VBR by default.

But is their some fundamental reason why nobody else insists on VBR?

No CD Audio comparison?

[Disco2k]

Does anyone else feel it would have been nice to see Red Book CD audio (16-bit 44.1KHz uncompressed) compared as a control? Seeing how "pure" audio compares to these compression standards could make the results seem more objective.

Ogg on Quicktime

[heroine]

Been using Ogg/Vorbis/Squish on Quicktime for a year. The Ogg/Vorbis/Squish codec got much better between 1.0rc2 and 1.0. At 128k it's already better than mp3 and the managed bitrate encoding is faster than the hard drive can read. The real value is of course, the ability to read these encoded files as long as there is UNIX. Mp3 is going to die and when it does there won't be any appliance makers interested in paying the $10,000 royalty to support mp3.

Re:Poor Steve Jobs...

[Greg W.]

AAC's performance at 64 kbps is not necessarily indicative of its performance at 128 kbps.

Newer algorithms of limited use to many

[dh003i]

To thsoe of us who just want to listen to music on a PC, the newest greatest best algorithms are always good (mp3pro, oggs, wma8). But for many, the goal is to put that music on a MP3Player and listen to it anywhere. I'll summarize the support of these various codecs by MP3Players, as well as mention whether or not my MP3Player (RioVolt SP100) supports them.

MP3PRO -- little support on MP3Players. Not supported by RioVolt SP100.

Oggs -- little/no support on MP3players. Not supported by RioVolt SP100.

WMA8 -- little support on MP3players, though many support older WMA's. Not supported by RioVolt.

So, in summary, all of these new formats are completely useless to me on my MP3Player. The one option they present is if I want to encode something in two formats -- one for my computer, and another for the MP3Player.

Personally, I think more work should go into fractal endcoding, as most music has fractal patterns in it (especially Bach's music).

Test compares codecs, not formats

[benwaggoner]

This is an interesting and relatively well done test (although it appears that the listeners knew which format they were listening to, so it wasn't truly double-blind, and a anti-MS and pro-Ogg bias can't be ruled out).

However, some discussions seem to be focusing on this saying AAC is bad or WMA is bad, when really it refers to the particular implementations in codecs of those formats.

For example, the Apple MPEG-4 AAC-LC encoder was used for AAC. This is a Low Complexity version of the format. Also, the Apple encoder has a strange limitation where it automatically converts 44.1 stereo to 32 stereo at that data rate. This isn't required by the AAC format. Other AAC encoders yield MUCH better results, and beat MP3 Pro in double-blind testing. I haven't seen any double-blind comparisons between AAC and Ogg.

Also, the WMA8 encoder is due to be replaced by the backwards-compatible WMA9 in early September. Of course, there may well be improved versions of the other encoders by then as well.