Slashdot Mirror

So some people will say Codec A sounds best. Some will say Codec B sounds best. Some will say that Codecs A and B suck donkey shit and Codec C sounds best. What exactly does this prove?

You're commenting on something you have no knowledge about.

This is a blind ABX listening test. You can go read about what that is here.

This is the program being used in this test.

It proves whether or not a statistically significant difference in perceived audio quality can be found between the various codecs by the group of testers. It is valid and scientific. If it wasn't, you wouldn't see the Hydrogen Audio forum associated with it, because they're actually interested in the real deal, and it's against their forum rules to make claims without evidence. (See here. Rule #8.) That would also be why many encoder and codec designers/developers are active in the forum, because it's where they get real empirical feedback.

If you dare, check out this page in order to train your ears to be more sensitive to lossy compression artifacts:
http://ff123.net/training/training.html

Interesting. The worst for me are the ones that sound like little skips in the rip, as in Blackbird and Your Latest Trick. A lot of the other artifacts, though, even when I could kinda-sorta hear them (usually they either sounded "underwater" or "less bright"), weren't all that distracting. I'd give up some compression to get rid of them if I cared about the songs, but I wouldn't reject the encodings if they were all I had. That is to say, I've heard way worse (like most YouTube audio).

Low quality mp3s sound more like you're listening to music with cotton in your ears.

That is the case for music which has been decimated with a low-pass filter (i.e. the high frequencies are not "passed" through). But there are other artifacts like pre-echo (before a sharp attack like from a cymbal or castanet there is kind of a echo or "smear" added to to the music).

If you dare, check out this page in order to train your ears to be more sensitive to lossy compression artifacts:
http://ff123.net/training/training.html

BTW, altering the overall volume down by 2dB will introduce significant undesirable quantization noise. I see no good reason why you should do so

The constant re-allocation of the available bandwidth among the sub bands, and the way the quantization noise follows it, is part of the problem. It sounds like subtle amplitude modulation of the signal to me, like with a badly set up multiband compressor, and you can hear parts of the ambient noise floor vanishing and re-appearing. Other distortions are caused by the ringing of the FFT filters.

Another sound containing short high energy impulses that MP3 encoders really don't like are clavs.
http://ff123.net/preecho.html

I bet I could make a sound combining simultaneous swept sines, clav like impulses and jangling keys that would obviously fox any mp3 encoder.
It's not really fair as it's not really music and is designed to reveal artifacts, but it would be a good educational tool.

..is 18.7%

http://ff123.net/abx/abx.php

For fuck's sake read my reply. The ITU standard implies a reference and a hidden reference (=> double blind): Take a look at this to get an idea about what the test looked like.

hmmm. I'm surprised no one mentioned this. You migh want to head on over to ff123's artifact training page and listen to the examples there if you aren't familiar with the kinds of artifacts that can occur with lossy compression of audio.

The truth is 90% of the people can't tell 256k mp3 from the original cd track.

Everything on MP3.com had to be encoded at 128 kbps. Some people put a lot of work into tweaking their encoder settings to get decent-sounding audio, but most didn't bother.

I agree that 256 kbps would be fine for downloads (with a good encoder like LAME or FhG). But if I was paying for the music, I'd want the option of getting a losslessly compressed file.

Fisher LSD is better than I thought, but you are still going about it the wrong way. The only way the Fisher LSD is protected is if you do the ANOVA first and it shows a significant difference between the groups, then you do the post hoc tests accordingly.

That is exactly what is happening here. A blocked ANOVA (each listener is a block) is used to determine if there is a significant difference anywhere. Fisher LSD is used if the ANOVA is significant (this is what it means to be protected).

As I mentioned in another message, if people really want to nitpick, they should take the raw data and run it through my bootstrap resampling program, which does not assume a normal distribution and corrects for multiple comparisons.

http://ff123.net/bootstrap/

However, you can't get the nice graphs that Roberto showed (tabular pvalues are the best you can get), and the difference in the conclusions will be something that is more conservative than the Fisher LSD. But again, who cares? So there is a bigger chance that you'll get a type I error. Get over it. It's not a black and white thing, where if you do it one way the results are totally wrong, like you're making it out to be. Are the conclusions reasonable? Yes. Are they significant to 95% confidence? Maybe if you use ANOVA/Fisher LSD. Maybe not (90% confidence?) if you use another, more conservative method. It's not the end of the world.

BTW, Blocked ANOVA with Fisher LSD is what the book, "Sensory Evaluation Techniques," by Meilgaard, Civille, and Carr recommends for subjective tests like these.

Also BTW, there really isn't one or two results which are strikingly different from the others. There was no low anchor included in this test, which would have produced such a result.

ff123

The intervals in the rating scale are 0.1 steps, which is close enough for argument's sake.

And ANOVA is a robust method as you've commented, so it's probably reasonable to assume normality. In any case, the raw data is available for any stats weenies to play with, and there are a couple of more conservative methods besides the Fisher LSD readily available to try, if anyone has an uncontrollable urge.

I point you to:

http://ff123.net/friedman/stats.html

where you can run a non-parametric analysis of the raw data using a web form if you like. Or download the program to correct the pvals for multiple comparisons using Tukey's Honestly Significant Difference instead of the Fisher LSD (either paramteric or non-parametric).

And if you want to really geek out, you can do a bootstap resampling method with different methods of correcting the pvals for multiple comparisons:

http://ff123.net/bootstrap/

But let's not lose the forest for the trees here. The blocked ANOVA/Protected Fisher LSD used for the test provides a reasonable (if not the most conservative) summary of the results.

A couple of more important weaknesses of the test are (and they're related):

1. Only 12 samples were used. Although this is probably close to the practical limit, more samples always provides a more comprehensive picture. The results are suggestive, but not definitive.

2. The selection of the samples makes a difference. If you listen mainly to classical for example, this test may not be representative for you.

The intervals in the rating scale are 0.1 steps, which is close enough for argument's sake.

And ANOVA is a robust method as you've commented, so it's probably reasonable to assume normality. In any case, the raw data is available for any stats weenies to play with, and there are a couple of more conservative methods besides the Fisher LSD readily available to try, if anyone has an uncontrollable urge.

I point you to:

http://ff123.net/friedman/stats.html

where you can run a non-parametric analysis of the raw data using a web form if you like. Or download the program to correct the pvals for multiple comparisons using Tukey's Honestly Significant Difference instead of the Fisher LSD (either paramteric or non-parametric).

And if you want to really geek out, you can do a bootstap resampling method with different methods of correcting the pvals for multiple comparisons:

http://ff123.net/bootstrap/

But let's not lose the forest for the trees here. The blocked ANOVA/Protected Fisher LSD used for the test provides a reasonable (if not the most conservative) summary of the results.

A couple of more important weaknesses of the test are (and they're related):

1. Only 12 samples were used. Although this is probably close to the practical limit, more samples always provides a more comprehensive picture. The results are suggestive, but not definitive.

2. The selection of the samples makes a difference. If you listen mainly to classical for example, this test may not be representative for you.

Anyways, here's listening tests at 64kbps. HE AAC beats Vorbis handily, QT AAC not so well... but that's at 64 kbps. 128kbps tests are here.

There's more here and here.

Overall, Vorbis seems to be great at low bitrates - 64 kbps, for instance. At higher bitrates, though, it doesn't keep up compared to AAC.

-T

All things being equal (source quality, etc.), which they probably aren't, AAC should beat out WMA handily at bitrates like what the iTMS and Wal-Mart are using. The only chance WMA would have of approaching AAC in quality at that bitrate would have been if Wal-Mart had used WMA Pro, but because of the lack of hardware player support for WMA Pro, that probably won't happen soon.

I haven't seen tests directly comparing AAC to WMA (non-Pro), but Roberto Amorim's testing at 128kbps with AAC and WMA Pro and ff123's testing of a different AAC codec against WMA non-Pro probably say enough.

Also, Apple has actually spoken about the quality of the sources that they encode from (the original masters rather than CDs themselves), and Wal-Mart hasn't.

I do hope that whoever elects to actually directly compare the quality of Wal-Mart's music to Apple's doesn't just look at frequency analysis to do it. Apple's AAC lowpasses at 16 KHz, but to use this as some sort of indication of quality is ludicrous.

Did you forget to credit ff123, without whom none of the tests would have been possible?

So what i'm asking you, BUSTER, is: has an "scientifically valid" "are differences audiable" experiment been done or not?

Yes. Individuals have performed such tests and established bit rates at which they became unable to distinguish the difference. Here is one such test. I have never seen any test results in which the listener could, with any statistical significance, identify a 320k MP3 vs. a CD of the same music.

But the important thing is whether you can hear the difference. If you are downloading music for your own enjoyment, it makes no difference whether I, some other random Internet user, or a panel of listeners could hear the difference or not. What matters is whether you can hear the difference. And the only way to be sure when the differences are subtle is with a double-blind test.

Your "power of suggestion" theory needs this experiment and yet you claim it has never been done.

It's not my "theory." It's accepted as scientific fact. Ever heard of "the placebo effect"? That's an example of the power of suggestion. Why do you think that clinical drug trials have a control group who receive placebos? It's so that the the power of suggestion does not cause a patient to imagine drug effects that are not there.

Why not show spectrum analasys of different songs encoded into the given formats too?

Perhaps I'm just an audio freak, but I would find that a lot more interesting than just ratings.

Learn why you shouldn't use spectral analysis to determine lossy codecs' quality.

The most respected technique is double-blind testing using an ABX tool such as PC ABX, WinABX or ABC/HR.

More info on conducting blind tests can be found at the PC ABX site.

mp4 is video.

You could start here or here.

I have only used bladeenc. Is there much of a difference?

Panos' audio test has been ridiculed all over the audio compression world. Look here for a review of some reviews, including the Panos' test (ekei.com)

Better tests to refer to would be ff123's test or the new C'T test (which IMHO isn't as good as ff123's). There were some complaints about C'T not using the optimal settings for Ogg Vorbis but it didn't method since Ogg Vorbis clearly won the test anyway.

c't is a good computer magazine, but it's not an professional audio magazine.
If you want to learn about audio encoding, listening tests and so on, visit audio-illumination.org and ff123.net

A discussion about heise's listening test can be found here.

128 kbps.

It's difficult to do fully double-blind tests, because many people who are interested in this kind of structured testing can identify a particular format by the type of artifacts it produces. Having said that here is a pretty rigorous test of a number of formats. Mp3pro comes out top, ogg a very close second, with aac and wma trailing. This only covered 64kpbs samples.

Note that mp3 wasn't even included - pretty much everyone interested in the topic already knows that mp3 doesn't match up to the best formats such as ogg or mpc (probably not even wma), so there's not a lot of point testing it. It's only use is its wide-ranging hardware compatibility (a good use, mind you :) )

Oh, and mp3 isn't free, the cost is just hidden.

Cheers, Paul

Well, for low bitrates: ff123's low bitrate test might be what you are looking for.

gnoshi

Note the tests at 128 kbps seem much more difficult to discern a clear winner without resorting to some statistical work.

Note the tests at 128 kbps seem much more difficult to discern a clear winner without resorting to some statistical work.

Note the tests at 128 kbps seem much more difficult to discern a clear winner without resorting to some statistical work.

mp3pro is better than wma8 (99.9% confidence)
mp3pro is better than aac (99.9% confidence)
oggq0 is better than aac (99.3% confidence)
oggq0 is better than wma8 (99% confidence)
ogg64 is better than aac (97.2% confidence)
ogg64 is better than wma8 (96.1% confidence)

Is 99% good enough for you? Or perhaps you should just take the two at 99.9%?

Dammit, the lameness filter is kicking in. No, these are *not* junk characters - I'm trying to show the peon some useful statistical information, you worthless piece of software. I've already removed all the hyphens, what the hell more do you want me to do? Is a percentage sign 'junk'? Is a question mark? Is a space? What the *fuck* use is this, when it doesn't stop all the crapflooders in any way whatsoever - they just flood with random gay/incest/beastiality sex stories instead... I've been posting on this site for years, and for my sins haven't crap-flooded once - give me a LITTLE FUCKING LICENCE TO POST MATERIAL.

Vorbis at 96kbps is usually somewhere between the quality of a good 112 or 128 cbr mp3, and I and quite a few other are already in the belief, after early testing from 1.0-ish CVS-code, that it is better than wma8 at 64kbps.

ff123 will be conducting a 64 kbits/s blind listening test where people will send in their results, and that will show how vorbis stacks up against the likes of wma8, mp3pro and quicktime-AAC.

IMO it doesn't really matter if it is better.. if it is at least comparable, than that should be enough for us to make the switch. Because besides being a flexible codec of high quality, it is open source AND completely free of patents (amazing!).. oh yeah, plus it has that really cool bitrate-peeling feature. Anyway, this is one of the few chances we have to get something right in the computer world (for a change!), so let's not blow it! Spread the word and take your hats off for xiph and vorbis!

The waiting is over people, at last we can start ogging for real! ^_^

• Unbiased listening tests prove OGG sounds better then WMA8 at 128Kbit/sec.
• Current software (not just the format) supports 255 channels, 24bit sample depth, 96KHz+ (though it's not tuned for >48KHz, it works fine and gives acceptable results)
• The Vorbis format is far more extensiable the the WMA8 format, there should be room for at least another 20% quality vs bitrate increase with no decoder change.
• Most modern mp3 players (the same ones which can play WMA) are technically able to be firmware upgraded; all thats missing is your demand.

* Much better than OGG and MP3
* Picture perfect at 128 kbit/s

And what is this comment based on? These results have been pointed out in comments for previous articles, but I'd like to mention them again. ff123 has been conducting double blind tests comparing various audio codecs, and the results are here.

The following is from the page:

Comparisons in red below are true as a group with 95% confidence.

ogg is better than wma8
mpc is better than wma8
ogg is better than xing
mpc is better than xing
aac is better than wma8
aac is better than xing
lame is better than wma8
lame is better than xing

Looks to me like WMA8 got beat by pretty much everything... But hey, what good is statistical analysis anyways...

The best tests we have at the moment were conducted by
ff123 at 128kpbs. There have been two so far (the second is technically still underway, although it's now based on outdated encoders, so I imagine a third will start fairly soon). The
first listening test compared RC2 Ogg Vorbis, LAME MP3, Xing MP3, Liquifier AAC, MPC, and WMA8. The formal analysis showed that, on the file compared, the encoders could be divided with 95% confidence into three groups (from best to worst):

1. MPC and AAC
   
2. WMA8 and LAME and OGG
   
3. XING
   

Now all we need is a third test with the latest updates of all the encoders - since we now have a new stable version both of Ogg Vorbis (RC3) and LAME (3.91).

The best tests we have at the moment were conducted by
ff123 at 128kpbs. There have been two so far (the second is technically still underway, although it's now based on outdated encoders, so I imagine a third will start fairly soon). The
first listening test compared RC2 Ogg Vorbis, LAME MP3, Xing MP3, Liquifier AAC, MPC, and WMA8. The formal analysis showed that, on the file compared, the encoders could be divided with 95% confidence into three groups (from best to worst):

1. MPC and AAC
   
2. WMA8 and LAME and OGG
   
3. XING
   

Now all we need is a third test with the latest updates of all the encoders - since we now have a new stable version both of Ogg Vorbis (RC3) and LAME (3.91).

The best tests we have at the moment were conducted by
ff123 at 128kpbs. There have been two so far (the second is technically still underway, although it's now based on outdated encoders, so I imagine a third will start fairly soon). The
first listening test compared RC2 Ogg Vorbis, LAME MP3, Xing MP3, Liquifier AAC, MPC, and WMA8. The formal analysis showed that, on the file compared, the encoders could be divided with 95% confidence into three groups (from best to worst):

1. MPC and AAC
   
2. WMA8 and LAME and OGG
   
3. XING
   

Now all we need is a third test with the latest updates of all the encoders - since we now have a new stable version both of Ogg Vorbis (RC3) and LAME (3.91).

The best tests we have at the moment were conducted by
ff123 at 128kpbs. There have been two so far (the second is technically still underway, although it's now based on outdated encoders, so I imagine a third will start fairly soon). The
first listening test compared RC2 Ogg Vorbis, LAME MP3, Xing MP3, Liquifier AAC, MPC, and WMA8. The formal analysis showed that, on the file compared, the encoders could be divided with 95% confidence into three groups (from best to worst):

1. MPC and AAC
   
2. WMA8 and LAME and OGG
   
3. XING
   

Now all we need is a third test with the latest updates of all the encoders - since we now have a new stable version both of Ogg Vorbis (RC3) and LAME (3.91).

If you want to take the listening test yourself, read the instructions and jump in. For now, there's also a page of interim results, but to quote ff123, "Major conclusion: I need more listeners!"

Monty

If you want to take the listening test yourself, read the instructions and jump in. For now, there's also a page of interim results, but to quote ff123, "Major conclusion: I need more listeners!"

Monty

ff123 is planning more tests at such bitrates - it's difficult to organise a test, gather results and publish findings on the same day as an RC.

This test is actually being analysed properly, using the insights gained from his previous 128kbps test, whose results you can find here). You can see the preliminary results here: http://ff123.net/128test/interim.html. In particular, look at the results of the last of the three test files. With overall 95% confidence we can say that on this test clip:
mpc is better than xing
ogg is better than xing
lame is better than xing
aac is better than xing
mpc is better than wma8
ogg is better than wma8
lame is better than wma8
aac is better than wma8

Aside from this, listening tests have previously shown that WMA8 is better than WMA7 at the low end (sub 64kpbs), at the expense of being worse at the middle to high end (112kpbs upwards). For listening to music on anything that costs more than $10, you would do best to stay away from WMA (and, looking at the other results on the page I previously mentioned, from Xing encoded MP3s as well...).

This test is actually being analysed properly, using the insights gained from his previous 128kbps test, whose results you can find here). You can see the preliminary results here: http://ff123.net/128test/interim.html. In particular, look at the results of the last of the three test files. With overall 95% confidence we can say that on this test clip:
mpc is better than xing
ogg is better than xing
lame is better than xing
aac is better than xing
mpc is better than wma8
ogg is better than wma8
lame is better than wma8
aac is better than wma8

Aside from this, listening tests have previously shown that WMA8 is better than WMA7 at the low end (sub 64kpbs), at the expense of being worse at the middle to high end (112kpbs upwards). For listening to music on anything that costs more than $10, you would do best to stay away from WMA (and, looking at the other results on the page I previously mentioned, from Xing encoded MP3s as well...).

This test is actually being analysed properly, using the insights gained from his previous 128kbps test, whose results you can find here). You can see the preliminary results here: http://ff123.net/128test/interim.html. In particular, look at the results of the last of the three test files. With overall 95% confidence we can say that on this test clip:
mpc is better than xing
ogg is better than xing
lame is better than xing
aac is better than xing
mpc is better than wma8
ogg is better than wma8
lame is better than wma8
aac is better than wma8

Aside from this, listening tests have previously shown that WMA8 is better than WMA7 at the low end (sub 64kpbs), at the expense of being worse at the middle to high end (112kpbs upwards). For listening to music on anything that costs more than $10, you would do best to stay away from WMA (and, looking at the other results on the page I previously mentioned, from Xing encoded MP3s as well...).

There recently was a post in for an audio test. Let's see... http://slashdot.org/article.pl?sid=01/10/31/212723 1&mode=thread, which leads to this: http://ff123.net/128test/instruct.html

I participated in this test. There were 3 samples of music, each encoded 6 different ways. I could not tell the difference between any of the 18 encoded files and the originals. No way, no how. As a caveat, I should mention that I was in the US Navy for 10 years and worked in an engine room for many of those years, so I was exposed to high decibels for long periods of time. I found a tone sweep somewhere and played it. I can't hear anything over about 13 kHz.

So, for me, this is overkill. I've been ripping my CD's to MP3's pretty steadily. I've found a few that sound bad. I'm marking them and when I get through with all of them, I'll re-encode those at a higher bit-rate. Total cost to me: 40GB hard disk - $200; Winamp - free; Lineout cables to preamp - $10. Already had the computer and sound card. At this point, I'm not sure 40GB is enough, but if it comes down to it, there's a lot of crap in amongst what I've ripped so far. (Not every song on every album is a good one.)

So, a word of advice from an old (38) geezer: Buy expensive stereo stuff when you're young. Also, don't blow out your ears too early - wear hearing protection at concerts, while shooting guns, perhaps at work if it's noisy there. You'll thank yourself later.

The tests that you link to were done incredibly badly, and should just be ignored.

Here is a test that, although not perfect, was at least semi-blind. The conclusions: at that bitrate, MPC ('MPegPlus' not 'MP3Pro') and AAC were the best, followed by LAME MP3, OGG & WMA8 all together, and finally the very worst was XING encoded MP3.

This test was run with Vorbis RC2. RC3 will be out in a week, with much improved noise masking. For a taste of RC3, you can check out the Vorbis CVS, which includes most of the RC3 fixes but encodes at a fixed rate of 128 kp/ps. This raises the low pass, improves the noise masking, and the stereo channel coupling code.

MPC has better licensing than Monkey's Audio: the *decoder* is open source (GPL even), so you will always be able to decode your music. *encoding* is only possible on Windows however (although there is an older binary version available for i386 Linux systems), and the encoder will be made shareware in the near future. This is a real pity, because tests have shown that even at 128kp/s MPC is up there with AAC (MPEG-4 audio).

If you're running Windows, you can get ABX from http://www.pcabx.com/. On UNIX systems, the LAME source code comes with an ABX program (in the misc/ directory, I think).

Here is an example of a test that took place using a slightly different testing methodology, more akin to MUSHRA (which is used to evaluate lots of encoders at the same time): http://www.ff123.net/128tests.html.

even though i also use 192kbps on my mixed song, i found that 128kbps on mp3 is pretty good enough for everyday listening.

check out http://ff123.net/cbr128.html