Slashdot Mirror
Audio Format Listening Tests Concluded
Pointing to the conclusions of this listening study, nullity writes: "The results are interesting, and show a high variation in the performance of the various codecs on different musical styles. Ogg seems to work well on dance music, WMA8 on chamber music, etc."
Well....not quite. There's a different frequency distribution between electronic, pop acoustic and classical music.
Specifically, electronic music, which most dance stuff is, has a very flat frequency distribution. See this for yourself - load your favourite media player, siwtch on the graphic equaliser graph and watch how basically nothing happens except in the mid-range.
Now try again with an orchestral piece. There will be much more variation, though in most it will tend towards the top end.
Now try again with rock. Tends towards the bottom and top, with middle frequencies missing.
Keep going with any format you feel like mentioning...you'll get the same.
Actually, this is a striking example of how recording techniques can ruin sound as well. Take a look at the Apollo 440 album - Gettin' High on Your Own Supply. A good mixture of guitars and electronics, right? Well, look at the frequency graph again. See how virtually every guitar frequency variation has been cut out: this music was recorded digitally, mostly using samples by the looks of it. The normal variations you'd associate with having guitars play live are all filtered out, and the graph goes back to the flat digital sound again.
Cheers,
Ian
that these codecs are lossy, and take advantage of the fact that the human ear is better at hearing certain things than others to pair out extraneous info and improve compression. IOW, it doesn't matter how technically different the new files are as long as they still sound the same to the human ear.
BlackGriffen
First of all, people should NOT confuse mp3 with mp3 pro.. mp3 pro has very little support, is only useful for lower bitrates, and is even more mangled than mp3 when it comes to patents and stuff. So even though it won by a small margin, it still doesn't stand a chance against vorbis in practice.
Besides, vorbis will improve at 64 kbps in the future, and probably beat mp3 pro one day. The vorbis-team has focused most on the 128-ish bitrates for now.
One more thing.. mp3 is not even included in this test.. but perhaps they should have, as I know from experience that mp3 at 64kbps sucks very much.. would it suck even more than wma8 and QT-AAC? Probably! But they should almost have included mp3 if only so some hesitant people could see how bad mp3s really are for streaming..
More tests will come in the future.. one at 128 kbps too I suppose.. and there I guarantee you that mp3 will be included and get its ass kicked by vorbis. Vorbis is a superior technology, don't get that confused. Its main competitor is aac, but aac is patented and not free, and currently not very tweaked.
Its not diferent formats. Just diferent encoder options.
The oggq0 entries are for music enconded in a Variable Bit Rate mode (oggenc -q0) -- the encoder defines a quality treshold and uses whatever bitrate necessary to keep it there.
The ogg64 entries are for music encoded with a nominal bitrate (oggenc -b 64 --manage) -- it atemts to keep the bitrate around 64 kbps without looking at sound quality.
Why did only Ogg Vorbis got to show these two modes? Because though the test focuses on 64kbps (nominal bit rate) encoding, its likely than most Ogg Vorbis users will use variable bit rate encoding with it. I know I do.
ABC/HR.. as in ABC/Hidden Reference... as in, there is a copy of the original track included as a hidden reference on every single trial.
The users are given 2 sliders per sample laid out on a panel. The samples are loaded in random order. On the sliders for each sample, one slider is for the original sample, and one is for the encoded. These are also randomized per sample. The user does not know which is which. If they happen to rate the original sample less than 5.0 (highest rating, meaning it should be transparent), then their results are disregarded entirely for that sample.
mp3pro is better than wma8 (99.9% confidence)
mp3pro is better than aac (99.9% confidence)
oggq0 is better than aac (99.3% confidence)
oggq0 is better than wma8 (99% confidence)
ogg64 is better than aac (97.2% confidence)
ogg64 is better than wma8 (96.1% confidence)
Is 99% good enough for you? Or perhaps you should just take the two at 99.9%?
Dammit, the lameness filter is kicking in. No, these are *not* junk characters - I'm trying to show the peon some useful statistical information, you worthless piece of software. I've already removed all the hyphens, what the hell more do you want me to do? Is a percentage sign 'junk'? Is a question mark? Is a space? What the *fuck* use is this, when it doesn't stop all the crapflooders in any way whatsoever - they just flood with random gay/incest/beastiality sex stories instead... I've been posting on this site for years, and for my sins haven't crap-flooded once - give me a LITTLE FUCKING LICENCE TO POST MATERIAL.
-- Help Digitise the Public Domain at DP.
Comparing 'pure audio' to the compressed version is inbuilt into the test methodology. I suggest you download the program used for the tests, and try it out before commenting further. Hopefully this will give you some semblance of knowing what you're talking about.
-- Help Digitise the Public Domain at DP.
-- Help Digitise the Public Domain at DP.
I am Karma Man, hear me Whore.
An honest double-blind listening test is extremely difficult to arrange, and there is no evidence whatsoever on such on the site.
This is how the test was conducted.
The test required access to a Windows machine (probably Win95 and up, didn't try with Win3.1) with a sound card. Users were required to download the ABC/HR "practice" Zip file, which includes the ABC/HR program, the Ogg Vorbis 1.0 command-line encoder and decoder, a LAME command-line encoder/decoder (I forget which version), a FLAC command-line decoder program, and a .flac sample file (the instrumental introduction to The Eagles' "New Kid in Town").
After unzipping this, the user had to run a batch file (encdec_foobar.bat) which un-FLACced the sample file, then encoded it with Ogg Vorbis and LAME, then decoded both of the resulting files back to .wav.
Then the user executed the ABC/HR program, which is a Win32 GUI application. After loading the sample into the application (pull-down menu and file selector dialog), the interface became a row of double-slider pairs. Below each slider was a "Play" button. Below each slider pair was a "Play Ref" button. Below that was a "Stop" button. There was a pair of sliders for each decoded sample -- so for the practice run, there were two pairs of sliders: one for file #1, and one for file #2. The user did not know which file was Ogg Vorbis, and which was LAME MP3.
The user then listened to the Reference file by clicking any of the "Play Ref" buttons. After hearing the Reference, the user could then click any of the normal "Play" buttons. The first task was to determine, for each pair of sliders, which one was the original and which one was the encoded file. Having determined that, the user used the slider (which went from 1.0 to 5.0 in increments of 0.1) to "score" the sample on the subjective quality of the result. There were also text labels on the slider: 4.0 was "perceptible but not annoying", 3.0 was "slightly annoying", 2.0 was "annoying" and 1.0 was "very annoying".
Finally, there was an ABX button, which launched a different window. In the ABX window, the user could select "Original", "Sample 1", or "Sample 2" for the "A" and "B" samples. Normal ABX testing proceeded from that point. (If you don't know what ABX is, go to pcabx.com.) I found that the ABX window sometimes helped me to focus on a specific sample so that I could find its flaws; armed with that knowledge, I was able to make a determination of which of the two sliders, right or left, was the encoded version.
Once a slider was pulled down from the default 5.0 position, another button became active above that slider. Clicking on it opened a new window with a text box, into which comments could be typed. When the user was finished with the test, the slider positions, the comments, and the ABX results (if any) were written to a plain text file (DOS CR/LF format), which was to be mailed to the test administrator. (Though, of course, you weren't supposed to mail the practice results.)
Now, that was just the practice session, which was a prerequisite for participation in the actual test. For the actual test, the process was similar, but differed in a few details.
The actual test samples included copyrighted, patented codecs for which there are no freely distributable decoders. Therefore, the WMA, AAC and MP3Pro samples were distributed as FLAC files, and decoded by the batch file. Since MP3 did not participate in the listening test, the LAME encoder was not used during the actual test. The Vorbis encoder, of course, was used twice: first with -q 0, and then with -b 64 --managed.
With 5 encodings per audio sample in the actual test, there were 5 pairs of active sliders instead of only 2 pairs. But otherwise, the actual test was exactly like the practice session.
(Personal note: I did 10 of the 12 samples, skipping the two classical ones. Out of 50 encoded versions of the 10 samples, there was only one case where I couldn't tell right from left -- "The Source", encoded with MP3Pro.)
Please define little support since WMA8 is supported by a bunch of devices less than 2 years old through a firmware upgrade. I've tried looking for a RioVolt that didn't support and the only ones were the Coke brand and the SP50. So there you go, the vast majority of players support both MP3 and WMA.
Check out Hydrogen Audio
Its pretty much the best audio discussion you can find on the 'net.