Slashdot Mirror
2nd Multi-Format 128kbps Public Listening Test
technology is sexy writes "Roberto Amorim has launched his latest public listening test evaluating the performance of different audio codecs at 128kbps, among them Apple's AAC implementation (used in iTunes), LAME, Ogg Vorbis fork auTuV, WMA, Musepack and even Sony's Atrac3 format, which is soon to be used in their own music store. Read more on Hydrogenaudio and check out the results of prior tests. As opposed to most evaluations of audio codecs, this is a scientific test adhering to ITU-R BS.1116-1 as much as possible while still allowing everybody to participate."
Never heard of it.
Ogg, ogg ogg. Ogg oggity ogg ogg!
Now that that's out of the way, let the insightful comments begin.
I know you can do frequency analysis on the output of these various codecs. Just compare that to the average human auditory capacity and you can get an objective measurement of the merits of these various compression methods.
So uh, why is this necessary, exactly?
128kbps doesn't cut it. It's an absolute lossy, disgusting bitrate, no matter what it's in. They should test similar file sizes instead of by bitrate, to determine whether something is good or not- this gives a better impression of quality vs size, instead of a purely comparison based test.
Great, now all the ____ fanboys are going to forge results to make their codec look good. Talk about useless tests.
Not possible. All you will get is a bunch of WAV-files, you have no way to tell which file belong to which codec.
That said, I don't care which codec wins the test because Vorbis is still the only one free from patents and the margins are so incredibly small.
Vorbis will win for me even in the unlikely scenario that it comes out last.
My other account has a 3-digit UID.
How do you bas a listening test on the web? People with crappy speakers are going to say that all of them sound bad yet the people that have the better speakers are going to have the better responses. This should be something that is done in a controled environment so that the hardware that is playing back the audio is standard.
Yes... certainly this kind of listening test is important to access the capabilities of each codec.
But in the real world other factors may be more important to chose a coded, like for example general acceptance, freely available code and specs, and a large content base available.
You see: performance will increase allways in all codecs with time... so this kind of testing is only a minute factor amongst others.
You cannot proceed from the informal to formal by formal means
Of course, if that turns out to be inferior to any of the other formats, it would prove that something's wrong with the tests.
Why does anyone still use 128kbps? I hate it when I download music (legal ;) and the only bitrate available for the song i want is 128. With 200GB+ hard disks being so affordable these days and everyone having high speed, I think everyone should encode their (mp3||ogg||aac) at 192 or 256.
BTW, I think the difference between MP3 and Vorbis at 128 kb/s is perfectly noticeable. MP3 sounds rather bad, vorbis sounds pretty good. And the point is precisely to tell which format sounds best, so you don't want to do 512 kb/s bitrate where all formats sound close to CD quality.
There used to be a great site called r3mix.net, which, IIRC, did some spectral analysis on some of the assorted compression algorithms (trying various different options for them). It was focused on the LAME mp3 encoder, but also looked at a few others.
They also had some great forums for info on music ripping/preferred encoding methods/CD burning/etc.
Now, that URL goes to some lame "sponsored mp3 links" site.
Anyone know why r3mix.net died and if there's any new site that makes a good replacement?
When you listen to compressed audio over inexpensive speakers / headphones, you can't hear the difference. With my Sony Studio Monitor headphones, I lost the difference at about 250k with mp3, so I started using 320K as that was the best at the time. Then I bought $2000 Martin Logan Mosaic Speakers, and the original CD was clearly better than even the 320K bitrate. So now I only do lossless compression. That's fine at home, but in any other environment, there's usually so much noise and distractions that even if you had excellent headphones or speakers, you wouldn't appreciate that little difference lossless brings over 256K or even 128K.
DON'T CLICK THE LINK!
The sad thing is that somebody went to the trouble of putting together a perfectly reasonable, logical post just to throw in a porn link. *sigh*
Karma: Segmentation fault (tried to dereference a null post)
Of course, proving the patent-freeness of Vorbis requires searching every single patent with a fine-toothed comb, further indicating how messed-up the whole patent system is at this point.
I just have to wonder how many companies are waiting to pounce on the first major commercial user of Vorbis with a patent suit. (Yes, I know there are commercial users of Vorbis, but none are really big enough to attract patent litigation, especially since none of them are wedded enough to vorbis that they wouldn't be able to just drop it for mp3 support with its well-known and, IMO fairly-reasonable, license fees.)
There are 10 kinds of people: ones who understand ternary, ones who don't, and ones who think this joke is about binary
I'd read the thread when they were discussing which version of Apple's ACC codec to use for the test, and concluded based on a few samples that the new version was subpar.
If they'd included both versions of iTunes/QuickTime in this test, perhaps they could have helped shame Apple into fixing what they broke.
A .wav file at 128kbps is going to sound absolutely awful. At 8 bits per sample (which sounds pretty bad no matter what), 128kbps gives you a sample rate of only 16khz, so any frequencies above 8khz will be lost. If you up the sample quality to 16 bit (CD quality), the sample rate goes down to 8khz (4khz frequencies).
And this is for monaural sound. If you want stereo, cut the sampling rate in half -- this might cut it for voice, but it won't work for anything else.
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
That is not lamer-proof.
One could just send in forms with the same ratings to manipulate the test arbitrary.
If you mod this up, your slashdot background will turn into a beautiful sunset!
The best replacement for r3mix.net in my opinion is HydrogenAudio . The forums are frequented by a lot of professionals, as well as developers of LAME, FLAC, Nero AAC, Musepack, Wavpack, and other codecs.
Not possible. All you will get is a bunch of WAV-files, you have no way to tell which file belong to which codec.
.ogg vorbis, an mp4 and 3 flacs. If you want to be biased either for or against mp3/oggvorbis/quicktime itunes AAC, you can.
Check the contents of the sampleXX.zip files; you actually get an mp3, an
SCO employee? Check out the bounty
You are 100% clueless, pardon my french.
.wav file is about 1.5Mbps.
The bit rate of
it's double-blind, so you don't know what you're testing. Good gear has practically no bearing on identifying compression artifacts - that you need good equipment to hear slight imperfections is a myth.
Jeremy
Vorbis does variable bit rate and you set the quality you want. That way you don't waste lots of bits where they are not needed. My 4MB ogg file sounds as good or better than my little brother's 6MB mp3. The difference is more songs on my 256MB compact flash card. Yes, it's easy to play that music on my Zaurus, which cost about as much or less than DRM gimped portable music players.
I hate it when I download music (legal ;) and the only bitrate available for the song i want is 128.
Cry me a river.
Friends don't help friends install M$ junk.
The r3mix tuning (--r3mix), while a small step forward, was inherently flawed because of his insistance on tuning based on pictures instead of acual listening tests. As a result, the --dm-presets were invented and improved by Dibrom (the HydrogenAudio founder) along with a multitude of testers. eventually those were included in LAME as the --alt-presets (and in the latest version they just replace the normal --presets). In short, Hydrogen Audio is THE place to go for this stuff now.
Jeremy
After a while, once you have weeded out bad ways, one is going to reach the following situation. Each algorithm will perform very well for a large set of music and poorly for some small set of music. Barring pathologies, The poor set will be assymtotically fixable by increacing the bit rate. By the way this is not just my opinion. Theres theorems that say this is true of any compression scheme when applied to all problems.
what does this mean? it means that the end user is never going to work at the truly low end of the bit rate specrrum because they want something that virtually always works. Plus they want a wee bit more just in case they have to transcode it. So if the recommended rate is 128 people will encode at 160.
So these comparisons need to be done not at the bitter edge where music flaws are easy to spot because NO ONE WILL ACTUALLY MAKE THAT THE OPERATING POINT THEY USE. That is to say everyone knows vorbis sounds so-so at 64KB while MP3 sound much worse. But no one wants So-So they want darn good. So they are going to recors their Mp3 at 160 and at 160 Ogg and Mp3 sound so close that the size of the test you'd have to do to pick up the difference is silly.
the proper way to do this is the following. Pick the gold standard format, say MP3 and its standard excellent operating point, say 160. now test all the others at lower bit rates than 160, and see which one has the lowest bit rate that scores as good as the Mp3 at 160.
comparing all methods at a constant bit rate, esepciall a low one, is stupid
Some drink at the fountain of knowledge. Others just gargle.
It is really refreshing to see someone so willing to demonstrate their wrongheaded ignorance. Saves us all a lot of trouble.
I've found most of the people on Hydrogenaudio to be incredibly pragmatic. Perfection isn't the only parameter of importance. If it were, they'd not be wasting time testing codecs at 128kbps, except to demonstrate their unsuitability compaired to losless formats. They'd not be wasting time letting phillistines with their waxy untrimmed ears particpate in listening tests with their $20 sony earbuds.
As for the vendors lauding useless gear, um, what vendors lauding useless gear?
But hell, why let any of that get in the place of a perfectly good piece of ranting rhetoric. Still, it would be better if you'd unloaded at a deserving target. There are certainly enough of them out there.
People are constantly comparing audio coding standards, but realize that most of the stuff you hear is marketing speak. Many companies have lots of IP in this area and they obviously want to make their solution the standard.
:) Those books can provide a good basis of how all the coders tested works.
What makes one codec sound different than others is the psychoacoustic measures implemented, quantization method, and the windowing scheme implemented before MDCT is performed. Note that all of the coders tested there do not use the same windowing method, but all of them use MDCT in a way.
MP3 is a subband coding, it slices the audio into sub bands before transforming them. AAC, OTOH, is not. AAC uses straight MDCT and does the filtering there. The criteria for filtering is still the same old, tho. That is part of the reason why AAC at 128 kbps way outperform MP3.
Psychoacoustic is not new, it's been described extensively in a book "Psychoacoustic" by Fastl. The catch is, audio coders have to take into account the complexity of performing the full model. MP3 uses a very simplified version of it, and it taxes the highest spec of its day. That is also the reason why AAC-LC (low complexity) is more popular than AAC-Main profile nowadays.
Vorbis can sound better because with new hardware, a more mathematical heavy version of psychoacoustic can be implemented today. Plus, they discard the notion of constant bitrate and use quantization quality instead. This is also evident in FAAC.
128 kbps stereo is practically the limit of almost-transparent quality audio now. 64 kbps mp3pro is just bull, it doesn't perform anywhere close to modern mp3 at 128 kbps. There is a limit on compression, and that is governed by the entropy (information content) of a signal. You go lower than entropy, you lose information, simple as that. Having said that, the only way to reduce entropy is using psychoacoustic models, and that also have a limit.
Note also that Dolby-AC3 that is used in DVD and movie theatre compresses 5.1 channels into 384 kbps, or roughly 150-ish kbps stereo. Again, the same lower limit is evident. They do compression by combining the high frequencies > 15 khz and ignore the phase information in that high frequencies. As you can probably tell, AC-3 sounds pretty good.
If you're interested in this area, I suggest the MPEG-4 book by Ebrahimi, Psychoacoustic by Fastl, Multimedia Compression by Gibson and DSP First by I forgot who