Listening Comparisons For Audio Codecs At 64kbps

waaka! writes "Hydrogenaudio has just wrapped up a listening test of various audio codecs at 64kbps. Check out the results, where Ogg Vorbis performed quite well, scoring significantly better than WMA, RealAudio and QuickTime AAC, and kept pace with MP3Pro and HE-AAC (AAC with the SBR extensions that MP3Pro uses). Clearly, though, no codec can honestly claim 128 kbps MP3 quality at 64 kbps. The charts at the end show entries for 128kbps LAME MP3 and 64kbps FhG MP3, but these are used as high and low anchors for reference, as MP3 is really out of its league at bitrates such as these."

Comments

Before anybody says..

[Anonvmous Coward]

... "Why not just encode it at 384 and be done with it?", consider that PDAs have 64 megs of RAM, cell phones get no better than 56k, and that not everybody has broadband.

Re:Before anybody says..

[Capt'n Hector]

That's what the 40 gig iPod is for.

But how...

[Atario]

...to compensate for everyone's crappy $1.99 computer speakers?

Sometimes you don't need high audio quality

[Anonymous Coward]

Sometimes, simple audio clips don't require the highest quality. Function triumphs over high performance hot-rodding.

CD

[Leffe]

Clearly, though, no codec can honestly claim 128 kbps MP3 quality at 64 kbps.

You used to compare against CD quality.

Oh well, times change, I guess it's time to throw all my CDs away and instead store all music in this new exciting digital format.

And seriously, does anyone listen to music encoded at 64 kbps? 128 is the bare minumum.

Re:I'll say this

[Anonvmous Coward]

"the purpose of PDAs is not to play music "

Yet they do, and people like to use them for that. Fascinating.

"the purpose of cell phones is not to play music"

Yet the holy grail of mobile computing is to merge the PDA (which can play music) and the cell phone.

You may like carrying around a cell phone, PDA, and iPod in your pockets, but I want one device that does it all.

MP3 is the standard.

[Schezar]

MP3s are ubiquitous. My computer, DVD player, portable audio player, and car stereo all support it. The same can't be said for other formats.

More to the point, why are all of these competitions at such low bitrates? The differences in quality between various types of audio compression become indistinguishable (and therefore irrelevant) as you raise the bitrate.

I just use good old variable bitrate MP3 and forget about it. Simple and standard.

Re:MP3 is the standard.

[molarmass192]

You're thinking in terms of music only. The MP3 patent (#5,579,430) only expires on Jan 26, 2015. So if you're a software/game maker with slim margins to start with, you'd want a format that's not patent encumbered or that cost less. Sure a $1 royalty fee for each decoder doesn't seem like much until you ship 500k units and have to cut that check to Fraunhofer. If the quality is comparable and a lower/no cost option is available, you'd have a pretty strong case for not using MP3.

Re:MP3 is the standard.

[StaticEngine]

The Game Industry has embraced OGG, although somewhat silently. With slim budgets, we're always looking for the cheap (and free) solution, and OGG is perfect when we want compressed audio at a good quality.

The sole deciding factor in whether or not compressed audio really gets used in a game is available minspec bandwidth. If marketing is forcing us to target a 500MHz machine, and decompressing OGG audio kills our framerate, then audio compression goes. It the sad truth that the tech heads do not call the shots in this department.

Audio Quality

[RobPiano]

Its nice to see ogg doing well, but ogg needs to start advertising. Nobody is going to give a shit about ogg if computer companies (apple) is distributing aac and their old napster mp3s don't play on an ogg player. Fact of the matter is that there is a LOT of power behind the MPEG (4, 7, 21) movement.

Have you seen the cuidado project? Have you read what companies publish? Why fight the good fight if you aren't getting the public hooked on it while its still the best option.

hmm
Rob

some good listening test material

[MarcoAtWork]

I saved this thread quite a while ago and I agree with several of the recommendations (notably with the 'Tori Amos' 'Boys for Pele' CD, not that it's the type of music I usually listen to, but I have to admit the production values are outstanding), after all using hyper-compressed (re: other slashdot articles) crappy source material is not that helpful in terms of figuring out how good the various encoders really are...

the thread on google

Personally I rip my own CDs with lame --alt-preset extreme (on said Tori Amos' CD it seems it hovers around 224kbps with -lots- of frames at 256 and 320), for fun I transcoded (I know, transcoding is bad, mmkay?) a few of them to vorbis 48kbps and it's amazing how good they sound at that low of a bitrate.

question

[zymano]

Are we at the LIMITS of compression technology ? Is there anything new being worked on by anyone ?

Re:question

[merlin_jim]

Are we at the LIMITS of compression technology ? Is there anything new being worked on by anyone ?

Well we are certainly near the limit of lossless compression. In that there is a well-studied field of computer science (information theory), which provides a framework to determine the theoretically maximum amount of lossless compression possible given a particular data sample, and the best lossless compression algorithms we can come up with are within a small percentage of that figure. FYI, a fundamental tenant of information science is that everything can be reduced to a certain atomic level of representation, and that this atomic piece is the "information" contained within "data"... and that one cannot convey "information" in less space than this atomic piece.

For instance, I've heard that common every day american english conveys approximately 1.2 bits of information per word... meaning that the least redundant approximation of human speech would need that bit rate to represent it.

As far as lossy compression, there might or might not be more work to be done. The problem is that the human ear and the human auditory nervious response are far from being fully characterized, though we do have a good start on it.

The idea of a lossy compression algorithm is to remove pieces of information that the human ear and/or auditory nervous response is not sensitive too... therefore increasing the theoretically possible maximum compression without adversely affecting the signal representation. As we as a species come to characterize these human responses, we will certainly see better codecs coming out. I do however believe that we're rapidly approaching an asymptotic level of understanding where further levels of effort and research into codecs is not economical with regard to expected payoffs...

Re:Streaming audio

[connect4]

Sometimes people try to encode stereo into less than 64kbs. It's just crazy. I've listened to mono 56kbps mp3 that sounds a lot better than stereo 64kbs.

I'd much rather listen to a favourite song in mono with reasonable reproduction quality, than in stereo that sounds like it's coming out of a tunnell.

Re:I'll say this

[Anonvmous Coward]

"actually, I carry a cell phone, PDA, gameboy advance, *AND* an iPod. You know why? I hate compromising. And like it or not, there aren't any decent "convergence" devices that can do 2 of those things well, let alone 4."

I agree with you from the gaming standpoint of it, but I don't on the rest of it. There's no reason why a single device can't be a PDA, phone, and Mp3 player. The storage is the problem now, and one day in the not too distant future, that'll be fixed.

Re:I'll say this

[mabinogi]

You see, that's why choice and diversity are a good thing.

You like having a dedicated device for everything, some people (me included) like having something small and convenient that'll do a fair job of a lot of things....

However, don't make the mistake of thinking that just because your preference means there is no need for something (in this case low bitrate digital audio) therefore means that no one has any need for it.

Okay, maybe I'm nitpicking...

[shirai]

Just some relevant data that doesn't appear on the front page is that the test is blind and they do compare the audio to an uncompressed reference. You have to click through to "Return to Roberto's Listening Tests page" to find this information though. Just thought I'd mention this because my first thought was what they are comparing the sound to.

Comparing without a reference reflects how much you like the encoding of the codec, not how accurate it is to the original. For example, if a codec boosts the bass or encodes slightly louder, you may interpret this as better sound. For example, when auditioning speakers, you must always balance the output of the speakers as most people will psychologically prefer the louder (most sensitive) speaker. This does not mean the speakers are accurate however.

At any rate, here is the relevant quote on that page:

One of the most acclaimed methods of comparing codec quality is by performing so-called "Double Blind Listening Tests". In this sort of test, the participant compares various encoded samples against each other and against an uncompressed reference sample. The blind part means that the participant doesn't know which sample was encoded by which encoder. That guarantees there'll be no psychological bias towards his/her favorite codec, or against the codec he/she dislikes.

Note that the quote (and here's the nitpick) suggests that double-blind means that the participant doesn't know which encoder is used. Double-blind means that both the participant and the person running the test don't know. By the way, this is, indeed as accurate as double-blind (since, well, the computer might know but surely doesn't care to influence the results). And I realize he doesn't say "double-blind means" but seems to suggest the definition of double-blind. Anyways, that's just the nitpick. Please don't mod me down for it. It's just an observation and I'm trying to build some Karma!

To paraphrase Gates

[commodoresloat]

Nobody will ever need less than 64kbps of audio.

Digital Music artist perspective

[merlin_jim]

As an artist that releases mainly online, I found these results very interesting, and thought I'd share my feelings with the slashdot community.

While MP3Pro and Vorbis were good competitors overall, and have a fairly good footprint to boot, I'd have to say that if I'm forced to encode to 64MBit/s, I'd absolutely choose Ahead HE AAC, if I'm judging solely on this comparison (which I am at this point in time...)

Why? Because there was no sample that Ahead HE AAC did POORLY at. MP3Pro and Vorbis (and all the other codecs) each had one or two samples that they just totally choked on, quality-wise. So if I was forced to use a 64 MBit/s codec, it would absolutely be Ahead HE AAC, because while it didn't score highest on every test, and the three codec were virtually tied across the whole competition, I would feel far safer trusting my best digital work to a codec that, according to this test, would have the least chance of representing it particularly poorly.

I wonder how these results compare to higher encoding rates; I could easily imagine that most codecs have a sweet spot, where the encoding quality/bitrate maximizes... it would be interesting to do some research to find this sweet spot.

Anyone want a quick way to slashdot a server? :D

Re:I'll say this

[rutledjw]

I think people will get over it. I volunteer with a Jr High youth group and there are 2 things I've noticed:

• Almost ALL of these kids have cell phones. These are middle class kids mind you.
• Almost ALL OF THEM message each other using that miserable little numeric pad.

The issue is that you're too old! ;) Don't take it personally, I am to, I don't like using the number pad either. But the kids have it memorized. They can fly through messages almost without looking. They'll do a couple quick checks, but that's it.

Pretty impressive really. I'm trying to change my ways and try using my itty-bitty phone for e-mail and whatnot. I mean gimme a break, I'm 29 and already getting outpaced...

Check yer fly, your ignorance is showing...

[CaptainPhong]

Ok, first of all, while the test was DISCUSSED on HydrogenAudio, and most of the participants are HA regulars, Roberto Amorim did all the hard work of organizing the test and compiling the results, dealing with complaints, etc. To not give him credit is not very nice.

Second, there was a 128kbps test a month or two ago (which for some reason got repeatedly rejected when submitted to slashdot). You can see it here. Unfortunately, the results there aren't quite as interesting (it was mostly a big tie). Unfortunately, tests at higher bitrates are difficult because detecting problems at, say, 160kbps often requires well trained ears and good audio equipment.

Third, it's a good idea when commenting on an article to actually read it and click around on a few links to actually have an idea what you are talking about. Many /.ers seem to be only half-literate (can write but not read). There is a hilarious number of denegrating comments here by people who know nothing about either statistics or psychoacoustic audio compression. ABC/HR type methodology is the standard for comparing the relative quality of audio sources. Also, a great deal of effort went in to assuring that the best settings for the best encoders for each codec were used for the test. A little reading of the pre-test discussions would reveal this. Further, HydrogenAudio is not a club of audiofools who spend zillions of dollars on fancy speaker cable without any science to back it up. It is an objectivist forum. Anyone who makes statements without backing them up (with something like ABC/HR or ABX results) gets flamed HARSHLY. Some of the regluars have PhDs on various audio topics. They know what the fuck they're doing.

Fourth, just because you don't have a use for 64k audio, doesn't mean the results are meaningless. Lots of people have small-capacity players, and some codecs can tolerate that bitrate for very casual listening (such as in the car). Lots of streaming audio sources are at this bitrate or lower. Satellite radio is at 64k or lower. Also, it's not a good idea to try to extend these results to other bitrates. MPC for example, isn't even worth considering at 64kbps, but at bitrates over about 140kbps, it will beat the pants off of anything else.

Finally, for those who want to know more, or want their audio collections to sound best, read the FAQs at HA. Many codecs have a preset where they are transparent for the vast majority of samples; usually a VBR setting that averages somewhere between 160 and 200kbps (such as lame --preset standard, mppenc --standard, oggenc around -q5 or -q6).

obligatory audiophile style rant

[plip]

I want to give full respect to the people who put all the research into creating these new audio formats. The results are truly phenomenal for 64kbps codecs. It's a fabulous academic demonstration.

However, each is what it is. A 64kbps codec.

I have about $6000 invested in my 2-channel +subwoofer setup here at home, and I consider that moderate compared to what you can truly achieve. I love listening to music, and it is completely remarkable when it is reproduced as realisticly as possible. So I go to painstaking methods to make sure the AC power is clean, the wiring is right, the distortion is low as possible. The signal to noise ratio is far between, with a good amp, and great speakers... I am especially pleased when the recording I am playing on my wonderful system is in the best production quality that it can possibly be.

As amazing as they are, these 64kbit formats are useless on a person like me. I crave LOSSLESS not LOSSY. I might as well be listening to music on a $60 AIWA boombox, since it would sound relatively similar either way. All the subtle beauty and realism of the music is completely wasted with destructive compression.

And for those of you that say it's for portable devices, It's not too unreasonable to get a portable player that plays high streaming VBR mp3s with some nice ~$100-$150 headphones. The small little investment to hear your music from 20hz-20khz flat response with low distortion is worth every single penny.

I simply do not understand the need to take our ever improving technology and lower the quality of the music. If anything, it should be increasing... higher resolutions. 24bit/192khz technologies, and wonderful DSP equalizers, large portable storage devices... they are all realities now, but nobody seems to care but the fanatics like me. I would think that techno geeks would care more about the music they love, but that does not seem to be the case. The only logic that I can fathom to explain why is that perhaps they don't even know what they're missing. I know I didn't, until I actually experienced how good sound quality can be on the right system.