Listening Comparisons For Audio Codecs At 64kbps

waaka! writes "Hydrogenaudio has just wrapped up a listening test of various audio codecs at 64kbps. Check out the results, where Ogg Vorbis performed quite well, scoring significantly better than WMA, RealAudio and QuickTime AAC, and kept pace with MP3Pro and HE-AAC (AAC with the SBR extensions that MP3Pro uses). Clearly, though, no codec can honestly claim 128 kbps MP3 quality at 64 kbps. The charts at the end show entries for 128kbps LAME MP3 and 64kbps FhG MP3, but these are used as high and low anchors for reference, as MP3 is really out of its league at bitrates such as these."

Comments

some good listening test material

[MarcoAtWork]

I saved this thread quite a while ago and I agree with several of the recommendations (notably with the 'Tori Amos' 'Boys for Pele' CD, not that it's the type of music I usually listen to, but I have to admit the production values are outstanding), after all using hyper-compressed (re: other slashdot articles) crappy source material is not that helpful in terms of figuring out how good the various encoders really are...

the thread on google

Personally I rip my own CDs with lame --alt-preset extreme (on said Tori Amos' CD it seems it hovers around 224kbps with -lots- of frames at 256 and 320), for fun I transcoded (I know, transcoding is bad, mmkay?) a few of them to vorbis 48kbps and it's amazing how good they sound at that low of a bitrate.

Re:some good listening test material

[default luser]

(I know, transcoding is bad, mmkay?)

Actually, it's not as bad as you think, given the circumstances.

The general problem with re-encoding audio is errors will become magnified versus a direct encode to the lower bitrate. If you take a 192k or 160k CBR mp3 encode and downsample it to some other format, it is going to sound like crap. But you have to remember that modes like LAME --alt-preset virtually eliminate errors in audio reproduction.

Sure, the inaudable tones have been removed, but every bit of the audible spectrum has been accurately rendered, making it nearly as good as the original source for the purposes of transcoding.

I rip all my albums using --alt-preset standard, and I transcode them to 128k ABR for my handheld mp3 player. I've never been able to hear any perceptible difference between this and a direct-from-CD 128k ABR encode.

Just Habit...

[Ironix]

I always encoded my MP3s 224 kbps and when iTunes came out I simply continued the tradition.

In any case, I can certainly notice the improvement from MP3s encoded at the same rate.

Re:Just Habit...

[hondo77]

Try 128 AAC. I can hear that it's better than 192 MP3 even through my dinky little headphones. Better sounds and smaller files make me a happy guy.

Why so low?

[Izago909]

Anything below 128k/s (in my opinion) is only good for streaming and embedding. Even 128 is the bare minimum for anything that sounds decent. Are there any comprehensive articles that deal with comparing high encoding rates (192+) of multiple formats?

It should also be noted that it is not recommended using CBR encoding with OGG. It is a native VBR codec that is only forced into CBR for steaming. The quality of CBR is much lower than VBR. It would be very nice to see a comparison that uses VBR for all codecs that stick to the same bitrate range.

Re:Why so low?

[Izago909]

Actually, even at 128 (at 48 kHz) you can tell a difference. With mp3, higher frequency sounds (ex. cymbal crashes) can artifact heavily. The more that's going on, the worse it gets. Higher range vocals also are affected. I have some bebop styled tracks that use a lot of the stand up bass and brass percussion. The vocals often sound very metallic, especially when she starts hitting the higher notes.

For most of my archival I use OGG at a quality setting of 7 (~224k/s) and transcode it to mp3 @ 128-192 when ever I need to play them on my portable. Eventually, when I quit my profession as a poor student, I might buy a portable that plays more than mp3's and wma's. Until then, I have noticed very little to no quality loss by transcoding to a lower rate. The only real problem I've noticed with OGG is that sometimes lower frequency sounds (60 -100 Hz) sometimes sound fuzzy, but noting to be too concerned with.

Re:Why so low?

[proxima]

Well, 64kbps is a good rate for streaming and low-capacity situations (like flash-based mp3 players). If ogg can manage to become more popular in hardware, it would make an excellent alternative to standard mp3 encoding.

That said, I've fallen down the quality slope - with hard drives so large now I've decided just to encode all my music with FLAC and have absolutely no quality loss (lossless compression; flac is to ogg as PNG is to JIF). Granted, I don't know if I can tell the difference between ~256kbps ogg (what I used to use) and what ends up being 900+kbps flac, but it's nice to know I can generate a practically perfect audio CD if I ever lose the originals.

WIth a decent pair ($30 and up) of headphones or a good system, some songs sound tremendously better going from 128 -> 192, and even 192 -> 256. Check out The Cranberries' "Time is Ticking Out" as a good example - the beeping at the beginning of the song is lost at 128 kbps mp3, it's poor at 192 kbps mp3, but the song sounds great compared to the wav at ~256kbps ogg (no, I didn't try 256 kbps mp3 for comparison).

Now, if NPR would start streaming programs in ogg, I could finally be rid of real player at work.

clearly though

[Anonymous Coward]

I don't think I am deaf or anything, but in all honesty I am totally incapable of distinguishing a quality 0.0 ogg from the 192kbit/s mp3 I converted it from, and even if I could, the resulting music still has the precise same quality of making me feel good and happy as it had before. If you encode it slightly higher, I believe the difference will be inaudible to anyone. So why waste space?

Links to the samples

[Anonymous Coward]

The URLs of the sample files are hidden in a text file in a zip file. I've extracted the links, and hyperlinked them, so you can download them easier.

BigYellow
DaFunk
EnolaGay
experiencia
gone
Illinois
mybloodrusts
NewYorkCity
Polonaise
riteofspring
Scars
Waiting

And to help reduce the load, Ive also got a mirror

The reality of tests is...

[overbyj]

that Joe Schmo out there with the Windows machine will be pretty much sticking to WMA. Sure hardcore audiophiles can tell a difference between formats but the average computer idiot doesn't care.

The saddest part of all is that WMA is a beast that is growing and will be hard to get rid of. Since MS has submitted this format for inspection for widespread adoption, they will continue to force their way into this becoming the de facto standard even though it sucks ass. More importantly, because of the draconian DRM, WMA is the format that the RIAA and other head asswipes at recording labels are drooling over. They could really care less about the quality of their digital music as long as they control the rights management on it.

Until more portable players support formats like ogg, WMA will be an immovable force, the 800 hundred pound gorilla that will be difficult to move.

Re:Procedural info would be appreciated

[Canar]

Yes, it was a double blind (ABC/HR) test. All the information is available in the links presented. The testers are presented with several groups of two samples, and are asked to subjectively compare the samples and state which one is the original and which is the encoded sample. Furthermore, one group contains two of only the non-compressed sample. Thus, information where one of the non-compressed samples is rated lower than the other can be easily discarded.

Digital Music artist perspective

[merlin_jim]

As an artist that releases mainly online, I found these results very interesting, and thought I'd share my feelings with the slashdot community.

While MP3Pro and Vorbis were good competitors overall, and have a fairly good footprint to boot, I'd have to say that if I'm forced to encode to 64MBit/s, I'd absolutely choose Ahead HE AAC, if I'm judging solely on this comparison (which I am at this point in time...)

Why? Because there was no sample that Ahead HE AAC did POORLY at. MP3Pro and Vorbis (and all the other codecs) each had one or two samples that they just totally choked on, quality-wise. So if I was forced to use a 64 MBit/s codec, it would absolutely be Ahead HE AAC, because while it didn't score highest on every test, and the three codec were virtually tied across the whole competition, I would feel far safer trusting my best digital work to a codec that, according to this test, would have the least chance of representing it particularly poorly.

I wonder how these results compare to higher encoding rates; I could easily imagine that most codecs have a sweet spot, where the encoding quality/bitrate maximizes... it would be interesting to do some research to find this sweet spot.

Anyone want a quick way to slashdot a server? :D

Re:MP3 is the standard.

[StaticEngine]

The Game Industry has embraced OGG, although somewhat silently. With slim budgets, we're always looking for the cheap (and free) solution, and OGG is perfect when we want compressed audio at a good quality.

The sole deciding factor in whether or not compressed audio really gets used in a game is available minspec bandwidth. If marketing is forcing us to target a 500MHz machine, and decompressing OGG audio kills our framerate, then audio compression goes. It the sad truth that the tech heads do not call the shots in this department.

SBR

[zurab]

It's worth pointing out that at least MP3Pro and HE-AAC from tested codecs use SBR. SBR is a method (mostly post-process) that allows transmission of lower half of audio spectrum, and have the decoder "guess" what the the other part of the spectrum would have been. While this allows for "cool-sounding" audio at low bitrates, the generated part of the spectrum is not actually an encoded original audio, but rather its "guessed" reconstruction. SBR is also patented.

Search for more info on SBR if interested, like this one.

Re:CD

[Anonymous Coward]

Landline telephones are approx 8KHz, and can be handled with very, VERY low bitrates acceptably... way below 64kbps.

It ain't CD quality, but it's perfectly clear and understandable.

Re:CD

[harlows_monkeys]

And seriously, does anyone listen to music encoded at 64 kbps? 128 is the bare minumum

If you mean "does anyone rip things at 64 kpbs?", then I'd guess mostly not. However, if you really mean what you asked, then plenty of people do.

Take a look at live365.com. A huge number of the streaming stations there are at 64 kbps or less.

I listen to filk radio via live365 a lot, for example, and it is below 64kbps.

64k and below can work fine for listenting to music. However, many people listen to the encoding, not the music, and for them it might be too painful.

BTW, I've noticed that if I listen to filk.com on my Linux box, my ears get worn out fairly quickly. On my Mac at work, however, it sounds a lot better. Same stream. I think Apple's doing some filtering or something to try to make low bitrate streams sound better.

Re:question

[merlin_jim]

lossy compression includes lossless compression... basically, you throw out information and then use a lossless compression algorithm to do the actuall compression.

And you can't cross-compress data. Remember, according to information theory, a particular piece of data has a certain amount of information in it that cannot be conveyed in less than a certain number of bits. All lossy compression does is get rid of some of that information before compressing.

But you can't take two different compression algorithms and cross compress and expect the final result to be significantly smaller...

It always pisses me off when someone zips of a GIF or JPG or MP3 or something and sends it to me. I will say that the compression algorithms used in these formats (especially GIF) is far from ideal, so you get SOME utility out of cross compressing them (you can inch towards the theoretical maximum compression of the original data that way) but it really just isn't worth it...

rjamorim conducted the test, not HA

[Compact Dick]

While Hydrogen Audio did provide the resources to host this test, the real work was done by Roberto Amorim, who organised this monster.

Credit where credit is due.

Re:rjamorim conducted the test, not HA

[tangent3]

Did you forget to credit ff123, without whom none of the tests would have been possible?

Re:An interesting intepretation

[ff123]

Also, don't be deceived by the "confidence intervals" shown in the graph. They're all drawn to the same widths for each set! At best, this is an approximation. At worst, the author is simply using a program that draws in some uniform (and meaningless) bars. Fear graphs.

The bars are not meaningless. The exact meaning of the bars is described in the results writeup. I suggest you read that writeup.

The exact procedure used to compare ratings is a blocked ANOVA, with a protected Fisher's Least Significant Difference to separate the means if the ANOVA says there is a significant difference somewhere. The Fisher's LSD yields a constant confidence interval for every mean. To get non-constant intervals, one would have to do something a lot more complicated (such as resampling). But then a graph couldn't tell the whole story (you'd need to be able to compare confidence intervals of one sample against every other sample), and we'd be stuck with a dreary matrix.

ff123