AAC Put To The Test

technology is sexy writes "Following the increasing popularity of AAC in online music stores and the growing amount of implementations in software and hardware, the format is now being put to the test. How well does Apple's implementation fare against Ahead Nero, Sorenson or the Open Source FAAC at the popular bitrate of 128kbps? Find out for yourself and help by submitting the results. You can find instructions on how to participate here. The best AAC codec gets to face MP3, MP3Pro, Vorbis, MusePack and WMA in the next test. Previous test results at 64kbps can be found here."

Comments

i prefer just to steal the music

[Anonymous Coward]

this makes the format rather irrelevant.

Re:i prefer just to steal the music

[PhoenixK7]

There's nothing inherent in AAC that makes it DRM friendly. Its the quicktime wrapper thats DRM friendly. It would not be difficult to slap some form of DRM on MP3s, you just wouldn't get that many users because none of the MP3 players would support it out-of-the-box. People want to be able to fire up their winamp (or *shudder* realplayer, wmp, etc) and play the files without hassal. SInce iTunes is the most popular mac audio player, its much easier to add DRM to a format that hasn't been used much. Players will implement fairplay and authorize tracks. Apple could have done the same damn thing with MP3s, called them something other tham mp3s and it would have worked perfectly fine.

AAC _is_ technically superior to MP3. The problem is we've had around 10 years now to refine and perfect our MP3 encoders while free/cheap AAC encoders are just coming onto the market. Give it time, once it reaches its prime it will provide quality that I'm sure will undeniably rival MP3.

Re:WTFDAACM ?

[Urthpaw]

AAC@Everything2

crap in, crap out

[larry bagina]

just remember, every codec depends on the quality of what it is encoding. I haven't heard any AAC encoded music myself (i use uncompressed wav or 256khz mp3 myself), but Apple allegedly uses the master recordings to encode their files.

Most mp3s or oggs you find out there are from someone's CD-Rom drive, who knows how the disc looked, or how much jitter there was. I have heard stories of people downloading songs to find a skip or two in the middle, or been an amalgam of two different files accidently spliced together.

I'd hazard a guess that most people that encode with ogg-vorbis do a better ripping and encoding job, though.

Re:crap in, crap out

[jpt.d]

256khz mp3? That is amazing, I only ever use 44khz

Re:crap in, crap out

[Shenkerian]

Granted it was probably mostly marketing bluster, but Steve Jobs did claim that Apple is encoding the original master recordings when they're available.

Re:crap in, crap out

[larry bagina]

Cdparanois uses the term "frame jitter" for block skewing. Out of respect for them, i use their terminology.

This is what the cdparanoia faq has to say about ripping...

I can play audio CDs perfectly; why is reading the CD into a file so difficult and prone to errors? It's just the same thing.

Unfortunately, it isn't that easy. The audio CD is not a random access format. It can only be played from some starting point in sequence until it is done, like a vinyl LP. Unlike a data CD, there are no synchronization or positioning headers in the audio data (a CD, audio or data, uses 2352 byte sectors. In a data CD, 304 bytes of each sector is used for header, sync and error correction. An audio CD uses all 2352 bytes for data). The audio CD *does* have a continuous fragmented subchannel, but this is only good for seeking +/-1 second (or 75 sectors or ~176kB) of the desired area, as per the SCSI spec.

When the CD is being played as audio, it is not only moving at 1x, the drive is keeping the media data rate (the spin speed) exactly locked to playback speed. Pick up a portable CD player while it's playing and rotate it 90 degrees. Chances are it will skip; you disturbed this delicate balance. In addition, a player is never distracted from what it's doing... it has nothing else taking up its time. Now add a non-realtime, (relatively) high-latency, multitasking kernel into the mess; it's like picking up the player and constantly shaking it.

CDROM drives generally assume that any sort of DAE will be linear and throw a readahead buffer at the task. However, the OS is reading the data as broken up, seperated read requests. The drive is doing readahead buffering and attempting to store additional data as it comes in off media while it waits for the OS to get around to reading previous blocks. Seeing as how, at 36x, data is coming in at 6.2MB/second, and each read is only 13 sectors or ~30k (due to DMA restrictions), one has to get off 208 read requests a second, minimum without any interruption, to avoid skipping. A single swap to disc or flush of filesystem cache by the OS will generally result in loss of streaming, assuming the drive is working flawlessly. Oh, and virtually no PC on earth has that kind of I/O throughput; a Sun Enterprise server might, but a PC does not. Most don't come within a factor of five, assuming perfect realtime behavior.

To keep piling on the difficulties, faster drives are often prone to vibration and alignment problems; some are total fiascos. They lose streaming *constantly* even without being interrupted. Philips determined 15 years ago that the CD could only be spun up to 50-60x until the physical CD (made of polycarbonate) would deform from centripetal force badly enough to become unreadable. Today's players are pushing physics to the limit. Few do so terribly reliably.

Note that CD 'playback speed' is an excellent example of advertisers making numbers lie for them. A 36x cdrom is generally not spinning at 36x a normal drive's speed. As a 1x drive is adjusting velocity depending on the access's distance from the hub, a 36x drive is probably using a constant angular velocity across the whole surface such that it gets 36x max at the edge. Thus it's actually spinning slower, assuming the '36x' isn't a complete lie, as it is on some drives.

Because audio discs have no headers in the data to assist in picking up where things got lost, most drives will just guess.

This doesn't even *begin* to get into stupid firmware bugs. Even Plextors have occasionally had DAE bugs (although in every case, Plextor has fixed the bug *and* replaced/repaired drives for free). Cheaper drives are often complete basket cases.

Rant Update (for those in the know):

Several folks, through personal mail and on Usenet, have pointed out that audio discs do place absolute positioning information for (at least) nine out of every ten sectors into the Q subchannel, and that my original stateme

Re:crap in, crap out

[Guppy06]

"I'd hazard a guess that most people that encode with ogg-vorbis do a better ripping and encoding job, though."

Only because right now you'd have to know a thing or two about the intricacies of digital music to have ever heard the phrase "ogg vorbis." If a big on-line music player were to standardize on this format instead of MP3 and it too becomes the common man's format, you can be sure the quality of ogg files will go down just as well.

Re:crap in, crap out

[Anonymous Coward]

Master recording? They'll use the CD like everyone else.

No. Apple doesn't actually make the compressed recordings they sell on ITMS. The record labels are responsible for doing that themselves. And the labels have access to the original master recordings. Some labels have chosen in some cases to go back to the masters when making their AAC's, though it's not widely known which labels made that choice or which songs were encoded that way.

Re:crap in, crap out

[73939133]

I haven't heard any AAC encoded music myself (i use uncompressed wav or 256khz mp3 myself), but Apple allegedly uses the master recordings to encode their files.

Unless you are psychic, you won't be able to tell the difference between an MP3 ripped from a "master recording" (whatever that may be) and an MP3 ripped from a CD. And unless you are an alien, a dog, or an infant, you are lucky to hear anything meaninful above 16khz, which means that 44khz sampling is plenty.

Most mp3s or oggs you find out there are from someone's CD-Rom drive, who knows how the disc looked, or how much jitter there was.

Yeah, those MP3s are even worse when people forget to clean and replace the stylus in their CD-ROM drive regularly. Those CD-ROM diamond styluses sure wear down fast. I expect you clean yours regularly, right?

Re:crap in, crap out

[nattt]

The fact is, that CD ripping works fine, and has done for many years. You can rip the same bit twice if you want and compare bit for bit to see that each rip is the same. CD Paranoia is just the word for it. For all their technical voodoo, they sound like they're talking bollacks. CD ripping is a perfectly mature technology that works well.

Re:crap in, crap out

[evilviper]

I have heard stories of people downloading songs to find a skip or two in the middle, or been an amalgam of two different files accidently spliced together.

These artifacts are almost always the result of early P2P networks, that would download a missing piece of a file from anything that had the same name... Of course you can find these now, because even with the improvements, people often keep their old files.

I will admit that some pops were the results of bad audio rippers, but that was in the infancy of CD ripping. These days, the most basic software can produce a perfect rip 100% of the time, and has great default settings for encoding. The only thing a luser could do to screw things up is to change the quality settings (which few bother with) or screw things up if they transfer the file.

Face it, the problems with audio quality was the software, and it's simply that the old files were still around in force until recently. Now, even the biggest idiots can encode perfect quality files, and can transfer them over P2P networks properly.

Re:crap in, crap out

[nattt]

Strange - ever since I got a CD-ROM drive for my Mac IICX, I've never had a problem getting decent audio files off a CD. Perhaps the problems are a PC phenonema?

Re:crap in, crap out

[raxx7]

Your comment, though correct, has absolutely nothing to do with the phenomena know as jitter related to CD Digital Audio Extraction.

Put simple, the problem lies in the fact that you can't accurately seek any bit on a CD-ROM (or HardDisk,etc for that matter). You need to add extra bits to the media so the drive can know where its at. Thats what syncronization headers are for.
In a CD data track, there are syncronization headers for every 1024 bytes block. But in Audio tracks, there isn't such thing. So, you can only accurately seek the begining of the track.
Therefore, the DAE process must be continous. If, by some reason (e.g., the host system can't sustain the I/O throughput), its interrupted it won't be possible to accurately resume it from where it was interrupted.
Innacurately resuming it leads to errors, known as _jitter_. The alternative is to restart the extraction from the begining of the track.

That said, let me point out: this phenomena isn't always present. It only shows up if there are problems: dirty/scrateched CDs, bad CD-ROMs or systems too busy to sustain I/O, etc.
So, with little effort its possible to achieve accurate extractions.

Re:crap in, crap out

[cens0r]

apparently you haven't taken any signals classes or DSP. :) There is a nice little rule, called the nyquist theroy that says if you sample at a frequency twice the highest frequency you need to capture, you will not loose any data. Since humans hear at about 20Hz to 20kHz, you would need to sample at 40kHz to capture everything. When coming up with the CD standard they put in a little extra headroom and used 44.1kHz which would capture all audio up to 22.05kHz.

An honest question - who cares?

[LeoDV]

This might get modded Troll, but it's an honest question. Whenever I rip a CD, I encode it into mathematically loseless MP3s, and with the cheapness of disk space these days, I can't stop being amazed at how many people don't do the same. If the quality can be compressed into something loseless from the original digital medium (the CD), then who cares if AAC sounds better than OGG sounds better than WMA sounds better than MP3 at 64 kb/s?

Please enlighten me, I'm actually, honestly, curious.

Re:An honest question - who cares?

[markv242]

"...I encode it into mathematically loseless [sic] MP3s..."

Not possible. MP3 by its very nature is a lossy encoding scheme, hence there will always be artifacts when you pass the audio through the encoder. You may not be able to hear the quality change (even after passing the files over and over and over through the encoder) but you will be generating noise.

As far as your original question, it all comes down to file portability. It takes people a bit longer to send a 65 meg wav to their friends, compared to a 6.5 meg mp3.

Re:An honest question - who cares?

[afidel]

Well it matters because some people are buying shitty 128k aac file from Apple (a LOT aparantly, they have sold almost 4 million tracks already). I too rip to high quality mp3 (~220kbps VBR mp3's from LAME which passed tripple blind with wav and ogg) but I guess it's usefull to know what quality you can expect from this service since Apple will be coming out with iTunes for windows later this year and bringing their online service to the masses.

Re:An honest question - who cares?

[Monkelectric]

You are grossly misinformed. MP3 and most other audio compression formats perform FFT's and throw away coefficents of the FFT that are least noticeable (thats a gross simplification).

There *are* lossless codecs like FLAC and SHN, but they generally achieve between 10 - 30% compression.

Re:An honest question - who cares?

[beans-n-rice]

Um, have you ever done raw video work?

At 190MB per second of 1920x1080 24fps (1080p HDTV standard) 16-bit YUV 4:2:2 video, even if you have a TB (~1024 GB), saving just the LoTR-FoTR (178 minutes) would require ~1.9 TB. And that's JUST the video...audio not included. Now granted, perhaps you didn't mean uncompressed at mastering quality, but 1080p is an eventuality and appears to be THE emerging mastering standard for film.

You'd need several terabytes to store more than a few movies at production quality raw...but why in the hell would you want to?

Re:An honest question - who cares?

[ahhhmytoes]

There *are* lossless codecs like FLAC and SHN, but they generally achieve between 10 - 30% compression.

Actually, the compression ratio for SHN is much better. As much as 74% compression can be achieved on techno and pop. I would call 55% typical for live shows from etree.org.

FLAC has similiar compression rates. FLAC's strengths lie in its ability to compress 24bit audio and built-in checksums.

Re:An honest question - who cares?

[gerardrj]

What is "triple blind?"

A blind test is where the test subjects don't know what specifically they are sampling. The researcher prepares the samples and knows what is going on.

Double-blind is where neither the researcher nor the test subjects know specifically what is being tested. The samples are prepared by a dis-interested third party and given to the researcher and test subjects without any identification. This eliminates researcher induced errors/data fudging.

There are no other parties to such tests, so I really am confused. Are you just making stuff up to lend credence to your arguments?

Re:An honest question - who cares?

[Alric]

I'm not sure what the original post meant, but from my days working in the psych labs, "triple blind" is a colloquial term for when the two parties are not aware of the test or the true nature of the test.

For example, I am the researcher, and you are the subject. I am giving you the Pepsi challenge. I do not know which container has Pepsi, and which one was Coke. I administer the test. However, Xavier, the research director has been slowly increasing the temperature in the room to observe if this affects your and my interpersonal communication.

1. You are ignorant of the test data.
2. I am ignorant of the test dat.
3. We are both ignorant of the true test.

aahh... AAC sux, anyway...

[Corporate Drone]

... especially if they allow Miami, BC, and Syracuse in...

DVDA

[Graspee_Leemoor]

I prefer DVDA.

What hyphen?

graspee

Re:Isn't AAC used for its DRM features?

[Anonymous Coward]

You mean the DRM features that allow me to rip my own CD's to AAC and copy the resultant files to any and all computers or players (that understand them) and play them back?

Or how about the DRM feature that allows me to export bought AAC's to aiff and then convert them to MP3/OGG/AAC/.wav/.au etc and do with them what I please?

True, Apple's TMS is selling AAC's that have a DRM-like "inconvenience protection" on them but it's not _inherent_ to the AAC format, nor does it affect the sound quality vs. file size questions.

(In any case, we _should_ be cheering for any company that's actually trying to give us quite reasonably limited freedom with copyrighted material, while satisfying the RIAA/MPAA etc.)

patent and the possibility of DRM

[SHEENmaster]

Patent encomberment is a serious deal. It means than a legal OSS player is nearly out of the questions. If I can't play the things on my iBook(Linux), iMac(Linux), server(Linux), palmtop(Linux), and at school (OS X) then I won't be using it. Quality is irrelevant at that point.

Ogg Vorbis, because of its openness and mpeg, becase people ignore the patent, are my best two options. AAC is not an option, so its quality means nothing.

Would you rather use a train that can safely travel at 100mph along prelaid tracks that don't follow your route or a car that can safely go 60mph along much more convenient roads?

(Oh yeah, Linux is a rocket car in the analogy because it has to be stuck in there somewhere. Windows is a horse in that it can go anywhere if at a crawling pace while shitting over everything, but a rocket car can go more places...

Re:patent and the possibility of DRM

[Anonymous Coward]

Patent encomberment is a serious deal. It means than a legal OSS player is nearly out of the questions.

No. Anybody who wants to can get a license, get the reference code, and write an open source player. (Or encoder, even.) There is no barrier here except cost.

Of course, in order for somebody to do that, to pay for a license I mean, they'd have to literally put their money where their open source mouth is. If it's sufficiently important, this shouldn't be a problem.

Why doesn't some enterprising individual buy a license, write an open source player, and then sell it (source and binary) to Linux users? Okay, maybe that's not a great idea, because my gut tells me that a person who did that would make enough to buy a pizza, but that's about it.

Why doesn't somebody start an open source player kibbutz and take donations? If everbody who wants a Linux player were to send in $10, the costs would be covered easily.

my iBook(Linux), iMac(Linux), server(Linux), palmtop(Linux)

You're running the wrong OS on three out of four of these things. Palmtops should run PalmOS. iBooks and iMacs should run OS X. Linux is not a good solution for any of those things.

Then again, Ogg is not a good solution for compressed audio, either, so maybe I'm seeing a pattern here.

Would you rather use a train that can safely travel at 100mph along prelaid tracks that don't follow your route or a car that can safely go 60mph along much more convenient roads?

You're missing the point of the car analogy. Ogg is a car that doesn't go where you want it to go. There's no Ogg support in QuickTime. There's no Ogg support on the iPod. It simply can't go places that people want to go.

Re:patent and the possibility of DRM

[PhoenixK7]

"You're missing the point of the car analogy. Ogg is a car that doesn't go where you want it to go. There's no Ogg support in QuickTime. There's no Ogg support on the iPod. It simply can't go places that people want to go."

Indeed. You've got a car, it can drive in any direction, but all of the terrain is impassable using said car. You either have to pave your own roads or wait until someone else does it because they've got enough spare time and want the road badly enough.

The paving materials and equipment are freely available, but someone has to invest the time to lay the infrastructure.

On the other hand you can pay some cash, get some other vehicle and use the roads/rails/whatever they've built using revenue from selling whatever vehicles they happen to be.

Frankly, I've got a mac. I've got an iPod. I already made the hardware investment (and software, but thats really just a sunk cost since the computer came with Jaguar, and the iPod came with its own little OS). Why not use it? AAC on high quality encoding at 128 kbits sounds pretty damn good to me. But then again, there's no reason you should take my word for it, as I don't have any "audiophile" equipment, and I've had a very mild case of tinnitus in my right ear (dammit!). It lets me store lots of music, and it sounds good to me. To boot, some other people seem to think it sounds pretty damn good to (and an equal if not greater number that dislike it either because it sounds bad to them or they've got some sort of political agenda that clashes with apple/dolby/patents/whatever).

MPEG2-AAC not too bad

[kungfoolouie]

I happen to own a Panasonic SV-AV30 (4-in-1), and music management app that comes with it has to options to save in MP3 or MPEG2-AAC. Lately, I have been transcoding the audio from mp3s and cd to 96kbps AAC and the results are surprisingly good. I play it in my car and have not really noticed a much of a difference.

Obviously, the tracks which were bad to begin with will be bad as AACs.

BTW, I have been playing/making music for 14 years and have a pretty good ear when it comes to tone and timbre. Hi-hats on CDs have always bothered me with the lack of warmth and fullness of timbre. So take my word for it if I saw it's not too bad :-)

Don't take 64 Kbps AAC results seriously

[benwaggoner]

I note that in the 64 Kbps test, they used the AAC-LC encoder from QuickTime 6.0. This was a pretty darn lousy one, lacking any ability to specify a sample rate at a given data rate, and had poor quality. The current version of QuickTime 6.3 (for Windows and MacOS X), has a much improved, more flexible AAC-LC encoder, so if they did that test today AAC would likely rank higher.

If using the Apple encoder, encode in "Better" mode with 16-bit source, and in "Best" mode with source that's more than 16-bits per sample (and hence isn't a CD rip). Support for mastering from 24-bit when running in "Best" is one of the reasons why the AAC-LC files as part of iTunes sound so good.

Pfffft...

[Squirrel of Doom]

In the Billy and the Boingers' "U Stink But I Luv U" encoding test, the OOP-AAC compression scheme won by a wide margin.

Sigh. Not a way to get good results

[Anonymous Coward]

Please note that the survey's host makes no claims that what he's doing is in any way scientific. Keep that in mind. The reasons why the results are to be taken with a grain of salt:

1. There is no guarantee of clean data - the users are expected to generate their own files. MIstakes happen.
2. The type of user who participates in this (and more likely in the OGG vs AAC coming debate) may have some predisposed bias. There is no way to weed out any placebo effects.
3. There is no way to weed out folks who have tin ears. I don't want some idiot who loves dance forming an opinion about Bach not sounding "boomy" enough

This may fly in the face of the /. crowd, but an open call to the masses to submit their opinions is not science nor does it have any scientific meaning.

And that wasn't a dis

[benwaggoner]

Also, I didn't mean that to be a criticism of the original test. 6.0 was the current version of QuickTime when they did the test, so it looks like a fair test for the state of the technologies at the time.

Re:I prefer analog

[rsidd]

Just remember that digital=loss

Well, any analog medium = much worse loss. LPs and cassette tapes can't approach the dynamic range of a CD. Plus you get noise, which gets worse on repeated playback.

The only lossless music is a live performance. But even then, you may crib about acoustics. Besides, you can't hire Brendel to play live for you whenever you feel like, and even if you could, he may not be in good form every day.

Not a good test of iTunes service

[benwaggoner]

It looks like they're working with 16-bit source, not the 24-bit source that most of the iTunes AAC files are ripped from. So this test, while certainly very interesting, won't be useful to determine the iTunes music store quality.

Re:I prefer analog

[afidel]

What you prefer the loss of the pits on your LP wearing as the needle passes through the grooves? Yeah spectral analysis shows that after three plays with a high end player the LP has already lost MORE dynamic range then a ADAT recording, and of course in hundreds of plays and a couple generations the digital copy is obviously superior, plus getting vinyl from mail order sucks, I know I DJ and the % of DOA stuff is way too high for my liking (especially if it's a white label or rare import, ususally means I get the insurance $ but never find the music again for a reasonable price). Analog has its place (like scratching, cd's just are not the same) but long term quality is not where it's at.

Heise did a public test about them one years ago

[Psychic Burrito]

Read more about the test here (german link).

With 6000 participants, the double-blind public test results were:

1. Ogg
2. WMA
3. RealAudio
4. Mp3Pro
5. MP3
6. AAC (Sic!)

Of course, this was crazy, with AAC even behind MP3, but these really were the results...

Ratings are nice, but...

[MP3Chuck]

Why not show spectrum analasys of different songs encoded into the given formats too?

Perhaps I'm just an audio freak, but I would find that a lot more interesting than just ratings.

Re:Sigh. When will people RTFA and get a clue?

[Anonymous Coward]

First, read this:

http://www.ff123.net/abchr/abchr.html

This describes the program and testing methodology used here, which, btw, is based on widely accepted perceptual testing conventions. And yes, by the scientific community. These are the same techniques used by the scientists that do the research and development on these formats. Please note the references at the bottom of the page.

1. Wrong, the MP4 files are already encoded and created for the user, stored in the .zip files.
2. Wrong, the Hidden Reference (ABC/*HR*, please read the page at the first link), ensures that if the user honestly cannot tell the difference but thinks that one exists (placebo), and rates the original lower than one of the encoded versions, that their results are discarded.
3. This is where the statistics come in. With enough listeners, the "noise" gets weeded out of relevant results. Most past tests using this methodology have been shown to provide highly relevant and fairly uniform results when all the data is factored together.

An open call to the masses is the only way to measure the perception of the masses, and if the test is performed properly (which it is in this case), then it *is* scientific.

Next time, please read up a little more on what is happening before jumping to all sorts of incorrect conclusions.

Some more 64 and 128 Kbit/s AAC listening results

[florin]

As was previously mentioned on Slashdot, a highly regarded German magazine called C'T dedicated an article to a similar comparison of various audio compression codecs last year.

They created fourteen different .WAV recordings containing 3 short excerpts from various CD music tracks (pop, classical and jazz) that had previously been encoded by 6 popular codecs, each at both 64 Kbit/s and 128 Kbit/s (or as close as possible for VBR-only encoders). For verification of the results, 2 of the recordings came directly from CD and had not gone through any encoding process. Because the .WAV files were all the same size, there was no way for the listener to know which encoder had been used on a particular file. Participants were asked to rank their preferences among these files. The encoders included MP3, MP3PRO, Ogg Vorbis, WMA, RealAudio and AAC.

Over 6000 people downloaded those tracks and submitted their preferences. Unfortunately, the results of that test were only published in print and I haven't been able to find an online version of it. A few noteworthy results are below however.

The percentages indicate how many people put a particular codec at a particular ranking:
MP3 64 KBit/s
1st place: 1 %
2: 1%
3: 1%
4: 1%
5: 2%
6: 4%
7th place: 90%

As might be expected for the oldest codec, almost everyone agreed that the file that had been run through MP3 at 64 Kbit was the worst sounding of all. At 128 KBit however, listeners were clearly divided on whether MP3 sounded worse or better than others:

MP3 128 Kbit/s
1: 11%
2: 14%
3: 15%
4: 15%
5: 16%
6: 16%
7: 14%

Now the AAC results. At 64 Kbit, it was ranked a slightly below average performer:
AAC 64 KBit/s
1: 7%
2: 12%
3: 17%
4: 26%
5: 22%
6: 14%
7: 2%

What's interesting is that at 128 Kbit/s, more people ranked AAC the worst sounding encoder than any other codec in the test including MP3!
AAC 128 KBit/s
1: 11%
2: 11%
3: 13%
4: 12%
5: 14%
6: 14%
7: 26%

Not surprisingly, the files that had been read directly from CD without any encoding steps done in between got the best rankings of all. Ogg Vorbis did very well indeed and came in second overall.

Sounds Better != High Fidelity

[Josuah]

This experiment is really designed to test which codec overall sounds better to the average user, for an arbitrary and inconsistent range of hardware setups, acoustic environments, and listening preferences (e.g. do I pay more attention to the primary beat or to the background harmony). I wouldn't place any value on this test other than to choose which codec I might choose if I wanted to please the ignorant consumer (a valid market, of course!). It does nothing to address how accurately a codec reproduces the artist's original sound.

I'll put a lot more stock in the Report on the MPEG-2 AAC Stereo Verification Tests put together by David Meares (BBC), Kaoru Watanabe (NHK), Eric Scheirer (MIT Media Labs) for the ISO. And the other MPEG Audio Public Documents.

How can this possibly be accurate?

[NeoOokami]

You're basically asking for a lot of people to submit their opinions. This will show you what the people who participated in it prefer, but it doesn't really reveal much in they way of actual sound quality. Everyone has their own opinions already about which audio codec is supperior. The only way you could rule out the placebo affect is to give the test blind, so that they have no clue which file is which. Even then since the results are being turned in on good faith, you have to accept that some people may simply lie about the results based on their own biases. You'd need an objectional third party to administer a test like this, and even then almost no one would agree on a third party in the end. If someone's favorite format lost they'd just bitch about the test being rigged. The only un arguable test would to actully compare the integrity of the audio to the original via an olliscope or some other device. Audio's not my area of expertise so I could be wrong there. It seems to me it's best to just not worry about it and use what you're happy with. Seeing a test like this wouldn't change my mind really. "Person A liked Audio B encoded with mp3 the best!" It just doesn't seem to hold that much sway over me.

Re:How can this possibly be accurate?

[Anonymous Coward]

It's rather obvious that you haven't bothered to read anything about the test, the program used, etc.

The test *is* blind, and it is based on widely accepted perceptual testing techniques. It uses hidden references (references to the original vs the encoded sample, on a per sample basis in which the user is not aware of which is which, thus if they rate the original as being worse than the encoded version, their result is discarded) as a control. The program devised has been developed by someone who has taken the time to do the proper research, read the appropriate papers and other sources, discuss the idea with developers of many different audio codecs (LAME, Vorbis, PsyTEL AAC, etc). The technique here works, and has been used many times before. It's not simply some amateurish scheme that someone who knew nothing about the appropriate sciences dreamed up simply because he wanted to find out if "Person A liked Audio B".

Re:How can this possibly be accurate?

[90XDoubleSide]

While the first post was way off-base, there are two massive problems with this testing method: first, there is no standardized reproduction equipment. If you wanted to test only the codec, the test would need to be performed with everyone listening on the same (reference-quality) equipment. Secondly, because the test uses a nonrandom sample of people rather than professional listeners, the test measures how good people think the music sounds, rather than which codec actually reproduces the original recording the best; because people will typically not pick faithful reproductions as the best, the test cannot be construed as a measure of which codec works the best, but only of which one produces the sound most desirable to the the nonrandom sample group over imperfect reproduction equipment.

I care.

[crapulent]

I care because I have not fallen for the "golden ears" fallacy. To me, 192kbps ABR lame-encoded sounds exactly like the original. I don't have super expensive speakers attached to the computer, nor do I have a fancy sound card (Creative Live 5.1.) Storing music losslessly is a waste of space to me. Sometimes I like to share music files and it's a heck of a lot easier and others are a heck of a lot more interested in trading compressed music compared to lossless files. And I can put a heck of a lot more of them on a CDR. And should I wish to listem to them in my MP3 player with limited memory, I'm sure as hell not going to use a lossless format.

If YOU want to use up your hard drive space, internet bandwidth, and blank media with huge lossless encoded files, feel free. But don't get all smug and proclaim to not have any idea why anyone would not want to waste their resources.

Oh, and I'm not going to touch that "mathematically lossless" crap, others have covered that already.

Re:I care.

[Admiral Burrito]

I care because I have not fallen for the "golden ears" fallacy. To me, 192kbps ABR lame-encoded sounds exactly like the original. I don't have super expensive speakers attached to the computer, nor do I have a fancy sound card (Creative Live 5.1.) Storing music losslessly is a waste of space to me.

The only problem is, if MP3 or AAC or whatever lossy format fall out of fashion (due to patent or whatever), you could end up with a bunch of files you can't play on the latest gadgets and software. Then you need to re-encode them in whatever the new format, which will add additional lossiness due to the transcoding (unless you re-rip everything, which is a major pain).

If you use a lossless algorithm, you can re-encode to whatever you like. For example, it wouldn't be terribly difficult to take a losslessly-encoded file, decompress it, and run it through a fast MP3 encoder as part of the process of copying the file to your portable. Then if ACC or OGG or whatever displace MP3, you can just change the script when you upgrade your portable.

I agree about the "golden ear" fallacy. Some people avoid lossy because they want "perfect" sound. But there is no "perfect", really... the original recording required sound travelling through air, which is lossy. The recording equipment was also lossy, as is your playback equipment. Your ears and mind are also lossy. Very lossy, in fact... Your ears being lossy is why these lossy audio codecs work as well as they do (they "lose" the parts that your ears/mind would lose anyway).

Re:WTFDAACM ?

[stagmeister]

THANK YOU for linking to e2!

Now for the next hour - or maybe the next few - I will waste my time going to pages such as this.

Once again, thanks!

--Jason

Sorry

[2nd Post!]

That was a flippant answer to his seeming flippant post.

I like Ogg fine. It is my codec of choice, except of course that no one bothers to support it for my OS of choice, OS X.

There's no good Ogg encoders that can interface with iTunes and support Unicode (yet, of course)
There's no Ogg codec for Quicktime on OS X 10.2.6 (yet, of course)

I much prefer Ogg, ideologically, but it's not something I can actually *live* with, because the support isn't there.

I have 100% support for MP3 and AAC.

Yes, I believe in fighting for causes I believe in. Right now Ogg is not one of those causes; maybe later. Right now I'm more concerned with my friends, my mortgage, and my state of unemployment, sorry.

Thank iTunes for the skips etc.

[SuperBanana]

I have heard stories of people downloading songs to find a skip or two in the middle

You can probably thank iTunes for that- I had numerous problems with encoding my CDs. Songs has skips, and more commonly, ended early- often by more than 15-20 seconds. It was extremely irritating.

Curiously, I never had such problems with Xing's AudioCatalyst, an awesome encoder for the Mac(it was, and I think still is, the only encoder for the Mac that can do live encoding from line-in). AudioCatalyst was also exceedingly fast on my powerbook- 4x encoding speed, and the rip of the CD was very, very fast.

If you want perfect rips of the audio to encode from, you don't need masters- you need a CD ripper that doesn't suck, like CDparanoia(although CDparanoia is very slow.)

I use uncompressed wav or 256khz mp3 myself

Assuming you mean 256kbit, that's an absurd waste of disk space- anything over 160 is. In fact, if you look at encodes done by "groups", the most they ever do is 192kbit, and usually only if the material is worth it- ie, it has really good production quality, the music is very nice, etc.

Personally, I wish people would take the disk space to do 160kbit- from most encoders, 128kbit files sound pretty bad on anything better than a $25 set of computer speakers.

Re:Thank iTunes for the skips etc.

[cide1]

I disagree on the 160 vs 256 kbps statement. I listen to mostly rock and punk, so I took a Thursday song, which is kindof in the middle of the two genres, and encoded it at 32, 48, 56, 64, 96, 112, 128, 160, 192, 256 and 320 kbps. I wanted to encode my whole CD selection (350 cds) at a bitrate that I couldn't hear the differance, and a bitrate that I could stream at decently. For streaming, 56 was the magic number. Any less and it sounded like crap, any more, and my DSL line couldn't host 2 streams at once. For music, 192 was good, but I could still hear the mp3 compression. I find that bass tends to get distorted in mp3s, and once I went to 256 this seemed to go away. I did all these tests with an audigy2 under windows XP, using Lame with q=9. Playback was through the Infinity HTS-20 Speaker System.

Re:Thank iTunes for the skips etc.

[lvdrproject]

Er? I'm not an "audiophile" or anything, but as far as i knew VBR is the best quality-for-size ratio you can get with MP3. I don't understand "cracking and whooshing". Can you explain? :/

Just to keep you intellectually honest...

[2nd Post!]

It doesn't sound like you use a Mac or the iTunes Music Store, so why do you say the AACs from Apple are shitty?

There are at least three distinct things to keep in mind:
MP3s encoded from your music using LAME at 220kbps VBR is one quality
AACs encoded with Quicktime 6.3 is one quality
AACs encoded from masters, ala iTunes Music Store, are another quality

You, in one sentence, mix all three quality levels as if they are currently comparable.

The music from the iTunes music store is encoded from a higher quality source, and can arguably be of higher quality than even your 220kbps mp3s. It's hard to make any educated guess because I don't know anyone who's done a comparison between AAC files ripped from masters vs MP3s ripped from CD.

The music you get from iTunes itself is based on Quicktime 6.3, and that *is* being compared and characterized in this test; this will probably illustrate the level of quality iTunes for Windows will have, and is more directly comparable to your 220kbps mp3s, but only *after* the test is performed.

it's fine to believe that your mp3s are better, but there is no proof yet.

Re:Heise did a public test about them one years ag

[mindriot]

The complete results can be found in issue 19/2002 of Heise's offline magazine C't. Along with the online public test, some 'experts' (such as some music producers, hobby listeners, a singer, and a young student and choir singer) were consulted.

In the online public test, the 64 kBit/s comparison yielded

1. Ogg
2. MP3Pro
3. WMA (WMA9 Beta)
4. AAC
5. RealAudio
6. MP3

The parent's results were the ones for 128 kBit/s. The eight experts compared the codecs on 160 kBit/s as well, with much more varying results (not much of a surprise). But on average, the results were

1. Ogg
2. AAC
3. WMA
4. Real
5. MP3
6. MP3Pro (sic)

As I said, those were an average, with the individual results of the eight experts strongly deviating. Ogg was placed once 1st, once 2nd, twice 3rd and 4th, and once 5th and 7th. (One had actually placed the plain wave reference 5th...)

Re-encoding

[cmason]

For what should be fairly obvious reasons, I'd rather see a comparison of encoders re-encoding AAC to MP3. I tried this several weeks ago using AudioHijack and the iTunes MP3 encoder, and the results were less than stellar.

I imagine that an encoder could be optimized for re-encoding. I wonder if anyone is working on this. I'd like to write a program which would automatically do this conversion in my music library, but currently I can't stand the loss of quality.

Re:Re-encoding

[cmason]

So a quick google search yielded iLoveMP3 which is able to re-encode encrypted AAC to MP3 using LAME. If it doesn't sound good using LAME, it probably won't sound good using anything else.

I'll post results when the encoding finishes.

Re:Re-encoding

[afidel]

Transcoding is never recommended as there are fundamental differences in the way that different encoders (even different implementations of the same format) will decide on what data is unneeded and so you will get more and more data thrown away in each step. There is no panacea in this regard so the only solutions are to reencode everything or rip to a lossless format in first place. More and more people I know are doing the latter so that they can encode to whatever codec happens to be popular this year.

Re:Re-encoding

[cmason]

This is obviously true. However, the reason I want to do this is to play iTunes Music Store purchased music on non-Apple hardware. I have no source material to rip from other than the AAC track I download.

I think LAME actually does a pretty good job of re-encoding AAC to MP3. At least, I can't tell the difference (unlike when using iTunes to re-encode, where I most definitely could). This is good enough for me.

-c

Ogg

[Anonymous Coward]

Well, I'm not using AAC until it supports Ogg.

Re:Ogg

[Ziviyr]

Funny?

Theres little stopping anyone from putting AAC in an Ogg stream.

Re:I prefer analog

[Achoi77]

You could always get a laser turntable if you don't want to scratch your record. Expensive, though.

Apple vs. punk rock ethic

[switcha]

for a company who sold one of their first computers for $666 dollars, I'd imagine some of the bands are very disappointed in them using such a good codec.

I mean, damn them! Nirvana didn't pay $606.17 to record Bleach so that some Corporate Asswipe could make a high fidelty copy of it!

The Ramones would be very peeved to find all the work they put into keeping most songs to three, dingy, distorted chords, ripped to a high fidelty format.

160 Kbps MP3 NOT very good!

[benwaggoner]

Over 160? Maybe you've been to too many Megadeth concerts or something, but 160 Kbps is quite audibly lossy in my experience. Now, I'm fussy about encoding artifacts, but 160 is the lowest I'll use for listening to on headphones on an airplane. It has to be at least 192 for me to not find artifacts distracting while listening on a good stereo (Paradigm Active/20 reference monitors attached to my video editing rig, in my case, self-powered with all XLR signal routing from the jukebox machine. I grant this is overkill for the casual listener).

Personally, I encode my library at 320 Kbps Normal Stereo without any filtering. This is overkill for listening, but that's enough data that I can recompress to another, more portable format like AAC on an iPod without windup up with a audible multigeneration artifacts.

All things being equal, I'd use FLAC, but I really really like the iTunes interface, and 320 MP3 is the best format it has historically supported. It now does 320 AAC, and I'm toying with switching to that (although I haven't yet, since the files won't be quite as widely interoperable).

Re:160 Kbps MP3 NOT very good!

[King_TJ]

Yep!! I encoded my entire CD collection at 192 bits, and I don't waste my time listening to anything less, if I can help it. I find it maddening that most people still encode at 128 bits and think it sounds "good enough". It only sounds ok on cheap speakers, or perhaps even moderate priced speakers + some audio processing enabled to "enhance" the sound.

Whenever I listen to 128-bit MP3s through my set of studio monitors, they sound "thin". Even in my car (I have a Rio MP3 car player), with Diamond Audio seperates and a Rockford Fosgate amp, 128-bit MP3s are noticeably poorer quality than ones encoded at higher bitrates.

I would have encoded above 192 bits, even, except I was trying to strike a good balance between optimal sound quality and conserving a little bit of drive space. (After all, if disk space is no object, then it makes no sense to use a compressed audio format to begin with!) I found that 192 bits was about as low as I could encode and still feel like I wasn't losing any significant amount of audio quality. (At worst, you might be able to hear very small details, such as a brush against a cymbal in an otherwise quiet passage, that would have sounded more "life-like" at 256 bit+ encoding than at 192. You'd almost have to listen to them side by side to even tell though.)

not likely to make a difference

[Trepidity]

The quality loss in dithering 24->16 is much less than the quality loss in doing a lossy encode to 128kbps AAC, by at least an order of magnitude.

Law & Order

[xihr]

The African American Congress? Okay, okay, I'll be the first to admit I watch too much Law & Order.

My own test

[withinavoid]

I did my own test of this a while back (AAC,MP3,OGG only). I didn't do 128K CBR but instead did 160K VBR.

My results were:
1. AAC
2. OGG Vorbis
3. MP3

Re:My own test

[evilviper]

Since you said it, it must be true...

I did my own test a while back. My results were:

1. Phonograph
2. DVD
3. Reel-to-reel
4. CD
5. SACD

Yes, I'm joking, but mod me up too... I post at +2, and have a good list of fans (unlike the parent), so I must be right...

Re:Fine print......

[salimma]

Gotta love the advertising line though, almost as bad as selling computers using P2P and music sharing as a 'feature'

Heh, here in UK I've been cringing everytime I saw a BT Internet advertisement, with people supposed to be users touting its use for downloading music.

In the worst ad, the girl even said she used to buy CDs but now she just downloaded them. Granted, there are ways to do it legally (hello Apple Store) but in UK and on a PC?

AAC 5.1 surround sound work great in matroska

[ChristianHJW]

You question the use of AAC audio ? why bother, if we have MP3 for music ? Well, as a quality freak i will only use musepack ( MPC ) for audio compression, if ever, but for DVD backups with the DivX or XviD codec, i am using AAC as standard now. Why ? well, the new Nero encoder will allow you to create high quality, true 5.1 surround AAC audio with a bitrate of about 180 - 240 kbps, from 5.1 AIFF or WAV files. Use the matroska container and DirectShow parser filter ( http://www.matroska.org ) to store your DivX video with AAC 5.1 and the CoreAAC DirectShow decoder filter to playback on Windows. A sample file ( 10 MB ) in incredible video quality, plus some documentation how to play and make such files, can be found here : http://corecodec.com/modules.php?op=modload&name=P NphpBB2&file=viewtopic&t=328

Re:Oh Brother...

[2nd Post!]

The only reason Apple is using AAC is because it is an obscure format which tends to lock users into a Mac/iPod solution.

No, the only reason Apple is using AAC is because it's an open, documented, non vendor-locked format that cannot be simply hijacked and manipulated by, say, Microsoft.

AAC is cross platform; in fact, AAC is the logical successor to MP3, so everything you love or hate about MP3, ideologically, should apply to AAC. To think otherwise seems silly and ignorant to me.

Re:Many flavors of bad

[benwaggoner]

You don't listen to a lot of classical, do you?

Bang a cymbal, and let it fade out into nothingness. You can definitely hit audible limits of 16-bit PCM in that case. PCM->FFT->PCM will make it worse.

Also, codec like Dolby Digital are capable of decoding in more than 16-bit, so with capable equipment, you're really able to take advantage of available dynamic range.

Patent license terms prohibit use in OSS

[yerricde]

Why doesn't some enterprising individual buy a license, write an open source player, and then sell it (source and binary) to Linux users?

The typical license for LZW data compression patents (the foreign counterparts to U.S. Patent 4,558,302 owned by Unisys, which expires in just over a week) do not allow redistribution of the encoder's source code and binaries. I'd guess that the typical licenses for software implementations of audio codec patents have similar terms; otherwise, somebody would probably have already donated an MP3 patent license to the LAME project.

Palmtops should run PalmOS.

That's like saying "Desktop computers should run BeOS." Palm OS is not the only PDA platform. For instance, Sharp Zaurus handheld computers do not ship with Palm OS; instead, they ship with a Linux OS.

iBooks and iMacs should run OS X.

What if the fastest available GUI for Linux runs more responsively on Linux than Quartz runs on Mac OS X on a given piece of Mac hardware?

There's no Ogg support in QuickTime.

I beg to differ, unless you're talking only about those QuickTime components shipped by Apple Computer.