AAC Put To The Test
technology is sexy writes "Following the increasing popularity of AAC in online music stores and the growing amount of implementations in software and hardware, the format is now being put to the test. How well does Apple's implementation fare against Ahead Nero, Sorenson or the Open Source FAAC at the popular bitrate of 128kbps? Find out for yourself and help by submitting the
results. You can find instructions on how to participate here. The best AAC codec gets to face MP3, MP3Pro, Vorbis, MusePack and WMA in the next test. Previous test results at 64kbps can be found here."
this makes the format rather irrelevant.
Most mp3s or oggs you find out there are from someone's CD-Rom drive, who knows how the disc looked, or how much jitter there was. I have heard stories of people downloading songs to find a skip or two in the middle, or been an amalgam of two different files accidently spliced together.
I'd hazard a guess that most people that encode with ogg-vorbis do a better ripping and encoding job, though.
Do you even lift?
These aren't the 'roids you're looking for.
This might get modded Troll, but it's an honest question. Whenever I rip a CD, I encode it into mathematically loseless MP3s, and with the cheapness of disk space these days, I can't stop being amazed at how many people don't do the same. If the quality can be compressed into something loseless from the original digital medium (the CD), then who cares if AAC sounds better than OGG sounds better than WMA sounds better than MP3 at 64 kb/s?
Please enlighten me, I'm actually, honestly, curious.
mmm... yeah... You see, we're putting the cover sheets on all TPS reports now before they go out...
I prefer DVDA.
What hyphen?
graspee
You mean the DRM features that allow me to rip my own CD's to AAC and copy the resultant files to any and all computers or players (that understand them) and play them back?
Or how about the DRM feature that allows me to export bought AAC's to aiff and then convert them to MP3/OGG/AAC/.wav/.au etc and do with them what I please?
True, Apple's TMS is selling AAC's that have a DRM-like "inconvenience protection" on them but it's not _inherent_ to the AAC format, nor does it affect the sound quality vs. file size questions.
(In any case, we _should_ be cheering for any company that's actually trying to give us quite reasonably limited freedom with copyrighted material, while satisfying the RIAA/MPAA etc.)
Patent encomberment is a serious deal. It means than a legal OSS player is nearly out of the questions. If I can't play the things on my iBook(Linux), iMac(Linux), server(Linux), palmtop(Linux), and at school (OS X) then I won't be using it. Quality is irrelevant at that point.
Ogg Vorbis, because of its openness and mpeg, becase people ignore the patent, are my best two options. AAC is not an option, so its quality means nothing.
Would you rather use a train that can safely travel at 100mph along prelaid tracks that don't follow your route or a car that can safely go 60mph along much more convenient roads?
(Oh yeah, Linux is a rocket car in the analogy because it has to be stuck in there somewhere. Windows is a horse in that it can go anywhere if at a crawling pace while shitting over everything, but a rocket car can go more places...
You can't judge a book by the way it wears its hair.
I note that in the 64 Kbps test, they used the AAC-LC encoder from QuickTime 6.0. This was a pretty darn lousy one, lacking any ability to specify a sample rate at a given data rate, and had poor quality. The current version of QuickTime 6.3 (for Windows and MacOS X), has a much improved, more flexible AAC-LC encoder, so if they did that test today AAC would likely rank higher.
If using the Apple encoder, encode in "Better" mode with 16-bit source, and in "Best" mode with source that's more than 16-bits per sample (and hence isn't a CD rip). Support for mastering from 24-bit when running in "Best" is one of the reasons why the AAC-LC files as part of iTunes sound so good.
My video compression blog
In the Billy and the Boingers' "U Stink But I Luv U" encoding test, the OOP-AAC compression scheme won by a wide margin.
Well, any analog medium = much worse loss. LPs and cassette tapes can't approach the dynamic range of a CD. Plus you get noise, which gets worse on repeated playback.
The only lossless music is a live performance. But even then, you may crib about acoustics. Besides, you can't hire Brendel to play live for you whenever you feel like, and even if you could, he may not be in good form every day.
It looks like they're working with 16-bit source, not the 24-bit source that most of the iTunes AAC files are ripped from. So this test, while certainly very interesting, won't be useful to determine the iTunes music store quality.
My video compression blog
What you prefer the loss of the pits on your LP wearing as the needle passes through the grooves? Yeah spectral analysis shows that after three plays with a high end player the LP has already lost MORE dynamic range then a ADAT recording, and of course in hundreds of plays and a couple generations the digital copy is obviously superior, plus getting vinyl from mail order sucks, I know I DJ and the % of DOA stuff is way too high for my liking (especially if it's a white label or rare import, ususally means I get the insurance $ but never find the music again for a reasonable price). Analog has its place (like scratching, cd's just are not the same) but long term quality is not where it's at.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
With 6000 participants, the double-blind public test results were:
- Ogg
- WMA
- RealAudio
- Mp3Pro
- MP3
- AAC (Sic!)
Of course, this was crazy, with AAC even behind MP3, but these really were the results...First, read this:
.zip files.
http://www.ff123.net/abchr/abchr.html
This describes the program and testing methodology used here, which, btw, is based on widely accepted perceptual testing conventions. And yes, by the scientific community. These are the same techniques used by the scientists that do the research and development on these formats. Please note the references at the bottom of the page.
1. Wrong, the MP4 files are already encoded and created for the user, stored in the
2. Wrong, the Hidden Reference (ABC/*HR*, please read the page at the first link), ensures that if the user honestly cannot tell the difference but thinks that one exists (placebo), and rates the original lower than one of the encoded versions, that their results are discarded.
3. This is where the statistics come in. With enough listeners, the "noise" gets weeded out of relevant results. Most past tests using this methodology have been shown to provide highly relevant and fairly uniform results when all the data is factored together.
An open call to the masses is the only way to measure the perception of the masses, and if the test is performed properly (which it is in this case), then it *is* scientific.
Next time, please read up a little more on what is happening before jumping to all sorts of incorrect conclusions.
As was previously mentioned on Slashdot, a highly regarded German magazine called C'T dedicated an article to a similar comparison of various audio compression codecs last year.
.WAV recordings containing 3 short excerpts from various CD music tracks (pop, classical and jazz) that had previously been encoded by 6 popular codecs, each at both 64 Kbit/s and 128 Kbit/s (or as close as possible for VBR-only encoders). For verification of the results, 2 of the recordings came directly from CD and had not gone through any encoding process. Because the .WAV files were all the same size, there was no way for the listener to know which encoder had been used on a particular file. Participants were asked to rank their preferences among these files. The encoders included MP3, MP3PRO, Ogg Vorbis, WMA, RealAudio and AAC.
They created fourteen different
Over 6000 people downloaded those tracks and submitted their preferences. Unfortunately, the results of that test were only published in print and I haven't been able to find an online version of it. A few noteworthy results are below however.
The percentages indicate how many people put a particular codec at a particular ranking:
MP3 64 KBit/s
1st place: 1 %
2: 1%
3: 1%
4: 1%
5: 2%
6: 4%
7th place: 90%
As might be expected for the oldest codec, almost everyone agreed that the file that had been run through MP3 at 64 Kbit was the worst sounding of all. At 128 KBit however, listeners were clearly divided on whether MP3 sounded worse or better than others:
MP3 128 Kbit/s
1: 11%
2: 14%
3: 15%
4: 15%
5: 16%
6: 16%
7: 14%
Now the AAC results. At 64 Kbit, it was ranked a slightly below average performer:
AAC 64 KBit/s
1: 7%
2: 12%
3: 17%
4: 26%
5: 22%
6: 14%
7: 2%
What's interesting is that at 128 Kbit/s, more people ranked AAC the worst sounding encoder than any other codec in the test including MP3!
AAC 128 KBit/s
1: 11%
2: 11%
3: 13%
4: 12%
5: 14%
6: 14%
7: 26%
Not surprisingly, the files that had been read directly from CD without any encoding steps done in between got the best rankings of all. Ogg Vorbis did very well indeed and came in second overall.
This experiment is really designed to test which codec overall sounds better to the average user, for an arbitrary and inconsistent range of hardware setups, acoustic environments, and listening preferences (e.g. do I pay more attention to the primary beat or to the background harmony). I wouldn't place any value on this test other than to choose which codec I might choose if I wanted to please the ignorant consumer (a valid market, of course!). It does nothing to address how accurately a codec reproduces the artist's original sound.
I'll put a lot more stock in the Report on the MPEG-2 AAC Stereo Verification Tests put together by David Meares (BBC), Kaoru Watanabe (NHK), Eric Scheirer (MIT Media Labs) for the ISO. And the other MPEG Audio Public Documents.
You're basically asking for a lot of people to submit their opinions. This will show you what the people who participated in it prefer, but it doesn't really reveal much in they way of actual sound quality. Everyone has their own opinions already about which audio codec is supperior. The only way you could rule out the placebo affect is to give the test blind, so that they have no clue which file is which. Even then since the results are being turned in on good faith, you have to accept that some people may simply lie about the results based on their own biases. You'd need an objectional third party to administer a test like this, and even then almost no one would agree on a third party in the end. If someone's favorite format lost they'd just bitch about the test being rigged. The only un arguable test would to actully compare the integrity of the audio to the original via an olliscope or some other device. Audio's not my area of expertise so I could be wrong there. It seems to me it's best to just not worry about it and use what you're happy with. Seeing a test like this wouldn't change my mind really. "Person A liked Audio B encoded with mp3 the best!" It just doesn't seem to hold that much sway over me.
I care because I have not fallen for the "golden ears" fallacy. To me, 192kbps ABR lame-encoded sounds exactly like the original. I don't have super expensive speakers attached to the computer, nor do I have a fancy sound card (Creative Live 5.1.) Storing music losslessly is a waste of space to me. Sometimes I like to share music files and it's a heck of a lot easier and others are a heck of a lot more interested in trading compressed music compared to lossless files. And I can put a heck of a lot more of them on a CDR. And should I wish to listem to them in my MP3 player with limited memory, I'm sure as hell not going to use a lossless format.
If YOU want to use up your hard drive space, internet bandwidth, and blank media with huge lossless encoded files, feel free. But don't get all smug and proclaim to not have any idea why anyone would not want to waste their resources.
Oh, and I'm not going to touch that "mathematically lossless" crap, others have covered that already.
That was a flippant answer to his seeming flippant post.
I like Ogg fine. It is my codec of choice, except of course that no one bothers to support it for my OS of choice, OS X.
There's no good Ogg encoders that can interface with iTunes and support Unicode (yet, of course)
There's no Ogg codec for Quicktime on OS X 10.2.6 (yet, of course)
I much prefer Ogg, ideologically, but it's not something I can actually *live* with, because the support isn't there.
I have 100% support for MP3 and AAC.
Yes, I believe in fighting for causes I believe in. Right now Ogg is not one of those causes; maybe later. Right now I'm more concerned with my friends, my mortgage, and my state of unemployment, sorry.
GPL Deconstructed
It doesn't sound like you use a Mac or the iTunes Music Store, so why do you say the AACs from Apple are shitty?
There are at least three distinct things to keep in mind:
MP3s encoded from your music using LAME at 220kbps VBR is one quality
AACs encoded with Quicktime 6.3 is one quality
AACs encoded from masters, ala iTunes Music Store, are another quality
You, in one sentence, mix all three quality levels as if they are currently comparable.
The music from the iTunes music store is encoded from a higher quality source, and can arguably be of higher quality than even your 220kbps mp3s. It's hard to make any educated guess because I don't know anyone who's done a comparison between AAC files ripped from masters vs MP3s ripped from CD.
The music you get from iTunes itself is based on Quicktime 6.3, and that *is* being compared and characterized in this test; this will probably illustrate the level of quality iTunes for Windows will have, and is more directly comparable to your 220kbps mp3s, but only *after* the test is performed.
it's fine to believe that your mp3s are better, but there is no proof yet.
GPL Deconstructed
The complete results can be found in issue 19/2002 of Heise's offline magazine C't. Along with the online public test, some 'experts' (such as some music producers, hobby listeners, a singer, and a young student and choir singer) were consulted.
In the online public test, the 64 kBit/s comparison yielded
The parent's results were the ones for 128 kBit/s. The eight experts compared the codecs on 160 kBit/s as well, with much more varying results (not much of a surprise). But on average, the results were
As I said, those were an average, with the individual results of the eight experts strongly deviating. Ogg was placed once 1st, once 2nd, twice 3rd and 4th, and once 5th and 7th. (One had actually placed the plain wave reference 5th...)
I disagree on the 160 vs 256 kbps statement. I listen to mostly rock and punk, so I took a Thursday song, which is kindof in the middle of the two genres, and encoded it at 32, 48, 56, 64, 96, 112, 128, 160, 192, 256 and 320 kbps. I wanted to encode my whole CD selection (350 cds) at a bitrate that I couldn't hear the differance, and a bitrate that I could stream at decently. For streaming, 56 was the magic number. Any less and it sounded like crap, any more, and my DSL line couldn't host 2 streams at once. For music, 192 was good, but I could still hear the mp3 compression. I find that bass tends to get distorted in mp3s, and once I went to 256 this seemed to go away. I did all these tests with an audigy2 under windows XP, using Lame with q=9. Playback was through the Infinity HTS-20 Speaker System.
-- the computer doesn't want any beer, no matter how much you think it does. NEVER, EVER feed your computer beer.
Well, I'm not using AAC until it supports Ogg.
I mean, damn them! Nirvana didn't pay $606.17 to record Bleach so that some Corporate Asswipe could make a high fidelty copy of it!
The Ramones would be very peeved to find all the work they put into keeping most songs to three, dingy, distorted chords, ripped to a high fidelty format.
You know what?
Over 160? Maybe you've been to too many Megadeth concerts or something, but 160 Kbps is quite audibly lossy in my experience. Now, I'm fussy about encoding artifacts, but 160 is the lowest I'll use for listening to on headphones on an airplane. It has to be at least 192 for me to not find artifacts distracting while listening on a good stereo (Paradigm Active/20 reference monitors attached to my video editing rig, in my case, self-powered with all XLR signal routing from the jukebox machine. I grant this is overkill for the casual listener).
Personally, I encode my library at 320 Kbps Normal Stereo without any filtering. This is overkill for listening, but that's enough data that I can recompress to another, more portable format like AAC on an iPod without windup up with a audible multigeneration artifacts.
All things being equal, I'd use FLAC, but I really really like the iTunes interface, and 320 MP3 is the best format it has historically supported. It now does 320 AAC, and I'm toying with switching to that (although I haven't yet, since the files won't be quite as widely interoperable).
My video compression blog
The African American Congress? Okay, okay, I'll be the first to admit I watch too much Law & Order.
I did my own test of this a while back (AAC,MP3,OGG only). I didn't do 128K CBR but instead did 160K VBR.
My results were:
1. AAC
2. OGG Vorbis
3. MP3