Slashdot Mirror
AAC Put To The Test
technology is sexy writes "Following the increasing popularity of AAC in online music stores and the growing amount of implementations in software and hardware, the format is now being put to the test. How well does Apple's implementation fare against Ahead Nero, Sorenson or the Open Source FAAC at the popular bitrate of 128kbps? Find out for yourself and help by submitting the
results. You can find instructions on how to participate here. The best AAC codec gets to face MP3, MP3Pro, Vorbis, MusePack and WMA in the next test. Previous test results at 64kbps can be found here."
this makes the format rather irrelevant.
Most mp3s or oggs you find out there are from someone's CD-Rom drive, who knows how the disc looked, or how much jitter there was. I have heard stories of people downloading songs to find a skip or two in the middle, or been an amalgam of two different files accidently spliced together.
I'd hazard a guess that most people that encode with ogg-vorbis do a better ripping and encoding job, though.
Do you even lift?
These aren't the 'roids you're looking for.
mmm... yeah... You see, we're putting the cover sheets on all TPS reports now before they go out...
I prefer DVDA.
What hyphen?
graspee
You mean the DRM features that allow me to rip my own CD's to AAC and copy the resultant files to any and all computers or players (that understand them) and play them back?
Or how about the DRM feature that allows me to export bought AAC's to aiff and then convert them to MP3/OGG/AAC/.wav/.au etc and do with them what I please?
True, Apple's TMS is selling AAC's that have a DRM-like "inconvenience protection" on them but it's not _inherent_ to the AAC format, nor does it affect the sound quality vs. file size questions.
(In any case, we _should_ be cheering for any company that's actually trying to give us quite reasonably limited freedom with copyrighted material, while satisfying the RIAA/MPAA etc.)
I note that in the 64 Kbps test, they used the AAC-LC encoder from QuickTime 6.0. This was a pretty darn lousy one, lacking any ability to specify a sample rate at a given data rate, and had poor quality. The current version of QuickTime 6.3 (for Windows and MacOS X), has a much improved, more flexible AAC-LC encoder, so if they did that test today AAC would likely rank higher.
If using the Apple encoder, encode in "Better" mode with 16-bit source, and in "Best" mode with source that's more than 16-bits per sample (and hence isn't a CD rip). Support for mastering from 24-bit when running in "Best" is one of the reasons why the AAC-LC files as part of iTunes sound so good.
My video compression blog
Not possible. MP3 by its very nature is a lossy encoding scheme, hence there will always be artifacts when you pass the audio through the encoder. You may not be able to hear the quality change (even after passing the files over and over and over through the encoder) but you will be generating noise.
As far as your original question, it all comes down to file portability. It takes people a bit longer to send a 65 meg wav to their friends, compared to a 6.5 meg mp3.
In the Billy and the Boingers' "U Stink But I Luv U" encoding test, the OOP-AAC compression scheme won by a wide margin.
There *are* lossless codecs like FLAC and SHN, but they generally achieve between 10 - 30% compression.
Religion is a gateway psychosis. -- Dave Foley
Well, any analog medium = much worse loss. LPs and cassette tapes can't approach the dynamic range of a CD. Plus you get noise, which gets worse on repeated playback.
The only lossless music is a live performance. But even then, you may crib about acoustics. Besides, you can't hire Brendel to play live for you whenever you feel like, and even if you could, he may not be in good form every day.
It looks like they're working with 16-bit source, not the 24-bit source that most of the iTunes AAC files are ripped from. So this test, while certainly very interesting, won't be useful to determine the iTunes music store quality.
My video compression blog
What you prefer the loss of the pits on your LP wearing as the needle passes through the grooves? Yeah spectral analysis shows that after three plays with a high end player the LP has already lost MORE dynamic range then a ADAT recording, and of course in hundreds of plays and a couple generations the digital copy is obviously superior, plus getting vinyl from mail order sucks, I know I DJ and the % of DOA stuff is way too high for my liking (especially if it's a white label or rare import, ususally means I get the insurance $ but never find the music again for a reasonable price). Analog has its place (like scratching, cd's just are not the same) but long term quality is not where it's at.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
With 6000 participants, the double-blind public test results were:
- Ogg
- WMA
- RealAudio
- Mp3Pro
- MP3
- AAC (Sic!)
Of course, this was crazy, with AAC even behind MP3, but these really were the results...First, read this:
.zip files.
http://www.ff123.net/abchr/abchr.html
This describes the program and testing methodology used here, which, btw, is based on widely accepted perceptual testing conventions. And yes, by the scientific community. These are the same techniques used by the scientists that do the research and development on these formats. Please note the references at the bottom of the page.
1. Wrong, the MP4 files are already encoded and created for the user, stored in the
2. Wrong, the Hidden Reference (ABC/*HR*, please read the page at the first link), ensures that if the user honestly cannot tell the difference but thinks that one exists (placebo), and rates the original lower than one of the encoded versions, that their results are discarded.
3. This is where the statistics come in. With enough listeners, the "noise" gets weeded out of relevant results. Most past tests using this methodology have been shown to provide highly relevant and fairly uniform results when all the data is factored together.
An open call to the masses is the only way to measure the perception of the masses, and if the test is performed properly (which it is in this case), then it *is* scientific.
Next time, please read up a little more on what is happening before jumping to all sorts of incorrect conclusions.
As was previously mentioned on Slashdot, a highly regarded German magazine called C'T dedicated an article to a similar comparison of various audio compression codecs last year.
.WAV recordings containing 3 short excerpts from various CD music tracks (pop, classical and jazz) that had previously been encoded by 6 popular codecs, each at both 64 Kbit/s and 128 Kbit/s (or as close as possible for VBR-only encoders). For verification of the results, 2 of the recordings came directly from CD and had not gone through any encoding process. Because the .WAV files were all the same size, there was no way for the listener to know which encoder had been used on a particular file. Participants were asked to rank their preferences among these files. The encoders included MP3, MP3PRO, Ogg Vorbis, WMA, RealAudio and AAC.
They created fourteen different
Over 6000 people downloaded those tracks and submitted their preferences. Unfortunately, the results of that test were only published in print and I haven't been able to find an online version of it. A few noteworthy results are below however.
The percentages indicate how many people put a particular codec at a particular ranking:
MP3 64 KBit/s
1st place: 1 %
2: 1%
3: 1%
4: 1%
5: 2%
6: 4%
7th place: 90%
As might be expected for the oldest codec, almost everyone agreed that the file that had been run through MP3 at 64 Kbit was the worst sounding of all. At 128 KBit however, listeners were clearly divided on whether MP3 sounded worse or better than others:
MP3 128 Kbit/s
1: 11%
2: 14%
3: 15%
4: 15%
5: 16%
6: 16%
7: 14%
Now the AAC results. At 64 Kbit, it was ranked a slightly below average performer:
AAC 64 KBit/s
1: 7%
2: 12%
3: 17%
4: 26%
5: 22%
6: 14%
7: 2%
What's interesting is that at 128 Kbit/s, more people ranked AAC the worst sounding encoder than any other codec in the test including MP3!
AAC 128 KBit/s
1: 11%
2: 11%
3: 13%
4: 12%
5: 14%
6: 14%
7: 26%
Not surprisingly, the files that had been read directly from CD without any encoding steps done in between got the best rankings of all. Ogg Vorbis did very well indeed and came in second overall.
This experiment is really designed to test which codec overall sounds better to the average user, for an arbitrary and inconsistent range of hardware setups, acoustic environments, and listening preferences (e.g. do I pay more attention to the primary beat or to the background harmony). I wouldn't place any value on this test other than to choose which codec I might choose if I wanted to please the ignorant consumer (a valid market, of course!). It does nothing to address how accurately a codec reproduces the artist's original sound.
I'll put a lot more stock in the Report on the MPEG-2 AAC Stereo Verification Tests put together by David Meares (BBC), Kaoru Watanabe (NHK), Eric Scheirer (MIT Media Labs) for the ISO. And the other MPEG Audio Public Documents.
I care because I have not fallen for the "golden ears" fallacy. To me, 192kbps ABR lame-encoded sounds exactly like the original. I don't have super expensive speakers attached to the computer, nor do I have a fancy sound card (Creative Live 5.1.) Storing music losslessly is a waste of space to me. Sometimes I like to share music files and it's a heck of a lot easier and others are a heck of a lot more interested in trading compressed music compared to lossless files. And I can put a heck of a lot more of them on a CDR. And should I wish to listem to them in my MP3 player with limited memory, I'm sure as hell not going to use a lossless format.
If YOU want to use up your hard drive space, internet bandwidth, and blank media with huge lossless encoded files, feel free. But don't get all smug and proclaim to not have any idea why anyone would not want to waste their resources.
Oh, and I'm not going to touch that "mathematically lossless" crap, others have covered that already.
It's rather obvious that you haven't bothered to read anything about the test, the program used, etc.
The test *is* blind, and it is based on widely accepted perceptual testing techniques. It uses hidden references (references to the original vs the encoded sample, on a per sample basis in which the user is not aware of which is which, thus if they rate the original as being worse than the encoded version, their result is discarded) as a control. The program devised has been developed by someone who has taken the time to do the proper research, read the appropriate papers and other sources, discuss the idea with developers of many different audio codecs (LAME, Vorbis, PsyTEL AAC, etc). The technique here works, and has been used many times before. It's not simply some amateurish scheme that someone who knew nothing about the appropriate sciences dreamed up simply because he wanted to find out if "Person A liked Audio B".
The complete results can be found in issue 19/2002 of Heise's offline magazine C't. Along with the online public test, some 'experts' (such as some music producers, hobby listeners, a singer, and a young student and choir singer) were consulted.
In the online public test, the 64 kBit/s comparison yielded
The parent's results were the ones for 128 kBit/s. The eight experts compared the codecs on 160 kBit/s as well, with much more varying results (not much of a surprise). But on average, the results were
As I said, those were an average, with the individual results of the eight experts strongly deviating. Ogg was placed once 1st, once 2nd, twice 3rd and 4th, and once 5th and 7th. (One had actually placed the plain wave reference 5th...)
I disagree on the 160 vs 256 kbps statement. I listen to mostly rock and punk, so I took a Thursday song, which is kindof in the middle of the two genres, and encoded it at 32, 48, 56, 64, 96, 112, 128, 160, 192, 256 and 320 kbps. I wanted to encode my whole CD selection (350 cds) at a bitrate that I couldn't hear the differance, and a bitrate that I could stream at decently. For streaming, 56 was the magic number. Any less and it sounded like crap, any more, and my DSL line couldn't host 2 streams at once. For music, 192 was good, but I could still hear the mp3 compression. I find that bass tends to get distorted in mp3s, and once I went to 256 this seemed to go away. I did all these tests with an audigy2 under windows XP, using Lame with q=9. Playback was through the Infinity HTS-20 Speaker System.
-- the computer doesn't want any beer, no matter how much you think it does. NEVER, EVER feed your computer beer.
Well, I'm not using AAC until it supports Ogg.
I mean, damn them! Nirvana didn't pay $606.17 to record Bleach so that some Corporate Asswipe could make a high fidelty copy of it!
The Ramones would be very peeved to find all the work they put into keeping most songs to three, dingy, distorted chords, ripped to a high fidelty format.
You know what?
Over 160? Maybe you've been to too many Megadeth concerts or something, but 160 Kbps is quite audibly lossy in my experience. Now, I'm fussy about encoding artifacts, but 160 is the lowest I'll use for listening to on headphones on an airplane. It has to be at least 192 for me to not find artifacts distracting while listening on a good stereo (Paradigm Active/20 reference monitors attached to my video editing rig, in my case, self-powered with all XLR signal routing from the jukebox machine. I grant this is overkill for the casual listener).
Personally, I encode my library at 320 Kbps Normal Stereo without any filtering. This is overkill for listening, but that's enough data that I can recompress to another, more portable format like AAC on an iPod without windup up with a audible multigeneration artifacts.
All things being equal, I'd use FLAC, but I really really like the iTunes interface, and 320 MP3 is the best format it has historically supported. It now does 320 AAC, and I'm toying with switching to that (although I haven't yet, since the files won't be quite as widely interoperable).
My video compression blog
"You're missing the point of the car analogy. Ogg is a car that doesn't go where you want it to go. There's no Ogg support in QuickTime. There's no Ogg support on the iPod. It simply can't go places that people want to go."
Indeed. You've got a car, it can drive in any direction, but all of the terrain is impassable using said car. You either have to pave your own roads or wait until someone else does it because they've got enough spare time and want the road badly enough.
The paving materials and equipment are freely available, but someone has to invest the time to lay the infrastructure.
On the other hand you can pay some cash, get some other vehicle and use the roads/rails/whatever they've built using revenue from selling whatever vehicles they happen to be.
Frankly, I've got a mac. I've got an iPod. I already made the hardware investment (and software, but thats really just a sunk cost since the computer came with Jaguar, and the iPod came with its own little OS). Why not use it? AAC on high quality encoding at 128 kbits sounds pretty damn good to me. But then again, there's no reason you should take my word for it, as I don't have any "audiophile" equipment, and I've had a very mild case of tinnitus in my right ear (dammit!). It lets me store lots of music, and it sounds good to me. To boot, some other people seem to think it sounds pretty damn good to (and an equal if not greater number that dislike it either because it sounds bad to them or they've got some sort of political agenda that clashes with apple/dolby/patents/whatever).