AAC Put To The Test

technology is sexy writes "Following the increasing popularity of AAC in online music stores and the growing amount of implementations in software and hardware, the format is now being put to the test. How well does Apple's implementation fare against Ahead Nero, Sorenson or the Open Source FAAC at the popular bitrate of 128kbps? Find out for yourself and help by submitting the results. You can find instructions on how to participate here. The best AAC codec gets to face MP3, MP3Pro, Vorbis, MusePack and WMA in the next test. Previous test results at 64kbps can be found here."

Comments

Re:Isn't AAC used for its DRM features?

[2nd Post!]

Well, if you don't care about the quality, why the heck does it matter if Ogg has good or bad quality?

All it needs to be is open and unencumbered, right?

Well, the AAC produced by Apple Quicktime isn't DRM burdened, even if it does have some patent stuff attached.

Re:Isn't AAC used for its DRM features?

[Anonymous Coward]

You mean the DRM features that allow me to rip my own CD's to AAC and copy the resultant files to any and all computers or players (that understand them) and play them back?

Or how about the DRM feature that allows me to export bought AAC's to aiff and then convert them to MP3/OGG/AAC/.wav/.au etc and do with them what I please?

True, Apple's TMS is selling AAC's that have a DRM-like "inconvenience protection" on them but it's not _inherent_ to the AAC format, nor does it affect the sound quality vs. file size questions.

(In any case, we _should_ be cheering for any company that's actually trying to give us quite reasonably limited freedom with copyrighted material, while satisfying the RIAA/MPAA etc.)

Re:An honest question - who cares?

[Anonymous Coward]

mathematically loseless MP3s

Oh really? At what bitrate or setting are you getting the exact same WAV files back out after an encode/decode cycle?

You want lossless, use a lossless codec (like FLAC). MP3 is not designed for that, and can't do it. You might not be able to hear the difference, but that doesn't make it anywhere near mathematically loseless.

patent and the possibility of DRM

[SHEENmaster]

Patent encomberment is a serious deal. It means than a legal OSS player is nearly out of the questions. If I can't play the things on my iBook(Linux), iMac(Linux), server(Linux), palmtop(Linux), and at school (OS X) then I won't be using it. Quality is irrelevant at that point.

Ogg Vorbis, because of its openness and mpeg, becase people ignore the patent, are my best two options. AAC is not an option, so its quality means nothing.

Would you rather use a train that can safely travel at 100mph along prelaid tracks that don't follow your route or a car that can safely go 60mph along much more convenient roads?

(Oh yeah, Linux is a rocket car in the analogy because it has to be stuck in there somewhere. Windows is a horse in that it can go anywhere if at a crawling pace while shitting over everything, but a rocket car can go more places...

Re:patent and the possibility of DRM

[PhoenixK7]

"You're missing the point of the car analogy. Ogg is a car that doesn't go where you want it to go. There's no Ogg support in QuickTime. There's no Ogg support on the iPod. It simply can't go places that people want to go."

Indeed. You've got a car, it can drive in any direction, but all of the terrain is impassable using said car. You either have to pave your own roads or wait until someone else does it because they've got enough spare time and want the road badly enough.

The paving materials and equipment are freely available, but someone has to invest the time to lay the infrastructure.

On the other hand you can pay some cash, get some other vehicle and use the roads/rails/whatever they've built using revenue from selling whatever vehicles they happen to be.

Frankly, I've got a mac. I've got an iPod. I already made the hardware investment (and software, but thats really just a sunk cost since the computer came with Jaguar, and the iPod came with its own little OS). Why not use it? AAC on high quality encoding at 128 kbits sounds pretty damn good to me. But then again, there's no reason you should take my word for it, as I don't have any "audiophile" equipment, and I've had a very mild case of tinnitus in my right ear (dammit!). It lets me store lots of music, and it sounds good to me. To boot, some other people seem to think it sounds pretty damn good to (and an equal if not greater number that dislike it either because it sounds bad to them or they've got some sort of political agenda that clashes with apple/dolby/patents/whatever).

Re:An honest question - who cares?

[afidel]

Well it matters because some people are buying shitty 128k aac file from Apple (a LOT aparantly, they have sold almost 4 million tracks already). I too rip to high quality mp3 (~220kbps VBR mp3's from LAME which passed tripple blind with wav and ogg) but I guess it's usefull to know what quality you can expect from this service since Apple will be coming out with iTunes for windows later this year and bringing their online service to the masses.

Re:I prefer analog

[rsidd]

Just remember that digital=loss

Well, any analog medium = much worse loss. LPs and cassette tapes can't approach the dynamic range of a CD. Plus you get noise, which gets worse on repeated playback.

The only lossless music is a live performance. But even then, you may crib about acoustics. Besides, you can't hire Brendel to play live for you whenever you feel like, and even if you could, he may not be in good form every day.

Sounds Better != High Fidelity

[Josuah]

This experiment is really designed to test which codec overall sounds better to the average user, for an arbitrary and inconsistent range of hardware setups, acoustic environments, and listening preferences (e.g. do I pay more attention to the primary beat or to the background harmony). I wouldn't place any value on this test other than to choose which codec I might choose if I wanted to please the ignorant consumer (a valid market, of course!). It does nothing to address how accurately a codec reproduces the artist's original sound.

I'll put a lot more stock in the Report on the MPEG-2 AAC Stereo Verification Tests put together by David Meares (BBC), Kaoru Watanabe (NHK), Eric Scheirer (MIT Media Labs) for the ISO. And the other MPEG Audio Public Documents.

Re:Sounds Better != High Fidelity

[Anonymous Coward]

Wrong. The experiment is designed to blindly test how well the listener can discern the encoded sample vs the original on a sample by sample basis. This has nothing to do with overall quality or arbitrariness, it has to do with the ability to reliabily pick out which is the encoded and which is the original, and then compare across all the encoded samples. The codec which they are able to distinguish from the original the most infrequently should get the highest rating, not simply which one they "like the sound of the best."

So, in actuality, it *does* address how accurately a codec reproduces the artist's original sound.

Also, the manner in which the testing is perfomed is very similar to that which is used by these groups which often do more "formal" tests.

As a side note, I'm amazed how many people I've seen jump to conclusions about the test here without bothering to research it even on a basic level first.

How can this possibly be accurate?

[NeoOokami]

You're basically asking for a lot of people to submit their opinions. This will show you what the people who participated in it prefer, but it doesn't really reveal much in they way of actual sound quality. Everyone has their own opinions already about which audio codec is supperior. The only way you could rule out the placebo affect is to give the test blind, so that they have no clue which file is which. Even then since the results are being turned in on good faith, you have to accept that some people may simply lie about the results based on their own biases. You'd need an objectional third party to administer a test like this, and even then almost no one would agree on a third party in the end. If someone's favorite format lost they'd just bitch about the test being rigged. The only un arguable test would to actully compare the integrity of the audio to the original via an olliscope or some other device. Audio's not my area of expertise so I could be wrong there. It seems to me it's best to just not worry about it and use what you're happy with. Seeing a test like this wouldn't change my mind really. "Person A liked Audio B encoded with mp3 the best!" It just doesn't seem to hold that much sway over me.

Re:How can this possibly be accurate?

[90XDoubleSide]

While the first post was way off-base, there are two massive problems with this testing method: first, there is no standardized reproduction equipment. If you wanted to test only the codec, the test would need to be performed with everyone listening on the same (reference-quality) equipment. Secondly, because the test uses a nonrandom sample of people rather than professional listeners, the test measures how good people think the music sounds, rather than which codec actually reproduces the original recording the best; because people will typically not pick faithful reproductions as the best, the test cannot be construed as a measure of which codec works the best, but only of which one produces the sound most desirable to the the nonrandom sample group over imperfect reproduction equipment.

Re:crap in, crap out

[Guppy06]

"I'd hazard a guess that most people that encode with ogg-vorbis do a better ripping and encoding job, though."

Only because right now you'd have to know a thing or two about the intricacies of digital music to have ever heard the phrase "ogg vorbis." If a big on-line music player were to standardize on this format instead of MP3 and it too becomes the common man's format, you can be sure the quality of ogg files will go down just as well.

Just to keep you intellectually honest...

[2nd Post!]

It doesn't sound like you use a Mac or the iTunes Music Store, so why do you say the AACs from Apple are shitty?

There are at least three distinct things to keep in mind:
MP3s encoded from your music using LAME at 220kbps VBR is one quality
AACs encoded with Quicktime 6.3 is one quality
AACs encoded from masters, ala iTunes Music Store, are another quality

You, in one sentence, mix all three quality levels as if they are currently comparable.

The music from the iTunes music store is encoded from a higher quality source, and can arguably be of higher quality than even your 220kbps mp3s. It's hard to make any educated guess because I don't know anyone who's done a comparison between AAC files ripped from masters vs MP3s ripped from CD.

The music you get from iTunes itself is based on Quicktime 6.3, and that *is* being compared and characterized in this test; this will probably illustrate the level of quality iTunes for Windows will have, and is more directly comparable to your 220kbps mp3s, but only *after* the test is performed.

it's fine to believe that your mp3s are better, but there is no proof yet.

160 Kbps MP3 NOT very good!

[benwaggoner]

Over 160? Maybe you've been to too many Megadeth concerts or something, but 160 Kbps is quite audibly lossy in my experience. Now, I'm fussy about encoding artifacts, but 160 is the lowest I'll use for listening to on headphones on an airplane. It has to be at least 192 for me to not find artifacts distracting while listening on a good stereo (Paradigm Active/20 reference monitors attached to my video editing rig, in my case, self-powered with all XLR signal routing from the jukebox machine. I grant this is overkill for the casual listener).

Personally, I encode my library at 320 Kbps Normal Stereo without any filtering. This is overkill for listening, but that's enough data that I can recompress to another, more portable format like AAC on an iPod without windup up with a audible multigeneration artifacts.

All things being equal, I'd use FLAC, but I really really like the iTunes interface, and 320 MP3 is the best format it has historically supported. It now does 320 AAC, and I'm toying with switching to that (although I haven't yet, since the files won't be quite as widely interoperable).

Re:160 Kbps MP3 NOT very good!

[King_TJ]

Yep!! I encoded my entire CD collection at 192 bits, and I don't waste my time listening to anything less, if I can help it. I find it maddening that most people still encode at 128 bits and think it sounds "good enough". It only sounds ok on cheap speakers, or perhaps even moderate priced speakers + some audio processing enabled to "enhance" the sound.

Whenever I listen to 128-bit MP3s through my set of studio monitors, they sound "thin". Even in my car (I have a Rio MP3 car player), with Diamond Audio seperates and a Rockford Fosgate amp, 128-bit MP3s are noticeably poorer quality than ones encoded at higher bitrates.

I would have encoded above 192 bits, even, except I was trying to strike a good balance between optimal sound quality and conserving a little bit of drive space. (After all, if disk space is no object, then it makes no sense to use a compressed audio format to begin with!) I found that 192 bits was about as low as I could encode and still feel like I wasn't losing any significant amount of audio quality. (At worst, you might be able to hear very small details, such as a brush against a cymbal in an otherwise quiet passage, that would have sounded more "life-like" at 256 bit+ encoding than at 192. You'd almost have to listen to them side by side to even tell though.)

Re:i prefer just to steal the music

[PhoenixK7]

There's nothing inherent in AAC that makes it DRM friendly. Its the quicktime wrapper thats DRM friendly. It would not be difficult to slap some form of DRM on MP3s, you just wouldn't get that many users because none of the MP3 players would support it out-of-the-box. People want to be able to fire up their winamp (or *shudder* realplayer, wmp, etc) and play the files without hassal. SInce iTunes is the most popular mac audio player, its much easier to add DRM to a format that hasn't been used much. Players will implement fairplay and authorize tracks. Apple could have done the same damn thing with MP3s, called them something other tham mp3s and it would have worked perfectly fine.

AAC _is_ technically superior to MP3. The problem is we've had around 10 years now to refine and perfect our MP3 encoders while free/cheap AAC encoders are just coming onto the market. Give it time, once it reaches its prime it will provide quality that I'm sure will undeniably rival MP3.

Re:Fine print......

[salimma]

Gotta love the advertising line though, almost as bad as selling computers using P2P and music sharing as a 'feature'

Heh, here in UK I've been cringing everytime I saw a BT Internet advertisement, with people supposed to be users touting its use for downloading music.

In the worst ad, the girl even said she used to buy CDs but now she just downloaded them. Granted, there are ways to do it legally (hello Apple Store) but in UK and on a PC?

It has nowhere to go but up

[Anonymous Coward]

"so if they did that test today AAC would likely rank higher."

Couldn't go down any. AAC ended up at the *bottom* of the rankings.

Enough with this bullshit that AAC works magic at 128kb. 128kb isn't enough with our current generation of encoders, and isn't likely to improve for 5 years at a minimum.

Meanwhile, suckers are paying $10 for albums with less fidelity than I can get from the used CD section at Amazon, and what I'm buying has no DRM restrictions at all.

And the iTunes fans have the balls to tell me what a great deal 128kb AACs are for $1? Sheesh. If Apple put shit on a stick, some of you would say it tastes better than "regular" food.