Audio Format Listening Tests Concluded

Pointing to the conclusions of this listening study, nullity writes: "The results are interesting, and show a high variation in the performance of the various codecs on different musical styles. Ogg seems to work well on dance music, WMA8 on chamber music, etc."

Comments

WMA8

[af_robot]

Ogg seems to work well on dance music, WMA8 on chamber music, etc.

Like requiem...

Re:WMA8

[Snork Asaurus]

Ogg seems to work well on dance music, WMA8 on chamber music , etc.

This might explain why I only think of WMA as being good enough for recording the sounds that I hear while sitting my "chamber" on my "throne".

(no offence to those sounds intended).

Sound Artifacts

[alatesystems]

While mp3s encoded at lower bit rates seem to have a tingly sound sometimes, almost every wma file I've gotten or encoded ALWAYS sounds like someone is blowing windchimes in my music. I can't stand it.

In my opinion, 192kbps MP3 is the way to go, but then what do I know? I can't hear about 13khz.

Chris

Re:Sound Artifacts

[funky womble]

I have problems with some 192Kbps CBR MP3s, I find --r3mix usually sounds a lot better (and takes up less space), though sometimes I neeed to bump it up to ABR (usually around 220Kbps if it's a difficult piece).

Re:Sound Artifacts

[13Echo]

I thought about switching to OGG, but I wouldn't find much benefit to it unless I wanted to go with smaller filetypes, which is where it really shines. But since I encode all of my MP3s in Lame r3mix VBR 128-320 (averages 192), I get the best sound possible.

Re:Sound Artifacts

[sheldon]

It depends on your listening environment.

Most computer audio systems introduce a ton of noise and distortion themselves. I had some ESS chipset in my computer at work, and everything sounded just horrible until I found this switch buried deep in the settings which was causing it to try to simulate surround sound on two channels.

There's also a high level of S/N, if the volume is over a certain threshold I hear a lot of hiss.

I've also had problems in the past with soundcards plugged into the PCI bus picking up excess noise from the video card.

It's like listening to music in a car... it works, and sometimes it sounds ok, but it's certainly not ideal.

Re:Sound Artifacts

[MrResistor]

If you really want to hear the "loss" in mp3 try a few Pixies songs. Jimi Hendrix would probably work, too. Both derived alot of their "feel" from overtones and harmonics, stuff that's really only noticable in it's absence, and apparently stuff the mp3 folks decided we could do without.

IMHO, nothing shows off the shortcomings of mp3 like the Pixies.

Re:Sound Artifacts

[Jucius Maximus]

"That is because you have crap hearing. Get a very cleanly (over) produced song - I recommend TLC - Unpretty (no, you don't have to like it. [...] If you can't hear this difference having done exactly what I've said then I suggest you are not qualified to ever post a comment to a thread discussing audio quality again."

And if you log onto kazaa and download the mp3 to and then attempt to do this quality comaprison, you are not qualified to ever post on slashdot again. :P

Re:Sound Artifacts

[Oculus Habent]

Most people are playing 128kbps MP3s through computer speakers. For the average person who, based on my personal experience, listens to music far louder than they should, the quality differences aren't noticable.

I encode at 160 usually, and a just accept that d/ls are 128 most of the time. I don't mind unless it's a really good song, and then (*note to RIAA*)I get the CD.

I think this survey was badly flawed. How many people listen to 64kbps MP3s? Streamed, maybe. They should use fair comparisons of the different formats. I suppose at higher bit-rates, the differences become more negligable to the average listener, though.

Of course, I also know people who say RealAudio sounds better than MP3.

Sigh.

Who cares about 64 kbps tests?

[splorf]

These tests are all at 64 kbps and most people use much higher bitrates for real music. I'd like to see comparisons at 128k bits minimum, and preferably 160k or 192k, which is what most quality mp3's are at, for direct comparison.

Re:Who cares about 64 kbps tests?

[keller]

People who wnat to stream audio of course! .K

Re:Who cares about 64 kbps tests?

[ProtoCat]

64Kbps is where the flaws of a codec are truely exposed. It's a great median between being too high to produce much results and too low where everything completely falls apart. You may not think any of this has any relevance to you as you're encoding above 128Kbps, but it actually does make a difference when you stress your encoder with a difficult piece of music.

However, if the difference between sounding 'good' and sounding 'accurate' mean little to you, as someone who'd make an argument of 64Kbps tests being worthless would, then you really aren't the intended audiance of such tests. You can merrily use any of those encoders at 128-192Kbps without ever really noticing or caring much.

I, personally, would like to see OGG1.0, MP3 Pro and WMA8 take on some real tough to beat codecs such as Dolby's AAC High-Complexity Mode (which no AAC freely available encoder supports, including QuickTime) and Sony's ATRAC3. But, that'd be kinda moot, because most people out there do not have access to those toys.

For now, I'm content to just watch people hop around and proclaim whatever they want as king of audio formats while sticking to 256Kbps Fraunhoffer MP3 (archival purposes) and 192Kbps LAME HQ MP3 (general usage) as something both widely supported and pratically indistinguishable from the source. Even if AAC-HC and ATRAC3 were freely available, it'd take an awful large effort to wean people off of MP3 so far as support base and to migrate them to a new format. New P2P programs, new players/plug-ins (in some cases) and new hardware players. Not gonna happen for a while.

Re:Who cares about 64 kbps tests?

[flipflapflopflup]

> Are you pondering what I'm pondering?

I thought your sig was part of your comment, and was going to agree entirely. 64 kbps tests are pointless because no-ones uses those rates, and at higher rates the differences become negligable.

At the kind of bit rates that real people actually use (160, 192, and up), it takes a real pro/audiophile/picky git to tell the difference. Which makes the whole thing seem a bit pointless really.

Chances are then, it's not going to be audio quality that makes or breaks these standards. Look at betamax...

Re:Who cares about 64 kbps tests?

[nagora]

64Kbps is where the flaws of a codec are truely exposed.

Running your car over a cliff is where the flaws in its safety system are truely exposed but I don't tend to drive over cliffs much.

However, if the difference between sounding 'good' and sounding 'accurate' mean little to you, as someone who'd make an argument of 64Kbps tests being worthless would, then you really aren't the intended audiance of such tests.

What do you mean by this? 64Kbits is worthless for listening to any music I own while 128 is good enough to not actually annoy me much of the time so why should I be interested in these tests? Are you saying that the intended audience for these tests are people that are not interested in the quality of the music they're listening to?

TWW

Re:Who cares about 64 kbps tests?

[forgoil]

64kbps are more interesting when it comes to streaming music, obviously, so the test is valid for that purpose. I would love to get internet radio at something better than 64kbps 22khz, mono mp3s that doesn't eat bandwidth.

But most importanatly, at which kbps does the codecs become equal? MP3s do sound a lot better at 192kbps, and surely will beat ogg at 64kps. The music won't be better with one format over the other, as long as the "I can't belive it is lossy" barrier has been reached. Which format will make the music sound as the original (all the others can be discarded now thank you), and of those, which does it at the lowest bitrate?

Re:Who cares about 64 kbps tests?

[TummyX]

64 kbps tests are pointless because no-ones uses those rates, and at higher rates the differences become negligable.


With MP3 maybe, but OGG, WMA, MP3Pro and ACC all aim to get MP3@128Kbs quality at 64Kbs. That's what this test is FOR! Try comparing a 128KBps MP3 with a 64KBps OGG file. You'll find the quality to be quite comparable. It's a pity the test didn't include MP3 (at 128KBps) so we could see how good these new codecs really are.

And BTW, you might as well say it is pointless to test 128, 160 or 192kbps and that we should all be testing uncompressed 1500Kbps audio.

Re:Who cares about 64 kbps tests?

[Jugalator]

like to see comparisons at 128k bits minimum, and preferably 160k or 192k, which is what most quality mp3's are at

Yes - mp3's. I thought both wma and ogg used lower bitrates because they were more efficient.

A bitrate isn't a measure of quality - it's more like a measure of consumed bandwidth.

Re:Who cares about 64 kbps tests?

[Jugalator]

... this is btw very similar to the pitfall MHz = peformance, just applied to sound.

Re:Who cares about 64 kbps tests?

[gleam]

You don't often intentionally hurl your car at 45 miles an hour into a steel box, either, but insurance companies do it all the time to see how well a particular car stands up to the abuse.

Even if you don't knowingly take the insurance institute's results (or federal crash-test ratings) into account, the company selling you insurance does, and your premiums will be higher.

To say "just because i'll never do something this way it has no merit" is silly. Performance in a 45-mile-per-hour offset crash will tell a car company how well it would stand to you accidentally bumping into the corner of your garage, or into the bumper of another car.

Tests like this are important because they're indicative of performance at all bitrates. If you want to know WHICH codec will sound the best at 128kbit, you should look at which codec sounds the best at 64kbit--the two are likely to be the same.

There are two intended audiences for this test: 1) people trying to decide which audio format to use for a stream (which are very often in the 32-64kbit range)

and

2) people who realize these tests can tell us much more than simply which codec performs best at 64kbit, and want to know how to maximize the quality-to-diskspace ratio on their own encodings.

Hope this clears something up for you.

-gleam

Re:Who cares about 64 kbps tests?

[jorleif]

What do you mean by this? 64Kbits is worthless for listening to any music I own while 128 is good enough to not actually annoy me much of the time so why should I be interested in these tests?

I suppose he meant that since at 128Kbps all codecs perform quite well. Therefore personal preferences affect the results too much, i.e. some people like bass boosts and lots of treble, although this is not accurate in the sense that it differs from the original recorded signal.

If 128kbps is 'good enough' for you then I too suppose you would fall into the sounding 'good' instead of sounding 'accurate' category.

Re:Who cares about 64 kbps tests?

[nagora]

If you want to know WHICH codec will sound the best at 128kbit, you should look at which codec sounds the best at 64kbit--the two are likely to be the same.

Any proof of that?

TWW

Re:Who cares about 64 kbps tests?

[nagora]

If 128kbps is 'good enough' for you then I too suppose you would fall into the sounding 'good' instead of sounding 'accurate' category.

If accurate is what you want you don't use any of these: just save raw samples! There is a balance in all compressed formats between good enough and accurate. Anyone reading this test is, by definition, looking for "good" rather than accurate.

TWW

Re:Who cares about 64 kbps tests?

[perlyking]

64 kbps tests are pointless because no-ones uses those rates, and at higher rates the differences become negligable.

Not quite true, I used 64kbps wma files for about a week and listening using really shitty earbuds and walking alongside noisy truck filled roads on the way to work I couldnt tell the difference :-) Now I have some decent portable earbuds though (etymotics) you wont catch me using 64k wma (or I suspect any bitrate of wma!).

Re:Who cares about 64 kbps tests?

[alanjstr]

The whole point of the test, and I say this without reading the article, was that with these codecs, you don't have to code at higher bitrates any more.

Re:Who cares about 64 kbps tests?

[13Echo]

Agreed. Even certain MP3 encoders very quite greatly, depending on the bitrates that are used. BladeENC really blows at less than 256 kbps, but shines at higher bitrates. Xing just blows all the time. LAME just rocks no matter what, and is especially nice with its VBR implementations. FIIS's officialy encoder is the best for lower bitrates.

They aren't always the same.

Re:Who cares about 64 kbps tests?

[Lumpy]

A better test would be to loop an encode. take a piece and encode/decode/encode/decode this will magnify the problems with the codec. granted encoding at a very low bitrate will do this also. but then some codecs (mp3) use different systems for different bitrates. and also the encoder has a really large part of it at least in the mp3 world. lame with the right settings is the absolute best mp3 encoder availabe short of the $25,000.00 hardware unit used by studios and radio station conglomerates. Yes it blows away every pay-for encoder on the market... hands down.

so I do agree that the 64K test is a bit inaccurate, but it was for a reason and unfortunately the testers were not educated enough in what they were testing in order to perform an accurate and meaningful test.

I would grade these results as valuable as that of an 8th grade physics student... there is some info there but nothing that makes a difference.

Re:Who cares about 64 kbps tests?

[karnal]

I somewhat agree.

I've gone over this in my head a little bit, but I think the solution I've found is pretty decent (to my ears). I set the low end rate to 160, and the high end rate to 320, set the quality to 0 (overkill, but hey, why not?) and disable the lowpass filter. My command line looks something like this:

-v -V 0 -b 160 -B 320 -k

if I recall correctly... :) To each his own though.

Re:Who cares about 64 kbps tests?

[orbital3]

I actually haven't done many tests with Ogg 1.0 yet, due to a lack of time, but in the future, when portable devices are available that play Ogg, I really might be interested in ~64 kbps rates. Right now I have a Rio Volt and encode with lame 3.92 set at vbr -q 9. The files usually average a tiny bit under 128kbps. No, they aren't perfect when listened to with a decent sound setup, but they're not real bad.

But in reality, 99% of the reason I bought the Rio Volt was to be able to take one or two CDs in my car and have a decent selection of music. My car has a pretty crappy sound system, and the whole thing is being played through a tape adapter which further mangles the sound, and to be honest, I really can't tell a difference when I'm driving with the noise of the road/wind/AC.

If Ogg made it possible to have 64kbit files at about the same quality as the mp3s I have now, then I could fit twice as much music on one CD, which would put it at around 25+ albums. It'd be fantastic to be able to carry around my entire CD library in one of those 10 or 20 CD cases. Portable devices are where filesize reigns king.

Re:Who cares about 64 kbps tests?

[Anonvmous Coward]

Oh but it is CD quality.

"We took 1,000 people and had them listen to an NSync song off a 64-kbit WMA file, and then off the CD. When asked 'Do you like the song any better?', 999 out of 1,000 people said "No. Can I go now?"

So yes, there is no quality difference between WMA and CD's, it's just a matter of asking the right question.

Re:Who cares about 64 kbps tests?

[Zeinfeld]

Everyone is saying 'you would need golden ears to tell the difference' yadda yadda. In my view the whole test is bogus because we don't have figures for the original CD track.

I would not be at all suprised to see people favor the compressed over the original. The fact is that a lot of so called audiophiles are really pretty ignorant gadget freaks. At university I knew a friend who made money by helping to repackage the components of a bog standard Philips CD player in a pretty box to sell to audiophiles for ten times the price.

I had a so called audiophile witter on for ages about how this was actually quite rational and how using a more stable motor with reduced wow and flutter dramatically improved the sound. He still does not believe that the quartz crystal controls the sound output rate.

Re:Who cares about 64 kbps tests?

[crisco]

Take a look around his site. He's done other listening tests, including two separate tests at 128 kbps, with MPC and AAC highest in the first and MPC and Ogg Vorbis highest in the second.

Note the tests at 128 kbps seem much more difficult to discern a clear winner without resorting to some statistical work.

Re:Who cares about 64 kbps tests?

[Jucius Maximus]

"At those bitrates you'd need a golden ear to tell the difference."

I'd be surprised if most people in the ~20-30 year age range, and especially females would not be able to tell the difference between Jesse Cook - Fall At Your Feet between 256kbit and 340kbit using oggvorbis1.0. The difference is amazing.

Age matters because as you get older, your ability to hear up to 20 khz becomes more and more impaired. Furthermore, the effect is much more rapid and the threshold decreases much more quickly in males than in females. A 60 year old male might only hear up to 14 khz, while being in the aforementioned group myself, I have been tested and can hear 18-19 khz.

Re:Who cares about 64 kbps tests?

[Oculus Habent]

Then they should have had 128kbps MP3 in the mix as well. Until a new study shows (with lots of people - it's hard to negotiate the personal preference differences in a small sample sizes) a comparison of these formats to 128kbps MP3, we don't really have a platform of comparison. They should compare CD-Audio and 128kps MP3 as test standards against which to measure the quality of the formats.

Until there's a comparison to something we can measure difference against, we have no quantitative data. Sure, it's nice to know that you are the 2nd best student in the class, but if the best student is still failing, where does that leave you?

--
Statistics are only as accurate as their intentions.

Re:Who cares about 64 kbps tests?

[nagora]

...in all *lossy* compressed formats, you mean.

Since you mention it, are there any non-lossy compressed audio formats that actually give reliable compression with real music?

TWW

Re:Who cares about 64 kbps tests?

[Skuto]

>Until a new study shows (with lots of people -
>it's hard to negotiate the personal preference
>differences in a small sample sizes)

If you had actually looked at the results, you would have seen that

a) the sample size was enough to draw 95% confidence level conclusions

b) CD audio _was_ included as a reference (5.0)

--
GCP

Re:Who cares about 64 kbps tests?

[Skuto]

>A better test would be to loop an encode. take a
>piece and encode/decode/encode/decode this will
>magnify the problems with the codec.

This isn't a valid way of comparing. The codecs are not designed to reencode music.

A codec may be perfectly transparent after a single encoding, and fall apart when comparing the second encoding with the original. (Think about this: the codec doesn't know what the _original_ signal was like when starting the second encode)

--
GCP

Re:Who cares about 64 kbps tests?

[nagora]

Define "reliable," I guess.

Results in compression at least 75% of the time would be a starting point. It would also need to be a reasonable degree (15%?) before it would be worth the time to encode it.

TWW

Re:Who cares about 64 kbps tests?

[Oculus Habent]

I don't see where the CD Audio is listed - they were all ripped directly from CD, but that is to be expected.

And the 95% confidence ratings:

Each vertical line segment represents the 95% confidence interval (using Tukey's Honestly Significant Difference) for each codec.

One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment.

It has nothing to do with sample size - it's a statement that if the lowest average a codec received was higher that the highest average another received, we can be reasonably assured that it's an issue of quality vs. sample group.

The mean sampling group was 29 (29.25) users. While that doesn't invalidate the findings, it does give you a consideration for the basis.

--
Statistics are only as accurate as their intentions.

Re:Who cares about 64 kbps tests?

[forgoil]

It was a general statement that a codec that is "less good" at the same bitrates can outshine the other if given a huge bitrate advantage. I am sure OGG will still outshine MP3s if both do have 192kbps, but I suspect that I will have a heard time hearing the difference.

I simply wanted to reverse the question somewhat. Instead of "how good at bitrate X" I wanted to know "how high bitrate do I need for quality X". The reason a lossy scheme is used in the first place is space. A 24bit/196khz PCM really kicks ass, but it would burn disk space like nothing else.

The reason for this line of questioning is, if I would go rip my CD collection, how much harddisk do I need to buy to encode it without me hearing that it is no longer the original CDs?

Well, sounding like the original means "I won't hear that it is encoded". I am not an audiophile, but I can hear a distinctive difference between my stationary DVD player and my computer playing WAVE files (through the digital out, I have no idea how this can sound worse), but I can't heard the difference between an 160kbps ogg and the wavefile comming from the computer.

Hmmmm

[cca93014]

Tests confirmed that attempting to encode "Aphex Twin" with any of these codecs caused the PC to tremble at a frequency that, when connected to a refracting laser stuck up Bill Gates' ass, had it spell out "we're all dead" on the nearest wall.

Re:Didnt read the article yet but....

[mccalli]

it seems that music you listen closely too sounds better with WMP, and fast, not listened to music sounds ok w/ ogg.

Well....not quite. There's a different frequency distribution between electronic, pop acoustic and classical music.

Specifically, electronic music, which most dance stuff is, has a very flat frequency distribution. See this for yourself - load your favourite media player, siwtch on the graphic equaliser graph and watch how basically nothing happens except in the mid-range.

Now try again with an orchestral piece. There will be much more variation, though in most it will tend towards the top end.

Now try again with rock. Tends towards the bottom and top, with middle frequencies missing.

Keep going with any format you feel like mentioning...you'll get the same.

Actually, this is a striking example of how recording techniques can ruin sound as well. Take a look at the Apollo 440 album - Gettin' High on Your Own Supply. A good mixture of guitars and electronics, right? Well, look at the frequency graph again. See how virtually every guitar frequency variation has been cut out: this music was recorded digitally, mostly using samples by the looks of it. The normal variations you'd associate with having guitars play live are all filtered out, and the graph goes back to the flat digital sound again.

Cheers,
Ian

Not surprising.

[neksys]

This isn't terribly surprising, as it was already known that the different formats have different frequency responses. More specifically, the way they compress the music dictates what frequencies are cut out. MP3s, for example, are notorious for removing high and low frequencies from music - not a big deal for casual listeners, but those with high-end stereo systems will definitely notice the lack of high overtones, and the "flat" low end bass response. WMA sort of sounds like certain frequencies are cut from the raw audio, leaving the rest to fill in as sort of an approximation of the original full sound - it sounds hollow and "chime-y". Ogg has its defining sound characteristics as well. Thus, it isn't surprising that different styles of music sound better encoded in different formats, as different styles take advantage of different frequencies. Rock music has high frequency cymbals and low frequency bass drums and guitars, as well as a very full mid-range, so a well-rounded encoding system works well. Classical is somewhat more compressed, as a result of the physical limitations in terms of sound reproduction of the instruments, so to the undiscerning ear, a format with especially good mids will suffice. The examples go on and on, but the point is that different tools are needed for different jobs - if nothing else, this study shows that having having a number of encoding tools on hand is actually a good thing. When you look in your tool box, you've got more than a couple Phillips-head screwdrivers - you should have enough tools to deal adequately with any job. The same applies to music.

Should compare Ogg as a single entry

[Russell Coker]

Let's assume that anyone who likes Ogg and is seriously into music will compress their music with both Ogg variants and use the best variant for each file.
Therefore we should also consider taking the best of the two results and comparing it to mp3.
From a quick look at the results it appears that Ogg will still be edged out by mp3 when analysed in such a fashion, but it's much closer.
Also a test on several bit rates would be useful.

Re:Should compare Ogg as a single entry

[Skirwan]

Let's assume that anyone who likes Ogg and is seriously into music will compress their music with both Ogg variants and use the best variant for each file.

Even better, let's assume that anyone who is seriously into music will compress their music with everything and use the best variant for each file.

Kinda defeats the purpose, eh?

--
Damn the Emperor!

Re:Should compare Ogg as a single entry

[Mr_Silver]

Let's assume that anyone who likes Ogg and is seriously into music will compress their music with both Ogg variants and use the best variant for each file.
Therefore we should also consider taking the best of the two results and comparing it to mp3.

No we shouldn't.

You would be chronically distorting the results by merging both Ogg variants. What you would end up with is MP3Pro vs. some super-Ogg thing that doesn't actually exist.

They should be tested seperately. If people are going to do what you suggest then, yes, they will get the best. The problem is that you're making additional assumptions about usage which will serve no other purpose than to boost Ogg's results.

Re:Should compare Ogg as a single entry

[V. Mole]

No, those of us who are seriously into music and audio quality will buy bigger disks and compress with a lossless encoder like flac.

So I guess its time for a new frontend

[CableModemSniper]

I guess grip will have to use Genre info from CDDB to decide what to encode the the files as now. I wonder if you coudl set up something to optimize individual tracks. Like scan a wav and pick the best codec for the frequencies used in the audio.

How about speech?

[GCP]

I'd be interested to know how these codecs perform when streaming things like news or talk radio or foreign language lessons. Clarity at a low bandwidth would support a lot of simultaneous listeners from a low-end server. Clarity at medium bandwidth could provide the extra sound quality needed for something like language learning/practice, again from relatively modest servers.

Re:How about speech?

[yoz]

Take a look at Speex, an open source codec project aimed at speech compression. It uses Ogg for its file format (but don't confuse it with Vorbis). The quality's pretty good already.

MP3PRO not MP3

[Anonymous Coward]

I noticed a number of confused posters here... The tested codecs were AAC/MP3PRO/OGG/WMA, not MP3. Had mp3 been tested, it would have lost every round as all of the tested codecs are vastly superior to plain MP3 at this bitrate.

It also should be noted that the only two samples that WMA beat OGG at (indeed the only ones that it didn't totally flop on) were two very simple samples that are demonstrations of two differnt weaknesses in the current revision of vorbis. Orignally the results page had some very interesting commentary from Monty on this, but it looks like it got pulled.

With the exception of those two samples, OGG clearly won. Even including those, it was only beat out by MP3PRO by a small margin. When you factor in that MP3PRO isn't available at anything but such low bitrates and that it's substantially more propritary then MP3, it seems like pretty much a no-contest.

Re:MP3PRO not MP3

[Mr_Silver]

When you factor in that MP3PRO isn't available at anything but such low bitrates and that it's substantially more propritary then MP3, it seems like pretty much a no-contest.

Except that to get the best quality with Ogg I have to encode in one of two ways depending on the type of music.

With MP3Pro, there is only one.

For the music purists and geeks, this would matter. But for everyone else on the planet they'd rather have one encoding that does well.

It would appear that MP3Pro is pretty close to that.

Problem Is

[BlackGriffen]

that these codecs are lossy, and take advantage of the fact that the human ear is better at hearing certain things than others to pair out extraneous info and improve compression. IOW, it doesn't matter how technically different the new files are as long as they still sound the same to the human ear.

BlackGriffen

Multi-Codec Codec anyone?

[BlackGriffen]

Considering that different codecs do better at different music w/ different frequency spreads, who else thinks that the next generation of audio codecs will be multi-modal; in effect, be several codecs in one. Then have each codec specialize on certain types of music. Perhaps even have them run in an advanced mode where they do a frequency analysis of whole songs, rather than just using genre, to automatically select the best codec for the job. Perhaps even use different codecs for different sections of the song. That would definitely help songs like Bohemian Rhapsody and orchestas with movements, etc.

Would this be too time consuming to implement or what?

BlackGriffen

Re:Multi-Codec Codec anyone?

[John_Booty]

Using different codecs for different parts of a single song would be technically possible, but I don't think it would be desireable.

Different codecs have different sound qualities, because of the different ways in which they discard information in order to achieve their lossy compression. So I think that using different codecs within the same song would result in a pretty wide variation of the way music sounded within a single compressed song...

Re:Maybe A design decision?

[SK-null]

Its not diferent formats. Just diferent encoder options.
The oggq0 entries are for music enconded in a Variable Bit Rate mode (oggenc -q0) -- the encoder defines a quality treshold and uses whatever bitrate necessary to keep it there.
The ogg64 entries are for music encoded with a nominal bitrate (oggenc -b 64 --manage) -- it atemts to keep the bitrate around 64 kbps without looking at sound quality.
Why did only Ogg Vorbis got to show these two modes? Because though the test focuses on 64kbps (nominal bit rate) encoding, its likely than most Ogg Vorbis users will use variable bit rate encoding with it. I know I do.

constant quality variable bit rate

[oliverthered]

I'd like to know if there are and codecs that support constant quality but a variable bit rate?

A codec with a target bitrate of 64k but maintains quality by channing between say 1k(for silence) and 100k through the streem would be nice.

Re:constant quality variable bit rate

[John_Booty]

The Ogg encoder supports this. Also, the LAME mp3 encoded can be configured to do this through some command-line trickery. I don't know about the other encoders; I don't use them.

What about VBR?

[xA40D]

Okay, so everybody has pointed out that a 64k bitrate is useless for their needs. Some claim 128K is the best, others 192K. Personally I prefer a variable bitrate - indeed OGG encodes VBR by default.

But is their some fundamental reason why nobody else insists on VBR?

Re:You may have been sarcastic

[nagora]

But obviously to some people its just about how much music they can hoard away.

My particular problem is that I hate the noise PC CD-ROMS make while spinning at 1x. Since I like to listen to music while working, I make recordings that play without that noise in the background (my PC fan is very quiet). Plus, I can play from a large subset of my CD collection from either machine at home or from my machine at the office.

TWW

Re:Didnt read the article yet but....

[hfranz]

Just a side note about the frequency distribution of different styles of music:

The reason why classical music generally compresses better is because the frequency distribution of the sound of natural instruments like for instance string instruments (including the human voice) is harmonic. This means that the sound spectrum consists mainly of a superposition of peaks at the base frequencies of the instruments played and their corresponding harmonics at higher frequencies.
If you were to make a two dimensional spectral analysis of a such sound recording with the time axis to the right, the frequency to the top and the amplitude as the color intensity of the point you would see a lot of wiggling lines at
regular distances. (BTW: this would make a great visualization plugin for xmms)
Since audio compression algorithms also make such a spectral analysis and after that discard some of the information below a threshold they can
reproduce a mainly harmonic spectrum easier than that found in pop or rock music, which is much more complex and more "noisy" because of the
use of distorting amplification and all kinds of
percussion.

Holger

Outiers skewing the results?

[Outland Traveller]

Looking at the data, it looks the two samples where Ogg performed poorly ended up being encoded at a significantly smaller average bitrate than any of the other encoders.

The table at the end lists LiszBMinor with an average ogg bitrate of 45 and BachS1007 with an average bitrate of 47. Since the other codecs encoded those samples at a bitrate 64 or higher, this may explain the results.

The results may point to a flaw in Ogg's VBR login rather than in the lossy compression scheme.

Re:Outiers skewing the results?

[jonathan_ingram]

If you read the thread on HydrogenAudio (which is the message board where most of these tests / codecs are discussed), you'll find the following information from Monty, the lead developer of Vorbis:

Ogg had a very low bitrate (in the forties) on all the classical samples, which is the way it should have been (Classical solos with their deep noise floors and simple harmonics are relatively easy). But the real reason Ogg scored so low in both (and Beauty Slept as well) was a) the tuning behind noise normalization is still not perfect. This is the very first release of that feature, and the test found flaws b) also the first release of new, more aggressive stereo modes and I think that they too need more analysis infrastructure driving them.

I expect Ogg's performance on Liszt and Bach to be very subpar NN performance. The poor performance on BeautySlept and Waiting was most likely insufficient stereo analysis. Ogg had the infrastructure to win those four samples, but the encoder didn't know how to do it yet (because I didn't know it would be necessary).

Yes, look at the SCORES

[tweakt]

OVERALL RANKINGS (12 SAMPLES)

mp3pro 49.00
oggq0 44.00
ogg64 40.00
wm8 24.00
aac 23.00

The AC above me speaks the truth. mp3PRO has no hope of gaining enough market share to become a worthy competitor. It's a very proprietary extention to MP3. OGG being open source and free (as in beer) has clear advantages for hardware vendors (where it really counts). Lets hope the codec is easy to embed into portable products.

I want my Portable OGG CD Player! I'll buy the first one that comes out. Could you imagine? Twice the capacity of normal players and it STILL sounds better (or same capacity truly indistinguishable from CD -- at only 128k). Right now I have to encode my mp3's at ~180-220kbit to get something acceptable. =/

Re:Yes, look at the SCORES

[micromoog]

mp3PRO has no hope of gaining enough market share to become a worthy competitor. It's a very proprietary extention to MP3.

mp3PRO has one very specific advantage over all the other formats on the market-share front. It has the characters m, p, and 3 in it. Everyone has heard of mp3, and people who don't care about the open source cause (read: the vast majority of people) will buy an mp3PRO device way before considering an Ogg Vorbis device.

As I've said before, name is really important when marketing comes into play. And Ogg Vorbis' name simply blows.

Re:Yes, look at the SCORES

[Greg W.]

Twice the capacity of normal players and it STILL sounds better (or same capacity truly indistinguishable from CD -- at only 128k).

Vorbis 1.0 does sound amazingly good at ca. 128 kbps (VBR -q 4). That's what I've been using lately for CDs that I rip. But it's not "indistinguishable from CD" in all cases. On at least one song ("Feed My Hungry Soul" by the Lords of Acid), I can differentiate Vorbis -q 4 from the original in ABX testing. And I'm not a trained listener, and not using high-end equipment.

I urge everyone to encode for themselves, using their favorite music CDs, and decide what works best for them. Some people are very sensitive to the lossy stereo separation that Vorbis RC3 and 1.0 employ at low-to-mid-bitrate settings. I was able to hear this clearly on several of the samples in the 64kbps test, though I'd never noticed it at higher quality levels.

oggq0's variable bitrate

[cornette]

I haven't seen anyone else mention this yet. At the end, he gives a table of the bitrates for each song for each codec. The one with the greatest variation appears to be oggq0. I noticed that for the songs where that codec did well, the bitrate was much higher, and where it did poorly, it was much lower. I don't realy understand how the bitrate is chosen, but as I understand it, the encoder chooses it automatically somehow, right? I wonder how effective that really is.

Re:oggq0's variable bitrate

[moncyb]

I've tried ogg quality 0. It is the lowest quality setting and it sucks. Use at least level 1.

You are correct in questioning the oggq0. Me thinks comparing a file encoded using a quality setting with one using a fixed bitrate setting is like comparing apples and oranges. I didn't check (as this seems to be a MS Windows only test), but I assume they used sample rates of 44100 Hz. I think they should've thrown in some 22050 Hz samples and tested those. I think encoding at 64 bits, the lower sample rates would perform better--that's how I encode many of my oggs...

Hopefully the ogg people will find a way to make the quality settings work well--it seems to be a new feature.

2 files were actually 45 and 49 kbits/sec

[Alsee]

The 60 files (12 songs * 5 formats) were all compressed at between 64 and 74 kbits/sec - except LiszBMinor and BachS1007 for OGG q0. They were actually stored at 45 and 49 kbits/sec respectively. No surprise the testers rated them low.

-

Re:No Control

[Anonymous Coward]

Here, there is no control - crappy experiment.

The participants could have just been scoring on "this is different to the unencoded track, therefore it must be worse".

So put a copy of the unencoded track as a test track and see if it gets marked down (and also, of course do NOT tell the participants that it is there).

Umm.. did you bother to read about the testing methodology before coming to your grand conclusion here?

ABC/HR.. as in ABC/Hidden Reference... as in, there is a copy of the original track included as a hidden reference on every single trial.

The users are given 2 sliders per sample laid out on a panel. The samples are loaded in random order. On the sliders for each sample, one slider is for the original sample, and one is for the encoded. These are also randomized per sample. The user does not know which is which. If they happen to rate the original sample less than 5.0 (highest rating, meaning it should be transparent), then their results are disregarded entirely for that sample.

Re:Sound test

[jonathan_ingram]

Man did you guys even do the test or are you just looking at the results?

Yes, all 12 samples, and my results broadly agree with the general conclusions (particularly on MP3pro doing much better than AAC and WMA on almost all the samples).

The only ones I had trouble with were the two oggs and on some files the mp3pro

In general, the results agree with you, so I don't understand why you're so annoyed by the test. When Vorbis was good, it was *very* good - and when it was bad, it was terrible. Luckily, Monty will use the results to help the 64kbits performance for the next version of Vorbis.

(OT: you *really* need to learn about paragraphs)

No CD Audio comparison?

[Disco2k]

Does anyone else feel it would have been nice to see Red Book CD audio (16-bit 44.1KHz uncompressed) compared as a control? Seeing how "pure" audio compares to these compression standards could make the results seem more objective.

Re:No CD Audio comparison?

[jonathan_ingram]

Comparing 'pure audio' to the compressed version is inbuilt into the test methodology. I suggest you download the program used for the tests, and try it out before commenting further. Hopefully this will give you some semblance of knowing what you're talking about.

Re:Stats note: 95% confidence level

[jonathan_ingram]

Plug the results into the linked non-parametric stats analysis page, and you find:

mp3pro is better than wma8 (99.9% confidence)
mp3pro is better than aac (99.9% confidence)
oggq0 is better than aac (99.3% confidence)
oggq0 is better than wma8 (99% confidence)
ogg64 is better than aac (97.2% confidence)
ogg64 is better than wma8 (96.1% confidence)

Is 99% good enough for you? Or perhaps you should just take the two at 99.9%?

Dammit, the lameness filter is kicking in. No, these are *not* junk characters - I'm trying to show the peon some useful statistical information, you worthless piece of software. I've already removed all the hyphens, what the hell more do you want me to do? Is a percentage sign 'junk'? Is a question mark? Is a space? What the *fuck* use is this, when it doesn't stop all the crapflooders in any way whatsoever - they just flood with random gay/incest/beastiality sex stories instead... I've been posting on this site for years, and for my sins haven't crap-flooded once - give me a LITTLE FUCKING LICENCE TO POST MATERIAL.

Ogg on Quicktime

[heroine]

Been using Ogg/Vorbis/Squish on Quicktime for a year. The Ogg/Vorbis/Squish codec got much better between 1.0rc2 and 1.0. At 128k it's already better than mp3 and the managed bitrate encoding is faster than the hard drive can read. The real value is of course, the ability to read these encoded files as long as there is UNIX. Mp3 is going to die and when it does there won't be any appliance makers interested in paying the $10,000 royalty to support mp3.

Re:No Control

[jonathan_ingram]

Please moderate the parent up - unlike the grandparent, this anonymous coward actually knows what he is talking about.

Re:You may have been sarcastic

[zbuffered]

Convenience, convenience, convenience. I've got an MP3 CD in my car that's been in there for nearly a month. It's got all my current favorite tracks/albums on it. When I get tired of it, I'll burn another CD that may last as long. Also, lossy compression is very acceptable for mediums such as speeches, books on ta--MP3, etc. I've got a road trip coming up, and I'm planning on listening to the Lord of the Rings Trilogy that I borrowed from a friend. At 64kbps, I can fit something like 21 hours of audio on one CD. As opposed to the 18 or so CDs that it would take if it were pure CD-Audio. You don't have to get rid of CDs, but MP3s have their place, and they're not going away. Lossy compression or not, you can make an MP3 sound as good as CD Audio. You can. Just turn up the bitrate, play with the settings. Use lossless compression if you want, but I guarantee you can rip an MP3 of almost any track that you won't be able to tell from a ripped wav file from the same disc. For me, the quality of the original sample is usually the smallest factor in the imperfections of the audio. Usually, static from the amplifier, and/or inadequate powering of the speakers, and/or crappy speakers, and/or background noise, are the biggest factors. If I were an audiophile freak, I wouldn't rip MP3, but I'm not. I live in the real world. MP3's imperfections are the least of my audio-related problems out here.

FLAC

[Graspee_Leemoor]

I use flac (lossless compression). What was the problem again ? ;)

Seriously, the reason I use flac, even though it takes up a shit load of space is that in the future, inevitably we will have more space to store everything. When we do, thousands will be cursing their crappy mp3s that they ripped at 128 to save space.

Of course, ahem, if you kept your original CDs to rip from then you can just re-rip them to flac or another lossless compression then, but still, why do it all twice ?

graspee

Am I a freak??

[squaretorus]

Personally, I can hardly tell the difference between MP3 and CD and my old vinyl. When I play something on CD, then MP3, then CD and listen carefully for the 'crappy bits' I can hear them - but they don't bother me in the slightest when listening to them.

Are my ears just a bit shite? Are most of you guys able to tell the difference - or are the audiophiles just more vocal?

Re:Am I a freak??

[squaretorus]

On a more personal note, I'll take vinyl over the alternatives any day of the week!


Ahhh - respect! You must be pretty knowledgeable to chose the oldest format as your favourite... ;-)

Re:Poor Steve Jobs...

[Greg W.]

AAC's performance at 64 kbps is not necessarily indicative of its performance at 128 kbps.

I dont see how it matters...

[NeoSkandranon]

I think, that for anyone who would actually be interested in which codec does best on which kind of music, it's a moot point, since by now they delete anything below 128kbps on sight

Re:I dont see how it matters...

[Skuto]

>I think, that for anyone who would actually be
>interested in which codec does best on which
>kind of music, it's a moot point, since by now
>they delete anything below 128kbps on sight

One word: Streaming

--
GCP

Opeth - Blackwater Park

[MicroBerto]

I am truly impressed with the selection of music on this test.

For those of you that don't know, Opeth's Blackwater Park is one of the most earth-shattering CDs I've ever been privy to witness. They are my favorite band. Check out the last song, Blackwater Park. Wow.

You can get a taste of them in #mp3_metal or #mp3_death in dalnet. type @locator opeth blackwater park and you'll get plenty of results.

Caution - very harsh grunting vocals. May take some time to get used to, but their musicianship is absolutely brilliant.

Newer algorithms of limited use to many

[dh003i]

To thsoe of us who just want to listen to music on a PC, the newest greatest best algorithms are always good (mp3pro, oggs, wma8). But for many, the goal is to put that music on a MP3Player and listen to it anywhere. I'll summarize the support of these various codecs by MP3Players, as well as mention whether or not my MP3Player (RioVolt SP100) supports them.

MP3PRO -- little support on MP3Players. Not supported by RioVolt SP100.

Oggs -- little/no support on MP3players. Not supported by RioVolt SP100.

WMA8 -- little support on MP3players, though many support older WMA's. Not supported by RioVolt.

So, in summary, all of these new formats are completely useless to me on my MP3Player. The one option they present is if I want to encode something in two formats -- one for my computer, and another for the MP3Player.

Personally, I think more work should go into fractal endcoding, as most music has fractal patterns in it (especially Bach's music).

Re:Newer algorithms of limited use to many

[jeeptj]

Please define little support since WMA8 is supported by a bunch of devices less than 2 years old through a firmware upgrade. I've tried looking for a RioVolt that didn't support and the only ones were the Coke brand and the SP50. So there you go, the vast majority of players support both MP3 and WMA.

Re:Newer algorithms of limited use to many

[dh003i]

Duh, they all support MP3. But few yet support MP3Pro. Even if they did support MP3Pro, that's also of limited use as we only have access to 64kbps compression w/ MP3Pro -- which sucks.

NONE support OGG that I can think of. OGG is the best format I've come accross -- and I'm speaking from listening experience here.

As for WMA8 -- so what? WMA8 sucks anyways. I'll be the first one to tell you I gave it a try, but after trying it out, I realize that for the most part its crap: like everything else produced by M$.

Re:Newer algorithms of limited use to many

[afidel]

I agree on WMA8, except for streaming applications it kicks plain mp3's arse. I did my own tripple blind testing WMA8@160, lame mp3@r3mix vbr avg 210, and the raw 16bit 44.1khz PCM wave. Of the three the only one I could pick out was WMA8, it sound ranged from noticibly different to such suckily bright highs that I literally ripped the headphones off my head. Since right now there are only 3 formats that have been used for streaming in any volume, mp3, wma, and ogg and ogg is the best according to this and other tests at low bitrate I will stick to high quality vbr mp3's for my ipod and pc and encourage ogg for streaming applications.

Re:No Control

[SloppyElvis]

The parent of all of this was certainly in err, but this still isn't a very good experiment. Look at the number of testers that were used. Most tests numbered in the 30s with respect to the number of subjects. While that number may be sufficient for small sample statistical tests, it is not a sufficient sample to test for such a normative value across the human population, such as judging music quality represents. Having achieved a small variance of opinion must not be determined to prove that the sample size was large enough to account for variance in opinion for the greater population, and while these tests are interesting, they are incomplete IMHO.

Re:This study tells nothing

[Greg W.]

I am Karma Man, hear me Whore.

An honest double-blind listening test is extremely difficult to arrange, and there is no evidence whatsoever on such on the site.

This is how the test was conducted.

The test required access to a Windows machine (probably Win95 and up, didn't try with Win3.1) with a sound card. Users were required to download the ABC/HR "practice" Zip file, which includes the ABC/HR program, the Ogg Vorbis 1.0 command-line encoder and decoder, a LAME command-line encoder/decoder (I forget which version), a FLAC command-line decoder program, and a .flac sample file (the instrumental introduction to The Eagles' "New Kid in Town").

After unzipping this, the user had to run a batch file (encdec_foobar.bat) which un-FLACced the sample file, then encoded it with Ogg Vorbis and LAME, then decoded both of the resulting files back to .wav.

Then the user executed the ABC/HR program, which is a Win32 GUI application. After loading the sample into the application (pull-down menu and file selector dialog), the interface became a row of double-slider pairs. Below each slider was a "Play" button. Below each slider pair was a "Play Ref" button. Below that was a "Stop" button. There was a pair of sliders for each decoded sample -- so for the practice run, there were two pairs of sliders: one for file #1, and one for file #2. The user did not know which file was Ogg Vorbis, and which was LAME MP3.

The user then listened to the Reference file by clicking any of the "Play Ref" buttons. After hearing the Reference, the user could then click any of the normal "Play" buttons. The first task was to determine, for each pair of sliders, which one was the original and which one was the encoded file. Having determined that, the user used the slider (which went from 1.0 to 5.0 in increments of 0.1) to "score" the sample on the subjective quality of the result. There were also text labels on the slider: 4.0 was "perceptible but not annoying", 3.0 was "slightly annoying", 2.0 was "annoying" and 1.0 was "very annoying".

Finally, there was an ABX button, which launched a different window. In the ABX window, the user could select "Original", "Sample 1", or "Sample 2" for the "A" and "B" samples. Normal ABX testing proceeded from that point. (If you don't know what ABX is, go to pcabx.com.) I found that the ABX window sometimes helped me to focus on a specific sample so that I could find its flaws; armed with that knowledge, I was able to make a determination of which of the two sliders, right or left, was the encoded version.

Once a slider was pulled down from the default 5.0 position, another button became active above that slider. Clicking on it opened a new window with a text box, into which comments could be typed. When the user was finished with the test, the slider positions, the comments, and the ABX results (if any) were written to a plain text file (DOS CR/LF format), which was to be mailed to the test administrator. (Though, of course, you weren't supposed to mail the practice results.)

Now, that was just the practice session, which was a prerequisite for participation in the actual test. For the actual test, the process was similar, but differed in a few details.

The actual test samples included copyrighted, patented codecs for which there are no freely distributable decoders. Therefore, the WMA, AAC and MP3Pro samples were distributed as FLAC files, and decoded by the batch file. Since MP3 did not participate in the listening test, the LAME encoder was not used during the actual test. The Vorbis encoder, of course, was used twice: first with -q 0, and then with -b 64 --managed.

With 5 encodings per audio sample in the actual test, there were 5 pairs of active sliders instead of only 2 pairs. But otherwise, the actual test was exactly like the practice session.

(Personal note: I did 10 of the 12 samples, skipping the two classical ones. Out of 50 encoded versions of the 10 samples, there was only one case where I couldn't tell right from left -- "The Source", encoded with MP3Pro.)

Re:You may have been sarcastic

[marmoset]

I cycle about 100 miles a week. My bike is probably where I end up listening to the most music. A good portable player (I use an iPod, but this would apply to some others as well) beats the crap out of a discman in this situation, because of the vastly superior portability. The iPod is about a third of the size of any portable CD player except for those useless 3-inch models, has an effectively infinite anti-skip buffer, and requires hauling around no bulky media.

As for theft, since the iPod is only the size of a cigarette pack, there's no reason to leave it in the car to be stolen when you go inside somewhere -- that's what pockets are for, chief.

Re:Poor Steve Jobs...

[cryptochrome]

or 80 and 96, comparable to 160 and 192 for mp3s respectively. Any lossy codec is going to break down at some point. The question is where and how quickly? One way is to design a codec to perform as well as it can at some bitrate, determined by needs. But for music, when your goal is to have audio that is virtually indistinguishable from the original rather than merely audible, the goal is for it to be as indistinguishable as possible for as low a bitrate as possible (which means you have tradeoffs). Below that, who cares? A codec that can do that is not necessarily going to scale properly to very low bitrates. And I have yet to hear of any codec that can handle near-indistinguishable music at 64.

Re:No Control

[nedron]

I have to agree. Not showing the rating of the control in the charts makes the chart pointless. It would be interesting to see how the user ratings for the control line up against the codecs.

It's possible that the data from the control showed that the listener preference had little to do with how the rated the codec reproduced the original.

Re:Ogg Vorbis

[Salsaman]

Isn't mp3pro limited to 64K ? Why would you want to use that, when you could create 128K ogg files that were much higher quality ?

It looked to me from the results that mp3pro performed about equal with ogg at 64K, but would you really want to play 64K files in your car ?

"LIVE" format is best...

[turnstyle]

Just worth mentioning that the single best audio format is "LIVE" - it sounds so good, it'll seem that you can reach out and touch the performers (please don't).

I know that the thread is about compression formats, but hey - go to a bar/club with "LIVE" music, pay $10 at the door, have a drink and a good time.

Hopefully, the guys playing are getting a percent of the door, and they'll be happy to see you in the audience. Feel that bass!

For Detailed Audio Discussion...

[aligas]

Check out Hydrogen Audio

Its pretty much the best audio discussion you can find on the 'net.

Rapida Carrera

[Mulletproof]

Apollo 440 is actually a prety damn good test bed for this stuff, especially (in my opinion) the Rapid Racer theme. Vocals, guitars, high frequency synth and ultra-low "theta-bass". Of course, you could have just used one of those Dynamic Frequancy test CDs, but what fun would that have been?

This is NOT when you expose car flaws

[ColGraff]

Running any car over a cliff will destroy it, regardless of safety systems. This is not a valid test of a car safety system.

Re:This is NOT when you expose car flaws

[nagora]

Running any car over a cliff will destroy it, regardless of safety systems. This is not a valid test of a car safety system.

Playing music at 64Kb will sound crap, this was not a valid guide to what is a good audio codec!

TWW

Re:You may have been sarcastic

[afidel]

Depends where he is cycling, around me we have some bike paths that run along old railroad lines. Outside the very popular parks where there are large numbers of parking spots for cars there aren't many people. In fact some of them are remote enough I have had complete 5 miles hikes where I have seen 2-3 people the whole 2 hours. At least around me most people who are serious about hiking or biking do it to get away from people, life etc as much as the exercise.

Re:OMG get it right

[big.ears]

Whatever. You claim vorbis is an audio codec and ogg a bitstream manager, but you don't tell us what the sound format is. Why is the executable called oggenc (ogg encoder) if ogg is not a sound format? Why are the files name .ogg by default if ogg is not a sound format? Why don't you go troll on the xiph mailing lists, because it appears that they are the ones who have perpetuated this.

Test compares codecs, not formats

[benwaggoner]

This is an interesting and relatively well done test (although it appears that the listeners knew which format they were listening to, so it wasn't truly double-blind, and a anti-MS and pro-Ogg bias can't be ruled out).

However, some discussions seem to be focusing on this saying AAC is bad or WMA is bad, when really it refers to the particular implementations in codecs of those formats.

For example, the Apple MPEG-4 AAC-LC encoder was used for AAC. This is a Low Complexity version of the format. Also, the Apple encoder has a strange limitation where it automatically converts 44.1 stereo to 32 stereo at that data rate. This isn't required by the AAC format. Other AAC encoders yield MUCH better results, and beat MP3 Pro in double-blind testing. I haven't seen any double-blind comparisons between AAC and Ogg.

Also, the WMA8 encoder is due to be replaced by the backwards-compatible WMA9 in early September. Of course, there may well be improved versions of the other encoders by then as well.

Re:Test compares codecs, not formats

[ff123]

I had several obvious choices for the AAC encoder: Psytel, the Quicktime, and Liquid Audio 5 (I hadn't looked into LA6). Liquid Audio 5 is another FhG low complexity encoder, but lowpasses at a lower frequency than the Quicktime. The Psytel encoder is worse-sounding than the Quicktime at 64 kbit/s. I did try to choose the best AAC implementation available to me (I do not have access to the latest and greatest implementations).

It's possible I could have set up the experiment as a Latin Square, and randomized which codecs any individual was comparing, but my home-grown statistical tools are not up to that task. That is, I can only perform balanced analyses, where N is the same for every codec.

ff123

Re:Test compares codecs, not formats

[Skuto]

>This is an interesting and relatively well done
>test (although it appears that the listeners
>knew which format they were listening to, so it
>wasn't truly double-blind, and a anti-MS and pro-
>Ogg bias can't be ruled out).

The test *was* double-blind. Some of the listeners had experience with some of the codecs involved, and were able to identify some codecs based on the flaws they heard.

--
GCP

Re:Test compares codecs, not formats

[benwaggoner]

Indeed. I wasn't critiquing your study, but pointing out that some folks were making the wrong interpretation of its subject.

How about uncompressed...?

[Rui del-Negro]

I'm sorry if this has been said before, but I couldn't find it:

Why didn't they also try playing the original, uncompressed music, to see how high it scored...?

RMN
~~~

Vinyl for dance

[Tokerat]

mmm the warm bass... only type of music i ever noticed a vinyl difference was for music with loud or enhanced bass range... it just seems "warmer" in vinyl.

I post, therefore I am. ;-)(

Re:This study tells nothing

[Skuto]

>Basically, after the first listening, you have
>already instinctively 'decided' how the sample
>sounds.

If this were true, it would never have been possible to make conclusions with 95% certainty, and there would have been no correlation between the results of the listeners.

Both are false, and hence, your unsubtantiated claim.

>How many times were the tests repeated?

One test per listener per sample.

>Was there a control group?

There was a control group in the form of 'trusted' listeners also participating in the tests. The results were similar to the general ones.

>Also, there is no information on the sound
>system used and no measurements on its effect on
>the test.

If you had bothered to actually look, you would see that each tester used the system he generally uses to listen to music on his computer, whatever that was.

>My suspicion is, if the test were repeated, the
>result would be completely different.

There is nothing that supports this claim and past tests contradict it.

--
GCP

Re:Those two oddball tests

[Skuto]

>I do not know why Vorbis tried to compress those
>two so much more than the others.

Vorbis with the -q mode tries to maintain a certain quality level. The other codecs try to maintain a certain bitrate.

On an easy clip, Vorbis uses less bits because it can do so and keep reasonable quality. The other codecs use more bits because they try to reach the certain bitrate.

--
GCP

No worse than "Em Pee Three"

[yerricde]

It would not matter if ogg was capable of reproducing music at any bitrate, no body will care with a name like that.

It would not matter if mp3 was capable of reproducing music at any bitrate, no body will care with a name like that. Heck, the middle syllable sounds like something you do in the bathroom.

The name of a product can make or break its success in the consumer world

Coca-Cola sells, even though the first half of its name is the first half of "cocaine". Trust me, any name can be promoted by the entertainment media.

As it is now, [the name of Ogg technology is] based on two bits of extremely nerdy trivia

And "Motion Picture Experts Group Audio Layer 3" isn't?

It has one big disadvantage at the moment

[hayden]

No integer only codec. This means no portable player support as they generally don't have a FPU.

Re:Didnt read the article yet but....

[cornflux]

The first thing I thought of when you were describing your 2-D representation is the The Music Animation Machine:

The Music Animation Machine display is a score without any measures or clefs, in which information about the music's structure is conveyed with bars of color representing the notes. These bars scroll across the screen as the music plays. Their position on the screen tells you their pitch and their timing in relation to each other. Different colors denote different instruments or voices, thematic material, or tonality. And each note lights up at the exact moment it sounds, so you can't lose your place.

As someone who knows little-to-nothing about the technical aspects of music, I find this relatively fascinating.