Vorbis And Musepack Win 128kbps Multiformat Test

technology is sexy writes "After 11 days of collecting results Roberto Amorim today announced the results of his 2nd Multi-Format listening test: Vorbis fork AoTuV scored the highest and ranks as the winner together with open source contender Musepack closely followed by Apple's AAC implementation and LAME MP3, which improved markably since last year thanks to further tunings of its VBR model done by Gabriel Bouvigne. Sony's ATRAC3 format ranks last after WMA on the third place. The suprising success of AoTuV (compared to last year's performance of Xiph.org's reference implementation) shows the potential of Vorbis and possible room for further tuning and improvments. Take a look at the detailed results and their discussion at Hydrogenaudio.org."

Comments

Striving for innovation

[MrIrwin]

So given Microsofts stated goal to bring us innovative technology, they should throw in the towel and ship OggVorbis and derivatives with Windows, right?

But does it matter?

[Gothmolly]

When everyone gets an iPod, dood, or the WinFooTunes player that you get with your Dell only works with WMA, or your in-dash CD player only groks 128kbps MP3s, whats the practical application of the other codecs? It's nice that we propeller-heads on Slashdot can smirk while we rip everything to FLAC and write custom Perl apps to transcode-on-the-fly to our wireless enabled MythTV box, but for John Q. Drone^H^H^H^H^HConsumer, none of this matters.

So how do we get the word out? How do we start the revolution? Open-Source hardware?

mp3 still defacto standard

[Random Web Developer]

No matter what researchers find the best format, the best format for users is what they can doubleclick to play, use on their el-cheapo portable mp3 player or whatever music device they own.

This might be of interest to musicians but the proverbial "jane doe" will keep using mp3 for quite a while

Re:mp3 still defacto standard

[Roland Piquepaille]

This might be of interest to musicians but the proverbial "jane doe" will keep using mp3 for quite a while

Actually it's not that simple. Jane and Joe Doe will start using Ogg, AoTuV or other TLA and ETLA compression schemes when their favorite music players feature them. In the case of Ogg, it's not going to happen anytime soom because:

1 - There's an entrenched MP3 market, as you said

2 - It's an open-source format, i.e. it reeks of piracy and hackers in the minds of music player manufacturers and of the public

3 - It doesn't have the backing of major industry players, being seen as a "maverick" effort to undermine other potentially money-making closed-source formats

4 - It certainly doesn't have the backing of the RIAA, because it doesn't have DRM and other in-the-customer's-face copyright protection schemes

In short, people using Ogg will be opensource-aware and advocates for a long time to come. As for other Apple customer-unfriendly sort of schemes, I'm not convinced the general populace has bought into the idea of paying for music tracks that can become unplayable at the next Apple format-change-du-jour, because they're copyright-protected and therefore impossible to convert to another standard (in theory).

So yes, you're right, MP3 will stay around for a long time. I certainly won't convert my collection anytime soon...

Re:mp3 still defacto standard

[Zilch]

2 - It's an open-source format, i.e. it reeks of piracy and hackers in the minds of music player manufacturers and of the public

Yeah - because piracy and hackers didn't have a hand in making MP3 popular.

Zilch

best vs popular

[trs9000]

i realize the geeks of the populace want the highest quality encoding to win. naturally. and it helps when something such as vorbis is rated so highly; it gives it even more geek cred.

however: as someone who studied music and audio, i am constantly surprised at what people will listen to. my friends (well some of them) have no problem cranking low quality mp3s of 50 cent, while i drop my jaw at the poor audio quality as a result of lost information. one time i even remarked to my dad "oh its an mp3" when he was playing something i had given to him which had been apparently later encoded. he wasnt sure (he didnt do the encoding) but doublechecked and yes it was mp3 (probably 160 kbps). he was impressed, when to me the timbral change in the cymbals was a dead giveaway. another time i asked a friend of mine if he was using aac to import all his cds in to itunes when he had been recently doing so. he looked at me blankly and said "whats aac?". which meant, yes he was.

i apologize for rambling, this is what im arriving at:
despite early adoption influence etc that geeks hold, how much does all of this really matter. most people dont care what format its in as long as they can listen to it. and often they cant discern loss of quality unless its extreme. so while i applaud these efforts, im simply wondering if -- aside from research -- they arent futile.

How much of this is just OGG fans voting?

[DrewBeavis]

I read some of the results, and I'm not a Vorbis hater or anything, but how much of this is open source fans voting for their favorite codec? I looked at the test just now, but can't tell if it was blind or not.

Re:How much of this is just OGG fans voting?

[technology is sexy]

It was a double blind test (ABC/HR) adhering to ITU-R BS.1116-1. Read more about the methodology in the initial announcement.
In addition to being double blind results were also encrypted so manipulation is very unlikely.

Re:Good.

[m0rbidini]

OK, you have a point. But check VorbisHardware for hardware with Ogg Vorbis support. Also, though Lame did well, MP3 is known to have some limitations. But if you have to use MP3, experiment --alt-preset standard in Lame. It was made to offer very good sound quality in bitrates that average below 200 kbps in most cases.

Regarding the results... It's a bit surprising that this third party tuning/tweak of Vorbis did so well. Which is great and I think Xiph should think about incorporating this work on their official encoder as soon as possible, in order to take advantage of its potential. You may be surprised about the relative low performance of AAC. This is partially due to the fact that the chosen AAC encoder was a CBR only encoder (because it was the best AAC encoder at this bitrate on a previous test - Nero encoder is also a good one and offers VBR encoder). With a good implementation of VBR AAC, it should be possible to get a better performance.

While most of the tested codecs/formats showed good performance at 128 kbps, this test alone shows that none can give transparency ( transparency == unability to distinct from the original source for most people and under good conditions) at this bitrate, contrary to what many think. People who think this is important should demand higher quality files from famous online music services (like iTunes Music Store).

People interested in lossy audio encoding should also try Musepack (file extension .mpc). It is considered by many of the hydrogenaudio enthusiasts as the best format at medium/high bitrates, offering transparency with bitrates normally lower (with standard preset ~170 kbps, typical 142 ... 184 kbps) than what is possible with other formats/codecs. It's now open source (LGPL, iirc). Its biggest disadvantage is the lack of support in portable players (though decoding musepack is faster than decoding the other formats in this test). There are plugins for almost every software player and foobar2000 (which I consider the best one) has native support for it

Blind testing? And "Best sound" or "Accuracy"?

[turnstyle]

Of course it matters! I took a quick look, but I didn't seem to find anything describing how the tests were performed.

IMHO, the best way to test is to provide an uncompressed source and a variety of compressed files, and ask "which most closely matches the uncompressed source" -- and NOT "which sounds best."

Years ago, I did an a/b switch test with a high-end audio engineer between a CD and a 128kbit/s MP3. Though we could both clearly hear a difference, he actually guessed wrong.

My point is: the test needs to be blind, and the test should be looking for compressed files that most closely sound like the uncompressed original -- and not the ones that "sound best."

Vorbis and MP3 formats from a techie view point

[tiger_omega]

Having designed and written a mp3 decoder and now working on a vorbis decoder I can't say I'm that suprised by vorbis coming out on top.

From a technological standpoint the Vorbis codec has 10 years of audio compression R&D in it since MP3 was invented.

MP3 is a subband DCT based codec using fixed window length. Vorbis is also DCT based but encodes an approximation to the orginal frame's spectral curve and also uses variable length window length.

In using the source from the vorbis library and the decoder specification to help guide its development I have to say it is a real joy to code. The people at xiph.org have really done a first class job and have approached some of the problems of audio codec design with some of the best lateral thinking that I have ever seen.

Believe me! Coming from me that is very rare praise.