Slashdot Mirror
Lossy Music Formats Compared
Nicholas writes: "Today's Washington Post has an article
detailing the results of having a "a diverse panel of listeners: two members of the National Symphony Orchestra, a high-end stereo salesman, a record producer, a composer and two guitarists" comparing MP3Pro and Vorbis formats. The punchline: "...felt Vorbis was the least realistic, with MP3Pro sounding better and Windows Media Audio best of all -- but none of these formats achieved CD quality.""
It still doesn't matter to me. If I could listen to WMA on my linux system(s) I would. If I could use WMA on my car mp3-cd player, I would.
I can't though, so it doesn't matter. I'm not a musician by any means, nor can I detect the difference between 160 and 192 mp3 compression. So I'll continue using my inferior, yet cross platform, non-license restricted, used-everwhere, mp3 format.
If you get serious with dither and error-feedback noise shaping, you can get whatever quality you like, including a characteristic very reminiscent of SACD, all from 16 bit audio CD.
Most CDs out there DON'T have acceptable dithering and the new ones that do, are usually crap in other ways!
I've written dithering that will literally resolve signals -156db down from CD audio, as long as they are bassy frequencies- the noise level increases with frequency like in SACD. Charts and details at http://www.airwindows.com/dithering/TechDetails.ht ml, the software itself is at http://www.airwindows.com/dithering/index.html.
This is not Linux software: it is Mac software written in an easier language than C. But it IS GPLed, and that's not an accident.
CD doesn't have to mean rotten quality, and this is an analog freak saying this. CD just has _historically_ meant crap quality, just as chrysalis describes. What's being described is truncation and bad word length maintenance, particularly if it involves cheap-ass DAWs or digital mixers.
You don't have to do that- though you're not likely to see anything but that from the major labels. They've been using cheaper and cheaper equipment for years, the latest trend is for everything to be all Pro Tools, and the poor mastering engineers are stuck trying to get a musical result out of that...
Combine that with no mention of the bitrates used and you essentially have a fluff piece.
I've only seen one test that might be called rigorous and that was a few years back. Does anyone know of quality listening comparisons between various codecs at various bitrates? Blind testing (Double Blind?), listeners with some musical background (musicians, recording engineers, even audiophiles), a wide range of music, maybe even different sound systems? I'm curious, almost (but not quite, I'm also lazy) enough to round up some friends and do my own sorry version of the test.
Chris Cothrun
Curator of Chaos
Bleh!
The editor used compression on the story.
--
--
Mod up a post Rob doesn't like and you'll never mod again
Exactly.
My suspicion - given the preference for MP3Pro over MP3 - is that it was MP3 at 64kbps vs. MP3Pro at 64kbps.
Even makes sense, after all. Including WMA in the study at all is weird -- and if they encoded the WMA and the MP3 using MSFT's built-in MP3 encoder, they'd have gotten the shittiest-sounding 64k MP3 on the planet.
MP3Pro is designed for streaming audio, and outperforms MP3 at extremely low bitrates (i.e. 64K).
The part in the article where the guitarist said he could hear the valves closing on the clarinet and bassoon... is utter bullshit. He probably heard some flanging artifact and thought it was a valve closure.
Finally - and this is another biggie - for the MP3 section, did they use Blade, LAME, Fraun, or Xing? Some rock on some forms of music, but utterly sux0r on others. And some encoders sound like dog shit on all forms of music.
Sounds to me like a puff piece designed to get people out of (ubiquitous) MP3 and into (proprietary, Windoze-only) MP3Pro and (DRM-encumbered and proprietary) WMA, with Ogg Vorbis thrown in for a "What's that?" appeal.
(Credit to them grokking that lossy compression in music often throws away stereo separation / spatial components, though.)
This study varies significantly from the study done in Heise link here . That study concluded that high quality mp3s were indistinguishable from CD recordings !!
I can only conclude that one of these two sets of tests were biased or did something wrong. They can't both be right and the quality of digital recordings can't decrease with time !
I spoke to the reporter who did these 'listening tests'... the tests were not structured, were not controlled, were not blind or double blind... in short, the reviewers knew exactly which codec sample they were listening to at all times. The reviewers also came into the test with already formed opinions about which codec was best and-- surprise--- the test seemed to confirm what they already 'knew'.
In ABX blind testing, Vorbis usually surprises the tester, for example, from our happy maniac friends at r3mix forum (MPEGplus and Vorbis won this ABX test first and second place over mp3pro, AAC, WMA8, TwinVQ and ATRAC3):
That is, he knew for sure... incorrectly. A brilliant example of the power of suggestion.In any case, after talking to the Post reporter he feels a little sheepish about the whole thing... he thought he balanced the article by mixing positive traits of the openness of the code with a critical quality review and has agreed to be more fair to the first 1.0 encoder release candidate.
Monty
If it were really diverse, they wouldn't just have music professionals on the board. They'd have a deaf person, some giggling schoolgirls, a super-intelligent killbot, and maybe a monkey.
Here's an interesting web site that lets you do the comparisons:
x .h tm
h tm
www.pcabx.com
Also see
www.pcavtech.com
for some results
and
oeonline.com/~djcarlst/abx.htm
for explanation of ABX in general.
This url:
http://www.pcabx.com/product/coder_decoder/inde
contains samples to compare FOR YOURSELF several codec variations.
This url:
http://www.pcavtech.com/play-rec/summary/index.
contains some results... Apologies for not posting with html...
Mark
Apart from the fact that, as other posters have noted, there's no information given on the bit rates or encoders used, the article straight-out admits that "the test sessions were done in a home environment with an ordinary stereo system".
Which, I'll warrant, means no double blinding and no level matching. Probably not even single blinding (where the testers know what's being listened to, but the testees don't). Level matching is essential; different encoding methods may play back at slightly different levels, and just turning up the volume a tad will convince a large proportion of casual or professional listeners that something's improved in the sound.
Without a proper scientific test, psychoacoustic effects can swamp even quite large real differences in sound, and can cause people to hear quite large differences that they wouldn't hear if they didn't know when they were listening to what.
As can trivially be demonstrated, when you look at the number of golden-eared but scientifically ignorant audiophiles who are utterly convinced that marble plinths for solid state equipment, little discs made of Mpingo ebony that you sit on top of your components, incredibly expensive special power cables and CD "demagnetisers" all make a clear and definite difference to the sound of their hi-fi system.
If you do a proper scientific test and find an audible difference between an amplifier whose transistors have had voodoo incantations spoken over them and another otherwise identical amplifier that has not been so treated, then I'll give you my rapt attention, despite the fundamental ridiculousness of the concept. If the evidence supports your contention, then your contention has value, by definition.
But if you don't do a proper test, your results are going to be random. From what little this article says about the test's methodology, I see no reason to believe a word of it.
So they make a comparisson about compression formats, get together several "experts" in the field and the writer sums up the conclusions in a few oversimplified statements for the lazy reader.
God forbid they actually told you ALL the aspects of the story, the complete facts: what're the compression rates? which one compresses more? is there a relation between file size vs. quality (well, of course there is, but is some form of compression significan enough to justify lower quality? or the other way around?)
Sorry for the ranting, but I've been trying to catch up on what's going on in the world today and the more I read the more frustrated I get with regular articles from regular sources (the CNN's and MSNBC's). And has anyone noticed that it's becoming less and less of a practice to actually link the original source of the information?
Oh, and why is slashdot so slow today? maybe I should have named this post "A bad day...".
I apologize again for the negative tone.
There are two kinds of people in the world: Those with good memory.
all of the good details. Like how they where encoded at what bit rate etc. Also why did the listeners know what they where listening to. The comments should have been sample a sounded better then sample b not about which format sounded better. Also they did not mention file size or any of the other reasons that people use lossy formats in the first place.
Cypherpunks: Civil Liberty Through Complex Mathematics. Those who live by the sword die by the arrow.
>>>>>
There is a noticable loss of sound quality with any compression technique and IMHO there is no comparison to the original.
>>>>>
Yes, thats what the audiophiles in the c't test (which was correctly conducted) said.
Unfortunately for them, they couldn't distinguish
mp3 @ 256kbps and the cd's AT ALL.
--
GCP
Well, besides the obvious missing details, there
is a lot that is wrong about this:
>>>>>>
The test sessions were done in a home environment with an ordinary stereo system. We focused most of our attention on MP3Pro and Vorbis, the two newest formats, with Windows Audio Media and MP3, the older and more familiar formats, given more limited tests.
>>>>>>
Notice 'a home environment with an ordinary stereo system'. So esentially any more subtle loss in sound quality should have been lost. Great environment for listening tests eh. Note that this isn't compensated by the fact that is what most people use. The distortion between such systems varies widely, and hence what sounds good on one system doesn't necessarily sound good on another.
>>>>>>>
We had them listen to digitally encoded versions of two songs: the opening of a recent recording of Stravinsky's "The Rite of Spring" and the Who's "Love Ain't for Keeping."
>>>>>>>
TWO songs? We have hundreds of music genres and they used two songs for a comparisation? Christ. Encoding Heavy Metal (very bitrate heavy) is a whole different job than encoding classical music(very sensitive for minor distortions).
>>>>>>>
Of the seven listeners, two couldn't discern much difference between MP3Pro and Vorbis. The other five felt Vorbis was the least realistic,
>>>>>>>
Discern difference between the codecs? The way
this paragraph is put makes it highly unlikely
they were doing blind A/B tests.
More likely they actually told the test subjects
which codes it was each time.
This is _always_ going to favor mp3pro., just
because of the name. Also, the point of a good
encoder is to replicate the original music,
not to make it sound good! That is what an
exciter is for.
>>>>
Most thought the beta version of the Vorbis encoder poorly represented the natural sounds of the individual voices or musical instruments.
(A few disagreed, saying certain instruments sounded more synthetic in MP3Pro.)
>>>>
This convices me even more the setup of the test was failed. As I stated in a previous post, mp3pro 'makes up' the high end of the music. This is why some people must have thought it sounded better, while the better trained ones where able to pick up the fact that the high end was artificial.
>>>>
giving higher tones a better ring.
>>>>
Yep. More ringing on the hing end. Sounds like mp3pro for sure. Nevermind that the original music doesnt have it.
>>>>>>
"In the compressed format it sounded as if they had all moved their chairs together," said Hubscher. He founded this especially troublesome in Vorbis.
>>>>>>
BONK. Vorbis is the only format that does NOT
use any kind of joint/intensity stereo coding.
(it will in the 1.0 release)
Then how can it ever get a smaller stereo image??
This isn't making any sense at all...
>>>>>>>
Vrbsky and Lipnick blamed this on the way digital compression shaves off the beginnings and ends of notes.
>>>>>>>
Hahaaaa. They heard something about temporal
masking I'm sure. Too bad they don't have a clue.
--
GCP (who did his own listening tests)
Transparency is great, but most people who know the contortions that a sound engineer goes through to create the illusion of stero sound will tell you, audio reproduction is still as much an art as it is a science, and sometimes a less true sound going to the speakers will result in you thinking you hear something more true. It's all smoke and mirrors.
If I were the boss of a project to design a new digital format for the Internet, the goal would not be to make it sound as much like a CD as possible. The goal would be to make it sound as good as possible. Who knows, maybe some mad genious out there will come up with a format that uses fewer bits, yet sounds better than CD's.
Then maybe someday people will be saying "sure that new CD format sounds okay, but it can't compete with MP8 quality, like you get from web streaming."
Information wants to be anthropomorphized.
If I had to guess they used a low bit rate for MP3Pro under the assumption that everyone will shoot for the same quality they do with MP3 but with the Pro versions greater compression. Personally I plan on doing all my futere encoding at the same bit rate looking for the (hopefully) improved quality over MP3. The same comment applies for Vorbis as well.
If you can't be good, be good at it!
As others have pointed out, the article lacks technical details. The most important detail of all... did the listeners KNOW which file format was being played? If they did then name branding could have had a play in some of the results. Can you imagine thoughts like...
"Ogg Vorbiss, what kind of name is that? Sounds like crap too." or...
"Microsoft, they got a lot of money for R&D. This should be good."
i know this is all fine and well, but who really wants to change to a different format if they have to completely re-encode their entire mp3 collection?
eat shit and die, Bambi!
Additionally, the goal of compressed audio is not to replace cd's, but to aid in the dissemination of audio data, or music. Whichever technology has grassroots appeal and is more importantly free, will be embraced by the millions of people who use audio compression everyday, leaving higher-end solutions to professionals and audiophiles. This is illustrated well in the audio hardware industry: How many people have 20 000 dollar sound systems, or DVD audio collections? Not enough to be more than a niche market. This has been previously discussed on Slashdot....
- If This Peace Is Fictious, I Shall Destroy It