Slashdot Mirror
Lossy Music Formats Compared
Nicholas writes: "Today's Washington Post has an article
detailing the results of having a "a diverse panel of listeners: two members of the National Symphony Orchestra, a high-end stereo salesman, a record producer, a composer and two guitarists" comparing MP3Pro and Vorbis formats. The punchline: "...felt Vorbis was the least realistic, with MP3Pro sounding better and Windows Media Audio best of all -- but none of these formats achieved CD quality.""
It still doesn't matter to me. If I could listen to WMA on my linux system(s) I would. If I could use WMA on my car mp3-cd player, I would.
I can't though, so it doesn't matter. I'm not a musician by any means, nor can I detect the difference between 160 and 192 mp3 compression. So I'll continue using my inferior, yet cross platform, non-license restricted, used-everwhere, mp3 format.
If you get serious with dither and error-feedback noise shaping, you can get whatever quality you like, including a characteristic very reminiscent of SACD, all from 16 bit audio CD.
Most CDs out there DON'T have acceptable dithering and the new ones that do, are usually crap in other ways!
I've written dithering that will literally resolve signals -156db down from CD audio, as long as they are bassy frequencies- the noise level increases with frequency like in SACD. Charts and details at http://www.airwindows.com/dithering/TechDetails.ht ml, the software itself is at http://www.airwindows.com/dithering/index.html.
This is not Linux software: it is Mac software written in an easier language than C. But it IS GPLed, and that's not an accident.
CD doesn't have to mean rotten quality, and this is an analog freak saying this. CD just has _historically_ meant crap quality, just as chrysalis describes. What's being described is truncation and bad word length maintenance, particularly if it involves cheap-ass DAWs or digital mixers.
You don't have to do that- though you're not likely to see anything but that from the major labels. They've been using cheaper and cheaper equipment for years, the latest trend is for everything to be all Pro Tools, and the poor mastering engineers are stuck trying to get a musical result out of that...
Combine that with no mention of the bitrates used and you essentially have a fluff piece.
I've only seen one test that might be called rigorous and that was a few years back. Does anyone know of quality listening comparisons between various codecs at various bitrates? Blind testing (Double Blind?), listeners with some musical background (musicians, recording engineers, even audiophiles), a wide range of music, maybe even different sound systems? I'm curious, almost (but not quite, I'm also lazy) enough to round up some friends and do my own sorry version of the test.
Chris Cothrun
Curator of Chaos
Bleh!
No, no, no. You're missing the point, and the previous poster to you was correct. To conduct a proper test for this, the test should have been set up according to the "sample a, sample b" idea. If the judges know beforehand what the format is for each sample, they can be biased against or for that format and reflect that bias in their responses. Have you ever taken a Practical Statistics class?
Hardly. Any musician, or any music fan with a good ear can distinguish between live music, CD-quality music, and MP3 at any commonly-used bitrate. There are significant audible artifacts in MP3-encoded music, particularly in the percussion sounds. Snare drums and cymbals seem to be most affected.
This is not like the subtle difference in sound quality that audiophiles used to claim from painting CD edges green - this is more like the difference between a well-miked drum kit and a pathologically distorted drum kit. Personally, I prefer the Ogg Vorbis sound at most bitrates. I haven't used WMA much.
I would like to see a test done with better controls and better reporting of test conditions.
-- Jeff Paulsen
The editor used compression on the story.
--
--
Mod up a post Rob doesn't like and you'll never mod again
When you're done, post it on www.kuro5hin.org
... Let people post with their votes on which sounds best. At the end, post the list of which sample came from which program, and a which bit-rate. At that point, post the compressed files as well so people can verify the results.
I'd suggest doing a blind test, compress the samples with various programs and bit-rates, then convert them back to wav files. Save them with names like "Rock-Sample1", "Rock-Sample2"
If you post that story (or email me for help in writing it and such) vote for it!
Actually, you may not be entirely correct. My fiancee and I have had quite a few discussions about high-end stereo equipment (like discrete amplified channels, speakers such as Definitive, Sonus Faber, and whoever it is that makes the filament speakers) as well as the quality of CD recordings vs. vinyl or tape or digital (DAT), etc. Now, mind you - she is an audio engineer, with a degree from one of the most reputed schools in the country (I think its in LA - I forget what its called).
One of the things she explains to me most often is that what you hear (on your CD) is NOT what was recorded in the studio, on stage at the symphony, etc. It is what has been cleaned and processed to sound best on a variety of common setups - like an average stereo, a boombox, and a common (stock) sound system in a common type of car.
So, while many folks believe they can hear more details than others (and admittedly, I'm sure recordings exist that still have these details in them), by and large - these details were lost on the studio post-processing floor. The goal of post-processing is apparently not to deliver the most hi-fi sound - its to deliver the music in a way that it sounds good in the most diverse environments.
To me, this sounds like what we get on CD is truly the lowest common denominator. At that point - encoding into MP3 or Vorbis doesn't seem to make more of a difference. I personally prefer Vorbis because (a) it sounds good (b) the file sizes are small and (c) it is patent free - that means a lot to me. Notice I didn't make any comparisons - I don't presume to say that it is better or smaller than anything else - I don't really care. Its the only format out there that allows me to enjoy my music in a format that is smaller than .wav files and more transportable than CDs whilst knowing that the algorithms are patent free to boot. Sweet deal if you ask me.
You're half correct. The release candidate for the version 1 *decoder* has been released, but the *encoder* is still at beta4. The release candidate for the encoder will be released at the end of the month, and that will be the first version supporting channel coupling. There are also several bugs in the psycho-acoustic model in beta4 that will be fixed in the release candidate.
Basically, they released the decoder as soon as they could, so other people can include a decoder in their applications that will play all version 1 Vorbis files.
So -- version 1 will have channel coupling, but it's not quite there yet.
-- Help Digitise the Public Domain at DP.
"What users want"? Maybe, if you want to write to the Least Common Denominator.... Anyway, trading off stability for "usability" is a mistake. A good SW Engr. group should be able to achieve both.
Maybe you missed the reason the AOL / MSFT talks broke down: the Dark Lords of Redmond wanted to tie all A-V formats to WMF with an exclusive contract, freezing out Real, MP3, Ogg Vorbis, etc. They still want to extend their monopoly any way they can.
I agree with this with some exceptions. If MSFT would compete solely on the quality of their products, I'd have no problem with them. So long as they are trying to extend their turf by leveraging their existing monopolies, and trying to get new monopoly positions, I've got no use for them.
--
"You've crossed my Line of Death!" "What? No! Where is it?" "Here in the fine print...."
When I tested this, I noticed that the player I used boosted the high end on Ogg and MP3, presumably to compensate for MP3 crushing it. Since Ogg didn't crush the high end, it sounded wrong. So, they need to choose good players.
Become a FSF associate member before the low #s are used
Why couldn't they test it with some good music? Like Britney Spears?
I could encode Britney's music with PGP and it would still sound just as good.
*dreams of bouncing breasts*
Actually, "cd quality" means poor quality. Honestly, a CD has a flat sound. Every CD sounds the same. Unrealistic.
12" vinyl records have much dynamic. More punch. If you listen to classic music on vinyl, you hear every instrument as if it was really nearby you. You don't get that feeling with a CD. Because a CD has only 16 bits of dynamic, music is overcompressed. Quiet music doesn't give anything on a CD. And when parts of the music become louder, there's a small difference of amplitude, but it has nothing to do with what real musicians played originally. Vinyl reflects this in a far better way.
Things may change with SCD (24 bits, 96 khz) . But the CD is definitely something lame for audiophile experts (nothing to do with the original music), for electronic music (the music isn't very punchy compared to vinyl), and for DJ's (I really hate mixing on CD, and I can't imagine hip-hop DJ's with CDJs) .
-- Pure FTP server - Upgrade your FTP server to something simple and secure.
{{.sig}}
Exactly.
My suspicion - given the preference for MP3Pro over MP3 - is that it was MP3 at 64kbps vs. MP3Pro at 64kbps.
Even makes sense, after all. Including WMA in the study at all is weird -- and if they encoded the WMA and the MP3 using MSFT's built-in MP3 encoder, they'd have gotten the shittiest-sounding 64k MP3 on the planet.
MP3Pro is designed for streaming audio, and outperforms MP3 at extremely low bitrates (i.e. 64K).
The part in the article where the guitarist said he could hear the valves closing on the clarinet and bassoon... is utter bullshit. He probably heard some flanging artifact and thought it was a valve closure.
Finally - and this is another biggie - for the MP3 section, did they use Blade, LAME, Fraun, or Xing? Some rock on some forms of music, but utterly sux0r on others. And some encoders sound like dog shit on all forms of music.
Sounds to me like a puff piece designed to get people out of (ubiquitous) MP3 and into (proprietary, Windoze-only) MP3Pro and (DRM-encumbered and proprietary) WMA, with Ogg Vorbis thrown in for a "What's that?" appeal.
(Credit to them grokking that lossy compression in music often throws away stereo separation / spatial components, though.)
This study varies significantly from the study done in Heise link here . That study concluded that high quality mp3s were indistinguishable from CD recordings !!
I can only conclude that one of these two sets of tests were biased or did something wrong. They can't both be right and the quality of digital recordings can't decrease with time !
And because of that, no one will ever use it. So Microsoft is effectively just burying technology.
Actually, a lot of people use it, just not for illegal file trading as is often seen with MP3.
The fact is -- some people actually only rip and encode their own CDs, and a good number of them people do so entirely with WMA.
This will become more prominent in Windows XP, and seeing as how Microsoft has crippled the included MP3 Encoder (or perhaps removed it altogether) WMA will be most people's choice, except for those of us who don't mind spending the high price for Fraunhofer encoder.
I rip and encode my own MP3s. That's a no-brainer. At the moment, MP3 is dominent.
I remember when GIF was the most popular image format, too. Does anybody remember VOC?
What about ARC, or LHA?
File formats shift through time. Some come, some go, and some remain in use to small groups. But in time I suspect even JPG will be replaced.
I don't see why MP3 is here to stay, even if WMA doesn't replace it.
"Everything you know is wrong. (And stupid.)"
"Everything you know is wrong. (And stupid.)"
Moderation Totals: Wrong=2, Stupid=3, Total=5.
I've noticed that some players provide clearer sound than others when playing the same file, so I would think that the same would apply to encoders.
This, of course, depends on the implementation of the algorithm used, whether a reverse-engineered algorithm is used or a native one is used, what kind of error correction is used, and so forth.
--Jon
Give me my freedom, and I'll take care of my own security, thank you.
???
I talked to him before the story went in the Post
(thus the little sentance about 1.0) and he said
it was neither blind nor structured (and I gave
him my concerns about that on Wednesday)...
OK, I better mail him again and see what's up.
Monty
I spoke to the reporter who did these 'listening tests'... the tests were not structured, were not controlled, were not blind or double blind... in short, the reviewers knew exactly which codec sample they were listening to at all times. The reviewers also came into the test with already formed opinions about which codec was best and-- surprise--- the test seemed to confirm what they already 'knew'.
In ABX blind testing, Vorbis usually surprises the tester, for example, from our happy maniac friends at r3mix forum (MPEGplus and Vorbis won this ABX test first and second place over mp3pro, AAC, WMA8, TwinVQ and ATRAC3):
That is, he knew for sure... incorrectly. A brilliant example of the power of suggestion.In any case, after talking to the Post reporter he feels a little sheepish about the whole thing... he thought he balanced the article by mixing positive traits of the openness of the code with a critical quality review and has agreed to be more fair to the first 1.0 encoder release candidate.
Monty
If it were really diverse, they wouldn't just have music professionals on the board. They'd have a deaf person, some giggling schoolgirls, a super-intelligent killbot, and maybe a monkey.
That always seemed to me to be the issue- If I copy a tape from a friend who copied it from another friend, each generation of copies degrades significantly. Every generation of copies of a digital file is identical to the first rip, aside from the usual Napster problem of losing the last 4 seconds of every song you download :-)
I do not deploy Linux. Ever.
Small or not it's quite perceptible. Play on my stereo an MP3 vs. a CD of the same song I will 100% pick out the CD as being better each time based on the sound alone, and I could point out the differences to anyone and make them see it too. Lots of people can; MP3's are simply no good for at home, pure listening purposes. For just about everything else, they're fine (computers, joggers, cars, DJ's, parties, etc...) for the most part.
I can't believe in this comparison they neglected one of the most unique formats of all time! I mean, this format has nearly zero compression, a limited spand, is fairly lossy in level (although it has a better frequency range), and is extremely easily corruptible and nearly impossible to repair.
I mean, the vinyl record didn't even earn its due in this battle! Although I understand that if you're willing to take a shorter piece of music, the media can be compressed to half its original diameter.
"I'm not even supposed to BE here today!"
Here's an interesting web site that lets you do the comparisons:
x .h tm
h tm
www.pcabx.com
Also see
www.pcavtech.com
for some results
and
oeonline.com/~djcarlst/abx.htm
for explanation of ABX in general.
This url:
http://www.pcabx.com/product/coder_decoder/inde
contains samples to compare FOR YOURSELF several codec variations.
This url:
http://www.pcavtech.com/play-rec/summary/index.
contains some results... Apologies for not posting with html...
Mark
Apart from the fact that, as other posters have noted, there's no information given on the bit rates or encoders used, the article straight-out admits that "the test sessions were done in a home environment with an ordinary stereo system".
Which, I'll warrant, means no double blinding and no level matching. Probably not even single blinding (where the testers know what's being listened to, but the testees don't). Level matching is essential; different encoding methods may play back at slightly different levels, and just turning up the volume a tad will convince a large proportion of casual or professional listeners that something's improved in the sound.
Without a proper scientific test, psychoacoustic effects can swamp even quite large real differences in sound, and can cause people to hear quite large differences that they wouldn't hear if they didn't know when they were listening to what.
As can trivially be demonstrated, when you look at the number of golden-eared but scientifically ignorant audiophiles who are utterly convinced that marble plinths for solid state equipment, little discs made of Mpingo ebony that you sit on top of your components, incredibly expensive special power cables and CD "demagnetisers" all make a clear and definite difference to the sound of their hi-fi system.
If you do a proper scientific test and find an audible difference between an amplifier whose transistors have had voodoo incantations spoken over them and another otherwise identical amplifier that has not been so treated, then I'll give you my rapt attention, despite the fundamental ridiculousness of the concept. If the evidence supports your contention, then your contention has value, by definition.
But if you don't do a proper test, your results are going to be random. From what little this article says about the test's methodology, I see no reason to believe a word of it.
Also... you are assuming all the world's music is already here, and in MP3. There'll be new music created that can be encoded using better formats, and of course not everyone has converted music to mp3 yet in the first place. There's plenty of room for new better formats.
I like paying taxes. With them I buy civilization -- Oliver Wendell Holmes
So they make a comparisson about compression formats, get together several "experts" in the field and the writer sums up the conclusions in a few oversimplified statements for the lazy reader.
God forbid they actually told you ALL the aspects of the story, the complete facts: what're the compression rates? which one compresses more? is there a relation between file size vs. quality (well, of course there is, but is some form of compression significan enough to justify lower quality? or the other way around?)
Sorry for the ranting, but I've been trying to catch up on what's going on in the world today and the more I read the more frustrated I get with regular articles from regular sources (the CNN's and MSNBC's). And has anyone noticed that it's becoming less and less of a practice to actually link the original source of the information?
Oh, and why is slashdot so slow today? maybe I should have named this post "A bad day...".
I apologize again for the negative tone.
There are two kinds of people in the world: Those with good memory.
Are they going to notice the difference? Probably not.
Granted this is nice information to have. But I don't see the average user caring much. The MP3 format itself was lossy but no one seemed to care.
Most MP3 compression utilities filter out frequencies above 20kHz or below 20Hz; CD-quality sound can produce frequencies at 44kHz and the inverse of the length of the sound. Clearly, the data filtered out is not tiny.
It is true that most low-grade speakers and headphones cannot produce frequencies above 20kHz or below 20Hz (check the frequency response), but higher grade equipment can. This is often the reason somebody can't tell the difference between CD sound and MP3.
Psychoaucustics models say that few people can hear frequencies, by themselves, above 20kHz. Suppose that you can't. However, multiple frequencies above 20kHz can produce frequencies below 20kHz. Try playing a root and a fourth on a very well-tuned string instrument. It should resonate and octave below the root. Playing a root and an octave above it also produces and octave below the root. Various other combinations produce different tones.
The frequencies above 20kHz do matter, whether you can hear them or not. These high frequencies often contribute to tone; differences in ambouchere, reed, mouthpieces, instruments for reed players, differences in strings, where the string is played, plucking/bowing style for string players.
Another way to think about it: try to produce a square wave with only sine waves. To reproduce it exactly, you need the wave that has the same period, then period/3, then period/5, then period/7, etc. When you take away the high frequencies, the wave isn't the same.
Higher bit rate MP3s are not CD quality sound. MP3 still distorts the original, and many find that objectionable when hearing it from quality equipment.
For actual CD-quality sound, I suggest a lossless algorithm such as shorten (.shn).
all of the good details. Like how they where encoded at what bit rate etc. Also why did the listeners know what they where listening to. The comments should have been sample a sounded better then sample b not about which format sounded better. Also they did not mention file size or any of the other reasons that people use lossy formats in the first place.
Cypherpunks: Civil Liberty Through Complex Mathematics. Those who live by the sword die by the arrow.
>>>>>
There is a noticable loss of sound quality with any compression technique and IMHO there is no comparison to the original.
>>>>>
Yes, thats what the audiophiles in the c't test (which was correctly conducted) said.
Unfortunately for them, they couldn't distinguish
mp3 @ 256kbps and the cd's AT ALL.
--
GCP
Well, besides the obvious missing details, there
is a lot that is wrong about this:
>>>>>>
The test sessions were done in a home environment with an ordinary stereo system. We focused most of our attention on MP3Pro and Vorbis, the two newest formats, with Windows Audio Media and MP3, the older and more familiar formats, given more limited tests.
>>>>>>
Notice 'a home environment with an ordinary stereo system'. So esentially any more subtle loss in sound quality should have been lost. Great environment for listening tests eh. Note that this isn't compensated by the fact that is what most people use. The distortion between such systems varies widely, and hence what sounds good on one system doesn't necessarily sound good on another.
>>>>>>>
We had them listen to digitally encoded versions of two songs: the opening of a recent recording of Stravinsky's "The Rite of Spring" and the Who's "Love Ain't for Keeping."
>>>>>>>
TWO songs? We have hundreds of music genres and they used two songs for a comparisation? Christ. Encoding Heavy Metal (very bitrate heavy) is a whole different job than encoding classical music(very sensitive for minor distortions).
>>>>>>>
Of the seven listeners, two couldn't discern much difference between MP3Pro and Vorbis. The other five felt Vorbis was the least realistic,
>>>>>>>
Discern difference between the codecs? The way
this paragraph is put makes it highly unlikely
they were doing blind A/B tests.
More likely they actually told the test subjects
which codes it was each time.
This is _always_ going to favor mp3pro., just
because of the name. Also, the point of a good
encoder is to replicate the original music,
not to make it sound good! That is what an
exciter is for.
>>>>
Most thought the beta version of the Vorbis encoder poorly represented the natural sounds of the individual voices or musical instruments.
(A few disagreed, saying certain instruments sounded more synthetic in MP3Pro.)
>>>>
This convices me even more the setup of the test was failed. As I stated in a previous post, mp3pro 'makes up' the high end of the music. This is why some people must have thought it sounded better, while the better trained ones where able to pick up the fact that the high end was artificial.
>>>>
giving higher tones a better ring.
>>>>
Yep. More ringing on the hing end. Sounds like mp3pro for sure. Nevermind that the original music doesnt have it.
>>>>>>
"In the compressed format it sounded as if they had all moved their chairs together," said Hubscher. He founded this especially troublesome in Vorbis.
>>>>>>
BONK. Vorbis is the only format that does NOT
use any kind of joint/intensity stereo coding.
(it will in the 1.0 release)
Then how can it ever get a smaller stereo image??
This isn't making any sense at all...
>>>>>>>
Vrbsky and Lipnick blamed this on the way digital compression shaves off the beginnings and ends of notes.
>>>>>>>
Hahaaaa. They heard something about temporal
masking I'm sure. Too bad they don't have a clue.
--
GCP (who did his own listening tests)
Transparency is great, but most people who know the contortions that a sound engineer goes through to create the illusion of stero sound will tell you, audio reproduction is still as much an art as it is a science, and sometimes a less true sound going to the speakers will result in you thinking you hear something more true. It's all smoke and mirrors.
If I were the boss of a project to design a new digital format for the Internet, the goal would not be to make it sound as much like a CD as possible. The goal would be to make it sound as good as possible. Who knows, maybe some mad genious out there will come up with a format that uses fewer bits, yet sounds better than CD's.
Then maybe someday people will be saying "sure that new CD format sounds okay, but it can't compete with MP8 quality, like you get from web streaming."
Information wants to be anthropomorphized.
Why? Well, it's quite good; apparently the billions and billions Microsoft has spent on research has finally resulted in a payoff. And Microsoft has managed to dominate market segments with complete shit products before; now that they actually have something decent, it's going to be difficult to stop them.
>said he could hear the valves closing on
>the clarinet and bassoon... is utter bullshit.
>He probably heard some flanging artifact and
>thought it was a valve closure.
Okay, if you read the article, you know that the classical piece they listened to was the beginning of Stravinsky's Firebird, which you SHOULD know begins with a BASSOON SOLO! I don't know what particular recording they listened to, but my Reference Recordings recording of the MN Orchestra playing this piece has very audible bassoon-key clicks at the beginning of this piece, and I imagine most other recordings would too. If you dont believe me, email me and I'll send you an MP3 of the beginning of this piece - the clicks are VERY audible, even in an MP3!
What is most surprising is that the listener said he COULDNT hear the clicks on the compressed audio, not that he COULD on the cd... they must have used a VERY low bitrate to drown out those clicks!
moo
If I had to guess they used a low bit rate for MP3Pro under the assumption that everyone will shoot for the same quality they do with MP3 but with the Pro versions greater compression. Personally I plan on doing all my futere encoding at the same bit rate looking for the (hopefully) improved quality over MP3. The same comment applies for Vorbis as well.
If you can't be good, be good at it!
also, how did these people encode their mp3/ogg/wma files? there's no indication of bitrate, sampling frequency, encoder, encoder settings, nothing!. without this info, these results are even more invalid, and misleading to people who believe them. for example: in mp3 encoding, a file encoded at 128 kbps with a newer version of lame will sound much better than a piece encoded at the same bitrate with bladenc. why? cuz lame is much better, that's why!
i'm not supporting one format over another, but please, please don't base anything on this crap. i don't know how this even got posted.
--
As others have pointed out, the article lacks technical details. The most important detail of all... did the listeners KNOW which file format was being played? If they did then name branding could have had a play in some of the results. Can you imagine thoughts like...
"Ogg Vorbiss, what kind of name is that? Sounds like crap too." or...
"Microsoft, they got a lot of money for R&D. This should be good."
i know this is all fine and well, but who really wants to change to a different format if they have to completely re-encode their entire mp3 collection?
eat shit and die, Bambi!
Additionally, the goal of compressed audio is not to replace cd's, but to aid in the dissemination of audio data, or music. Whichever technology has grassroots appeal and is more importantly free, will be embraced by the millions of people who use audio compression everyday, leaving higher-end solutions to professionals and audiophiles. This is illustrated well in the audio hardware industry: How many people have 20 000 dollar sound systems, or DVD audio collections? Not enough to be more than a niche market. This has been previously discussed on Slashdot....
- If This Peace Is Fictious, I Shall Destroy It
I think these guys are full of it. The amount of data that is lost in MP3 compression is tiny, and is mostly sounds out of range of human hearing. And if the files are recorded at higher bit rates CD quality sound can be achieved. I think these guys just felt they had to say something to the reporter, so as not to look like idiots. And at the end of the article, when the guy "worries" that we are losing our appreciation of music because of MP3s, I realized that this was an article about nothing, with quotes by people who knew nothing.
---
Hey man, can I bum a sig?