Dial-Up Audio Public Listening Test Opened

CaptainCheese writes "Hydrogenaudio.org's Roberto Amorim just announced the opening of their 32kps multi-format listening test, intended to test the current 'dial-up' quality codecs. From the Announcement: "The formats featured are Nero Digital Audio (HE-AAC+PS), Ogg Vorbis, WMA9 Std., MP3pro, Real Audio and QDesign Music Codec. Lame MP3 is being used as low anchor, and a lowpass at 7kHz is being used as high anchor." These codec tests are unusual in that they adhere to ITU-R BS.1116-1. The test is open until July 11th and all are invited to participate. There's more info in the original test discussion, which indicates the originator is interested in 'testing formats working on dial-up streaming bitrates' - the test page notes: 'The real arena where codecs are competing, and most development is going, is at low bitrates.'"

Comments

Get the news out to portable music player

[ralphart]

Now if only the companies who manufacture digital players would take a look and see that there is life beyond MP3. Nice that a few are starting to offer Ogg Vobis, but they are few and far between.

Re:Get the news out to portable music player

[glassjaw rocks]

I really think that the mp3 format is going to be around for quite a while, mainly just because of how widely-used it is. Even though I know the wma format is superior to mp3, I still encode all of my cds to mp3, more or less out of habit.

Is this a double-blind test?

[Anonymous Coward]

If it isn't, you'll only find out the most popular format, not the best.

Re:Is this a double-blind test?

[sploo22]

AS a matter of fact, it certainly is. Follow the link at the bottom of the page (to here) and you'll see this information:

One of the most acclaimed methods of comparing codec quality is by performing so-called "Double Blind Listening Tests". In this sort of test, the participant compares various encoded samples against each other and against an uncompressed reference sample. The blind part means that the participant doesn't know which sample was encoded by which encoder. That guarantees there'll be no psychological bias towards his/her favorite codec, or against the codec he/she dislikes.

Re:Is this a double-blind test?

[TommydCat]

On this site there is a useful little utility written by Arny Krugar to let you do your own DBT testing at any bitrate with any codec you want.

It does take a bit of preperation, but the results are legit. Not really suited for large organized polls, but fine to see your personal tastes and to understand exactly what a double-blind test is and how it works and why it is the only valid way to scientifically test.

What's the point?

[TommydCat]

I personally don't get the goal of a test like this. Listening at that low quality doesn't have as much commercial, and quite frankly, personal appeal as it did back in the 90's.

I've seen the double-blind tests done at 128kbps and again fail to see the point.

What I really want to see is a rating of codecs that are able to achieve DBT-proven audible transparency and see them rated in terms of storage space (thus allowing the VBR schemes to finally compete).

Of course FLAC would come in last (considering WAV is the 'source'), but can my high quality VBR LAME MP3 pass for the original and take less space than MPC?

Re:What's the point?

[Nakito]

Testing at low bitrates emphasizes certain weaknesses of a codec. At high bitrates, it takes a more sensitive and trained ear to detect the artifacts and flaws. But as you reduce the bitrate, the differences become apparent. If you've never tried experimenting with it you might find it interesting, because the various codecs produce very different sounds with the same source file when the bitrate is drastically reduced. But I wonder if this is a proper way to evaluate the best design overall, since some of these codecs are certainly not optimized for low bitrates, and I do not know if there is necessarily a correlation between the flaws of a codec at low bitrates and those at higher bitrates.

Re:Results may be flawed

[FlipmodePlaya]

Good point. Can a study that will probably have relatively small survey size of an opinionated tech crown likely to exhibit bias be trusted? I don't know too much, admittedly, but wouldn't an automated test that just compared the output of a compressed audio track to the original be more accurate? Or is there more truth than I think to certain frequencies being worthless and inaudible by human ears?

Slashdot Low Bitrate Ethnocentrism

[neildiamond]

Wow! I thought people on this site would have been a little more understanding. Believe it or not there are other places in the world (such as Africa) where high-speed Internet is not the norm or even available. Plus if you stream audio, any attempt to lower bandwidth is a plus as it lowers your bills.

Get over yourselves please.

By the way, did you ever notice the lack of multimedia even on this site? Why might that be? Hmmm...

Re:Slashdot Low Bitrate Ethnocentrism

[GregChant]

By the way, did you ever notice the lack of multimedia even on this site? Why might that be? Hmmm...

Because high bandwidth multimedia has no place on this type of news site; because it's unbelievably annoying; ...

Re:Why bother?

[evilviper]

no matter how good the codecs are, 32kb/s music just doesn't cut it.

Damn, this is the kind of crap that gets modded-up these days...

Codecs continue to get better and better. Vorbis is pretty good even at 48K (artifacts are subtle). And even if this was 1997, and 32K sounded like crap with current codecs, you're statement is just like the famed "640K is enough for anybody", and "there is a world market for maybe a dozen computers". It's absolutely guaranteed to be proven wrong with time.

Re:Why bother?

[geeber]

Cancel the Test! TexasDex has no use for low-bitrate music. Since he clearly speaks for everyone, the entire affair is clearly a frivolous waste of time.

Go about your business, people.

Still too bandwidth intensive to be useful?

[Wild Bill TX]

No matter how optimized it is, won't it will still use too much bandwidth for dial-up users who actually want to do something else with their connection? All of the streams I ever tried to listen to, including the 8kbps ones, gladly used all of my available bandwidth. I don't know about anybody else, but I'm not interested in only getting a fraction of my 2 KB/sec max for browsing, using chats, or other tasks.

Interesting results

[sploo22]

I just took the test with sample 9, one of the speech ones, and it's amazing how much variability there is in the various codecs. One of them was so good I could only reliably hear the difference after a dozen repeated listenings, and another sounded like a cellphone in a tunnel. I'll be interested to see the results in a week or so.

Re:Phone line are shit

[julesh]

Err, no. No matter how good the line is, sending a well-designed lossily-compressed digital signal over it at the maximum bitrate that can be supported by the line is guaranteed to give you a better result than sending an analogue signal over it. Information theory requires it.

Re:Results may be flawed

[izx]

The problem is that "automated comparisons" don't mimic human system responses (the ear, or the eye for video). Take video: the eye would finds grainy VHS tape more pleasing than a digital video that displayed some blocking. The blocked digital video, mathematically, is much closer to the original than the the VHS with its added noise...

These types of psychovisual (or psychoacoustic) responses are what make automated tools almost useless for judging the perceived quality of any lossy encoder. Perceived, that's the key word....it may not be mathematically up to scratch with the original, but if you PERCEIVE it to be as good as the original, thats what matters (this is of course for CD-quality high bitrate tests).

More interested in 32kbps speech

[Dr. Spork]

I am in the process of starting a project which needs accurate speech encoding at 32kbps. For now we're going with LAME at --alt-preset -b 32 -a --resample 22 --lowpass 6 -Z based on informal tests we did (ideas also came from here), but I'd love to see something more formal.

Notice all the different non-standard switches I had to use, which together help noticably. That's the sort of stuff you need to do to LAME before it produces acceptable results at very low bitrates. It is optimized only for 44.1KHz, so we should keep that in mind when we see the results. Notice now that none of these switches are being used for this test, so I'm almost certain that LAME will come out looking much worse than it is.

I would love for there to be a LAME-based encoder that is optimized for speech, low bitrates and sample rates. If it is made, I am prepared to re-encode all the readings that are (and are about to be) posted on my site.