Dial-Up Audio Public Listening Test Opened

CaptainCheese writes "Hydrogenaudio.org's Roberto Amorim just announced the opening of their 32kps multi-format listening test, intended to test the current 'dial-up' quality codecs. From the Announcement: "The formats featured are Nero Digital Audio (HE-AAC+PS), Ogg Vorbis, WMA9 Std., MP3pro, Real Audio and QDesign Music Codec. Lame MP3 is being used as low anchor, and a lowpass at 7kHz is being used as high anchor." These codec tests are unusual in that they adhere to ITU-R BS.1116-1. The test is open until July 11th and all are invited to participate. There's more info in the original test discussion, which indicates the originator is interested in 'testing formats working on dial-up streaming bitrates' - the test page notes: 'The real arena where codecs are competing, and most development is going, is at low bitrates.'"

Comments

Results may be flawed

[Atario]

...since I'm liable to vote for whichever one sounds most like the Centurions from Battlestar Galactica, or the voice communications from THX-1138. Not best quality, not most understandable, just coolest.

Re:Results may be flawed

[FlipmodePlaya]

Good point. Can a study that will probably have relatively small survey size of an opinionated tech crown likely to exhibit bias be trusted? I don't know too much, admittedly, but wouldn't an automated test that just compared the output of a compressed audio track to the original be more accurate? Or is there more truth than I think to certain frequencies being worthless and inaudible by human ears?

Re:Results may be flawed

[Grant29]

Yeah, and probably some people have some crappy speakers. Some people would probably even try it with thier on-board speaker...

--
Only 5 Gmail invitations left!

Re:Results may be flawed

[izx]

The problem is that "automated comparisons" don't mimic human system responses (the ear, or the eye for video). Take video: the eye would finds grainy VHS tape more pleasing than a digital video that displayed some blocking. The blocked digital video, mathematically, is much closer to the original than the the VHS with its added noise...

These types of psychovisual (or psychoacoustic) responses are what make automated tools almost useless for judging the perceived quality of any lossy encoder. Perceived, that's the key word....it may not be mathematically up to scratch with the original, but if you PERCEIVE it to be as good as the original, thats what matters (this is of course for CD-quality high bitrate tests).

Re:Results may be flawed

[zoeblade]

...wouldn't an automated test that just compared the output of a compressed audio track to the original be more accurate? Or is there more truth than I think to certain frequencies being worthless and inaudible by human ears?

The whole idea behind lossy audio codecs is that the human brain and ear aren't that good at what they do :) As was pointed out on the Ogg Vorbis mailing list a looong time ago, technical tests like you're proposing would only tell you what computers would find more pleasant to listen to, not what humans would. So yeah, there's more to certain frequencies being inaudible to human ears :)

Get the news out to portable music player

[ralphart]

Now if only the companies who manufacture digital players would take a look and see that there is life beyond MP3. Nice that a few are starting to offer Ogg Vobis, but they are few and far between.

Re:Get the news out to portable music player

[glassjaw rocks]

I really think that the mp3 format is going to be around for quite a while, mainly just because of how widely-used it is. Even though I know the wma format is superior to mp3, I still encode all of my cds to mp3, more or less out of habit.

Re:Get the news out to portable music player

[2nd Post!]

Eww, skip the wma, please, for my sake.

The less the world is tied to Microsoft standards, the better off we'll all be, I think.

Re:Get the news out to portable music player

[Haydn Fenton]

Although many people are opposed to any microsoft anything, I have to say that their WMA format is very very useful at low bitrates.

I can use [a trial] of WMA Workshop to compress music files to as low as 2kbs. That's nothing special within itself, however, what is impressive (no matter how its done, IMO), is the fact that you can hear (and pretty clearly too) both the music AND words to the vast majority of songs. Which makes it perfect for sending my friends ultra-small previews (normally around 200kb in size) of full songs, so they know what they sound like - no doubt we've all told somebody to downl.. *cough* buy a song, but they hesitate because they think it will sound crap.

Note to members of the RIAA: The above statement is purely fictional. I have never and would never even consider the illegal distribution of music.

Is this a double-blind test?

[Anonymous Coward]

If it isn't, you'll only find out the most popular format, not the best.

Re:Is this a double-blind test?

[sploo22]

AS a matter of fact, it certainly is. Follow the link at the bottom of the page (to here) and you'll see this information:

One of the most acclaimed methods of comparing codec quality is by performing so-called "Double Blind Listening Tests". In this sort of test, the participant compares various encoded samples against each other and against an uncompressed reference sample. The blind part means that the participant doesn't know which sample was encoded by which encoder. That guarantees there'll be no psychological bias towards his/her favorite codec, or against the codec he/she dislikes.

Re:Is this a double-blind test?

[TommydCat]

On this site there is a useful little utility written by Arny Krugar to let you do your own DBT testing at any bitrate with any codec you want.

It does take a bit of preperation, but the results are legit. Not really suited for large organized polls, but fine to see your personal tastes and to understand exactly what a double-blind test is and how it works and why it is the only valid way to scientifically test.

Re:Is this a double-blind test?

[Matchstick]

It only needs to be a blind test, unless you worry that the computer administering the test is biased.

What's the point?

[TommydCat]

I personally don't get the goal of a test like this. Listening at that low quality doesn't have as much commercial, and quite frankly, personal appeal as it did back in the 90's.

I've seen the double-blind tests done at 128kbps and again fail to see the point.

What I really want to see is a rating of codecs that are able to achieve DBT-proven audible transparency and see them rated in terms of storage space (thus allowing the VBR schemes to finally compete).

Of course FLAC would come in last (considering WAV is the 'source'), but can my high quality VBR LAME MP3 pass for the original and take less space than MPC?

Re:What's the point?

[Sam3.14]

I agree that there isn't that much point to testing such low bitrates. People are quickly switching to DSL and Cable, And I think that there won't be many Dial-up users in 2010.

Re:What's the point?

[Nakito]

Testing at low bitrates emphasizes certain weaknesses of a codec. At high bitrates, it takes a more sensitive and trained ear to detect the artifacts and flaws. But as you reduce the bitrate, the differences become apparent. If you've never tried experimenting with it you might find it interesting, because the various codecs produce very different sounds with the same source file when the bitrate is drastically reduced. But I wonder if this is a proper way to evaluate the best design overall, since some of these codecs are certainly not optimized for low bitrates, and I do not know if there is necessarily a correlation between the flaws of a codec at low bitrates and those at higher bitrates.

Re:What's the point?

[bcombee]

Actually, low bitrates will be more important in the near future, as more people use streaming audio over PCS data services. For example, I listen to low-bitrate streams over Shoutcast several times a week on my PalmOne Treo 600, and 32Kbps streams much better than 64Kbps, while higher than that just isn't feasible on Sprint's PCS network. While this isn't as much of an issue for home users, mobile devices on relatively low-speed networks are going to be big.

Re:Why bother?

[ralphart]

Actually, with talk radio, the lower the bit rate, the better, if you catch my drift.

Slashdot Low Bitrate Ethnocentrism

[neildiamond]

Wow! I thought people on this site would have been a little more understanding. Believe it or not there are other places in the world (such as Africa) where high-speed Internet is not the norm or even available. Plus if you stream audio, any attempt to lower bandwidth is a plus as it lowers your bills.

Get over yourselves please.

By the way, did you ever notice the lack of multimedia even on this site? Why might that be? Hmmm...

Re:Slashdot Low Bitrate Ethnocentrism

[GregChant]

By the way, did you ever notice the lack of multimedia even on this site? Why might that be? Hmmm...

Because high bandwidth multimedia has no place on this type of news site; because it's unbelievably annoying; ...

hm... the days of dialup

[vmircea]

I remember back when I had dialup, it totally sucked, it took me absolutely forever to download ANYTHING, I would read a magazine while I used the computer because it would take so long for pages to load... when I got broadband at home it was a very happy day... but lots of people I know still don't have broadband in their area, which is why I think it is nice for people to do things like this, but also I think you could use this for voice over ip... just my two cents

Re:hm... the days of dialup

[eeg3]

Satellite is available anywhere there is uninhibited view of the sky, which is most places that don't have cable or DSL. However, satellite is expensive to set up... $500 installation fee from DirecTV last I checked.

Secondly, dial-up is not that bad, and it's definitely not as bad as your exaggeration. It's not comparable to broadband, but it's not unbarable. To further speed up dialup browsing, one should use a web cache, which is very helpful.

Re:hm... the days of dialup

[YankeeInExile]

Spot-beam satellite (i.e. DirecTV's offering, the British company whose name escapes me) is not available anywhere there is an uninhibited view of the sky. If you look at the contour maps for those products, you will see they are pretty tightly focused on their target market. I suspect it's even more of a problem on the uplink side -- those systems are running with really tight link budgets, and I don't think you're going to get an acceptable uplink BER if your antenna is 10dB off boresite.

While there are VSAT products that are available virtually anywhere, they are orders of magnitude more expensive. (For E1 speed in, say, Nigeria, figure $45K US per month, plus hardware costs in the several-kilobuck range.)

There is also an Intelsat data product, that last time I checked was about USD $7 per minute for DS0. A subrate option was available (9.6k) for about USD 2 per minute.

Re:Why bother?

[evilviper]

no matter how good the codecs are, 32kb/s music just doesn't cut it.

Damn, this is the kind of crap that gets modded-up these days...

Codecs continue to get better and better. Vorbis is pretty good even at 48K (artifacts are subtle). And even if this was 1997, and 32K sounded like crap with current codecs, you're statement is just like the famed "640K is enough for anybody", and "there is a world market for maybe a dozen computers". It's absolutely guaranteed to be proven wrong with time.

Re:Why bother?

[geeber]

Cancel the Test! TexasDex has no use for low-bitrate music. Since he clearly speaks for everyone, the entire affair is clearly a frivolous waste of time.

Go about your business, people.

Still too bandwidth intensive to be useful?

[Wild Bill TX]

No matter how optimized it is, won't it will still use too much bandwidth for dial-up users who actually want to do something else with their connection? All of the streams I ever tried to listen to, including the 8kbps ones, gladly used all of my available bandwidth. I don't know about anybody else, but I'm not interested in only getting a fraction of my 2 KB/sec max for browsing, using chats, or other tasks.

Interesting results

[sploo22]

I just took the test with sample 9, one of the speech ones, and it's amazing how much variability there is in the various codecs. One of them was so good I could only reliably hear the difference after a dozen repeated listenings, and another sounded like a cellphone in a tunnel. I'll be interested to see the results in a week or so.

Re:Phone line are shit

[julesh]

Err, no. No matter how good the line is, sending a well-designed lossily-compressed digital signal over it at the maximum bitrate that can be supported by the line is guaranteed to give you a better result than sending an analogue signal over it. Information theory requires it.

More interested in 32kbps speech

[Dr. Spork]

I am in the process of starting a project which needs accurate speech encoding at 32kbps. For now we're going with LAME at --alt-preset -b 32 -a --resample 22 --lowpass 6 -Z based on informal tests we did (ideas also came from here), but I'd love to see something more formal.

Notice all the different non-standard switches I had to use, which together help noticably. That's the sort of stuff you need to do to LAME before it produces acceptable results at very low bitrates. It is optimized only for 44.1KHz, so we should keep that in mind when we see the results. Notice now that none of these switches are being used for this test, so I'm almost certain that LAME will come out looking much worse than it is.

I would love for there to be a LAME-based encoder that is optimized for speech, low bitrates and sample rates. If it is made, I am prepared to re-encode all the readings that are (and are about to be) posted on my site.

This sounds terrible!!!

[rice_burners_suck]

I just listened to this, and I can tell you that it sounds totally like crap. Here at home, I have a sound room with a bitchen 100% analog system... Vacuum tube amplifiers, gold wiring, the works. I play my records on this thing and they have that wonderful warm sound. But this 32 kps sound sounds like garbage.

one use: books!

[timothy]

A lot of people are complaining here that low-bitrate recording is useless / stupid / so 1988 etc. Fine for them :) I'm glad that people are fanatic about sound quality and that storage prices make it reasonable for many people to use nothing but lossless codecs etc., and to care about the difference between 192kbps and 256kbps MP3s. Lossless is certainly a good storage answer for the long term, as the file can be inflated and re-squashed with the latest n' greatest lossy codecs as appropriate.

However, there are reasons and times where the lossy stuff, even hugely lossy stuff like this listening test focuses on, makes a lot of sense and has no big downside. For me, squashing audiobooks is this way. I can fit about 30 hours of book into one CD-R size chunk of hard drive as a series of (extremely listenable) quality zero mono ogg vorbis files. Beats carrying 30 CDs around.

timothy