After 4 Years, HydrogenAudio Opens New 128kbps Listening Test

kwanbis writes "After more than four years, a new MP3@128kbps listening test is finally open at HydrogenAudio.org! The featured encoders are: LAME 3.97, LAME 3.98.2, iTunes 8.0.1.11, Fraunhofer IIS mp3surround CL v1.5, and Helix v5.1 2005.08.09. The low anchor is l3enc 0.99a. The purpose of this test is to find out which popular MP3 VBR encoder outputs the best quality on bitrates around 128 kbps. All encoders experienced major or minor updates that should improve audio quality or encoding speed, and we have a totally new encoder on board. Note that you do not have to test all samples — it is a great help even if you test one or two. The test is scheduled to end on November 22nd, 2008."

Comments

use the cans, luke

[PhrostyMcByte]

good headphones are a must for such close listening tests. you'll only be able to hear really major differences with most speakers.

Re:use the cans, luke

[TheRealMindChild]

I call horseshit. I only care about the differences *I* can hear with the speakers/headphones *I* have. Isn't that the whole point? The shortcuts I can take without noticing a difference...

Re:use the cans, luke

[Jah-Wren Ryel]

I call horseshit. I only care about the differences *I* can hear with the speakers/headphones *I* have. Isn't that the whole point? The shortcuts I can take without noticing a difference...

It is a heck of a lot easier to upgrade your equipment than it is to re-encode your audio, assuming you even have the original sources around.
What sounds fine today on your current system may sound poor on your next system tomorrow.

Re:use the cans, luke

[rtollert]

A lot of very high quality encoder tuning has been done with $30 headphones on laptops. Concentration and patience is more important than equipment here.

Re:use the cans, luke

[lysergic.acid]

it really depends on the application. for music, headphones are probably a much better value and give a better listening experience. but if you're building a home theater setup for watching movies, obviously speakers are the way to go. you just can't properly enjoy 5.1 surround sound with stereo headphones. but at the same time, most recording studios probably work with headphones far more than they work with speakers.

and in this day and age you shouldn't have to spend $200 to get a decent pair of headphones. likewise, quality speakers shouldn't cost tens of thousands of dollars, much less hundreds of thousands of dollars. i mean, how much did a top of the line sound system cost in the 70's? even by then most high end audio equipment probably exceeded normal human hearing ability. i would hope that today's mid-level consumer audio equipment would be able to at least match the state-of-the-art from over 3-and-a-half decades ago.

Re:use the cans, luke

[aywwts4]

Assuming equal quality speakers (which indeed cost a lot more than headphones) The difference is your room. Many people don't enjoy the sound of being in an isolation chamber with your headphones, An effect that would be mimicked listening to speakers in a sound dampened room. A lot of people enjoy the effects of sound unfolding like in a music hall.

I'm sure the best sound in the world comes from those ridiculously expensive listening rooms you see, baffled and shaped like an opera house. But if you put the same expensive speakers in the average living room with vaulted ceilings hard angles and whatnot, the sound will not be equal in all places, there will be echos and some sounds will be scientifically and measurably deadened. Speakers are always at the whim of their environment, while headphones are only manipulated by your ear canal.

Re:use the cans, luke

[arth1]

Good headphones are nice in so far as they block ambient noise and allow you to hear any artifacts easier, but since MP3 is a perceptual encoder it is actually more likely that artifacts are audible on "defective" hardware.

Good headphones seldom block ambient noise. Some of the very best headphones out there are open, including (but not limited to):
AKG 701/702
Grado (all models)
Beyer DT880
Sennheiser HD600/650

They're made, not to isolate you from the environment and prevent sound from escaping either way, but to replace loudspeakers with the best sound possible, with no regard to (or for) the environment.

My beef with this test is that they use 128 kbps VBR.
These days, space isn't as such a premium as it was a few years ago, and few use 128 kbps anymore, unless it's the default for their encoder or they haven't bothered changing.
I would think that 160, 192, 224, 256 and even 320 kbps are more common than 128 kbps these days.

Then there's VBR. And VBR in itself is by many considered evil -- yes, you cram in more data that way, but you end up with a sound stream that switches back and forth between different qualities, which is more apparent to the ear than if it was all at the lowest quality. It's like listening to a radio where the FM stereo kicks in every now and then. Yes, that is quantifiably a better quality than listening to it in mono, but I still prefer switching to mono to get a worse, but stably worse, sound.
The same piano key hit multiple times can end up sounding different with VBR. First you get an awesome 224 or 320 kbps note, then another, but then omgwehaveusedupallthebandwidth you get a 80 kbps note that just doesn't sound similar. Overall, the quality has gone up, but the net effect is that it sounds jarring.

Personally, I have enough disk space, and use FLAC when I can. When I can't, I use 224 kbps CBR, because at that high bitrates, I can't really tell any difference, and I avoid the whole VBR bitrate-changing problem.

Re:use the cans, luke

[Cowclops]

Actually, the point of VBR is to keep quality close to constant, as some audio frames are more easily compressed than others. Constant bitrate actually gives you variable quality. Variable bitrate gives you near constant quality. If you "hear" the quality changing in a VBR recording, theres something wrong with the encoder.

ugh

[Anonymous Coward]

Wow, what a mess. Download this package. Now download fourteen more packages (DownThemAll is the only reason I didn't give up right then). Y'know, I'm kinda interested in this subject, as I have no trouble hearing artifacts in most 128kbps CBR MP3s, but this is just a huge pain in the ass. Wouldn't a simple Flash app have made things so much easier?

Re:ugh

[mollymoo]

That hardly solves the problem. The applet should be embedded in the web page and download all the samples automatically, on demand. Why the stupid rigmarole of doing everything yourself? It's a ridiculously complex process. I gave up when I discovered that "OS X users are asked to handle decoding of samples themselves" what does decoding the samples involve? I haven't a fucking clue, because that's all it tells me.

Painfully painful test

[Anonymous Coward]

You know what, I thought I'd be nice and give this a shot, but the amount of effort involved just isn't worth it. If it isn't 'click on this link, listen, rate', it's too much work. Download x, install x, email x - way, way, way too much work for what is being given in return.

Outdated?

[Max Romantschuk]

Umm... 128 Kbps? Seriously? And no Ogg Vorbis, AAC etc... If you're bothering to set up a listening test, why limit yourself to 128 Kbps MP3?

Also, this should really be set up as a blind test, you get to listen to two clips, and have to choose which is better. The clips are randomized, of course... I could go on, but I'd just make myself sound even more arrogant. ;)

Re:Outdated?

[darien]

this should really be set up as a blind test, you get to listen to two clips, and have to choose which is better.

I agree entirely. They should also include different bitrates - do many people still use 128kbps? - and versions which aren't compressed at all. Hopefully the results might shut up the audiobores who keep insisting that MP3 isn't good enough for their precious ears.

Re:Outdated?

[Chairboy]

For the same reason you do performance testing on slow machines. It makes it easier to detect differences in sound quality (or slow code in performance testing) and the results scale smoothly upwards.

By limiting the bitrate to 128, you're more likely to get good data instead of just guesses.

Re:Outdated?

[hedwards]

Indeed, after reripping my entire collection for like the 3rd or 4th time because it got corrupted randomly, I switched to just ripping everything to lossless. If I need a copy for multiple MP3 players and such or change my mind about what compression rate or type I want it's a task my computer can handle without me swapping tons of discs.

It's always been easier to encode to a lower quality than to a higher quality. And in a strict sense the latter isn't really even possible.

You only need to download *two* files

[rtollert]

The ABC/HR zip, and one sample zip. Each sample zip is a separate test that can be run completely separately from the others. Testing each sample may take quite some time (it took 1-2 hours for a single sample last night for me) - so splitting this up actually does make a bit of sense. That said, even on Windows this test has been plagued with problems. I've had to downgrade to Java 1.5 to avoid a crash.

Re:What kind of music is involved

[rtollert]

That's great if you're trying to use a codec for a purpose it was never designed for and nobody actually uses. Would you choose a JPEG codec based on its ability to encode/decode raw audio? Would you choose a car based on its ability to traverse the English Channel?

Re:What kind of music is involved

[rtollert]

Spectral music might make for great samples for this kind of testing, but your assertion is ultimately unsubstantiated unless you can provide real listening test results that show it makes for a more sensitive test. There are all kinds of subtle things going on that might seem to make for great encoder testing, but largely turn out to make an imperceptible difference. Just because so many overtones exist (99% of which do not exist in msot acoustic music, btw!) doesn't mean they are necessarily audible if they are perturbed. More specifically, I'd anticipate that most FFT-based imaging techniques would hammer encoder lowpasses very hard, but would not be nearly as hard on preecho performance or stereo imaging artifacts. In a lot of situations, the preecho is a lot more important than the lowpass.

Why are we still focusing on 128kb?

[The Optimizer]

It was 10 years ago when I bought a Rio PMP-300, the first readily available flash-based MP3 player. It came with 32MB of internal memory, and would accept a single SmartMedia card, 32MB max in size (which I quickly went out and bought).

Back then, the size of your MP3 files mattered a whole lot more. At 128K CBR, I could fit 6 to 9 songs on each bank, depending on how long they were. The artifacts were noticeable to even my poor hearing. So I then stepped up to 160kb CBR and then LAME -remix (VBR, average ~ 190K ) encoding setting. I will make a note here that not all MP3 encoders are created equal - there is no fixed encoding standard, just for decoding.

With the VBR files, I could only fit 3-6 songs per bank of the Rio, so yea, it mattered then. If I wanted a specific CD to take to the gym with me, I had to think about what I put on the Rio. Often I couldn't fit the whole CD on the device or I had to swap play order to better use the slack space in each memory partition.

Can you even buy a MP3 player with less than 1GB of internal flash memory today? Skip past something like the iPod shuffle or equivalent at 1 and 2 GB, and you are quickly looking at 4GB, 8GB, 16GB or more.

I just encoded my copy of Linkin park's Minutes to Midnight CD that I bought with LAME 3.97 high quality VBR and it came out to 77.6 MB for the whole thing with the average bit rates in the 230kb/s to 270kbs. It wouldn't fit on the RIO at this quality. On the cheapest iPod Shuffle, I could fit 13 similarly sized CDs at this quality encoding. On the cheapest iPod Nano, around 100 similarly sized and encoded CDs.

My point??

128Kbs is sooo 1990s.. We've moved on. Storage, be it flash or Hard drives, has gotten order(s) of magnitude cheaper and bigger. So why aren't we moving our mindset about default MP3 quality UP to reflect the change? Make very High quality VBR the default and raise the average quality bar.

you didn't say anything about the effect on sound

[Trepidity]

You said that if you look at a spectrum as an image you can spot degradation in a lossy encoder. That's entirely irrelevant, since mp3 is a perceptual encoder; it deliberately loses information that, according to its predictive model, listeners cannot actually hear. To show that it fails to do this, you need to conduct a blind listening test with the encoded and un-encoded version, and actually reliably (i.e. significantly more than 50% of the time) pick out the encoded one.