[Ross]

I recently bought an iPod. I bought the 40Gb version because I wanted to listen to uncompressed WAV files only. I had heard some MP3s on my notebook and they sounded awful. I figured I could get about 60 uncompressed CDs into my iPod, which would be fine.

Naturally, when I got the iPod, I decided to conduct a little experiment, and recorded a number of tracks to MP3 settings at various settings, some with LAME and some without. The results were that I thought there were clear differences between WAV and MP3 files, and I decided to stick with WAV. I posted the results in a thread a little while ago which generated a certain amount of controversy. With this in mind, I decided to repeat the experiment, using a number of pieces of familiar music, recorded in MP3-APS and WAV. It was a "blind" experiment (in that I did not know which type of recording was being played) though hardly a scientific one. It changed my views on a number of things.

First, I downloaded the tracks into my iPod, with one set of tracks marked with a number to indicate if they were WAV or MP3. The tracks were solo instruments (piano and guitar), some orchestral and some vocal works. With all tracks appearing in the menu and the iPod facing away from me, I ran my finger randomly around the circular scroll pad and then hit the play button. I listened to the tracks (using ER4Ps) until I felt comfortable I could identify whether it was MP3 or WAV. Then I recorded the result following which I looked at the iPod menu to see whether it was WAV or MP3.

I did this about 30 times. Not a huge sample, but it took a couple of hours. The results were that I got it right (and also wrong) exactly 50% of the time. This is of course consistent with the idea that MP3-APS recordings are indistinguishable from WAV recordings, at least on my equipment and with my ears (although the exactness of the result was itself improbable!). This is very different from the results I posted earlier, where I was certain I could hear a clear difference between MP3s and WAV files.

Nevertheless, despite these findings, I was still convinced that I could hear some kind of difference which was a subtle subliminal one; I was sure that MP3s sounded somehow less musically satisfying. When I looked back at the result after each test I "knew" that the MP3 track was a little different, perhaps a little veiled, a little harder, that something was missing, even if I couldn't accurately identify the type of recording. To test this I swapped back and forth between some extended tracks in both formats, and in doing so I knew that one of them - the one I'd identified as the WAV track - was just more satisfying than the other, even if I couldn't quite put my finger on why.

Later I went back to my computer to remove the excess files and I discovered something. I'd reversed the numbers I'd put on each track to identify whether it was WAV or MP3. The tracks I'd felt were more musically satisfying were in fact MP3-APS files, not WAV.

The result is that in something that resembled a blind test (though it was nowhere near as rigorous as a proper double blind experiment), I was unable to distinguish the WAV from MP3 files on my iPod. Even worse, I actually felt that the MP3s were more satisfying. I take this not to mean that MP3s are more satisfying, merely that my inherent prejudices against MP3 were preventing me from hearing them accurately.

The other result is that I will now be ripping my CDs to MP3-APS files instead of WAVs, and loading about 400 CDs into my iPod instead of 60, with no material loss in sound quality. I still find this result surprising, and it goes strongly against the grain, but it would be foolish to ignore the results of these non-scientific but interesting tests.

 

[blessingx]

Thanks for posting. We all need to reevaluate our findings from time to time, instead of sticking to the party line.

There's been a few posts in the past, that have indicated the artifacts introduced in psycho-acoustic modeling, may actually "warm" or make the sound more pleasing to some listeners. If the iPod is your only player, have you considered conducting an informal experiment between Quicktime/iTunes (or Nero presets) AAC and LAME MP3? I think they do sound different, and it would be interesting to hear which you prefer.

 

[Oddball]

Would you consider comparing other presets? Some say that standard is better than extreme or insane. Is there a particular reason why you chose APS?

 

[phillevy]

Strangely I've always found mp3/ogg more enjoyable to listen to than CD's (even played through my main hi-fi) and the way i have interpreted this is that being a vinyl addict, I always found CD's harsh and less musical to vinyl. Perhaps the compression introduced into a CD wav file being encoded into mp3 removes the elements of a CD's sound that I never liked to start with (even with the losses). On this basis if I encoded vinyl (which I haven't done to any extent yet) I would expect to always prefer the original wav file to the mp3 in that case. Pure theory but who knows!

 

[lan]

In an unideal sitation as portable listening, I am less inclined to think that one could find differences between MP3 and WAV. Sure there are differences but used in a realistic situation as being outside in the city doesn't warrent using WAV for me.

 

[kwhead]

Excellent post Ross! I haven't done my own blind tests yet, but your results aren't too surprising to me. So many audio effects are really just in our heads, so to speak. Your experience with having had the files swapped is a clear demonstration of the "experimenter expectancy effect" which has a deceptively strong influence on our senses. Many a scientist has had their reputation destroyed over the years because of it. Unfortunately in the audio world such things seem to persist for decades. So many audio magazines have completely uncontrolled setups and basically report total garbage as a result. It is one thing for you or me to kid ourselves, it is another to publish it and charge money for the bogus conclusion! Anyway, great work.

By the way, there was a 14.4% chance that you'd get a perfect 50/50 distribution in your tests. And a 41.4% chance that you'd end up within one choice of 50/50 - so it is not all that unlikely that you had perfect half and half.

Well, now I'm jazzed to do some of my own blind experiments. Enjoy your iPod with 400 CDs

.

 

[br--]

But this experiment really only applies to the iPod, not MP3 vs WAV in general. The difference is painfully obvious when listening to them on good speakers (see my sig).

Many people say the iPod as a source is unstatisfactory in terms of audio quality. Perhaps this is why you could not hear the difference?

 

[kwhead]

Quote:

Originally posted by br-- But this experiment really only applies to the iPod, not MP3 vs WAV in general. The difference is painfully obvious when listening to them on good speakers (see my sig). Many people say the iPod as a source is unstatisfactory in terms of audio quality. Perhaps this is why you could not hear the difference?

That very well could be, but to me the real interesting thing to take away from this is that previously Ross posted he heard a painfully obvious difference on his iPod. With proper testing his opinion changed. Perhaps you should try the same with your nice speakers?

 

[Dylan]

That's a great story and is similar to results I've had when I do tests that eliminate my bias.

This is why I shake my head when people make comments about high bitrate MP3s being "unlistenable", having horrible artifacts, being able to detect compressed recordings instantly, etc. etc.

The same surprising results happen when you remove bias from all kinds of audio comparisons. Competently designed pieces of solid state equipment, when operating within their design parameters, just don't sound much (if at all) different from each other.

Regarding bitrates, I used to fret about perceived compression artifacts in my music. I was considering re-ripping my entire 3500 song library into a lossless format. I wasn't willing to buy music from iTunes because of the compression. I was convinced that I was hearing lots of compression artifacts.

Then I spent some time doing blind tests of different bitrates. Like you, I wouldn't hold mine up to scientific scrutiny but I'm satisfied that I removed any bias. Here's what I found when I compared different formats to WAV.

128 AAC - I could pick it out every time (5/5) but the difference was very subtle and it took extremely close attention to detect it.

192 AAC - Indistinguishable to me.

LAME --aps - I got it right more times then wrong (7/10) and I think I was getting the hang of recognizing it. But I was getting bored with the exercise by this time and it was close enough that I didn't care anymore.

I did my listening with Sennheiser 580's. It's possible I would hear more differences with more detailed headphones like Ety's. Though the differences I could detect were not specific moments but more of a vague feeling that someting was wrong.

It can be hard to do tests like this. I try to put myself in the mindset that no result is more right or better then the other. Otherwise my natural reaction is to fight to preserve my existing way of thinking. (Especially if I've spent a lot of money to support my existing notions.) But if I can keep myself open and non-judgemental I always appreciate in the end that I have a rational result.

 

[AndyH]

Quote:

Originally posted by Oddball Would you consider comparing other presets? Some say that standard is better than extreme or insane. Is there a particular reason why you chose APS?

From his original post:
http://www5.head-fi.org/forums/showt...threadid=63803

Quote:

Originally posted by Ross alt-present-standard - this is where things started to get interesting. Here the difference between WAV and the LAME setting was significantly reduced. The difference was still clearly noticeable, but instruments now had some body and colour. alt-preset-extreme - better than alt-preset-standard (obviously), even fuller sounding; piano sounds still sounded thin and tinkly, and strings could still sound a little scratchy, and the overall image was "small" compared to WAV, but perhaps this was starting to sound acceptable. alt-preset-insane - in some ways this sounded better than alt-preset-extreme, but on every track I ripped using this setting, there was a weird reverberation effect (e.g. solo piano recordings sounded like they were recorded in a cave), which ruled this setting out.

James and I also noticed this "reverberation effect" with the alt-preset-insane encodings as well, here's that message thread:
http://www5.head-fi.org/forums/showt...threadid=65576

 

[Ross]

Quote:

If the iPod is your only player, have you considered conducting an informal experiment between Quicktime/iTunes (or Nero presets) AAC and LAME MP3? I think they do sound different, and it would be interesting to hear which you prefer.

I'd like to do this comparison, but unfortunately I can't use iTunes because I use Windows ME, which iTunes doesn't support (and I can't be bothered changing). I've tried using AAC files with other software, but the iPod won't recognise it.

Quote:

Would you consider comparing other presets? Some say that standard is better than extreme or insane. Is there a particular reason why you chose APS?

Yes. As I posted in the original thread I referred to, and as James has also posted in a separate thread, insane has a clearly audible "fluttering" effect with the iPod. Extreme does this to a lesser extent, and I have had good results with APS, so I was happy to continue using it.

Quote:

But this experiment really only applies to the iPod, not MP3 vs WAV in general. The difference is painfully obvious when listening to them on good speakers (see my sig).

Absolutely. There are a number of qualifications to the experiment: (i) it only used the iPod and a specific pair of headphones; (ii) it had only one subject (me!); (iii) the sample was limited and carried out over a short period; and (iv) the methodology of comparing track A to track B is not one that I think is necessarily the best way to test this effect. If we were to design a true scientific experiment, it would have hundreds of subjects and be carried out over months, and may not even test whether you can hear a difference. (If I were designing an ABX test, I would test hi fi usage over a period of at least a year and, after randomly swapping CDs with MP3 recordings on identical discs but for extended periods of weeks, I would test how many hours the subjects listened to music when they had MP3s and when they had WAVs. This would not test whether they can hear a difference, but whether the type of recording subconsciously altered their listening habits.) It is certainly true that with other equipment and under different conditions - or with a different subject - the results might have been quite different. In fact, my biases are still hoping the results would be different in a properly designed experiment.

Quote:

Many people say the iPod as a source is unstatisfactory in terms of audio quality. Perhaps this is why you could not hear the difference?

Perhaps. The iPod through Etys is nowhere near as good as my Naim CDS2/XPS through my Headroom Maxed Home and ER4S or Senn 650s. However, I think the iPod is a fine sounding player: it is a little soft and has restricted dynamics, but it is detailed and smooth, and to my ears sounds excellent despite its (obvious) limitations. The purpose of this experiment was not to demonstrate in absolute terms that WAV and MP3 are indistinguishable (which it certainly doesn't), but merely to identify which types of recordings I should use on my iPod. For this latter purpose, the experiment was very useful.

 

[austonia]

Ross, thanks for sharing this data. I am setting up my own double-blind sound quality test with several players using EAC/LAME - APX files.

 

[WmAx]

Ross, I congratulate you. Most audio lovers never get to appreciate or understand the power of psychological effects on human perception.

-Chris

 

[TonyTripleA]

Ross, a nice bit of double blind testing with a twist, or is that double, double blind. Anyway congrats.

It reinforces my own unscientific opinion that I cannot differentiat WAV from AAC at 320 on the Ipod with etymotic 4P phones. I don't claim to be an audiopile though.

I'm not so sure I would call your sample size insufficient. Many a meaningful experiment in biology was designed to have sample sizes of around 30. The minimum sample size can be calculated but at first glance I would suggest (as an old ex-scientist) that your results indicate no significant difference with a high degree of certainty.

Great work. I (and/or others) really must find time to replicate it (and isn't that the true peer review of scientific findings).

Cheers,

TonyAAA

 

[JaZZ]

Thanks, Ross, for your revision, which was fascinating to read! And my respect for publishing your change of perspective. I was about to think my ears are getting old (actually they are, BTW), so I'm a bit more comfortable now. Of course there will most likely be audible differences between (even) high-quality MP3s and the original in a more reveiling system than a DAP and particularly with speakers (!), but I've always found that for portable use data compression is no problem to me, and there's no need to bother with Flac or Wave -- the limiting factor for sound quality is by far the restrictions of portable design (size, weight, price, headphones...).