Head-Fi.org › Forums › Equipment Forums › Sound Science › REMINDER: MP3 vs. CD Test - closing at end of January 2013!
New Posts  All Forums:Forum Nav:

REMINDER: MP3 vs. CD Test - closing at end of January 2013!

post #1 of 12
Thread Starter 

Happy New Year everyone!

I posted an invite to an MP3 vs. FLAC test here in head-fi about a month back.

 

A reminder with plenty of time to go before closing the data collection.

--- Originally posted on AudioAsylum ---
Since opening the high bitrate MP3 vs. CD test on Dec. 11th, I have received 41 responses so far to the detailed survey. I will not analyze the data until the end of January, but just eyeballing the spread of results (Set A, B, "same") is already quite interesting. The respondents have come from 4 continents so far and reading some of the comments, I really appreciate the time people have put into this!

Furthermore, it's great to see a nice spread of equipment used from inexpensive (but good) headphone gear all the way to megabuck $50K+ systems.

Although we "shoot the breeze" around here and have great discussions around the hardware (sometimes inflaming arguments), it is infrequent that we actually do something like this where we stand up and be counted based on the actual experience of listening. I know that this isn't strictly "scientific" and many variables cannot be controlled in an open test like this, but for us "non-pro's", this could be the closest we get to participating in something which I hope is educational and (hopefully) fun as a hobbyist beyond theoretical discussions.

If you haven't given this a try, I encourage you to take some time and give it a shot. Be involved in a simple "blind test" (perhaps the only time in one's life) knowing you've tried something like this and contributed to the data set (whether one believes it's significant or not).

Thanks again AbeC. for hosting the fast link! Much appreciated, bro.

PS: One request - could some of you who participate in audiophile discussions in Asia (India, China, Japan, Malaysia, Singapore, Indonesia, Philippines, etc...) spread the test around. Would love to get some data from those folks!

 

Get the test & instructions here:

http://archimago.blogspot.com


Edited by Archimago - 1/5/13 at 4:51pm
post #2 of 12

Done and answered survey !

post #3 of 12
Thread Starter 
Quote:
Originally Posted by nick_charles View Post

Done and answered survey !


Thanks Nick!

post #4 of 12
Thread Starter 

Hi guys - thanks for all the entries so far! As of today, I've got 100 respondents for the MP3 survey / blind test now! Asia now "on line" with a few testers as well.

 

 

Still 1.5 weeks so there's time to get more results in... IMO the numbers are looking interesting and I would very much like to see more results to test significance!

 

A quick question for those who may have looked into the jitter issue with digital. Compared to the amount of "error" created by the MP3 algorithm tossing out bits ~320kbps, do you believe that in a standard system with say a Stereophile-measured jitter of <1ns for the digital component (pretty well ALL decent systems!), that jitter creates more or less difference / degradation than MP3? (ie. Would you be more concerned about the audibility of MP3 or potential jitter issues from the computer/cables/DAC...?)

post #5 of 12
Thread Starter 

Study complete!  Total respondents - 151.

Follow on my blog as I post the procedure and analysis in the days ahead.  Thanks for all the input.

post #6 of 12

Good write up and some interesting results. I was seeing the same trend in one of the files on my test where people were ranking the low bitrate file the highest(mine was at 128kbps though). But my sample size was much too small to actually draw any conclusions.

 

One thing I'm wondering though, is that by disabling the lowpass filter on the mp3's, it would require it to change more of the actual audible parts of the files resulting in an easier to notice difference. The degredation in sound quality is measured to be around the same, but I wonder if the degredation in audible sound quality is greater in your files. I know I can't hear past 18kHz so the lowpass filter seems like an easy way to compress for me.

 

It looks like you did comment on that though.regular_smile%20.gif


Edited by chewy4 - 2/4/13 at 1:38pm
post #7 of 12

Let me confirm this:

 

- More people mis-identified the mp3 set than got it right?

 

- More people with expensive equipment got it wrong than those with cheaper equipment?

post #8 of 12
Quote:
Originally Posted by scuttle View Post

Let me confirm this:

 

- More people mis-identified the mp3 set than got it right?

 

- More people with expensive equipment got it wrong than those with cheaper equipment?


Looks like it.

That too on a variety of equipment. 

 

Wow. 

post #9 of 12

Hats off to Archimago, that was a great study.  Thanks for your efforts.  beerchug.gif

 

Quote:
Originally Posted by scuttle View Post

Let me confirm this:

 

- More people mis-identified the mp3 set than got it right?

 

- More people with expensive equipment got it wrong than those with cheaper equipment?

 

The results are hilarious tongue.gif  So much for the sonic benefits of lossless audio files.  I'm especially amused by the fact that people with >$6000 equipment chose mp3 over lossless as the superior format by an almost 2 to 1 ratio, compared to just over 50% people with <$500 equipment.  I wonder if that's because people who pay dearly have more powerful expectations getting in the way of experience or if they're just older on average and can no longer hear the high frequencies where additional encoding makes a difference..  Too bad the survey didn't ask for people's age.

post #10 of 12
Quote:
Originally Posted by eucariote View Post

The results are hilarious tongue.gif  So much for the sonic benefits of lossless audio files.  I'm especially amused by the fact that people with >$6000 equipment chose mp3 over lossless as the superior format by an almost 2 to 1 ratio, compared to just over 50% people with <$500 equipment.  I wonder if that's because people who pay dearly have more powerful expectations getting in the way of experience or if they're just older on average and can no longer hear the high frequencies where additional encoding makes a difference..  Too bad the survey didn't ask for people's age.

 

 

What's really interesting is that people preferred the more processed sound - if bad hearing was too blame then we'd have seem something closer to 50-50.

post #11 of 12
Quote:
Originally Posted by eucariote View Post

The results are hilarious tongue.gif  So much for the sonic benefits of lossless audio files.  I'm especially amused by the fact that people with >$6000 equipment chose mp3 over lossless as the superior format by an almost 2 to 1 ratio, compared to just over 50% people with <$500 equipment.  I wonder if that's because people who pay dearly have more powerful expectations getting in the way of experience or if they're just older on average and can no longer hear the high frequencies where additional encoding makes a difference..  Too bad the survey didn't ask for people's age.

 

I'd need to run the calculations to confirm (and don't feel like doing it now, would probably need to reference some things anyway), but I'd think that the preference difference between the >$6000 and <$500 groups would not be statistically significant.  Or at least, it's borderline, dependent on pretty liberal interpretations of the data.  There weren't too many respondents in the >$6000 group.

 

That said, if there is some difference, I think your explanation based on age is a good bet.

post #12 of 12

^ you're right.  I just calculated the statistical difference between two proportions.

 

with z value for alpha=.05  being 1.65, proportion(rich)=.65 and proportion(poor)=.54, n(rich)=23 and n(poor)=48
90% confidence intervals are = (0.65-0.54) +/- 1.65 * sqrt(0.65*(1-0.65)/23 + 0.54*(1-0.54)/48)
=.11 +/- 1.65*0.122
=.11 +/- 0.2025
= (-0.0925, 0.3125)
 
We can be 90% confident that true proportion of rich lovers of mp3 files is between 9% lower and 31% higher than poor lovers of mp3s.  Since the latter is included within the former, the difference is not significant.  Rich owners of audio gear do not prefer mp3s over lossless significantly more than poor owners of audio gear.
 
Somebody can check the math in case I borked it.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › REMINDER: MP3 vs. CD Test - closing at end of January 2013!