Head-Fi.org › Forums › Equipment Forums › Sound Science › Should we have a stickied post/link about blind testing?
New Posts  All Forums:Forum Nav:

Should we have a stickied post/link about blind testing?

post #1 of 23
2/9/12 at 1:00pm
Thread Starter 

I was wondering if we should have a stickied post/link to put with the glossaries of terms in the Useful Links section. I'm a strong proponent of blind testing and using more objective measures for audio comparisons, but it seems like a lot of people do blind testing incorrectly and concluding false things as a result. The Sound Science forum shouldn't have so many incorrectly done blind tests. 

Reply
post #2 of 23
2/10/12 at 12:48pm

What's the proper way to do this?

Reply
post #3 of 23
2/10/12 at 12:50pm

Would be helpful to have it cross-posted across every single forum. 

Reply
post #4 of 23
2/10/12 at 1:10pm

Before we even get to blind testing, how about level matching?

 

Then again, it's the non-level matched, sighted A/B-tested (maybe from prior recollections done a long time ago) fantastical gear reviews that drive sales on head-fi...

Reply
post #5 of 23
2/10/12 at 1:19pm
Thread Starter 
Quote:
Originally Posted by junkers View Post

What's the proper way to do this?


As mikeaj mentioned, level matching is one important consideration to make. I'll admit I don't know everything about what is required for a useful blind test, but I know what things certainly bias one. The "proper way" would be to ideally consider all potential biases, e.g. volume differences, headphone/equipment appearances, knowledge about price, and construct the test in such a way that the subjects are not exposed to them.

 

Reply
post #6 of 23
2/10/12 at 1:49pm

actually I don't really get why level matching is needed for cases like lossy vs lossless, as long as the lossy is directly encoded from the lossless...

Reply
post #7 of 23
2/10/12 at 1:51pm

Quote:
Originally Posted by kn19h7 View Post

actually I don't really get why level matching is needed for cases like lossy vs lossless, as long as the lossy is directly encoded from the lossless...


Well yeah for that, the levels should already be matched unless there's something funky going on in the encoding.

 

 

Reply
post #8 of 23
2/10/12 at 1:59pm
Thread Starter 
Quote:
Originally Posted by kn19h7 View Post

actually I don't really get why level matching is needed for cases like lossy vs lossless, as long as the lossy is directly encoded from the lossless...


I've heard on these forums that you can experience lower volumes if you convert to rather low bitrate lossy formats (such as 128 kbps). Is that not true?

Reply
post #9 of 23
2/10/12 at 2:09pm

hmm... ok, its the first time I heard of this ==

 

reduced dynamic range and other artifacts are possible though, which may result in perception of volume difference...?

 

Reply
post #10 of 23
2/10/12 at 2:20pm

If the lossy encoding involves lowpass filtering and there's significant energy in the higher frequencies that are being rejected, then there could easily be a change in volume.  There could be other things going on as well, but this is the most obvious as far as I can see.

Reply
post #11 of 23
2/10/12 at 5:30pm

OK, so let's say I'm using foobar with LAME and converting my FLACs to something like V5, do I need to do anything else to get a 'proper' mp3? Would the resulting file be OK to ABX?

Reply
post #12 of 23
2/10/12 at 9:34pm


 

Quote:
Originally Posted by mikeaj View Post

If the lossy encoding involves lowpass filtering and there's significant energy in the higher frequencies that are being rejected, then there could easily be a change in volume.  There could be other things going on as well, but this is the most obvious as far as I can see.


 

to me thats also the difference to test for, as its the result of the lossy encoding

 

if replaygain is applied, the averaged volume difference will be spread all over the song instead of just those parts with rejected high frequencies, which doesn't seems to make things better to me either.

 

...and also apply gain to volume won't revive the rejected high frequncies, it's just applied on the low-passed frequencies isn't it?


Edited by kn19h7 - 2/10/12 at 9:49pm
Reply
post #13 of 23
2/11/12 at 12:00am

I seem to have been answering practical questions as if they were hypothetical.  As far as I can tell, you probably wouldn't want to level match lossy vs. lossless.  You want the lower frequencies that have not been filtered out, to be the same volume more or less as the original, which should be what you get if you don't do any normalization.

Reply
post #14 of 23
2/14/12 at 7:48am

So what's the proper way to convert from FLAC to mp3 to properly ABX test them? I think most people will use the foobar ABX plugin, so renaming/resizing them shouldn't matter.

Reply
post #15 of 23
2/15/12 at 7:09am

Level matching refers to the equipment, not the encodings.  Amp A/Headphone A should have the same volume as Amp B/Headphone B.

If your encoded music is not level matched, you did it wrong.

 

Anything less than a double blind with less than 5% error is junk.  Meaning your tester and your subjects can't know what your testing and you will need at least 384 test runs.

 

 

 


Edited by pyramid6 - 2/15/12 at 7:11am
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › Should we have a stickied post/link about blind testing?