[Sonic Defender]

I'll be brief because I'm in a hurry. In a well enough controlled experiment 7 subjects did multiple listening trials of the same song, with the same gear. I controlled everything and I was extremely careful not to bias the testing. Participants were played 1:30 seconds of the same very well recorded song Holy Cole - One Trick Pony in paired sequence. One play was the lossless, the other was the lossy 320mp3 file made from the original lossless copy via JRiver MC 20.
 
The files were played through JRiver MC as well. Each participant helped calibrate the volume they were comfortable with and all trials were left at that volume and the headphones were not moved between trials.
 
There was no difference in the two files in terms of volume either. The participants were simply asked after hearing each pairing if they had a preference for either A or B. I varied when the lossless versus lossy file was first or second in the pairing order. Each pairing was one lossy, one lossless.
 
After 7 participants going through between 4 and 6 pairing trials not one person did better than 50% in terms of selecting the lossless version as sounding better.

 

[sonitus mirus]

What version mp3 encoder was used?
 
Can you provide any additional details about the ABX testing tool and how the volume was matched?  
 
My experience has been that when I use dBPoweramp (rc 15) to directly rip a CD track to Lame 3.99.5 -vbr 0, I'm unable to identify a difference using the Foobar ABX plugin with replay gain utilized.  Generally speaking, I can successfully pass a lot of ABX tests between 160kbps CBR and lossless, but even then there are several songs that don't sound different at all to me at that low of a bit rate.
 
For low bit rates, you should give Opus a try, it will probably surprise you.  I consider it to be the best of the bunch at 96kbps and lower in the small amount of self-tests I administered.  At the highest bit rates available, AAC, MP3, OGG all seem very capable at achieving transparency to my ears.
 
I've been extremely happy with the sound quality of Google's streaming service.  Absolutely no distortion heard and it sounds phenomenal to my ears.  At this point, considering the cost and the excellent features available, I'm done searching for a source for my music, and I am now simply enjoying the tunes.

 

[cjl]

Asking whether they had a preference for A or B won't be a very sensitive test. You'd be better off giving them A and B (and even telling them which is which) and then asking them to match a third sample, X, to A or B. That way you're doing a comparative test rather than a preferential one.

 

[Sonic Defender]

  Asking whether they had a preference for A or B won't be a very sensitive test. You'd be better off giving them A and B (and even telling them which is which) and then asking them to match a third sample, X, to A or B. That way you're doing a comparative test rather than a preferential one.

I don't think it matters as preference is subjective, as is asking people to match X sample against A or B; there is no more inherent accuracy in that as far as I can tell. The rationale for my choice of asking what people preferred is that we all know people believe that they prefer a higher quality signal and the purported weakness is that a 320 MP3 file will sacrifice audible content, so transitively the lossless file should be preferred. Regardless of how you design a listening experiment, you are asking the subject to provide a subjective impression be it through matching or expressing a preference. Unless there is some kind of scientific evidence suggesting that matching is somehow a more accurate gauge of our ability to discriminate differences I think asking what file somebody prefers is appropriate.

 

[Sonic Defender]

  What version mp3 encoder was used?
 
 

It is the LAME encoder built into JRiver. We didn't bother using any tools, which set things by ears. A bunch of us listened to the two files before the experiment at the same volume on the test rig and nobody felt there was any difference in volume between the files. We then ensured that for each participant we left the volume the same throughout the trials.
 
When I listen to people tell how easily they can tell the difference between lossy and lossless files it seems like they feel it is very obvious. In fact, two of the test subjects had assured me they had been through similar tests they conducted on their own and they were very sure they could discriminate between a lossy and lossless file. One other subject only owned high resolution music swearing he found even redbook CD not as good. Ironically he did the worst and expressed a preference for the lossy file more than 50% of the time.

 

[sonitus mirus]

The methodology seems appropriate for what you are setting out to accomplish.  Awareness is a big factor.  
 
You should try taking a piece of music and dropping the bit rate down to a level where you can pass an ABX and then let others hear this.  Even that would amaze some people and would most likely not be a night/day difference that everyone could identify.  It could also show that the testing works, and that there are noticeable differences to be heard, but perhaps not at the higher qualities.  Maybe some of the folks that completely disregard ABX tests would feel differently about the matter; because, if it works between 128kbps and lossless, why would the test be invalid between 320kbps and lossless?  Probably would not help, but you never know. 

 

[castleofargh]

I don't think it is enough to test for audible differences(rapid switching works way better for that). but as an eye opener to show people how they misjudged what they thought they could hear, it probably works just fine.
the trap here is that with random luck, someone could have passed the test if it was limited between 4 and 6 attempts. but I guess listening to at least 15*2 full songs per person wouldn't be much fun.

 

[Sonic Defender]

  I don't think it is enough to test for audible differences(rapid switching works way better for that). but as an eye opener to show people how they misjudged what they thought they could hear, it probably works just fine.
the trap here is that with random luck, someone could have passed the test if it was limited between 4 and 6 attempts. but I guess listening to at least 15*2 full songs per person wouldn't be much fun.

I think it is a perfectly valid way of exploring subjective opinion, and yes people could have guessed correctly, but statistically speaking the odds are low that multiple people will guess successfully. As I said before, regardless of the methodology you are asking people to make a subjective evaluation so no matter what you do, the results are only as reliable as we accept subjective reports to be. If people can't tell after listening to multiple trials then the differences if any exist at all are not audible enough to have practical significance. The debate isn't whether or not people with good hearing and plenty of training can't distinguish small differences, I think we can accept for some people that is possible; however, people do not experience music like that, people listen to music so in practical terms, when just listening people can't reliably hear the differences.
 
It is like that in drug trial testing. Sometimes there is a detectable effect, but it is so small that it has no practical value and hence the drug is not approved as it would be an in-effective treatment. It is the same thing here, if an average person, under normal listening conditions can't hear a difference, then practically speaking there is no difference. And yes, doing more trials simply runs the risk of bringing listener fatigue into play which would call into question the results even further. I found after 4 trials some people were getting bored. That is why at our next meet in 5 months we are doing this again with another group of subjects, expanding the data set. At that point I will combine the data and show it here. As it is now the sample is too small, but I think the trend will continue regardless of the size of the sample.

 

[cgtyguner]

I don't think it is enough to test for audible differences(rapid switching works way better for that). but as an eye opener to show people how they misjudged what they thought they could hear, it probably works just fine.
the trap here is that with random luck, someone could have passed the test if it was limited between 4 and 6 attempts. but I guess listening to at least 15*2 full songs per person wouldn't be much fun.:confused_face:

i agree to that rapid switching would work better. Press play exact the same time and start to listen two songs but always one is muted and the other is not. And periodically change the muted one. After song is finished,ask "did you heard any diferences?/any unusual thing?/Which one was better/which one was lossless? Maybe we can make different experiments and compare them. if there's a meaningful diffrence in rates of guessing right, we can say that this test setup is better than the other ones etc. But comapring experiments is is not easy. i think first, we should do the same experiment with same participants on different times. Between this two, participants sould not know anything about the results. im just thinking loud

 

[Budgie]

I believe your test was successful in that it eliminated confirmation bias as a variable. My opinion is confirmation bias is one of the biggest factors in audio preferences.

 

[Budgie]

Lol- Your testing confirms my confirmation bias about confirmation bias!

 

[castleofargh]

the objectivist's paradox. how not to be biased against bias?

 

[bigx5murf]

I think the age of the participants matter here. I know people around my age grew up on lossy mp3, and what lossless CDs we did have, we heard through crappy discmans like devices, with crappy headphones. After a decade or two of primarily listening to lossy formats. It's easy to become conditioned to actually preferring the sound of a lossy format such as mp3.  I know this was the case with me. With my old setup, klipsch promedias, analong output from onboard sound. Using Tidal's lossy vs lossless tool, to pick out lossy vs lossless was random for me. When I started using an external DAC, and some passive speakers powered by an external amp. I started choosing lossy as my preference 5/5 times over and over.That's when I became aware, that I was choosing the lossy formats for the distortion it was doing to the end of vocals, and the hard hitting, but consistent tone of the bass hits. I was conditioned to prefer these forms of distortion.
 
I decided to switch to Tidal from Spotify. I copied my playlist over one at a time, choosing tracks from the same albumn version best I could.I had two main playlists I used. One that I primarily listened to at the gym, through my smartphone and IEMs, and one that I primarily listened to at my desktop through speakers. I couldn't believe it, but songs I was listening to for many years. I was starting to notice subtle differences in what I was expecting. Such as a pitch of a particular vocal not hitting as high or as low as I was anticipating.