[xenoVa]

Hello everyone

It is 2019 and people are still debating if certain things make a difference or not (cables, 16 vs 24 bit, FLAC vs WAV and many more. The problem is, people still discuss things subjectively and make subjective tests instead of doing simple experiments which will give objective results.

I designed a very simple experiment, it will tell us if a certain parameterer makes a difference or not. Two identifical audio systems will be used, everything will be the same expect the parameter we are researching for. In my example it is the cable. The sound outputs will be recorded and analyzed.

Then we will compare results of the two analysis. If they are the same or the difference is
indistinguishable small, we will declare that there isn't a sonic difference between the cables.

If analyses show a difference, we will repeat the experiment many times with various gears. If the results always show a certain difference (i.e highs being louder with Cable B) we will declare that there is a difference, highs sound louder with Cable B.

This experiment can be used for testing any parameter. Want to test 16 vs 24 bit? Set everything the same and play 16 bit audio from the one system and 24 bit from the other. Then compare the results.

Easy, isn't it? . Do I miss something or no one ever thought making such an experiment?

 

[tomb]

Yes - you missed the human ear and brain.

2nd, if you've determined the everything is the same except for the cables or the source files, you can delete the rest of the equipment and simply measure electronically at the output of the device (or cable, if you're measuring that). Most everyone is already doing that.

All that is moot, though, because it assumes that the measuring equipment is able to measure everything that the ear can detect or the brain can perceive.

We can measure frequency response, distortion, S/N, and other commonly standardized attributes of a music signal, but that only grossly approximates certain aspects of what the human ear and brain are experiencing. The differences in cables and things like audio capacitors are smaller than the differences those measurements can detect. Everyone discounts the human ear and brain as very imprecise instruments. That may be true in a classical measurement regime, but the reality in the perception of music is something very different.

 

[castleofargh]

Hello everyone

It is 2019 and people are still debating if certain things make a difference or not (cables, 16 vs 24 bit, FLAC vs WAV and many more. The problem is, people still discuss things subjectively and make subjective tests instead of doing simple experiments which will give objective results.

I designed a very simple experiment, it will tell us if a certain parameterer makes a difference or not. Two identifical audio systems will be used, everything will be the same expect the parameter we are researching for. In my example it is the cable. The sound outputs will be recorded and analyzed.

Then we will compare results of the two analysis. If they are the same or the difference is
indistinguishable small, we will declare that there isn't a sonic difference between the cables.

If analyses show a difference, we will repeat the experiment many times with various gears. If the results always show a certain difference (i.e highs being louder with Cable B) we will declare that there is a difference, highs sound louder with Cable B.

This experiment can be used for testing any parameter. Want to test 16 vs 24 bit? Set everything the same and play 16 bit audio from the one system and 24 bit from the other. Then compare the results.

Easy, isn't it? . Do I miss something or no one ever thought making such an experiment?

you can and will sometimes have to do this. but it will depend on the question we're trying to answer. because if the question is about audible difference, then you're pretty much never going to end a debate by showing measurements. those who know enough to properly interpret and draw conclusions from measurements are likely to already know if a given cable will matter or not. and everybody else is just going to argue about why it's not conclusive for audibility(for good or super dumb reasons).

second limitation is that when you involve the headphone and a microphone, the resolution of what you can measure instantly takes a massive hit(distortions from both transducers, ambient noise in the room). for something like a FR, so long as you don't move either between measurements, you're probably good. but for anything relatively small(like 16 vs 24bit), what you're recording won't have even 16bit of resolution.

otherwise, depending on the question we're asking, your idea will work pretty well and I've been doing that on a few occasions with some success(not in ending the debate, but some success in getting my answer ^_^). the basic principle at least is correct, reduce the extra variables as much as possible.

ps: I think "identical" is the word you want. je suis ceinture jaune d'Anglais.

 

[castleofargh]

Yes - you missed the human ear and brain.

agreed, but in some cases we could imagine running the recorded samples through an ABX test(after matching time and amplitude) if audibility is the main question.

2nd, if you've determined the everything is the same except for the cables or the source files, you can delete the rest of the equipment and simply measure electronically at the output of the device (or cable, if you're measuring that). Most everyone is already doing that.

again it's going to depend on what we're looking for. a headphone/IEM cable could be a pretty good example of something susceptible to show increased impact in the complete circuit instead of plugged straight into some analyzer. but in several cases, you're right that a more direct approach may help improve the resolution of the measurements(whatever they are).

All that is moot, though, because it assumes that the measuring equipment is able to measure everything that the ear can detect or the brain can perceive.

We can measure frequency response, distortion, S/N, and other commonly standardized attributes of a music signal, but that only grossly approximates certain aspects of what the human ear and brain are experiencing. The differences in cables and things like audio capacitors are smaller than the differences those measurements can detect. Everyone discounts the human ear and brain as very imprecise instruments. That may be true in a classical measurement regime, but the reality in the perception of music is something very different.

I don't get this. I would expect a human ear to detect something we fail to measure only when we measure the wrong stuff(or if the measurement rig is crazy bad). I have yet to see a case where someone can conclusively identify a difference by ear in a blind test but we can't find a clearly measurable difference in the signals. if we have no idea what we're looking for, we can just go with a simple null of the 2 signals. there will be many situations where we'll end up with a measured difference that's inaudible, but I'd be very curious to see a null showing nothing and people actually hearing a difference. and I mean hearing as a fact here, not as an opinion from sighted impressions.

 

[tomb]

This is obviously a rabbit hole and you (castleofargh) are an expert at it, but blind testing is inconclusive in audio. Auditory memory is too brief and too indiscriminate. It takes practice. This is at least anecdotally true in that you have to live with an amp/source for a while to be able to tell the differences, when slight. There are many studies about the ephemeral nature of auditory memory in humans. This one compares that directly to sight: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2667065/. The very title of "blind testing" is a bit ironic, because it implies that the superior sense is a foregone conclusion: sight, not hearing.

 

[xenoVa]

This is obviously a rabbit hole and you (castleofargh) are an expert at it, but blind testing is inconclusive in audio. Auditory memory is too brief and too indiscriminate. It takes practice. This is at least anecdotally true in that you have to live with an amp/source for a while to be able to tell the differences, when slight. There are many studies about the ephemeral nature of auditory memory in humans. This one compares that directly to sight: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2667065/. The very title of "blind testing" is a bit ironic, because it implies that the superior sense is a foregone conclusion: sight, not hearing.

Yes that's what I am talking about , a human's ability to test audio is not very good so I designed an expetiment which is completely objective and doesn't have any human elements.

 

[castleofargh]

This is obviously a rabbit hole and you (castleofargh) are an expert at it, but blind testing is inconclusive in audio. Auditory memory is too brief and too indiscriminate. It takes practice. This is at least anecdotally true in that you have to live with an amp/source for a while to be able to tell the differences, when slight. There are many studies about the ephemeral nature of auditory memory in humans. This one compares that directly to sight: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2667065/. The very title of "blind testing" is a bit ironic, because it implies that the superior sense is a foregone conclusion: sight, not hearing.

sight is our main source of information, the one we will typically trust over any other(thus how often we can so easily "hear" differences when we see 2 different products, but suddenly have a harder time when we can't see what we're hearing). it's also the biggest area of the brain compared to any other senses. that's exactly why we try to remove sight from any experiment about hearing because otherwise it's not much of an hearing test. so I'm not going to argue about that paper or your general position about sight. and I also clearly agree about memory's fallibility. that is why most listening test will try when it's possible, to use short samples and instant switching to limit to a minimum the interference of memory and its obvious imperfections. so I guess we agree with what a human is, but not on what to do about it. ^_^

now about your remark that it takes time to learn about an amp and maybe notice things that a rapid test might have missed. I'm inclined to agree for at least some aspects. but if after knowing your amp real well over a long period of time, you aren't able to pass some blind test with the sound samples of your choice, my conclusion is going to be that you can't hear any difference and all those you thought you've learned to notice were BS made up by the brain. in the end I'm a simple guy, I know that a controlled test can prove audibility(disprove the null hypothesis) with very high confidence. and I know that uncontrolled impressions can lead to report of audible differences even when there is none to be heard, making it completely inconclusive when small differences are at play. those two very clear facts makes it easy IMO to determine what to trust about audibility. unless I see a controlled experiment demonstrating audibility, I'm going to assume that claims of audible differences are beliefs instead of facts and reject any attempt to draw conclusions based on those beliefs.
if a listener believes that he needs specific conditions to pass a listening test(very long samples, test performed over 5 weeks, his favorite chair and a glass a bourbon, playing the same sample in a loop while he's asleep, etc), so long as it doesn't clearly interfere with the purpose of the test which is confirming audibility instead of eyesight or preconceptions, I'm OK with that. but, no repeatable demonstration, no cookie. random audiophile claiming that he can hear this and that, so long as I have no mean to tell when it's correct and when it's total audiofool crap, I'm going to give it a zero in confidence.

 

[Sgt. Ear Ache]

This is obviously a rabbit hole and you (castleofargh) are an expert at it, but blind testing is inconclusive in audio. Auditory memory is too brief and too indiscriminate...

this statement makes no sense. In fact, the last sentence is exactly the reason blind ABX testing matters so much. Auditory memory is so brief and indiscriminate that any conclusions based upon comparing something NOW to something from the past are rendered meaningless. So, buying a nice new DAC, connecting it to your system and then deciding after listening for a while that it's just "night and day" better than the DAC it replaced and that you now haven't listened to for several hours or days to make the comparison is totally meaningless. Unless you are comparing those two components (or cables, or amps or files or whatever) one against the other real time blind volume-matched, you aren't actually comparing them at all.

 

[tomb]

this statement makes no sense. In fact, the last sentence is exactly the reason blind ABX testing matters so much. Auditory memory is so brief and indiscriminate that any conclusions based upon comparing something NOW to something from the past are rendered meaningless. So, buying a nice new DAC, connecting it to your system and then deciding after listening for a while that it's just "night and day" better than the DAC it replaced and that you now haven't listened to for several hours or days to make the comparison is totally meaningless. Unless you are comparing those two components (or cables, or amps or files or whatever) one against the other real time blind volume-matched, you aren't actually comparing them at all.

You missed the point. The ear transducer and auditory memory are largely linearly sequential, and passed through a single pipe, as opposed to the eyes, which contain many receptors and a corresponding nerve bundle and brain nuclei.

So … it takes practice to develop a memory of complex music passages and frequency response. Go to a headphone meet and you may be impressed with an amp or DAC that peaks at a few strategic frequencies along the audio band. Your ears will tend to remember and focus on that, because the task is very limited. Buy or take that device home and you'll probably get sick of the emphasis before long. Similarly, if you had blind testing of how two devices reproduce a single frequency, it might be valid. Compare each under those conditions and auditory memory and hearing discrimination are easier to achieve.

Why do people review audio equipment by listening to certain passages of music? Because they've learned and practiced remembering the complex music passages so that the ear and the brain can focus on the discriminating parts.

Anyway, thanks to you and castleofargh for digging the rabbit hole. Unfortunately, I suspect the OP's thread will not end the debates.

 

[castleofargh]

the devil is in the Just Noticeable Difference.

 

[Sgt. Ear Ache]

You missed the point. The ear transducer and auditory memory are largely linearly sequential, and passed through a single pipe, as opposed to the eyes, which contain many receptors and a corresponding nerve bundle and brain nuclei.

So … it takes practice to develop a memory of complex music passages and frequency response. Go to a headphone meet and you may be impressed with an amp or DAC that peaks at a few strategic frequencies along the audio band. Your ears will tend to remember and focus on that, because the task is very limited. Buy or take that device home and you'll probably get sick of the emphasis before long. Similarly, if you had blind testing of how two devices reproduce a single frequency, it might be valid. Compare each under those conditions and auditory memory and hearing discrimination are easier to achieve.

Why do people review audio equipment by listening to certain passages of music? Because they've learned and practiced remembering the complex music passages so that the ear and the brain can focus on the discriminating parts...
.

which, again, only matters if those complex music passages are compared on equipment blindly and concurrently..."remembering" complex music passages is a terribly flawed way to compare equipment - especially if the person doing the comparison has any sort of vested interest in one piece of gear being an improvement over another such as having spent significant money on the new item.