Testing audiophile claims and myths

[559938]

I get all that, no problem. But what interests me here isn’t the comparison of the original with the CD output.
What interests me are the differences measured between the CD output with and without the “tweaks”. That was repeatable and consistent across time and environment. That differences were also measured on three different CD players of varying engineering depth would also seem to indicate, combined with repeatability, that the results are unlikely to be down to system self-noise.

My point here is that this article (and it is an article, not a scientific paper) is food for thought for further investigation, isn’t it? It’s (possibly maybe) an example of where measurements might be able to take us in our understanding of the role of power in our hifi systems.

We know noise from household appliances can disrupt or hifi. That’s a given. And those of living in old apartment buildings k ow that what our neighbours do impacts us all day every day. But I’m not at all sure we know how deep that rabbit hole goes. This article seems to start trying to find out.

I’m very happy with what’s proof and what isn’t, but I’m also inquisitive. Theres’s a long way to go.

On ABX and DBT - it doesn’t bear out the effect of long term listening, especially in the domain of the fatigue brought about by excessive electrical noise polluting the system. Like going from a PC or MacBook to a dedicated streamer. I worry that rapid testing under pressure isn’t designed to detect very small differences that only make themselves apparent over longer periods. Much like the quality in video monitors that seem negligible until you have to look at it for hours on end, or the noisy office that your brain filters out but is still there making you concentrate harder are wearing you down.

I don’t think that there’s zero merit to the article, and it has flaws, but isn’t it enough to get interested?

 

[559938]

@VNandor ”In short, I think that either the tested playback chain, their recording equipment or their signal processing is broken.”

Why? Because the results don’t correlate with previous experience? Sorry if I’ve misunderstood.

 

[bigshot]

You might want to look at the first post in this thread. It has links to pages that explain why ABX is the best way to conduct listening tests. Long term listening is a poor way to compare sound samples. Auditory memory is notoriously short for similar sound samples- as short as a few seconds. With this particular test, I doubt any of this would be anywhere close to audible.

You asked what I think of this article… whether by accident or design it omits the most important factors and focuses on tiny details. It doesn’t prove anything, in fact it raises questions about why they structured this test the way they did. As I’ve said twice already, everyone knows different components measure differently. We all agree on that. The real question is any of this audible in normal use. Without a DBT, this paper proves nothing. It’s just another manufacturer’s spec sheet with abstract numbers given no real world context. My vote goes for zero merit.

 

[559938]

You might want to look at the first post in this thread. It has links to pages that explain why ABX is the best way to conduct listening tests. Long term listening is a poor way to compare sound samples. Auditory memory is notoriously short for similar sound samples- as short as a few seconds. With this particular test, I doubt any of this would be anywhere close to audible.

You asked what I think of this article… whether by accident or design it omits the most important factors and focuses on tiny details. It doesn’t prove anything, in fact it raises questions about why they structured this test the way they did. As I’ve said twice already, everyone knows different components measure differently. We all agree on that. The real question is any of this audible in normal use. Without a DBT, this paper proves nothing. It’s just another manufacturer’s spec sheet with abstract numbers given no real world context. My vote goes for zero merit.

I read the ABX things, cheers.

 

[VNandor]

If VNandor is correct, then they didn’t even do 2 correctly.

@VNandor ”In short, I think that either the tested playback chain, their recording equipment or their signal processing is broken.”

Why? Because the results don’t correlate with previous experience? Sorry if I’ve misunderstood.

I was so bothered by the figure they gave i actually did a test using their method and it turned out I was wrong. I got pretty much their results, my measured peak difference was -19dB which is about 11%. I checked the difference file and most of the difference comes from the lowest octave and the highest octave. I think the low frequency difference is caused by the DC protection filter and the high frequency difference is caused by the anti-imaging filter. I don't have any real way to know if most of these differences would disappear once I tried to compensate for it. The programs I use don't have a built-in way to try and equalize these differences out.

The reason I thought this must be completely wrong is because I ran a bunch of null tests ~10 years ago comparing different DAC outputs (from phones and DAPs) and the biggest difference I could find was an order of magnitude below this. It seems like the difference between a mid tier phone's DAC and a dedicated player's DAC is smaller than the difference between my current DAC and an ideal "perfect" DAC is which I didn't think would be the case.

i checked my other claim about mp3. With the file I used I got a peak difference of 16% with the average difference between the mp3 and uncompressed file being 1%.

 

[bigshot]

You were nulling the original PCM source file against the file run through the DAC and back through an ADC?

 

[VNandor]

You were nulling the original PCM source file against the file run through the DAC and back through an ADC?

Yes, this is exactly what I was doing. Something to note is that I only have 1 cable for this, so I used 1 channel of the PCM source. I double checked I was comparing the correct channels.

 

[bigshot]

How can you isolate differences caused by the DAC as opposed to the ADC? Is there some sort of calibration?

 

[VNandor]

How can you isolate differences caused by the DAC as opposed to the ADC? Is there some sort of calibration?

I have not made any effort to isolate any of the variables in the chain. The calibration would only work if I had a reference that is verifiably more accurate than the ADC I use. The difference could come from the input for all I know or even from the cable I used to connect the input and the output with. The null test isn't meant to discern between these. I think I got a better result when testing the different DACs because the recordings I took went through the same input so in that case the effect of the input on the measurement was greatly reduced. In this case, both the input and the output can (and will) make the null worse.

 

[bigshot]

I wonder why they used a null test when they couldn’t isolate the variables. For a music listener, the playback would be more pertinent than the capture, but it seems there’s no way to know what that is doing the way they tested it. I wonder if some sort of waveform comparison would be more relevant.

 

[559938]

I wonder why they used a null test when they couldn’t isolate the variables. For a music listener, the playback would be more pertinent than the capture, but it seems there’s no way to know what that is doing the way they tested it. I wonder if some sort of waveform comparison would be more relevant.

Isn’t the waveform comparison further on into the article? Where they stop bothering with frequency domain and work with time instead, comparing those errors to the 22 microsecond CD sample intervals?

Thanks for the responses so far.

 

[bigshot]

Time shouldn’t be an issue. If it can reproduce 20 to 20, the time factor should be fine. I suspect the ADC is more subject to error than the DAC.

 

[danadam]

On ABX and DBT - it doesn’t bear out the effect of long term listening, [...] I worry that rapid testing under pressure isn’t designed to detect very small differences that only make themselves apparent over longer periods.

I read the ABX things, cheers.

Really? So where does it say that ABX/DBT can't last long?

 

[bigshot]

Or that it's performed rapidly under pressure...

I think he read the info on ABX and realized that he was misinformed about it. That's fine. He knows now.

 

[559938]

I’m happy to admit that I’m not sure it’s a good thing to do forced choice testing for differences that are so small/subtle, under conditions that can produce changes to participants’ psychological and physiological states - we all know how much our musical experiences are affected by mood, sleep, alcohol, stress, time pressure, diet…

But returning to the interesting idea in the article, the measurements and discussion in the second half pertaining to the addition of the full suite of tweaks and also each tweak individually seem plausible. I can see that it doesn’t, however, give any evidence as to whether those variances would be audible.

However, the scope of the article wasn’t to determine that. It was merely asking, I hear differences that the usual measurements I see don’t account for, so is it possible to find a different system of measurement that might start to explain what I hear? (I’m obviously paraphrasing).

Viewed from that perspective it seems interesting and has piqued my intellectual curiosity. It certainly seems to suggest more investigation would be worthwhile to follow the rhesus through to a point where a solid conclusion could be drawn.

Thanks to those who tried to help my understanding!