Testing audiophile claims and myths

[jg1337]

Hmmm...I didn't struggle with the length of jg1337's post.

That said, some salient points to provide a Cliff's Notes version.

- Computer-controlled, automated A/B tests
- Levels automatically and electronically matched using a two separate Earthworks Audio M30 professional measurement microphones
- Completely dark room (devices and LEDs of components all patched so there's really no visible light in the room and certainly not at the listening position)
- Test results in the PHP web app. A-B-X, statistically significant at >90% recognition rate in 20+ tests
- Statistically significant results and can discern 3 types of RCA cables (some probably more expensive than your car ), differentiate amps SUPER easily (esp. with tubes involved) and am apparently able to tell the difference in 320 kbps vs 1411 kbps PCM/DSD within seconds

Rather than debate a misunderstanding/miscommunication and redirect what could be an interesting discussion, it would seem to make sense to focus on the science of his testing and the results!

Thank you for summarizing a few points. I felt the need to elaborate on a few side notes and facts since these results "shouldn't be" (given the general tone of this sub forum). Maybe I should record a video demonstrating the facilities or similar. In the end, maybe I should make that very clear, I seek answers to my undoubted experience in a blind test. I am curious to find evidence at which point my test is faulted, may be subject to pure luck (a lot, I guess!) or maybe... in the end... finding some people (given certain circumstances) are able to reliably pass an ABX test.

 

[KeithPhantom]

What's challenging, however, is to explain the test results I and two (non-audiophile) friends got. I must have gotten something very basic, very obviously wrong (I don't see it, though!), otherwise how could it be that I do in fact get statistically significant results and can discern 3 types of RCA cables (some probably more expensive than your car ), differentiate amps SUPER easily (esp. with tubes involved) and am apparently able to tell the difference in 320 kbps vs 1411 kbps PCM/DSD within seconds, really.

Please present your data, assumptions on data sampling and statistical treatment.

That's my provable experience and I welcome anybody to visit me, inspect the circumstances, watch me pass the test and then tell me what's wrong.

No, this is how it doesn’t work. You have to prove to be repeatable and falsifiable. That means other people should see your observations, hypothesis, data collection, experiment design, statistical treatment, and data interpretation in order to criticize your efforts. In other words, you should be able to go to another people’s also-valid experiments of either the same thing or derivations of your topic in order to prove your reliability in detecting differences.

What shocks me is the part between digital-to-analog converters and especially cables. Null tests show that most of the differences are about -90 dBFS. For now and until you present your data and methods, I cannot take you at face value and cannot take you as an exception of the rule until you build enough reliability by repeating the experiments under the design of other fellow scientists and your and their data has been collectively evaluated.

 

[GearMe]

Yes...the data and methodology for the tests on cables and dacs would be interesting to see!

The tube based amps not so much.

 

[sander99]

What is also important is which cables, DACs and amps @jg1337 could (allegedly or really) discern and how they were used.
Because there can of course be cables, DACs and amps that are not audibly transparent.
And in the case of cables that would also depend on output- and input impedances of the connected devices.
And in the case of amps:
-tube amps certainly can be non-audible transparent
-amps driven "out of spec" (unsuitable load, too loud/clipping) can or will be non-audible transparent (under those conditions)

About measurements:
I would expect that in all cases where @jg1337 can really hear a difference there will be a measurable difference.

Another thing: we would assume that no volume riding or gain staging tricks are used. For example using a recording made at -80 dB or a digital attenuation of -80 dB and turning the analog volume control after the DAC wide open.

@jg1337: For me the key point of interrest would be:
Can you really discern differences between two items that both are designed to be audibly transparent and both measure good enough to be considered audibly transparent according to todays scientific consensus?
That is what you sort of suggest by adding your remark about measurements.
But that would be extremely surprising.

 

[bfreedma]

IG1337… By “preventing perceptual error” I mean level matching and direct A/B switching to avoid a preference for a louder sample and error due to the shortness of auditory memory. I may come back and read more of your post, but it’s WAY too long. You should try to be more concise if you want people to read what you write. And rushing to judgement when you don’t even understand what I’m saying doesn’t encourage me to dive in and plow through your word salad.

While I suspect you’re correct and these tests are flawed, I believe you (or anyone else) lose a lot of credibility when you refuse to read posts. A one page equivalent post isn’t unreasonable, particularly when we ALWAYS demand to know testing methodology in detail when these types of claims are made.

 

[bigshot]

I'm afraid I don't know anything about the tests he's citing. I didn't get that far.

First off: "Perceptual error" is an ideologized and pejorative term, as clearly there is no such thing as a definite or reference perception you could possibly compare against - you could only compare against the emitter's emission, but this is not a perception (as the name 'emission' indicates). Perception includes another entity outside of the emitter which can introduce more than what can be measured from the emission. Consequentially, there is no such thing as 'perceptual error'. That's the root of my previous message, you seem to think you know a lot more than you really do (and especially about things you can't possibly be aware of, i.e. other people's perception and feelings).

IG337, I'll explain why I didn't read further than this first paragraph. It started out with a simple misunderstanding of what I meant by "perceptual error". This could have been solved very easily by simply asking me what I meant by that term. But instead, you built a whole argument around your misunderstanding, turning my words into something I never said, and concluding by telling me that I didn't know what I was talking about.

That's starting out on the wrong foot. I don't understand how you could expect me to read further than that first paragraph up there. If someone comes to me with respect, I'll reciprocate. If they come to me twisting what I say to quickly set up a straw man argument, knock it down and do a victory lap, I'm not going to waste my time reading anything more they have to say. That first paragraph is very important. It's where you establish your position. If your position is based on a logical fallacy, I may just dismiss your whole post out of hand. Be aware that I'm going to judge whether I'm interested in communicating with you by the argument you put forward in your first paragraph. Tone also matters. I totally have the right to dismiss people who address me without respect. Your apology would have been enough to get us back on the right track, but then you had to ruin it with this concluding paragraph...

I am genuinely sorry that your short-sighted mind does not allow for reading and understanding more complex statements spanning merely an entire A4 page in response to your very superficial and less than elaborate messages. If you really think a debate is or should be limited to overly simplified short statements, well, then you disqualified yourself from any 'scientific' type of debate all together. What a poor response from somebody who barks so loudly.

I'm sorry. These are the sorts of things a jerk says, and I have no interest in suffering fools gladly. I'm not an admin here and I'm not getting paid for it. I'm here to help people and you clearly aren't interested in my help. So why should I bother? This isn't my first rodeo. I've seen people behave like you around here before and it is never worth my time to engage with them. You'll march around and make a big fuss for a while, and then you'll go away and I'll still be here. Sound Science is a community and you don't go into a new community with this kind of attitude and expect to be welcomed.

Now, if you'd like to start over fresh and drop the rudeness and intellectually dishonest argumentative techniques, we can proceed. The ball is in your court.

 

[Joe Bloggs]

I have set up quite some computer-controlled A/B tests because I wanted to find out about cable, DAC, power supply and other highly controversial impacts on the system, as you said, eliminating visual and other biases. First off: "Perceptual error" is an ideologized and pejorative term, as clearly there is no such thing as a definite or reference perception you could possibly compare against - you could only compare against the emitter's emission, but this is not a perception (as the name 'emission' indicates). Perception includes another entity outside of the emitter which can introduce more than what can be measured from the emission. Consequentially, there is no such thing as 'perceptual error'. That's the root of my previous message, you seem to think you know a lot more than you really do (and especially about things you can't possibly be aware of, i.e. other people's perception and feelings).

As previously stated, I have conducted a number of A/B tests even going so far to buy some components multiple times and implementing Raspberry Pi-based circuits to switch test scenarios without interaction, leaving no room for human test procedure control error. I am a software engineer and highly interested in electrical engineering, so that was quite fun and not at all challenging to build. What's challenging, however, is to explain the test results I and two (non-audiophile) friends got. I must have gotten something very basic, very obviously wrong (I don't see it, though!), otherwise how could it be that I do in fact get statistically significant results and can discern 3 types of RCA cables (some probably more expensive than your car ), differentiate amps SUPER easily (esp. with tubes involved) and am apparently able to tell the difference in 320 kbps vs 1411 kbps PCM/DSD within seconds, really.

I still wonder what causes this - I did eliminate all influences I could have, or at least that I am aware of. Completely dark room (devices and LEDs of components all patched so there's really no visible light in the room and certainly not at the listening position), levels automatically and electronically matched using a two separate
Earthworks Audio M30 professional measurement microphones, all components far behind the listening space in an acoustically professional mastering studio room (aside from the 610Bs, 1176s, LA2A, Avalons, etc. in front of the desk) using reference monitor speakers ($15,000/pair). A raspberry automatically and randomly (as much as randomness is possible using standard computers) controls which set of components play (depending on the test, e.g. testing 3 different DACs), a PHP-based web interface is where I enter my perceptions during execution of the test. PHP reads from an XML file which describes the test in an abstract language, which in turn is also used by the Raspberry to perform the desired actions. Finally, you can see your test results in the PHP web app. A-B-X, statistically significant at >90% recognition rate in 20+ tests.

That's my provable experience and I welcome anybody to visit me, inspect the circumstances, watch me pass the test and then tell me what's wrong.

Finally, the gist of your postings boils down to 'it does not exist unless you can measure it'. I truly understand and value your fact-driven approach (that is why I came up with building my own reliable tests). However, that would logically expand to that every assumption of anything ever in existence simply did not happen until proven; that's quite ridiculous given that the scientific approach throughout human history has always and inevitably been based on experience, based on discovering an unexplained phenomenon which ultimately led to scientific discovery. Lastly, there is no such thing as 'the science [says]', there's generally nothing absolute (but you act as if) - scientific discovery is not something given by a god or nature but purely realized by human beings. Dude, at this point we don't even know how we came to exist.

It may have been the loudness matching method used.

The things you are comparing against are electronic in nature and not really requiring a microphone in room to measure their loudness. Using a mic to measure and compensate loudness introduces a whole lot of random variables in your room that may actually mess up the volume matching beyond what it may be without any effort at matching at all (I'm not saying that that's what's definitely happened, but I don't know what your method for mic based volume matching was, and can definitely see this as a possbibility).

What should have been done, was electronic readings done at the speaker terminals in parallel with the actual speakers, and setting a reference voltage there.

Also, what were the actual means by which volumes were automatically adjusted?

 

[Joe Bloggs]

It may have been the loudness matching method used.

The things you are comparing against are electronic in nature and not really requiring a microphone in room to measure their loudness. Using a mic to measure and compensate loudness introduces a whole lot of random variables in your room that may actually mess up the volume matching beyond what it may be without any effort at matching at all (I'm not saying that that's what's definitely happened, but I don't know what your method for mic based volume matching was, and can definitely see this as a possbibility).

What should have been done, was electronic readings done at the speaker terminals in parallel with the actual speakers, and setting a reference voltage there.

Also, what were the actual means by which volumes were automatically adjusted?

@jg1337 ...?

 

[bigshot]

I don't think he's interested in talking. He just came here to use another sock posting ID to yank my chain and I called him on it. If anyone thinks he's smarter than he actually is, it isn't me. Like I say, this isn't my first rodeo.

 

[Blackwoof]

What's challenging, however, is to explain the test results I and two (non-audiophile) friends got. I must have gotten something very basic, very obviously wrong (I don't see it, though!), otherwise how could it be that I do in fact get statistically significant results and can discern 3 types of RCA cables (some probably more expensive than your car ), differentiate amps SUPER easily (esp. with tubes involved) and am apparently able to tell the difference in 320 kbps vs 1411 kbps PCM/DSD within seconds, really.

And yet folk like you never show any proof of being to tell 320kbit/s MP3. Since Video games, Streaming, TV still use lossy under 256kbps.

 

[sander99]

I don't think he's interested in talking. He just came here to use another sock posting ID to yank my chain and I called him on it. If anyone thinks he's smarter than he actually is, it isn't me. Like I say, this isn't my first rodeo.

You could be right of course. But it is also possible that he is sincere, made "honest mistakes" in his listening tests and maybe also tested some odd not audibly transparent items. And maybe he comes back in the weekend because he has a busy day job during the week...

 

[bigshot]

I'm sure he'll be back. I'm just not sure what the posting ID will be.

 

[Hamltnblue]

This thread caught my eye tonight. Entertaining for sure.
I didn‘t read the entire 977 pages but the old saying “owning a race car does not make one a race driver” definitely applies here. Many experts can state facts, specs, blind tests etc, but have no little idea what they are talking about.
For instance, many think all cables are the same. They are not. They are however a part of the overall amplifier circuit up to the point of connection at the speaker or headphone. Changes in conductor metals, insulation materials, twists per foot, and even soldering techniques can affect the circuit. Differences in capacitance can have a significant effect on individual sounds. Anyone who ever studied basic electronics has been exposed to the formula for capacitive reactance.

The biggest thing missed is that hearing is a learned skill. Frequency range is just a small part of it.
The brain is the processor and has to learn or be taught.
Many audiophiles simply do not hear all of the available sounds, and wouldn’t know if they changed or not.
Put a dozen people in a room with good equipment and have them write down what they hear in a recording. You’ll get plenty of variance.

 

[Mehran]

I suggest audiophiles take the time to actually enjoy music once in awhile. It can help the. neurosis.

 

[71 dB]

This thread caught my eye tonight. Entertaining for sure.
I didn‘t read the entire 977 pages but the old saying “owning a race car does not make one a race driver” definitely applies here. Many experts can state facts, specs, blind tests etc, but have no little idea what they are talking about.
For instance, many think all cables are the same. They are not. They are however a part of the overall amplifier circuit up to the point of connection at the speaker or headphone. Changes in conductor metals, insulation materials, twists per foot, and even soldering techniques can affect the circuit. Differences in capacitance can have a significant effect on individual sounds. Anyone who ever studied basic electronics has been exposed to the formula for capacitive reactance.

The biggest thing missed is that hearing is a learned skill. Frequency range is just a small part of it.
The brain is the processor and has to learn or be taught.
Many audiophiles simply do not hear all of the available sounds, and wouldn’t know if they changed or not.
Put a dozen people in a room with good equipment and have them write down what they hear in a recording. You’ll get plenty of variance.

Nice of you as a new member of the forum with a dozen posts to just casually say people here do not know what they are talking about. You don't even need to read "the entire 977 pages." How much did you read? One page? Three pages?

Cables are the same in the sense that as long as they work and do their job they don't stop you enjoying the music. The potential improvements in sound quality are so increadibly tiny, that for a normal person or even an audiophile it is meaningless. Other things are massively more important such as ergonomics or the frequency response of the headphones. I write this as someone with a university degree in electrical engineering.

Even if we admitted cables do matter, what are people supposed to do with it? Buy ALL cables in the word and select the best 10 out of them to be used in different situations: One cable for rock music, another for classical music etc. Is that what you want? Sorry, but I am not interested of cables. My cables works and I enjoy music. I don't see the point of thinking about some other cables being better. There is always something better. If I keep chasing for better I can never be happy for what I have now. It's not 1948 anymore when audio gear was bad. Decent audio gear in 2021 is so good, that music listening experience is bottlenecked by bad music production. Well produced, mixed and mastered recordings sound phenomenal if you have decent audio gear. The best cables in the World don't help you at all if the recording is ruined by lousy production and loudness war.

The ways we see the World are not the same. Differences in attitude and beliefs can have a significant effect on happiness of individuals. Whenever something in our lives is "good enough" (such as "mediocre" headphones cables) we better take them as victories in life to be happy about, because there are so many things in our lives that are not "good enough" and also not easily improved. Thank God at least normal audio cables don't suck!