[Joe Bloggs]

There are a number of possible reasons for this but the main one is that the person hasn't isolated the exact difference heard & identified a section of audio where this difference can be routinely identified. Until he does this he isn't ready for an ABX test as this is what is needed to do a "genuine" ABX test.

If the tester had come across a real difference he could hear in sighted testing, why wouldn't he have picked it out as his basis for the blind test?

Does this mean that there isn't a difference that we can hear, no? This binary view of audio that uses a flawed test as the basis for its conclusions, is obviously badly mistaken.

This well tested view of audio that uses as good a test as there has ever been as the basis for its conclusions is obviously well correct. :rolleyes:

Ooh look, I can do adjectives! :rolleyes:

 

[mmerrill99]

There are a number of possible reasons for this but the main one is that the person hasn't isolated the exact difference heard & identified a section of audio where this difference can be routinely identified. Until he does this he isn't ready for an ABX test as this is what is needed to do a "genuine" ABX test.

If the tester had come across a real difference he could hear in sighted testing, why wouldn't he have picked it out as his basis for the blind test?

Because the ABX test is a statistical test that requires at least 16 trials for any statistically significant result. We don't normally listen forensically - it's a different approach to listening - so we can come to conclusions about two different devices/tracks sounding different without identifying a very specific difference. Doing an ABX test requires a very different approach than our normal listening & tests us in different ways to our normal listening - ways that you seem to not understand or not accept.

I've given ultmusicsnob's posts of his positive ABX results as an example of just what's needed to do a valid test - half-hearted listening or listening for a somewhat vague & non-specific difference is a sure way to get a null result, as he outlines

Let's look at Archimago's recent ABX tests of MQA where he gets 8/10 correct trials -he didn't do 16 trials (the recommended minimum) & many would reject this ABX test as invalid for the reason. But look at what he says about this "However, I did try to be more deliberate and "serious" than last time and took a number of minutes at the start to listen between the two files and pick a place to start where I thought I could familiarize myself with the slightly different sound."In other words he "trained himself" for this test

In his last ABX test of a 16/44 file VS MQA file (but not decoded through MQA) he achieved 7/10 correct (one less than above) & he concludes this "a quick and dirty ABX focused on a short passage just with standard DirectSound output at 16/44 setting (oops, forgot to tell foobar to use ASIO!). Evidence that it's audible but not an obvious difference."

So, he has admitted that he wasn't serious about this test, he used DS output & yet he states "Evidence that it's audible but not an obvious difference"

Please, this is the "science" you talk about? This is the "evidence" you are presenting? This is the "scientific approach" you are suggesting? This is kindergarten posing -"look I do tests & get evidence on which I base "conclusions" - see how scientific I am?"

Does this mean that there isn't a difference that we can hear, no? This binary view of audio that uses a flawed test as the basis for its conclusions, is obviously badly mistaken.

This well tested view of audio that uses as good a test as there has ever been as the basis for its conclusions is obviously well correct. :rolleyes:

Ooh look, I can do adjectives! :rolleyes:

Please, just because you state this doesn't add any credence to it as having any veracity!

 

[Joe Bloggs]

Please, this is the "science" you talk about? This is the "evidence" you are presenting? This is the "scientific approach" you are suggesting?

In other words, you utterly reject the empirical fact that one gets a big chance at scoring those nonsignificant results via a series of coin tosses?

Please, just because you state this doesn't add any credence to it as having any veracity!

Do any of your statements lend credence to themselves in the first place? :rolleyes:

Does this mean that there isn't a difference that we can hear, no? This binary view of audio that uses a flawed test as the basis for its conclusions, is obviously badly mistaken.

"Please, just because you state this doesn't add any credence to it as having any veracity!"

Doesn't seem to stop you from making hundreds of such statements though :rolleyes:

 

[Joe Bloggs]

Given how easily one passes "sighted" AB tests and the credence you lend toward them, why should I accept any blind ABX test results where the tester got even one wrong out of a hundred?

All the tester has to do is to carry the obvious differences he "heard" while sighted--and carry them over to the blind tests! :rolleyes:

 

[mmerrill99]

Ah well, let's bring this regression into pantomime posts to a close - it has become farcical

 

[Joe Bloggs]

Yes, what a farce :rolleyes:

 

[gregorio]

Nope, I'm pointing out the weaknesses of ABX testing & the absolutely flawed use of it in a home run environment.

 
Flawed? Yes. Absolutely? No! Thanks for proving with your reply exactly what I was asserting. Either you've read and understood science 101, in which case you are trolling or you need to read it and stop being a hypocrite!
 
You're free to run your sighted tests if that's what convinces you or hell, just accept marketing "evidence" if you wish but don't come here trying to push your "science is flawed, therefore it's worthless" BS. Save it for audiophiles, creationists and flat-earthers who'll praise for your eloquence, rather than laugh at you!
 
G

 

[mmerrill99]

Nope, I'm pointing out the weaknesses of ABX testing

Flawed? Yes. Absolutely? No! Thanks for proving with your reply exactly what I was asserting. Either you've read and understood science 101, in which case you are trolling or you need to read it and stop being a hypocrite!

You're free to run your sighted tests if that's what convinces you or hell, just accept marketing "evidence" if you wish but don't come here trying to push your "science is flawed, therefore it's worthless" BS. Save it for audiophiles, creationists and flat-earthers who'll praise for your eloquence, rather than laugh at you!

G

What is evident from the replies here is the overwhelming desire to fit me into some category that can then be denigrated & dismissed.

Here's my summary of the position I see represented here - some of you state that "yes, ABX blind testing is flawed but not as flawed as sighted listening" So now it's just a matter of establishing which results are more flawed. These would be the results that I would give less credence to. I have given all my reasons for why home based ABX "tests" are, in my opinion, the more flawed.

I don't really see anything being advanced about the flawed nature of knowledge/sightedness. Some don't even seem to know that expectation bias is what's at play here.

I asked a simple question about expectation bias - is it ALWAYS at play in sighted listening - if it is then the results of a blind test are irrelevant as once you stop listening blind & revert to sighted listening you will also revert back to your biased impression of what you hear.

If expectation bias isn't ALWAYS at play in sighted listening then there is some sighted listening that isn't affected by this bias i.e it is just as valid as blind listening. People don't want to admit to this because they have adopted an unthinking binary view of the matter.

So now we have some blind listening is flawed & some sighted listening is fine.

You just have to work out your own way through this non-binary situation.

What I have been arguing against is the lazy-thinking, view that blind testing must be better than sighted listening

 

[spruce music]

What I have been arguing against is the lazy-thinking, view that blind testing must be better than sighted listening

And that is where you are making your mistake.  One you argue around the world to dodge having to admit that it is a mistake. 
 
The central issue which you will no doubt refuse to acknowledge once again is the effects of bias.  Even if some sighted listening can be unbiased, sighted is the most easily biased method, the method subject to the most routes for bias to creep in, and gives the least consistent results.  The simplest act of blinding removes or reduces at least some of those.  Yet your curious reasoning is akin to sighted listening is biased, blind listening can be biased too, so sighted listening is at least as good as blind listening.  Sorry, your spiel does not make any sense. It is a naked rationalization to reach a conclusion you preferred to reach at the outset.

 

[Joe Bloggs]

I don't really see anything being advanced about the flawed nature of knowledge/sightedness. Some don't even seem to know that expectation bias is what's at play here.

No, expectation bias isn't even half of what blind testing is fighting against. We get "I never expected it to make any difference, but now that I heard it with my own ears, this [paperweight audio scam doodad] really works!" all the time.

When one runs a sighted "test", anytime you begin to imagine A possibly having a certain characteristic over B, your brain starts looking for all sorts of "evidence" to confirm this "observation" in a self-enforcing cycle. The initial tipping point may have been caused by you slightly unseating your headphones or moving in your seat, changing the acoustics from A randomly over B; thereafter, no such actual difference in stimuli is needed anymore as your brain takes over.

What the audio world has demonstrated to us over and over again is, even a person who was in flat-out denial about there possibly being any difference in two different colored digital cables of the exact same construction will have at least a 50% chance of fooling himself into hearing night and day differences between them at some point--unless he has a rock solid understanding of both the physical facts of the two items being identical, AND the illusory effects of self-priming confirmation bias playing havoc with his perceptions.

THAT's why blind tests are the only way to go.

 

[mmerrill99]

What I have been arguing against is the lazy-thinking, view that blind testing must be better than sighted listening

And that is where you are making your mistake.  One you argue around the world to dodge having to admit that it is a mistake. 

The central issue which you will no doubt refuse to acknowledge once again is the effects of bias.  Even if some sighted listening can be unbiased, sighted is the most easily biased method, the method subject to the most routes for bias to creep in, and gives the least consistent results.  The simplest act of blinding removes or reduces at least some of those.

The "simplest act of blinding" is not what I would call an ABX test where you have to do at least 16 trials & for each one identify if X is A or B. But there you go ignoring all that & calling it the simplistic.

Yet your curious reasoning is akin to sighted listening is biased, blind listening can be biased too, so sighted listening is at least as good as blind listening.  Sorry, your spiel does not make any sense. It is a naked rationalization to reach a conclusion you preferred to reach at the outset.

Nope, sighted listening CAN be biased towards false positive results, ABX blind CAN be biased towards false negative results - it's a matter of personal judgement which is more appropriate for your needs

 

[KeithEmo]

This whole thread really is getting silly - so much so that it's starting to be impossible for it to accomplish anything.
 
This thread is NOT the latest journal of some science association. It is merely a thread for audiophiles to discuss a specific issue based on science rather than pure opinion, or pseudo-science, or "subjectivism". And, to be honest, I see part of the purpose of this thread to be to encourage people to perform their own tests, following reasonable scientific protocols, and eliminating as much obvious error as possible. So, yes, claims based solely on opinions should be discouraged, and it's perfectly reasonable to criticize specific procedures and methodologies, but, even though your favorite standards organization may have decided that an ABX test must have at least 16 runs to satisfy THEIR idea of being a "good" ABX test, the fact remains that any double-blind test is still better than a sighted test, and even four or five runs is still more information than none.
 
Yes, of course, the more times you perform a test, the more you can eliminate the effects of randomness, and so the more statistically significant your results will be. However, statistical significance itself is a continuum. Tossing a coin and getting three heads in a row is statistically different than you would expect from random chance, but it isn't MUCH different; ten heads in a row is a lot further from what you would expect by random chance; and, when you apply things like standard deviation, you will find that you can quantify how much less likely ten heads in a row is than three heads in a row... but there isn't some line where "anything above x is significant". In fact, if you actually research the subject, you will find that the amount of deviation that's required to consider something significant itself varies depending on the test. (If you flip a coin 1000 times, a result that's more than 5% from the norm may be very unusual; however, if you count the traffic through a certain intersection between 4 PM and 6 PM, you may find that a 20% variation from day to day is quite normal. And, if a hundred people each flip that coin five times, and 75 of them come up with four or more heads, then, in the aggregate, that is indeed a significant result.)
 
My point is that, just because your favorite standards organization has decided that an ABX test requires at least sixteen runs to satisfy the criteria they've chosen for that particular type of test does NOT mean that an ABX test with only five runs "isn't valid" or "isn't worth anything"... It simply means that the results of a single test with five runs are much less conclusive, or less meaningful, than the results with 16 runs. This is especially significant since we're talking about a group discussion here, and so we may get meta-data from multiple tests. So, for example, if twenty people each do an ABX test with five runs, and fifteen of them get a correct answer 4 times out of 5 or better, then, collectively, that data may actually be quite significant... because, while the odds of a single person getting 4 out of 5 may not be especially significant, the odds of fifteen people out of twenty each getting 4 out of 5 may be much higher - and so that result may be much more significant.
 
Therefore, to put it bluntly, I think the goal of rational discussion is much better served if we DON'T discourage everyone from performing a test just because their methodology, while reasonable, doesn't live up to some arbitrary standard... I'd much prefer to see a lot of people performing "pretty good" tests for themselves, than see them discouraged from doing so... as long as we all understand the limitations and significance of all the results..... (And, to put it even more bluntly, the discussion itself will help people with no experience to learn how to determine for themselves the difference between a really good test, and a pretty good test, and one that really is total junk - or pseudo-science.)
     
To me, the only appropriate criticism of doing an ABX test with only five trials would be:
 
"Five trials is a relatively low number, which means that, even though you've got the right idea, and your results are suggestive, you'd need a larger number of runs to produce a compelling result, and to more completely rule out the possibility that your results were due to random variations in results produced by random chance."

 

[mmerrill99]

I don't really see anything being advanced about the flawed nature of knowledge/sightedness. Some don't even seem to know that expectation bias is what's at play here.

No, expectation bias isn't even half of what blind testing is fighting against. We get "I never expected it to make any difference, but now that I heard it with my own ears, this [paperweight audio scam doodad] really works!" all the time.

When one runs a sighted "test", anytime you begin to imagine A possibly having a certain characteristic over B, your brain starts looking for all sorts of "evidence" to confirm this "observation" in a self-enforcing cycle. The initial tipping point may have been caused by you slightly unseating your headphones or moving in your seat, changing the acoustics from A randomly over B; thereafter, no such actual difference in stimuli is needed anymore as your brain takes over.

And what guides you to "imagine A possibly having a certain characteristic over B" - just random thoughts?

What the audio world has demonstrated to us over and over again is, even a person who was in flat-out denial about there possibly being any difference in two different colored digital cables of the exact same construction will have at least a 50% chance of fooling himself into hearing night and day differences between them at some point--unless he has a rock solid understanding of both the physical facts of the two items being identical, AND the illusory effects of self-priming confirmation bias playing havoc with his perceptions.

Ah, "]self-priming confirmation bias" what is this a confirmation of - an expectation, perhaps?

THAT's why blind tests are the only way to go.

Of course, you won't find many industry insiders talking about this, because it's apt to shut down 99% of the industry's revenue if everybody took this seriously. I only say these things here because at this point there's not a snowflake in hell's chance of anybody other than you taking this discussion seriously anymore.

OK, so let's all have a laugh & agree to disagree, right?

 

[L8MDL]

I put my Pono on random and read every message in this thread while it played a varied selection of cd and hires selections through my Sennheiser HD600's with a Fiio E12. Each time I thought it was playing a hires file, I looked for the "blue light ". I got it right 64 out of 72 times. Does this prove anything?

 

[Joe Bloggs]

And what guides you to "imagine A possibly having a certain characteristic over B" - just random thoughts?

It was right on top of your head--"The initial tipping point may have been caused by you slightly unseating your headphones or moving in your seat, changing the acoustics from A randomly over B;", although random mood swings can serve just as well

Ah, "]self-priming confirmation bias" what is this a confirmation of - an expectation, perhaps?

Not an expectation in the sense of your expectation of whether there is a difference or not going into the test--but the expectations that can be set up by the random process described above--always favoring a difference. Once a difference appears because of some random fluctuation, your brain latches onto it, expands on it and continues to insist on it long after the initial difference has disappeared into the noise.