[mmerrill99]

I don't give equal weight to flowery prose making claims about how something improves the sound quality over any ABX test.

I don't blame you & I similarly downgrade the report depending on the level of realistic impressions - I generally look for examples of music & what was specifically heard in them. I have no problems with descriptions that include phrases like "a more defined sound stage" or "more realism to the sound" as I know someone is trying to describe the illusion that is created by a (2 channel) playback system of an auditory scene & these are elements that are part of the psychoacoustics of auditory scenes. It is particularly difficult to describe these things & so they might appear to be less than rigorous descriptions

Some home based self-administered tests are done more right than others.

How do you judge this? Have false negatives been tested for & what were the results?

 Some scientists that do these kind of things for a living sometimes get it wrong.

There is nothing sure in life but the scientific protocol attempts to allow evaluation, cross-checking & repeating of the same experiment  

Both need to be properly vetted to offer any reasonable proof.

I would be interested in how you vet a home-based self-administered ABX test

 

[watchnerd]

I have no problem with ABX testing done right but home based self-administered ABX testing is not even a test so please don't confuse issues.

 
What is "done right" in a home context?

 

[RRod]

We're faced with the conundrum that:
a) a legit attempt at an ABX is a better test, period, than a sighted evaluation
b) we have no way of verifying, on the interwebz, that any given test was legit
 
So on one hand we encourage people to attempt the test because it might save them $$. On the other hand, when someone comes back with a positive result we have to both take their word on it and go through the whole rigmarole of verifying every possible thing that might have gone wrong, and we all know how crappy online tech support. It also puts us in the precarious position of looking like hypocritical a-holes (the French moderator doesn't help 

)
 
Seems this is just the way it is. I'm super happy if I can keep someone from spending $10^4 on the latest R⁴R DAC with Gigatap™☭ filter technology when they can't blindly tell it apart from their Realtek. I'm also not keen on wasting effort on ferreting out every possible mistake someone could have made in a test, especially when dishonesty *does* in fact happen†
 
†Do remember the time that vaccines led to autism.

 

[watchnerd]

  We're faced with the conundrum that:
a) a legit attempt at an ABX is a better test, period, than a sighted evaluation
b) we have no way of verifying, on the interwebz, that any given test was legit
 
So on one hand we encourage people to attempt the test because it might save them $$. On the other hand, when someone comes back with a positive result we have to both take their word on it and go through the whole rigmarole of verifying every possible thing that might have gone wrong, and we all know how crappy online tech support. It also puts us in the precarious position of looking like hypocritical a-holes (the French moderator doesn't help 

)
 
Seems this is just the way it is. I'm super happy if I can keep someone from spending $10^4 on the latest R⁴R DAC with Gigatap™☭ filter technology when they can't blindly tell it apart from their Realtek. I'm also not keen on wasting effort on ferreting out every possible mistake someone could have made in a test, especially when dishonesty *does* in fact happen

 
You don't even need to get into all that.
 
Just ask them why they're wasting time on a "top-tier" DAC when they haven't tackled room correction yet.

 

[mmerrill99]

I have no problem with ABX testing done right but home based self-administered ABX testing is not even a test so please don't confuse issues.

What is "done right" in a home context?

Exactly!! It isn't likely - which is my whole point

 

[watchnerd]

Exactly!! It isn't likely - which is my whole point

 
But the point is silly.
 
Everyone knows that home ABX testing is for hobbyist and educational reasons, not for submission to a peer-reviewed paper.

 

[mmerrill99]

We're faced with the conundrum that:
a) a legit attempt at an ABX is a better test, period, than a sighted evaluation
b) we have no way of verifying, on the interwebz, that any given test was legit

So on one hand we encourage people to attempt the test because it might save them $$. On the other hand, when someone comes back with a positive result we have to both take their word on it and go through the whole rigmarole of verifying every possible thing that might have gone wrong, and we all know how crappy online tech support. It also puts us in the precarious position of looking like hypocritical a-holes (the French moderator doesn't help  )

Seems this is just the way it is. I'm super happy if I can keep someone from spending $10^4 on the latest R⁴R DAC with Gigatap™☭ filter technology when they can't blindly tell it apart from their Realtek. I'm also not keen on wasting effort on ferreting out every possible mistake someone could have made in a test, especially when dishonesty *does* in fact happen[COLOR=252525]†[/COLOR]

[COLOR=252525]†Do remember the time that vaccines led to autism.[/COLOR]

So, in the "sound science" section a test is being suggested & in many cases demanded, for which there is no possibility of verifying the legitimacy of the test - I call this cargo cult science - a pretence of being scientific

There are 2 things we know - sighted listening biases towards false positive results, blind listening tends towards false negative results. We don't know the extent of the bias in each case so why would one test be preferred over another? Yes they will give diametrically opposite results from each other because of their bias but why would this favour one over the other unless a bias towards a particular outcome was desired?

 

[mmerrill99]

Exactly!! It isn't likely - which is my whole point

But the point is silly.

Everyone knows that home ABX testing is for hobbyist and educational reasons, not for submission to a peer-reviewed paper.

And yet people treat it as such a legit test that it is referred to & features very prominently in the sound "science" section. Some people try to use it as "proof" of something. So if it's just a diversion & not to be taken seriously, why all the fuss?

 

[RRod]

So, in the "sound science" section a test is being suggested & in many cases demanded, for which there is no possibility of verifying the legitimacy of the test - I call this cargo cult science - a pretence of being scientific

There are 2 things we know - sighted listening biases towards false positive results, blind listening tends towards false negative results. We don't know the extent of the bias in each case so why would one test be preferred over another? Yes they will give diametrically opposite results from each other because of their bias but why would this favour one over the other unless a bias towards a particular outcome was desired?

 
Well more and more I'm inclined to suggest that we simply guide people towards ABX, for the benefits of blind testing, with the caveat that their results are for them and them alone. Nothing non-sciency about that, it's just being realistic.
 
Your second comment there is the very opposite of science. If a person wants to test two pieces of equipment, then *at the very least* they need to verify volume matching AND not be biased by knowing which piece of equipment they are listening to when doing the actual comparison. Please tell me cases where intentionally leaving in these biases improves matters scientifically?

 
You don't even need to get into all that.
 
Just ask them why they're wasting time on a "top-tier" DAC when they haven't tackled room correction yet.

 
Unfortunately convincing people to use even a bit of EQ seems even harder than convincing them to take blind tests.

 

[watchnerd]

And yet people treat it as such a legit test that it is referred to & features very prominently in the sound "science" section. Some people try to use it as "proof" of something. So if it's just a diversion & not to be taken seriously, why all the fuss?

 
It's a data point. From a sample size of one.  And less-than-pristine circumstances, even when done with the sincerest intent.
 
So it's better than nothing.
 
But anyone who doesn't have an axe to grind and/or remembers their science education knows the proper response is, "Okay, you might have found something interesting. But to know for sure we're going to have to repeat that test a lot more times and more controlled circumstances."
 
Seems pretty obvious to me.

 

[watchnerd]

 
Unfortunately convincing people to use even a bit of EQ seems even harder than convincing them to take blind tests.

 
They don't even have to use EQ...just take some measurements and use passive treatments!
 
P.S. The 'no-EQ' stance cracks me up because of the huge amounts of EQ used during the recording and mastering process. It's not like most recorded music is raw, unaltered mic feeds.

 

[RRod]

   
They don't even have to use EQ...just take some measurements and use passive treatments!

 
I agree but we're in headphone land, so it's more like convincing someone to try EQ instead of taking apart their $300 headphones to put in cotton balls.

 

[mmerrill99]

So, in the "sound science" section a test is being suggested

Well more and more I'm inclined to suggest that we simply guide people towards ABX, for the benefits of blind testing, with the caveat that their results are for them and them alone. Nothing non-sciency about that, it's just being realistic.

And I would also include in your suggested use of ABX as a personal test that the intricacies of perceptual testing be explained then & using ABX as a "burden of proof" test would disappear, right? I like your optimism but I find it naive as there are many who don't share your view of ABX tests.

Your second comment there is the very opposite of science. If a person wants to test two pieces of equipment, then *at the very least* they need to verify volume matching AND not be biased by knowing which piece of equipment they are listening to when doing the actual comparison. Please tell me cases where intentionally leaving in these biases improves matters scientifically?

If you understood my point your question translates to - "if I remove a bias towards false positives but introduce a bias towards false negatives - how can I then know which listening result is closer to giving the correct view"?

 

[RRod]

And I would also include in your suggested use of ABX as a personal test that the intricacies of perceptual testing be explained then & using ABX as a "burden of proof" test would disappear, right? I like your optimism but I find it naive as there are many who don't share your view of ABX tests.
If you understood my point your question translates to - "if I remove a bias towards false positives but introduce a bias towards false negatives - how can I then know which listening result is closer to giving the correct view"?

 
Again, the point of suggesting ABX is the requisite control of bias sources that goes along with doing it right. Bias is not the same as failing to reject the null hypothesis; you are conflating the two concepts.

 

[sonitus mirus]

   
They don't even have to use EQ...just take some measurements and use passive treatments!
 
P.S. The 'no-EQ' stance cracks me up because of the huge amounts of EQ used during the recording and mastering process. It's not like most recorded music is raw, unaltered mic feeds.

 
Is there any evidence that indicates treatment improves sound quality for a typical residential room environment?  This is one area where Ethan Winer stands to make some money from the sale of bass traps, but I don't see any conclusive listening test studies that show they really help in any way.  I don't give him a pass any more than I would for Bob Stuart's projects that he pushes for financial gain.