ABX Blind testing - the Ins & outs
Jan 18, 2016 at 2:30 PM Post #31 of 64
  Well, I won't tell you how to enjoy your hobby if you don't tell me how to enjoy mine. Your closing statement says that my way is incorrect as I won't be "enjoying the hobby." 
 
I know how I am "enjoying the hobby." I have equipment that meets my expectations in terms of delivering sound from storage to my ears. I am now on the lookout for good, well-made music that I like, so I can purchase them and enjoy listening at home, on the go, or at the office. I enjoy my hobby by tinkering with DIY projects, because I like building things that I can use. I enjoy my hobby by researching the science of sound, including psychoacoustics and ABX/DB testing because I find those topics interesting. I might even enjoy my hobby by purchasing another set of transducers to, you know, enjoy a different sound through enjoying my hobby. The differences between my HE560's and say Senneheiser HD650 or HD800S are measurable and audible.
 
If I enjoy the sound then I will buy the headphones, and here we meet. I am enjoying my hobby my way. I hope you are also enjoying your hobby, in the way that you enjoy it.

 
My enjoyments are similar, but I add in making recordings.
 
There really is no reference for telling you how good your system is like a recording you made yourself.
 
Jan 18, 2016 at 2:57 PM Post #32 of 64
..... There are many examples of this with musicians filling in gaps for missing notes, but the most impressive example is Poppy Crum's demonstrations with listening to records backwards.  When coached to hear it as noise, people here noise.  When coached to hear it as Satanic speech and shown lyrics, people hear it as speech and the language parts of their brain light up.

 
...
In most hobbies there are BS claims - it's relatively easy to decide what is BS, what is iffy & what is worth looking into.

 
ROFL, yeah not too difficult to decide on that one
biggrin.gif

 
Jan 18, 2016 at 3:23 PM Post #33 of 64
All this is similar to the proposition "well if you hear a difference then there must be a difference in the analogue waveform" Yes, absolutely agree with this but unless you know how auditory perception works then you don't know what differences to measure


Scarily, it's not as easy as this:

Because all sound is processed by your brain, you can "hear" things that aren't there, and verify it by looking at brain scans, if you are predisposed to do so.  In other words, you can "hear" (i.e. perceive) things that aren't even in the waveform.

There are many examples of this with musicians filling in gaps for missing notes, but the most impressive example is Poppy Crum's demonstrations with listening to records backwards.  When coached to hear it as noise, people here noise.  When coached to hear it as Satanic speech and shown lyrics, people hear it as speech and the language parts of their brain light up.

Yes but you are not talking about auditory perception - auditory perception is a study of the mechanisms & processing by which the perception of real auditory signals takes place & results in an ever changing auditory scene - not in imaginings/illusions/false perceptions
 
Jan 18, 2016 at 3:27 PM Post #34 of 64
As I said before home ABX testing will hide differences so it's a great way to avoid changing your system

Well, I won't tell you how to enjoy your hobby if you don't tell me how to enjoy mine. Your closing statement says that my way is incorrect as I won't be "enjoying the hobby." 
Yes, fair enough & that's all that is really being asked for by many people - allow anecdotal subjective reports the same standing as anecdotal ABX "proofs" - I've no problem with that

I know how I am "enjoying the hobby." I have equipment that meets my expectations in terms of delivering sound from storage to my ears. I am now on the lookout for good, well-made music that I like, so I can purchase them and enjoy listening at home, on the go, or at the office. I enjoy my hobby by tinkering with DIY projects, because I like building things that I can use. I enjoy my hobby by researching the science of sound, including psychoacoustics and ABX/DB testing because I find those topics interesting. I might even enjoy my hobby by purchasing another set of transducers to, you know, enjoy a different sound through enjoying my hobby. The differences between my HE560's and say Senneheiser HD650 or HD800S are measurable and audible.

If I enjoy the sound then I will buy the headphones, and here we meet. I am enjoying my hobby my way. I hope you are also enjoying your hobby, in the way that you enjoy it.
Sure, absolutely no problem with anything you say
 
Jan 18, 2016 at 3:29 PM Post #35 of 64
 
Well, I won't tell you how to enjoy your hobby if you don't tell me how to enjoy mine. Your closing statement says that my way is incorrect as I won't be "enjoying the hobby." 

I know how I am "enjoying the hobby." I have equipment that meets my expectations in terms of delivering sound from storage to my ears. I am now on the lookout for good, well-made music that I like, so I can purchase them and enjoy listening at home, on the go, or at the office. I enjoy my hobby by tinkering with DIY projects, because I like building things that I can use. I enjoy my hobby by researching the science of sound, including psychoacoustics and ABX/DB testing because I find those topics interesting. I might even enjoy my hobby by purchasing another set of transducers to, you know, enjoy a different sound through enjoying my hobby. The differences between my HE560's and say Senneheiser HD650 or HD800S are measurable and audible.

If I enjoy the sound then I will buy the headphones, and here we meet. I am enjoying my hobby my way. I hope you are also enjoying your hobby, in the way that you enjoy it.


My enjoyments are similar, but I add in making recordings.

There really is no reference for telling you how good your system is like a recording you made yourself.

Yes, completely agree but I don't record myself - it's something we should all be able to use as a reference
 
Jan 18, 2016 at 3:40 PM Post #36 of 64
.....
There are many examples of this with musicians filling in gaps for missing notes, but the most impressive example is Poppy Crum's demonstrations with listening to records backwards.  When coached to hear it as noise, people here noise.  When coached to hear it as Satanic speech and shown lyrics, people hear it as speech and the language parts of their brain light up.

I missed this when you posted it - yes the brain is a pattern matching machine that can't really help itself doing what comes naturally to it - it's both it's strength & it's weakness. But it's biology's answer to how to make instantaneous decisions about what is being heard & goes to the heart of the kernel of this sense of hearing. It has reached a balance between speed & accuracy in reaching decisions - obviously we wouldn't have survived, if the speed of working out what we heard took too long - we would have already been eaten by the tiger or bitten by the snake, etc. Obviously we wouldn't have survived if we regularly mistook what we heard - again survival etc.

So, we can reckon that this sense is fit-for-purpose but in areas of subtle differences it needs time - these are usually not critical to survival.

...

In most hobbies there are BS claims - it's relatively easy to decide what is BS, what is iffy


ROFL, yeah not too difficult to decide on that one :D
 
Jan 18, 2016 at 3:50 PM Post #37 of 64
Yes, fair enough & that's all that is really being asked for by many people - allow anecdotal subjective reports the same standing as anecdotal ABX "proofs" - I've no problem with that

 
And here, we part, again. I, in my enjoyment of my hobby, do not need to "allow anecdotal subjective reports the same standing". You are again trying to force your method of enjoyment on me.
 
Jan 18, 2016 at 4:07 PM Post #38 of 64
Yes, fair enough


And here, we part, again. I, in my enjoyment of my hobby, do not need to "allow anecdotal subjective reports the same standing". You are again trying to force your method of enjoyment on me.

In saying that, all I'm asking for is that people who enjoy their hobby using completely subjective evaluation should be allowed to express their impressions without someone telling them they are delusional or that they need to provide blind testing "proof" of the listening impressions they are reporting - simple as that.
 
Jan 18, 2016 at 5:12 PM Post #39 of 64
Yes but you are not talking about auditory perception - auditory perception is a study of the mechanisms & processing by which the perception of real auditory signals takes place & results in an ever changing auditory scene - not in imaginings/illusions/false perceptions

 
Oh I think it's highly relevant, as audiophilia is full of suggestions and imaginary things.
 
Jan 18, 2016 at 5:18 PM Post #40 of 64
   
but if it's reported as night and day difference, then indeed an ABX should be trivial to pass.
wink_face.gif

There are two points to be made on this:
- there's no accounting for the over-enthusiastic hyperbole that people report - it happens in every hobby when people discover something they like
- auditory perception isn't so simple as this - what we can perceive as a widespread & significant effect on what we are hearing, can often become difficult to identify, isolate & focus on in a blind test - I gave ultmusicsnob's description of this as examples of just how difficult it is to find a "Tell" that we can use in blind testing even though in our normal listening we can constantly perceive one piece as better than another. I have tried to detail in my previous posts how this issue arises - it's a combination of the nature of auditory perception - it is at best a constantly updated best guess interpretation of the signals coming from the ears - combine this with a test that takes away a major source of extra data used by auditory processing (a blind test) & we get doubt, insecurity & second guessing about what we were so sure of in normal listening. As I said if we pinch our noses a lot of food will taste the same - that doesn't mean the food "actually" really tastes the same now, does it?

All this is similar to the proposition "well if you hear a difference then there must be a difference in the analogue waveform" Yes, absolutely agree with this but unless you know how auditory perception works then you don't know what differences to measure & what sort of test signals will best reveal these differences. For instance, it's not necessarily best to use a measurement tool that uses averaging like FFTs - it's not necessarily best to use sine wave test signals, etc, etc. Unless you have a theory of approximately what you are testing for it's difficult to create meaningful tests.
I know that I look like I'm pushing too hard for ABX(or blind tests in general), but it often was the only way to test audibility for myself, and doing ABX(or variations of it) has been a real eye opener in my audio journey. I seriously pity those who have to make that journey with sighted evaluation only(but then again, maybe we could go for "ignorance is bliss"). you keep going with your weird excuses that a lie is truth if it's always there and that sighted evaluation is ok with biases as we have them all the time, but I don't wish to live that kind of life(welcome into the matrix).

I'm sorry but to use your categorisation, you are only substituting one lie for another lie when engaging in such flawed blind tests - yes some people think they are fun (I find them boring), some swear by them ("has been a real eye opener in my audio journey") but the reality is that they are just as flawed towards null results as sighted tests are towards positive results (although I gave the reasons I think long term sighted listening is better than one-shot blind testing)
I have no doubt that without blind testing when it comes to audible differences, I would still be way more ignorant and less humble than I am now. I don't know it all, far from it, often I didn't get answers and that's super frustrating, but many many times it gave me at least a reason to be less gullible and I'm grateful for that. so I say to people, go ABX! get used to it, learn about the pros and cons, and make it your own tool.

Yes, you are more comfortable with this particular lie that is ABX testing, nobody is in any doubt about that but recognise it for what it is - just another anecdotal listening impression (an unnatural one where the participant has to have all sorts of expertise/focus/motivation & determination to pass the test's bias towards a null result).

Of course it has a pseudo-scientific ring about it but only to those who aren't scientific & don't know that perceptual tests are fraught with so many problems that only experienced practitioners can produce rigorous & trustworthy results.
anytime you argue that the results of a personal ABX are meaningless, I half agree. at the community level it's not 100% meaningless, but very close. why would I trust the results of a guy on the net? in fact I don't bring my abx results as evidence, I have never done that. at best I mention what happened to me when the subject is brought up, and that's it. it's my anecdotal story and people do what they want of it.

Yes, but an anecdotal story of someone hearing a particular audible artifact when playing song X with device Y has just the same evidentiary value - people can try this themselves - it doesn't require any unnatural way of listening or special determination to overcome
but at the same time, I feel like you're telling people to miss out on all the good it can do, while giving a free pass to all the night and day BS claims we have to face all year long. in short, I feel like you're pushing for ignorance as you're taking down ABX brick by brick, but have yet to bring something usable in it's stead.

The problem you seem to have is that you are not willing to face up to the fact that there are great issues to be dealt with in useful perceptual testing - it's not something that can be done informally & invariably gives a null result unless carefully administered by people knowledgeable in this field - yet you want to suggest this as something everybody should try or they will "miss out on all the good it can do"

In most hobbies there are BS claims - it's relatively easy to decide what is BS, what is iffy & what is worth looking into.

As I said before home ABX testing will hide differences so it's a great way to avoid changing your system & convince yourself that your system is perfect - but I doubt it's a great way of enjoying the hobby.

 
you accept that serious blind tests with trained professionals can get some valid results. I do my own newbie tests, they're not perfect but I do try my best. I probably still leave some out at times, and I probably see a little more null than I should. but then I don't remember getting results that would contradict most of the AES papers on the matter I'm testing. so either my tests are more useful than you make them to be. or those guys don't know how to test something. but then who is? the amateur sighted listener? he's got the 2 things you don't like, unreliable method and non professional subject/tester. ^_^

 
Jan 18, 2016 at 6:28 PM Post #41 of 64
In saying that, all I'm asking for is that people who enjoy their hobby using completely subjective evaluation should be allowed to express their impressions without someone telling them they are delusional or that they need to provide blind testing "proof" of the listening impressions they are reporting - simple as that.

Well, I have not gone into the other parts of Head-Fi and demanded ABX be accepted. Nor have I asked if they really heard whatever they heard or not. The mods on the other fora have been pretty vigilent for such ramblings, so I doubt that the frequency of posts like the one you described is very high. I would bet it is close to zero these days on Head-Fi.
 
You are currently posting in Sound Science, so here, posters will ask for proof of audibility, especially for testable statements about near inaudible phenomenon. Post in Sound Science and say "I am hearing a buzz, but it only shows up when I am playing music and only in my left ear. If I flip the cups, I hear it in my right ear." Pretty much NO ONE will say, "Yo! You didn't post ABX or DBT results. We don't believe you heard a buzz."
 
Come in and post that they taped some "magic quantum crystals that absorb bad frequencies" to their headphone cables and then proclaim their HD800s suddenly have rumbling bass will require a much higher burden of proof. That poster should expect calls for DBT / ABX if they post in the Sound Science Forum.
 
Jan 18, 2016 at 6:49 PM Post #42 of 64
In saying that, all I'm asking for is that people who enjoy their hobby using completely subjective evaluation should be allowed to express their impressions without someone telling them they are delusional or that they need to provide blind testing "proof" of the listening impressions they are reporting - simple as that.

Well, I have not gone into the other parts of Head-Fi and demanded ABX be accepted. Nor have I asked if they really heard whatever they heard or not. The mods on the other fora have been pretty vigilent for such ramblings, so I doubt that the frequency of posts like the one you described is very high. I would bet it is close to zero these days on Head-Fi.
I wasn't saying you specifically did this but here's a post made 30 mins ago on another thread I'm posting in "I missed the part where he proved the problem was audible with double blind testing. It was a nice anecdote for the subjectivists though..."

You are currently posting in Sound Science, so here, posters will ask for proof of audibility, especially for testable statements about near inaudible phenomenon. Post in Sound Science and say "I am hearing a buzz, but it only shows up when I am playing music and only in my left ear. If I flip the cups, I hear it in my right ear." Pretty much NO ONE will say, "Yo! You didn't post ABX or DBT results. We don't believe you heard a buzz."

Come in and post that they taped some "magic quantum crystals that absorb bad frequencies" to their headphone cables and then proclaim their HD800s suddenly have rumbling bass will require a much higher burden of proof. That poster should expect calls for DBT / ABX if they post in the Sound Science Forum.
You mean you look for the results of a known flawed test as "proof" of audibility - why? I thought the idea of science was to better understand matters - that would include auditory perception & the testing of it.
 
Jan 18, 2016 at 7:20 PM Post #43 of 64
 
[CONTENTEMBED=/t/795020/abx-blind-testing-the-ins-outs/15#post_12263075 layout=inline] [/CONTENTEMBED]
but if it's reported as night and day difference, then indeed an ABX should be trivial to pass. :wink_face:

There are two points to be made on this:

- there's no accounting for the over-enthusiastic hyperbole that people report - it happens in every hobby when people discover something they like

- auditory perception isn't so simple as this - what we can perceive as a widespread
you accept that serious blind tests with trained professionals can get some valid results.
Sure, because there are published standards for running these tests that recognise the issues with perceptual testing that I have enumerated & if these guidelines are followed there is a much better chance of some valid results. As I said, it's a very difficult area to test & full of hidden pitfalls that no home tester can avoid even with the best will in the world
I do my own newbie tests, they're not perfect but I do try my best. I probably still leave some out at times, and I probably see a little more null than I should. but then I don't remember getting results that would contradict most of the AES papers on the matter I'm testing. so either my tests are more useful than you make them to be. or those guys don't know how to test something. but then who is? the amateur sighted listener? he's got the 2 things you don't like, unreliable method and non professional subject/tester. ^_^
Sure, I said that these home blind tests produce mostly nulls when it comes to differences which aren't grossly obvious but they are still being used in demanding "proof" - just encountered it on another thread .
 
Jan 18, 2016 at 7:25 PM Post #44 of 64
Yes but you are not talking about auditory perception - auditory perception is a study of the mechanisms


Oh I think it's highly relevant, as audiophilia is full of suggestions and imaginary things.
You are just muddying the waters of what auditory perception is about - first you figure out the mechanisms of it's operation & then you look into the possible anomalies of the mechanisms.
 
Jan 18, 2016 at 8:51 PM Post #45 of 64
You are just muddying the waters of what auditory perception is about - first you figure out the mechanisms of it's operation & then you look into the possible anomalies of the mechanisms.

 
The mechanisms are complex and related to all sorts of other cognitive elements....which is why we have DBT.
 

Users who are viewing this thread

Back
Top