[mmerrill99]

And what guides you to "imagine A possibly having a certain characteristic over B" - just random thoughts?

It was right on top of your head--"The initial tipping point may have been caused by you slightly unseating your headphones or moving in your seat, changing the acoustics from A randomly over B;", although random mood swings can serve just as well

Ah, "]self-priming confirmation bias" what is this a confirmation of - an expectation, perhaps?

Not an expectation in the sense of your expectation of whether there is a difference or not going into the test--but the expectations that can be set up by the random process described above--always favoring a difference. Once a difference appears because of some random fluctuation, your brain latches onto it, expands on it and continues to insist on it long after the initial difference has disappeared into the noise.

If it's just random events, as you suggest, then we would see just randomness in people's sighted listening & no particular device/protocol would be favoured more than any other!!

 

[mmerrill99]

I put my Pono on random and read every message in this thread while it played a varied selection of cd and hires selections through my Sennheiser HD600's with a Fiio E12. Each time I thought it was playing a hires file, I looked for the "blue light ". I got it right 64 out of 72 times. Does this prove anything?

I bet there are many here who will argue that your test proves nothing because it was not rigorous enough - we don't know if you actually were able to see the blue light out of the corner of your eye prior to making your determination, we don't know if you already know which of your tracks are high-res & which RB, etc.

So when a test returns positive results the rigour of the test is examined but when it returns null results my querying of the underlying rigour of the test is questioned.

 

[KeithEmo]

Expectation bias is ALWAYS in play.... most of what we experience in our lives is affected to some greater or lesser degree by what we expect to experience. Expectation bias is in fact a general term; in audio, it could mean that we expect a more expensive product to sound better, so we imagine it does; it could also mean that we like that new song by our favorite group just a little bit more than if we didn't know who was playing it; and it can be the opposite - it can mean that we hear a difference between two things, but, since we're convinced that they should be the same, we decide that we only imagined it. There are even arguably several different mechanisms involved. However, what they have in common is that they all cause our perceptions to be "skewed" in one way or another rather than being based entirely on reality.
 
The whole point, in the specific context of identifying whether differences between specific audio products are audible or not, is that we want to eliminate any and all forms of bias so we can determine what the actual differences are in the product itself - and we can do this most effectively simply by not knowing which product we're listening to. (And, yes, it would be quite possible for a NEGATIVE expectation bias to lead to a false null. However, at least in this discussion, the goal of the test is usually to debunk a difference that someone claims is audible. And, if the difference only exists when we know which product we're listening to, then we can logically infer that the difference is in our mind and not in the actual product, and so that it is due to some sort of bias.)
 
Quote:

What is evident from the replies here is the overwhelming desire to fit me into some category that can then be denigrated & dismissed.

Here's my summary of the position I see represented here - some of you state that "yes, ABX blind testing is flawed but not as flawed as sighted listening" So now it's just a matter of establishing which results are more flawed. These would be the results that I would give less credence to. I have given all my reasons for why home based ABX "tests" are, in my opinion, the more flawed.

I don't really see anything being advanced about the flawed nature of knowledge/sightedness. Some don't even seem to know that expectation bias is what's at play here.

I asked a simple question about expectation bias - is it ALWAYS at play in sighted listening - if it is then the results of a blind test are irrelevant as once you stop listening blind & revert to sighted listening you will also revert back to your biased impression of what you hear.

If expectation bias isn't ALWAYS at play in sighted listening then there is some sighted listening that isn't affected by this bias i.e it is just as valid as blind listening. People don't want to admit to this because they have adopted an unthinking binary view of the matter.

So now we have some blind listening is flawed & some sighted listening is fine.

You just have to work out your own way through this non-binary situation.

What I have been arguing against is the lazy-thinking, view that blind testing must be better than sighted listening

 

[mmerrill99]

Expectation bias is ALWAYS in play.... most of what we experience in our lives is affected to some greater or lesser degree by what we expect to experience. Expectation bias is in fact a general term; in audio, it could mean that we expect a more expensive product to sound better, so we imagine it does; it could also mean that we like that new song by our favorite group just a little bit more than if we didn't know who was playing it; and it can be the opposite - it can mean that we hear a difference between two things, but, since we're convinced that they should be the same, we decide that we only imagined it. There are even arguably several different mechanisms involved. However, what they have in common is that they all cause our perceptions to be "skewed" in one way or another rather than being based entirely on reality.

The whole point, in the specific context of identifying whether differences between specific audio products are audible or not, is that we want to eliminate any and all forms of bias so we can determine what the actual differences are in the product itself - and we can do this most effectively simply by not knowing which product we're listening to. (And, yes, it would be quite possible for a NEGATIVE expectation bias to lead to a false null. However, at least in this discussion, the goal of the test is usually to debunk a difference that someone claims is audible. And, if the difference only exists when we know which product we're listening to, then we can logically infer that the difference is in our mind and not in the actual product, and so that it is due to some sort of bias.)

Keith,
I'll put the question I posed before more precisely - if by ALWAYS you mean that we are therefore slaves to our expectation bias & the only way to avoid this is by blind listening, then there is no point in these tests as when we resumed listening normally (sighted) we will be back to hearing what our bias determines we hear.

If, however, by doing a blind test we now have removed the influence of the bias & we can now listen sighted but bias free, please explain the mechanism by which the expectation bias no longer operates

The final possibility is that the blind test changes our positive expectation to one of a negative expectation - i.e we imagine that the two device now sound the same because we couldn't determine any difference in the ABX blind test (no matter how flawed this test is)

 

[JWolf]

I bet there are many here who will argue that your test proves nothing because it was not rigorous enough - we don't know if you actually were able to see the blue light out of the corner of your eye prior to making your determination, we don't know if you already know which of your tracks are high-res & which RB, etc.

So when a test returns positive results the rigour of the test is examined but when it returns null results my querying of the underlying rigour of the test is questioned.

 
This is exactly why ABX is worthless. If it turns out that the results favor Hi-Res, then the results are questioned. STOP IT!
 

 

[dprimary]

Quote:
 
Originally Posted by L8MDL /img/forum/go_quote.gif

I put my Pono on random and read every message in this thread while it played a varied selection of cd and hires selections through my Sennheiser HD600's with a Fiio E12. Each time I thought it was playing a hires file, I looked for the "blue light ". I got it right 64 out of 72 times. Does this prove anything?

I bet there are many here who will argue that your test proves nothing because it was not rigorous enough - we don't know if you actually were able to see the blue light out of the corner of your eye prior to making your determination, we don't know if you already know which of your tracks are high-res & which RB, etc.

So when a test returns positive results the rigour of the test is examined but when it returns null results my querying of the underlying rigour of the test is questioned.

Maybe it proves something maybe it doesn't. Without the details it is meaningless.  Asking for more information is not shooting down the possibility that they hear a difference. I am hoping they had "hi res" and "low res versions of the same song. If not then it might prove they can remember which songs are "high res".
 
If it is the same song is it from the same master? I have heard "high res" versions that only was it not the same master it was not the same mix and did not seem to even be the same take, of a song that was pretty well known.  
 
If it is from the same source, how was it encoded? If it was encoded to MP3 at 128k using a 15 year old version of winamp. I think I might have good chance of picking out the MP3 having never even heard the original. Is it comparing raw 16/44.1 to raw 24/96? then that is completely different.
 
Then you get to - how did the samples get prepped? are they the same level? was one clipped, eq'ed, distorted, compressed? There is plenty of ways make a mistake.
 
If you are comparing a 16/44.1 MP3 at 256k to a FLAC 24/96 you are not really comparing 16/44.1 to 24/96 you are comparing lossy to lossless. Most people will not be able to detect a difference. A few will depending on the song. Even so not like you are going to be able to detect the difference riding on a subway.
 
Comparing 16/44.1 AAC at 320k to 16/44.1 lossless is not impossible it is pretty difficult. You really have be completely focused in low noise room. If it is night and day as if often claimed then I would suspect the decoder to be broken or you are not testing it by listening but by some other means.

 

[sonitus mirus]

From a technical perspective, it may be possible for some listeners to identify the FR rolloff when playing a FLAC rip of a CD using 44.1kHz sample rate. (see Fig. 3 in the link below)
 
http://www.stereophile.com/content/pono-ponoplayer-portable-music-player-measurements#XTOUiBXGPWaEDsOm.97
 
This is simply a poor characteristic of the Pono player itself, which was presumably created to cater to folks using files with higher sample rates, but it makes any comparison with CD quality unfair with regards to the format.

 

[dprimary]

  From a technical perspective, it may be possible for some listeners to identify the FR rolloff when playing a FLAC rip of a CD using 44.1kHz sample rate. (see Fig. 3 in the link below)
 
http://www.stereophile.com/content/pono-ponoplayer-portable-music-player-measurements#XTOUiBXGPWaEDsOm.97
 
This is simply a poor characteristic of the Pono player itself, which was presumably created to cater to folks using files with higher sample rates, but it makes any comparison with CD quality unfair with regards to the format.

That could be noticeable to some people. It is not up to the level of what I expect on a mid level piece of equipment. I was really interested in the Pono when it came out since it has balanced output and I need to interface to professional equipment most of the time. After many attempts to get complete specifications on the device or even find someone that had tested the balanced interface feeding pro gear I gave up and bought something else. 

 

[spruce music]

 
That could be noticeable to some people. It is not up to the level of what I expect on a mid level piece of equipment. I was really interested in the Pono when it came out since it has balanced output and I need to interface to professional equipment most of the time. After many attempts to get complete specifications on the device or even find someone that had tested the balanced interface feeding pro gear I gave up and bought something else. 

Did you find a similar device with balanced out, and if so what was it?

 

[watchnerd]

   
This is exactly why ABX is worthless. If it turns out that the results favor Hi-Res, then the results are questioned. STOP IT!
 

 
I'm confused. Stop what?

 

[OddE]

 
This is exactly why ABX is worthless. If it turns out that the results favor Hi-Res, then the results are questioned. STOP IT!

 
-Guess why?
 
A result favoring hi-res flies in the face of established science. Hence, one would be more inclined to ask questions as to methodology, source material and any biases which may have influenced the result than if the result was 'Wasn't able to tell any difference.'
 
The flip side being that if questions were answered satisfactorily and results proved reproduceable, it would be a significant discovery.
 
Sounds like a fair tradeoff to me.

 

[gregorio]

I asked a simple question about expectation bias - is it ALWAYS at play in sighted listening - if it is then the results of a blind test are irrelevant as once you stop listening blind & revert to sighted listening you will also revert back to your biased impression of what you hear.

 
I don't know for certain expectation bias is ALWAYS at play, although I strongly suspect that it probably is in the vast majority of cases, whether we are consciously aware of it or only sub-consciously. Your assertion that after an ABX test we will still have expectation bias would therefore be correct. However, the next part of your assertion is ridiculous!
 
Let's take a simple example, let's say I'm testing two sets of speakers, A & B. Maybe I have an expectation that set A is better, maybe I don't know if set A is better but just hope they are. I do a sighted test and indeed I do have a preference for A. Then I do an ABX and discover that actually I can't discern any difference between them. After both the sighted and ABX tests I indeed still have expectation bias. However, my expectation is different, it has changed, it does NOT "revert back to my biased impression" of what I heard before the ABX test. In this example, if my expectation were unaffected in anyway by the result of the ABX then I would be a fool!
 
The reality of the situation is not quite so absolute as my example indicates however. Anyone with a reasonable knowledge/experience of audio knows there are a lot of variables at play and even laboratory conditions ABX testing is not generally an absolute answer. In the example above, speakers A may, for example, have a slightly different dispersion pattern which may cause them to perform noticeably better (in an ABX test) than B, given different positioning or different test material or, my ABX test might have been flawed in some other way. ABX is simply additional (and all else being equal, more reliable) information, rather than absolute proof. I'm not sure why you find this so difficult to understand. Maybe you are just a fool, or maybe you do understand but are trolling or have some other agenda? Either way, your arguments are nonsensical/ridiculous and serve no purpose for either you or us. Again, why don't you try say the Cables forum, where your assertions are true, where ABX is indeed an impediment to the ethos and progress of that forum.
 
G

 

[mmerrill99]

I asked a simple question about expectation bias - is it ALWAYS at play in sighted listening - if it is then the results of a blind test are irrelevant as once you stop listening blind

I don't know for certain expectation bias is ALWAYS at play, although I strongly suspect that it probably is in the vast majority of cases, whether we are consciously aware of it or only sub-consciously. Your assertion that after an ABX test we will still have expectation bias would therefore be correct. However, the next part of your assertion is ridiculous!

Let's take a simple example, let's say I'm testing two sets of speakers, A & B. Maybe I have an expectation that set A is better, maybe I don't know if set A is better but just hope they are. I do a sighted test and indeed I do have a preference for A. Then I do an ABX and discover that actually I can't discern any difference between them. After both the sighted and ABX tests I indeed still have expectation bias. However, my expectation is different, it has changed, it does NOT "revert back to my biased impression" of what I heard before the ABX test. In this example, if my expectation were unaffected in anyway by the result of the ABX then I would be a fool!

Right & there you have an example of expectation bias being nullified or changed. You are now listening sighted without this bias affecting what you hear. So why can't you do this prior to doing a blind test? Why do you have to do a blind test in order to achieve this - is there something magical about a blind test? No? It's just a case of being aware of your expectations. They are not the overriding, constant & ubiquitous influence that many here try to make them out to be.

Of course the other possibility is not that a blind test is nullifying your expectation bias but that it's changing it into a negative expectation bias i.e that you are now convinced & expect that you won't hear any difference between A & B in sighted listening - guess what, you won't.

And as I said already, I'm far more willing & at ease with the possibility that some subconscious influence is affecting my conclusions in sighted listening than introducing all the influences & possible new biases that blind listening entails

The reality of the situation is not quite so absolute as my example indicates however. Anyone with a reasonable knowledge/experience of audio knows there are a lot of variables at play and even laboratory conditions ABX testing is not generally an absolute answer. In the example above, speakers A may, for example, have a slightly different dispersion pattern which may cause them to perform noticeably better (in an ABX test) than B, given different positioning or different test material or, my ABX test might have been flawed in some other way. ABX is simply additional (and all else being equal, more reliable) information, rather than absolute proof. I'm not sure why you find this so difficult to understand. Maybe you are just a fool, or maybe you do understand but are trolling or have some other agenda? Either way, your arguments are nonsensical/ridiculous and serve no purpose for either you or us. Again, why don't you try say the Cables forum, where your assertions are true, where ABX is indeed an impediment to the ethos and progress of that forum.

G

Well you can call me a fool & a troll & my arguments nonsensical but I'll let non-biased readers (is there such a thing?) make up their own mind. Again, I'll say it, home based ABX testing is fraught with so many flaws that it is risible anybody would categorise it as scientific or of any worth

 

[castleofargh]

I put my Pono on random and read every message in this thread while it played a varied selection of cd and hires selections through my Sennheiser HD600's with a Fiio E12. Each time I thought it was playing a hires file, I looked for the "blue light ". I got it right 64 out of 72 times. Does this prove anything?

taken from stereophile:

 

Fig.3 Pono PonoPlayer, frequency response at –12dBFS into 100k ohms with data sampled at: 44.1kHz (left channel cyan, right magenta), 96kHz (left green, right gray), 192kHz (left blue, right red) (0.5dB/vertical div.).

it may not be the only reason, because some design choices on the pono are special. and maybe the highres one takes a little longer to load and gives you a hint, or as suggested, maybe some visual cue?. but the roll off when playing 16/44 files is big enough to potentially be audible. so as far as I'm concerned I believe you could identify the highres files. I'm just not sure it has much to do with the quality of the file itself.
 
you could just upsample your 16/44 and try to see if you still can tell the difference as if it was highres music.

 

[dprimary]

 
Did you find a similar device with balanced out, and if so what was it?

I gave up on a pocket sized device and bought apogee convertors that would connect to my existing phone, tablets, and computers. It was double the money but it has mic preamps and line inputs as well.