Head-Fi.org › Forums › Equipment Forums › Sound Science › Research that ABX/DBT testing reflects real world listening?
New Posts  All Forums:Forum Nav:

Research that ABX/DBT testing reflects real world listening? - Page 2

post #16 of 62
I just don't like it when audiophiles like the Stereophile crew get beat up for their passion, as misdirected as it might end up being. OTOH it's not nice to bankrupt people by encouraging them to spend all their money on ghosts, either. Especially when a transient weekend flirtation in Portugal might have been cheaper, more fun and more memorable. smily_headphones1.gif

I did enjoy reading Stereophile, if only as a grand fiction I was willing to part with $12/yr for.
post #17 of 62
Thread Starter 
Quote:
Originally Posted by stv014 View Post

ABX does not reflect real world listening if you consider bias to be an important part of the listening experience. After all, regular listening is normally sighted, and is therefore affected by many factors not directly related to actual (objective) sound quality, like (visual) aesthetics, brand loyalty, etc. However, it is still important to be able to separate real audible differences (or lack of thereof) from these, since not everyone has the same biases; for example, someone might not be aware of the alleged flaws of an amplifier using negative feedback, and thus fail to "hear" the benefits of a no feedback one.

I agree with that. But I'm looking at something different. It's the idea of whether or not critically evaluating headphones in ABX test is equivalent to the listening experience--outside of those biases. The end goal is a pleasurable listening experience; good SQ as determined by ABX testing, it is being assumed, is reflective of that pleasurable experience.

Here's an example. I have an MA in literature. By the end of the program, I critically evaluated any movie I saw to such an extent that most movies I watched were no longer enjoyable in the same way--and many were no longer enjoyable at all. Took about five years after that before I would allow myself to "experience" the narrative rather than evaluating it. So in a testing situation, which is very clinical--everyone is aware they are evaluating something--the listening experience as an aesthetic experience may not be the same. That's different from the biases you are mentioning. In other words, the test must necessarily bias someone in an aesthetic experience toward a critical evaluative experience that may or may not be the same as how they would otherwise experience things.

I think that term "aesthetic experience" may be throwing people off here, too. An aesthetic experience is subjective, but it has more meaning that science gives it when speaking of the subjective.
post #18 of 62
Quote:
Originally Posted by cel4145 View Post

I critically evaluated any movie I saw to such an extent that most movies I watched were no longer enjoyable in the same way--and many were no longer enjoyable at all. Took about five years after that before I would allow myself to "experience" the narrative rather than evaluating it.

 

I work in the film business and that is the price you pay to be a creator, not just a consumer. 

post #19 of 62
Quote:
Originally Posted by cel4145 View Post

I'm looking at something different. It's the idea of whether or not critically evaluating headphones in ABX test is equivalent to the listening experience--outside of those biases. The end goal is a pleasurable listening experience; good SQ as determined by ABX testing, it is being assumed, is reflective of that pleasurable experience.

 

I disagree! While the "end goal" of someone taking an ABX test may be to achieve a pleasurable listening experience, that's not what an ABX test designed to achieve. All an ABX test is designed to achieve is whether or not the person taking the test can detect a difference between the items being tested. When we start looking at SQ in terms of "good", "bad" or "pleasurable experience" we enter the realm of individual perception and therefore by definition, individual bias. ABX testing is the best method we have of eliminating an individual's biases. To a certain extent ABX testing allows us to make a distinction between hearing and sound perception. Making a qualitative judgement appears on the face of it to be relatively simple, do we like something better than something else? In reality though it's a complex sequence of processes but the first of these processes is always; can we detect a difference? Unfortunately, the entire process of making a qualitative judgement can drastically affect whether or not the very first step is achieved accurately. ABX testing is designed as far as possible to eliminate the subsequent steps in the process of making a qualitative judgement and forces us to concentrate specifically on the first step. So I disagree with your statement, because ABX testing does not exist to determine good SQ, infact it could be said to exist specifically to avoid qualitative determinations!

 

Of course, all this raises various questions regarding the properties of sound, the human ears' ability to respond to those properties and the perception of sound. We have developed a branch of science specifically to deal with all this, psychoacoustics. The properties and behaviour of sound are very well understood, as is the physiology of the ear and how/what it is able to respond to, but while there have been great advances in psychoacoustics, there are still many more questions than there are answers when it comes to the perception of sound. While it would be nice to have a simple explanation of how we perceive sound, I don't agree with the original post, that scientists generally tend to believe that the perception of sound or the aesthetic experience of listening to music is in anyway trivial or that science has all the answers. In fact, if anything, the opposite is more true! We can make some very broad generalisations about sound and music perception and indeed many of these generalisations have been known about and consciously employed by composers for centuries but when we look in finer detail, it turns out that perception of sound is orders of magnitude more complex than the man-in-the-street (or even many music professionals) realise. Take the example of pitch detection, which on the face of it seems relatively simple but in psychoacoustics turns out to be a bit of a nightmare! All naturally occurring sounds contain a variety of frequencies, which we describe as a fundamental plus a series of harmonics. It is generally taught that the frequency of the fundamental defines the pitch of the note and indeed this is the basis upon which tuning and pitch detection software operates. We can prove this supposition by fabricating a single frequency (a sine wave) without any harmonics and the brain can indeed identify pitch. However, we can also take a complete note (fundamental + harmonic series), entirely remove the fundamental and still the brain can determine the pitch. So, we can take a note with no harmonics and determine pitch and we can take a note with no fundamental and determine pitch but we know there is no other component to pitch other than a fundamental + harmonics, so how the hell do we hear pitch?! It would seem that pitch is more of a human perception than of any specific quality of sound waves, even though pitch perception is directly related to frequency content. Psychoacoustics does have various theories but none of them fully account for pitch perception (see here for more details). I use this example of pitch perception to illustrate that even some apparently basic perceptions are not fully understood, let alone the issue of qualitative judgement of those perceptions. This raises the question of why perception even exists as something which is different to just hearing.

 

Science has discovered that the body's sensory organs produce many times more data than the brain could possibly process. To get round this problem, the brain essentially discards approximately 90% of this sensory data and then reconstructs or interpolates the remaining data to create an impression or perception of this data. Exactly how this process operates is not fully understood, although in general terms the brain appears to throw away large blocks of data and replace it with simple assumptions based on experience. This is the basics of how and why visual and aural illusions exist, by using the brain's assumption/experience data reduction technique to fool the perception. In some cases, knowing how the illusion works changes our knowledge/experience so we can perceive the illusion for what it is. In other cases, even knowing how an illusion works is not enough to stop us from experiencing the illusion, this suggests some sort of complex "weighting" between assumption and knowledge/experience, where one can overide the other and vice versa. What's also quite intriguing is that the concept of separate senses is in fact itself an illusion as it seems that the perception which results from the data reduction process is not separate at all but entirely interlinked. In other words, what we perceive as hearing is in fact a combination of all our senses plus a very significant dose of assumption and personal experience and, that this is true of all the senses. What we see is affected by what we hear, what we taste, what we assume and what we feel like. What we taste is affected by what we see, what we hear, what we assume and what we feel like and what we feel like is affected by what we hear, see, smell, taste, assume, etc. This makes a mockery of audiophiles who suggest there must be a fault with digital audio as a recording never sounds the same as the live performance. Understanding a few of the basics of audio perception suggests that even a theoretically absolutely perfect recording and playback chain will never provide more than a tiny fraction of the entire sensory data necessary, let alone the expectation and other assumption/experience components necessary to recreate an identical perception. If you can find it, watch the BBC Horizon episode - Is Seeing Believing.

 

We may one day understand perception better and discover ways to improve the ABX test so we can test more of the complexities of perception and of qualitative judgement but for the time being, ABX represents the best method we have of eliminating as much of the individual's assumption/experience errors (implicit in the perception process) as possible by asking the most basic question: Without knowing which is which, can we detect any sort difference between two items?

 

G


Edited by gregorio - 11/9/13 at 3:58am
post #20 of 62
This is exactly what I was trying to get at elsewhere in the forum, thank you! I've enjoyed studying the principles of acoustics, psychoacoustics and electronic sound for years as a hobby, and am no expert, but it has at least become clear to me that 1) The experience of sound is as much a neurologically-generated thing as it is a result of sensory input -- there's probably a complex, ever-changing dynamic feedback loop between reception and interpretation; and 2) Therefore the best thing audio science can do for the sake of improving audio equipment is create the scientific constraints, such as ABX testing, that can at least isolate and describe those things in the physical design of the equipment that will lead to the most accurate reproduction. Accuracy in this case meaning, repeatable, consistent correlation between information input, and information output.

Beyond that, we can then recognize there are phenomena which may yet be objectified with further science, such as psychoacoustics; and others that "exist" in the subjective but are out of the reach so far of our scientific analysis, involving future knowledge about neurological processes.

Some initial use of psychoacoustics in its evolution are at least clearly being used by companies like Sennheiser and Shure to take the next steps forward in transducer design.

It's all very exciting and interesting stuff!
Edited by Copperears - 11/9/13 at 4:13am
post #21 of 62
Thread Starter 
Quote:
Originally Posted by gregorio View Post

I disagree! While the "end goal" of someone taking an ABX test may be to achieve a pleasurable listening experience, that's not what an ABX test designed to achieve. All an ABX test is designed to achieve is whether or not the person taking the test can detect a difference between the items being tested. When we start looking at SQ in terms of "good", "bad" or "pleasurable experience" we enter the realm of individual perception and therefore by definition, individual bias. ABX testing is the best method we have of eliminating an individual's biases.

Don't shoot the messenger. I agree that ABX testing doesn't seem designed to do this. LOL

Harman engineers are the ones making the claims that people prefer a flat frequency response based on ABX testing. My point is that the ultimate goal of all this research is to find out what speaker attributes result in a pleasurable listening experience, and that's exactly one of the things that I'm questioning--whether or not ABX testing can work for that. Which I was asking about the research on this in my first post.

And ironically, the research Harman is doing is even getting misapplied. The other day on another forum, someone responded to me in a discussion and stated that this research proves that both trained listeners and regular people prefer flat speakers. The poster wouldn't listen to me when I told him the study doesn't claim that, that they would need to sample a much larger range of speakers to try to assert that.

Note that Harman is now doing something similar with headphones, using ABX testing to claim that a neutral frequency response is preferred.
post #22 of 62
The problem with coloration introduced into the signal path is that it is "one color" applied to everything (to get kinesthetic for a moment).

It may sound appealing at first, but eventually becomes noticeable and biases the listening experience around that coloration, thus, in my opinion and experience at least, flattening the overall variety of the musical experience possible.

The cynic in me also says it guarantees companies can sell overpriced equipment, as owners get tired of one type of coloration and inevitably seek another. smily_headphones1.gif Or at the other extreme, use the pursuit of accuracy to identify marketable engineering ideas about the presence of noise and distortion that are not discernible but people will pay top dollar for if sold on the possibility of this terrible impurity in their lives they can eradicate with more $$$$$.....

I know, just being cynical. I'd still love a complete Naim system. smily_headphones1.gif
post #23 of 62
Quote:
Originally Posted by cel4145 View Post


Don't shoot the messenger. I agree that ABX testing doesn't seem designed to do this. LOL

Harman engineers are the ones making the claims that people prefer a flat frequency response based on ABX testing. My point is that the ultimate goal of all this research is to find out what speaker attributes result in a pleasurable listening experience, and that's exactly one of the things that I'm questioning--whether or not ABX testing can work for that. Which I was asking about the research on this in my first post.

And ironically, the research Harman is doing is even getting misapplied. The other day on another forum, someone responded to me in a discussion and stated that this research proves that both trained listeners and regular people prefer flat speakers. The poster wouldn't listen to me when I told him the study doesn't claim that, that they would need to sample a much larger range of speakers to try to assert that.

Note that Harman is now doing something similar with headphones, using ABX testing to claim that a neutral frequency response is preferred.

 

Just a comment here:  My understanding is that Harman are not that people prefer flat or neutral in headphones, but rather that they prefer the sound of good speakers in a good room, which is close but somewhat warmer than flat or neutral.  The original study at the NRC in the '80s searched for what was pleasing on speakers, and they found that the pleasing response was somewhat warmer than flat...but it was close to flat.

post #24 of 62
Thread Starter 
Quote:
Originally Posted by Tyll Hertsens View Post

Just a comment here:  My understanding is that Harman are not that people prefer flat or neutral in headphones, but rather that they prefer the sound of good speakers in a good room, which is close but somewhat warmer than flat or neutral.  The original study at the NRC in the '80s searched for what was pleasing on speakers, and they found that the pleasing response was somewhat warmer than flat...but it was close to flat.

Thanks! I'm new to this, so I don't know all the research (I'd like to learn more). What I'm going on is Olive's test here claims, "The highest rated loudspeakers had the flattest measured frequency response maintained uniformly off axis" (from the abstract; the conclusion discusses the correlation found for this). Then Toole's older research (1986) which is often cited by proponents of flat speaker measurements that says, "Given the proper circumstances, experienced listeners with normal hearing prefer loudspeakers with wide bandwidth, flat and smooth amplitude response, and uniformly wide dispersion." In his blog post about recent headphone research, Olive has a whole section supporting the heading claims that "Listeners Prefer Headphones With An Accurate, Neutral Spectral Balance."
post #25 of 62
And then there's phase coherence; that was a big fad at one point, which morphed into the theory behind the superiority of bi-polar D'Appolito Arrays...... ah, memoriez.... wink.gif
post #26 of 62
Quote:
Originally Posted by cel4145 View Post


Thanks! I'm new to this, so I don't know all the research (I'd like to learn more). What I'm going on is Olive's test here claims, "The highest rated loudspeakers had the flattest measured frequency response maintained uniformly off axis" (from the abstract; the conclusion discusses the correlation found for this). Then Toole's older research (1986) which is often cited by proponents of flat speaker measurements that says, "Given the proper circumstances, experienced listeners with normal hearing prefer loudspeakers with wide bandwidth, flat and smooth amplitude response, and uniformly wide dispersion." In his blog post about recent headphone research, Olive has a whole section supporting the heading claims that "Listeners Prefer Headphones With An Accurate, Neutral Spectral Balance."

 

Um...you are correct, but my understanding is that one needs to understand that once you put a flat and even responding speaker with good off-axis performance in a room, the acoustic power response in the room has significant low frequency gain due to boundary gain and the bass-heavy off-axis radiation of the speaker. A flat speaker in an anechoic chamber sounds too lean for most listeners. So while the speaker response itself may be flat, the heard signal has more bass energy in a well designed room. I guess the point here is that the room and the speaker have to be considered simultaneously when considering pleasing speaker response.

 

That's my understanding anyway. And that's why I said the pleasing speaker response is close to flat and neutral, but not quite the whole story.

 

BTW, I'm in the midst of reading FLoyd Toole's book, "Sound Reproduction: The Acoustics and Psychoacoustics of Loudspeakers and Rooms", which is a brilliant dialog about this subject, and an eminently readable narrative. I'm constantly struck by the ability of these researchers to tease out---sometimes quite counterintuitive---truths about the listening experience. (For example: speech intelligibility of reproduced audio is aided by early reflections, which one might otherwise assume would be confusing not helpful.)

post #27 of 62
Quote:
Originally Posted by cel4145 View Post

Note that Harman is now doing something similar with headphones, using ABX testing to claim that a neutral frequency response is preferred.

 

 

That is not entirely fair  - this is what Dr Olive says

 

 

Quote:
The results provide evidence that trained listeners preferred the headphones perceived to have the most neutral, spectral balance. The acoustical measurements of the headphone generally confirmed and predicted which headphones listeners preferred....it is important for the reader not to draw generalizations from these results beyond the conditions we tested.
 
One audio writer has already questioned whether headphone sound quality preferences of trained listeners can be extrapolated to tastes of untrained younger demographics whose apparent appetite for bass-heavy headphones might indicate otherwise. We don't know the answer to this question. For younger consumers, headphone purchases may be  driven more by fashion trends and marketing B.S. (Before Science) than sound quality.  While this question is the focus of future research, ...the preliminary data suggests  in blind A/B comparisons kids pref headphones with accurate reproduction to colored, bass-heavy alternatives.  This would tend to confirm findings from previous investigations into loudspeaker preferences of high school and college students (both Japanese and American) that so far indicates most listeners prefer accurate  sound reproduction regardless of age, listener training or culture.

 

Note how carefully he frames the conclusions and speculations and draws on evidence from other studies, Olive never says that the research "proves" anything , he allows himself a bit of speculation but that is also carefully framed as such based on current evidence. His language is cautious as it should be.


Edited by nick_charles - 11/9/13 at 12:34pm
post #28 of 62
Thread Starter 
Quote:
Originally Posted by nick_charles View Post

Note how carefully he frames the conclusions and speculations and draws on evidence from other studies, Olive never says that the research "proves" anything , he allows himself a bit of speculation but that is also carefully framed as such based on current evidence. His language is cautious as it should be.

No doubt. I didn't say he was "proving" anything there. Reread my statement:
Quote:
Originally Posted by cel4145 View Post

In his blog post about recent headphone research, Olive has a whole section supporting the heading claims that "Listeners Prefer Headphones With An Accurate, Neutral Spectral Balance."
post #29 of 62
Quote:
Originally Posted by cel4145 View Post

Here's an example. I have an MA in literature. By the end of the program, I critically evaluated any movie I saw to such an extent that most movies I watched were no longer enjoyable in the same way--and many were no longer enjoyable at all. Took about five years after that before I would allow myself to "experience" the narrative rather than evaluating it. So in a testing situation, which is very clinical--everyone is aware they are evaluating something--the listening experience as an aesthetic experience may not be the same. That's different from the biases you are mentioning. In other words, the test must necessarily bias someone in an aesthetic experience toward a critical evaluative experience that may or may not be the same as how they would otherwise experience things.

I think that term "aesthetic experience" may be throwing people off here, too. An aesthetic experience is subjective, but it has more meaning that science gives it when speaking of the subjective.

 

Listening experiences can be different, listening for differences as in ABX is just one possible mode out of many others which can still incorporate critical elements such as the movie experience you mention. Some people listen as an intellectual exercise, some for relaxation, some to experience beauty, some have music as a background activity and so on, you are still making it a binary distinction. That said your premise that ABX is flawed because it is always a critical activity does not hold water for two reasons. First somebody may have that kind of approach for normal listening , if you read some of the horrendously florid essays in the subjectivist subforums it looks very much like some folk listen just to hear technical quality and/or differences so they are already doing the critical evaluative thing by default. Secondly even if the "listening for pleasure" (I use the term for simplicity) and critical listening modes (allowing your dichotomy for now) are different it does not in any way give you any serious evidence that "listening for pleasure" has greater value in allowing us to accurately establish that any differences in the listening experience between A and B are wholly due to the "real" differences between A and B and not due to any number of other uncontrolled variables i.e. we are back to more unprovable hypotheses. In other subforums this is meal du jour (and that is fine) , but here we seek for better evidence and these wholly subjective experiences are non-transferrable and thus of more limited utility.

 

The only other implication from your stance is that there is no point in any attempt to critically assess differences using the human experience, which is fine but this then allows the charlatans free reign to put forth any nonsense knowing it cannot be challenged, at which point let me introduce my invisible dragon !


Edited by nick_charles - 11/9/13 at 1:18pm
post #30 of 62
Thread Starter 
Quote:
Originally Posted by Tyll Hertsens View Post

Um...you are correct, but my understanding is that one needs to understand that once you put a flat and even responding speaker with good off-axis performance in a room, the acoustic power response in the room has significant low frequency gain due to boundary gain and the bass-heavy off-axis radiation of the speaker. A flat speaker in an anechoic chamber sounds too lean for most listeners. So while the speaker response itself may be flat, the heard signal has more bass energy in a well designed room. I guess the point here is that the room and the speaker have to be considered simultaneously when considering pleasing speaker response.

That's my understanding anyway. And that's why I said the pleasing speaker response is close to flat and neutral, but not quite the whole story.

BTW, I'm in the midst of reading FLoyd Toole's book, "[URL=http:]Sound Reproduction: The Acoustics and Psychoacoustics of Loudspeakers and Rooms[/URL]", which is a brilliant dialog about this subject, and an eminently readable narrative. I'm constantly struck by the ability of these researchers to tease out---sometimes quite counterintuitive---truths about the listening experience. (For example: speech intelligibility of reproduced audio is aided by early reflections, which one might otherwise assume would be confusing not helpful.)

Regardless of which Harman is leaning, the "speakers are best when measured flat" group always cite those first to pieces, which is partially when I'm interested in all this.

I wish I could read Toole's text, but I suspect it would over my head smily_headphones1.gif
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › Research that ABX/DBT testing reflects real world listening?