[castleofargh]

Listening tests are untrustworthy. The factors I feel have to do with the subjective interpretations each different listener perceives. Add to that how each person remembers what they have just perceived. Tests on visual cognitive (episodic visual memory) show that the brain files images away in accordance to previously understood visual parameters. Thus that tea pot you saw in the last room may be remembered as having a round knob on the lid simply due to the fact that most tea pots have round knobs, when reality proves the knob in truth was square.

But from the earliest of audiophile days, quality has been based on how true to life the reproduction is. And of course there are scientific tests which show us when an audio response differs from what is regarded as the correct baselines. This is how we got to where we are!

Could distortions in the right places make a headphone, amp or DAC sound more musical? Yes.
Could there be finite responses from gear that still remain slightly out of reach of being quantified by measuring equipment?
And finally I would rather live with a technically imperfect system which allowed me hours of casual listening over a perfect specification reproduction that left the music lifeless and cold.


But in ending the proof comes with time. Obviously the colored system is going to get boring over time once the magic wears off and the brain figures out that there are frequencies enhanced or left out in listening. But I tend to look at it like cars. Car engineers use the cutting edge of computer design combined with the complete science and physics history of mankind. They have big bucks on the line and all the money in the world for R&D. Yet somehow we have models and year makes of great cars by different manufactures and cars that come out either normal or inferior. Cars now in relationship to cars from the 1960s are completely different machines. The level of tolerances and intelligence of design have made today's cars faster, more comfortable and finally more dependable than car makers of the 1960s could ever have dreamed of. The designs tested right before anything went into production.

Still it's the daily use of that car........the humanity of that new car.......it's personality in the end so to speak, which over time creates a relationship. It's this relationship which provides the consumer with how good the product is. Electrical equipment for the reproduction of audio follow the same parameters in the end.

you always want to bring everything back to the emotional response. that's your own choice where you answer your own question "how do I feel?". just realize that we're trying to identify sound differences by ear, not feelings. feelings can be born out of virtually anything. audible differences can only be born from sound differences. if you keep changing the question for the one you like, of course we'll never agree on anything. we end up in some comedy where I ask how many fingers and you reply that you feel fine.

as for physoacoustic, echoic memory, and biases, there is a vast amount of literature and evidence. research done in the 70's was already suggesting some of the stuff we talk about, and since it never really stopped. it's not intuitive(not everything in life has to be), and because we are strictly limited to how our body works, we come to a crucial moment where 2 opposing principles fight each other. longer time to observe something will let us notice more, but longer time and more data also leads to less reliable memories of the experience. you can easily find examples to agree with each principle and decide that you've proved it true, but what do you know about the ideal moment between both for echoic memory? well people have been at it for a long time and have tested a lot of stuff. their conclusions consistently favored very short samples and consistently agreed that after a few seconds, the brain already starts to mess with the audio memory.

also here we're not trying to learn a piece of music by heart, count all the instruments, name them, etc. we're trying to check if our ears notice a change between 2 DACs. there is no need to analyze anything or think about it, it's our most primal function. if a 10khz tone almost instantly turns into a 8khz tone, you will notice a change. doesn't matter that you don't know what. doesn't matter that you need more time to tell the frequency of the change, realize that it's a lower tone, that it reminds you about jellyfish for some reason. and of course it doesn't matter if you prefer 10khz. those are not the questions we're trying to answer in this particular situation. was there an audible difference? yes. can you consistently notice it in a blind test? yes. bravo you've done it those 2 sounds were audibly different to you.
later if you want to get into identifying the difference, or simply go with your guts and pick the DAC you like more because that front panel reminds you of the day you listened to Stravinsky with your dad as a kid, well go for it. just understand that it's a different quest, and everybody isn't trying to always answer that one quest for you. sometimes, we care about the actual sound(because after all, we think we're paying for that).
your mistake isn't to care about the emotional response of music, we all care about that. your mistake with us at least, is to always try to bring everything back to it. a section that centers on objective approach isn't the best place to justify everything with emotions. ^_^

 

[Redcarmoose]

You don't trust your own ears?! Do you trust your philosophy and biases more than your ability to hear?

I actually do and only go by my subjective listening tests. And I do understand about short duration sound bite listening. Leaning if I could hear a difference between 320kbps and 16/44.1 FLAC, I set up a set of files where the same songs were in random in Foobar. After randomly choosing the same song title twice I could not differentiate between MP3s and lossless sound files. That was probably due to reading right here in SS, as I was sure lossless was the better way. In the process I did small 2 second listens between the beginning of songs I had had a long history with.

What I mean when describing personal equipment reviews, I myself need to spend a long time with equipment till I learn it's true nature. I have done reviews calling headphones as sounding natural, when 1/2 a year later it became apparent that the headphones were the farthest from natural sounding. So it never ceases to amaze me how undependable our hearing can be as far as testing equipment.

you always want to bring everything back to the emotional response. that's your own choice where you answer your own question "how do I feel?". just realize that we're trying to identify sound differences by ear, not feelings. feelings can be born out of virtually anything. audible differences can only be born from sound differences. if you keep changing the question for the one you like, of course we'll never agree on anything. we end up in some comedy where I ask how many fingers and you reply that you feel fine.

as for physoacoustic, echoic memory, and biases, there is a vast amount of literature and evidence. research done in the 70's was already suggesting some of the stuff we talk about, and since it never really stopped. it's not intuitive(not everything in life has to be), and because we are strictly limited to how our body works, we come to a crucial moment where 2 opposing principles fight each other. longer time to observe something will let us notice more, but longer time and more data also leads to less reliable memories of the experience. you can easily find examples to agree with each principle and decide that you've proved it true, but what do you know about the ideal moment between both for echoic memory? well people have been at it for a long time and have tested a lot of stuff. their conclusions consistently favored very short samples and consistently agreed that after a few seconds, the brain already starts to mess with the audio memory.

also here we're not trying to learn a piece of music by heart, count all the instruments, name them, etc. we're trying to check if our ears notice a change between 2 DACs. there is no need to analyze anything or think about it, it's our most primal function. if a 10khz tone almost instantly turns into a 8khz tone, you will notice a change. doesn't matter that you don't know what. doesn't matter that you need more time to tell the frequency of the change, realize that it's a lower tone, that it reminds you about jellyfish for some reason. and of course it doesn't matter if you prefer 10khz. those are not the questions we're trying to answer in this particular situation. was there an audible difference? yes. can you consistently notice it in a blind test? yes. bravo you've done it those 2 sounds were audibly different to you.
later if you want to get into identifying the difference, or simply go with your guts and pick the DAC you like more because that front panel reminds you of the day you listened to Stravinsky with your dad as a kid, well go for it. just understand that it's a different quest, and everybody isn't trying to always answer that one quest for you. sometimes, we care about the actual sound(because after all, we think we're paying for that).
your mistake isn't to care about the emotional response of music, we all care about that. your mistake with us at least, is to always try to bring everything back to it. a section that centers on objective approach isn't the best place to justify everything with emotions. ^_^

I'm the farthest from a scientific mind, and we all know that. The profession I worked in was retail. In retail we use a lot of study of the mind and the differences between buyers. It's useful because each member of the general public think, comprehend and purchase in very different ways. So the folks that were scientists are the personality profile we would call structured. The other types of scientific personality profiles would be called paced. The structured are the researchers and they could have complete notebooks of information before they felt self qualified to make a decision. The paced also had a specific time frame which delineated when the right time to purchase was. The final two personality profiles were dominate and ego driven personalities. The dominate buyer was normally self employed, they owned their own business, they talked in short direct questions and wanted short direct answers. For the direct the results were the end all concern.This process in learning basic personality profiles was so that we could both understand a member of the general public on another level, as well as it gave us parameters which helped us communicate with them in such a way that they could relate. The process enhanced communication with, and the understanding of, strangers.

The ego driven personality was super concerned with how they were perceived as in public. Normally their purchase could be even justified as to how it make them look to friends, or how the purchase made them look to themselves.

You also have to realize that no one person is 100% of the four categories. You will get a bunch of ego/dominates and you will see a bunch of paced/structured types. Rarely you will get 50% dominate and 50% structured.

After 20 years of this personality profiling all of us came to conclusions. The conclusion was 100% of buying decisions were emotional. Much of the time the structured buyer had way too much information collected to digest. In some situations distilling the facts down to a reasonable list packet was the only way to move the emotions forward.

Speaking of DACs and the NASA scientist researching the different choices, your going to have the structured personality profile study the different DAC methodologies and brand reputations. The NASA scientist is going to look over the percent rates of distortion. The scientist is going to research if he needs DSD and what his bit-rate cap will need to be. He may go as so far as to check the quality of the level of components like capacitors and volume knobs in relation to choices.

Still after all is done and he is ready to buy a DAC, that purchase will be an emotional decision.

I'm not the scientist and I know close to nothing of the electrical engineering that it takes to make the 1s and 0s into an analogue signal. Also my learning curve understanding new equipment is very long. So I can try the short listening tests, but no matter what it seems to take months before I truly think I know what a piece of equipment is truly about.

Emotion is maybe the highest level human trait in the world of audio second only to thoughts of curiosity. Seeing every headphone review I have ever watched on You-tube was drenched with emotion. And your right, when you hear that emotion in their voice describing the headphone as the best ever, it lessens the trust level your at. A slow unemotional and methodical research methodology always goes over better when studying audio.

And in the end my statements about taking time with a product is to actually counter the emotional-new-toy responce which always happens to some degree with an acquisition of new gear.

 

[gregorio]

[1] Listening tests are untrustworthy.
[2] But from the earliest of audiophile days, quality has been based on how true to life the reproduction is.
[2a] And of course there are scientific tests which show us when an audio response differs from what is regarded as the correct baselines.
[2b] This is how we got to where we are!
[3] Could distortions in the right places make a headphone, amp or DAC sound more musical? Yes.
[4] Could there be finite responses from gear that still remain slightly out of reach of being quantified by measuring equipment?
[5] And finally I would rather live with a technically imperfect system which allowed me hours of casual listening over a perfect specification reproduction that left the music lifeless and cold.
[6] Electrical equipment for the reproduction of audio follow the same parameters in the end.

1. That's an absolute assertion, simplified to the point of it being nonsense! There are a great many factors which can influence listening tests: For example, what we are testing for, how well the test is controlled (how many biases it eliminates) and sample size. The level of trust we can have with listening tests therefore varies from extremely low/almost none to extremely high/virtually certain.
2. Yep, that's one of the biggest fallacies/myths in the audiophile world, apparently stemming from a complete ignorance of how recordings are actually created/produced. In virtually all cases there is no "true to life" in the first place and in most cases, "true to life" is the very last thing we wish to achieve. So, audiophiles are judging quality based on a comparison of real life with the reproduction of a recording which is intentionally not true to life, which is obviously a nonsense comparison!
2a. There are simple tests which can extremely accurately identify the difference between an input signal to a piece (or pieces) of equipment and the output. The less of a difference, the more faithful (high fidelity) that piece of equipment.
2b. No, how we got where we are is due to audiophile marketing practices, the perversion of the facts, for financial gain and audiophiles' willingness to accept them.
3. Obviously not! Think about your answer logically for just a moment. Is the musicality of say Ace of Spades (Motorhead) the same as the musicality of Clair de lune (Debussy)? Would you want the musicality of Ace of Spades applied to Clair la Lune, or vice versa? How would an amp or DAC know what sort of music I'm listening to and how would it apply the appropriate musicality? Obviously this assertion is complete and utter nonsense. If you mean "euphonic distortion" rather than "musicality" then your assertion is somewhat less ludicrous but still, is the same "euphonic distortion" equally appropriate for all recordings, as appropriate for Ace of Spades as for Clair la lune?
4. No! Not even close to being a maybe. However, there are certainly human perceptual responses to the "finite responses from gear" which are entirely out of the reach of measuring equipment.
5. And this is the crucial point! If a system had "perfect specification reproduction" and the music sounded "lifeless and cold" it's because the artists created that way. What you're saying is that you don't want to hear what the artists created, you want your system to change what the artists created into whatever it is you think sounds better to you. Personally that's the exact opposite of what I want! If the artists created/intended "lifeless and cold", the very last thing I want is for my amp or DAC to try and turn it into "full of life and warm"! Of course it's your choice but if you do choose a system which changes what the artists created then you obviously cannot describe that system as high fidelity, transparent or neutral but unfortunately, that's exactly what most audiophiles do! This indicates they have absolutely no idea what the terms high fidelity, transparent or neutral actually mean, which effectively brings us back to point 2b.
6. Shouldn't "equipment for the reproduction of audio" try to accomplish the actual task for which it exists as well as possible, that of reproducing the audio? Maybe you're getting your terminology wrong, maybe you mean; audiophile electrical equipment for modifying audio, rather than "reproducing" audio?

G

 

[JaeYoon]

1. That's an absolute assertion, simplified to the point of it being nonsense! There are a great many factors which can influence listening tests: For example, what we are testing for, how well the test is controlled (how many biases it eliminates) and sample size. The level of trust we can have with listening tests therefore varies from extremely low/almost none to extremely high/virtually certain.
2. Yep, that's one of the biggest fallacies/myths in the audiophile world, apparently stemming from a complete ignorance of how recordings are actually created/produced. In virtually all cases there is no "true to life" in the first place and in most cases, "true to life" is the very last thing we wish to achieve. So, audiophiles are judging quality based on a comparison of real life with the reproduction of a recording which is intentionally not true to life, which is obviously a nonsense comparison!
2a. There are simple tests which can extremely accurately identify the difference between an input signal to a piece (or pieces) of equipment and the output. The less of a difference, the more faithful (high fidelity) that piece of equipment.
2b. No, how we got where we are is due to audiophile marketing practices, the perversion of the facts, for financial gain and audiophiles' willingness to accept them.
3. Obviously not! Think about your answer logically for just a moment. Is the musicality of say Ace of Spades (Motorhead) the same as the musicality of Clair de lune (Debussy)? Would you want the musicality of Ace of Spades applied to Clair la Lune, or vice versa? How would an amp or DAC know what sort of music I'm listening to and how would it apply the appropriate musicality? Obviously this assertion is complete and utter nonsense. If you mean "euphonic distortion" rather than "musicality" then your assertion is somewhat less ludicrous but still, is the same "euphonic distortion" equally appropriate for all recordings, as appropriate for Ace of Spades as for Clair la lune?
4. No! Not even close to being a maybe. However, there are certainly human perceptual responses to the "finite responses from gear" which are entirely out of the reach of measuring equipment.
5. And this is the crucial point! If a system had "perfect specification reproduction" and the music sounded "lifeless and cold" it's because the artists created that way. What you're saying is that you don't want to hear what the artists created, you want your system to change what the artists created into whatever it is you think sounds better to you. Personally that's the exact opposite of what I want! If the artists created/intended "lifeless and cold", the very last thing I want is for my amp or DAC to try and turn it into "full of life and warm"! Of course it's your choice but if you do choose a system which changes what the artists created then you obviously cannot describe that system as high fidelity, transparent or neutral but unfortunately, that's exactly what most audiophiles do! This indicates they have absolutely no idea what the terms high fidelity, transparent or neutral actually mean, which effectively brings us back to point 2b.
6. Shouldn't "equipment for the reproduction of audio" try to accomplish the actual task for which it exists as well as possible, that of reproducing the audio? Maybe you're getting your terminology wrong, maybe you mean; audiophile electrical equipment for modifying audio, rather than "reproducing" audio?

G

That is another issue. Number 5.
I asked several audiophiles what is the standard measurement in way they class dac/amp products in High fi or "top of the line". Some of them outright told me it's because of price. So more expensive the better it is.

That's a good question. Shouldn't products reproduce audio. A lot of companies do color the sound intentionally whether it's done in amp or not. It's definitely not accurate but more for a specific experience

 

[RRod]

That's a good question. Shouldn't products reproduce audio. A lot of companies do color the sound intentionally whether it's done in amp or not. It's definitely not accurate but more for a specific experience

If an album was mixed/mastered on extremely "colored" equipment, then technically you need to reproduce that coloring. This is why Olive made his post about the 'audio circle of confusion': without objective standards for the creative environment, how can we possibly quantify fidelity of the listening environment?

 

[Redcarmoose]

That is another issue. Number 5.
I asked several audiophiles what is the standard measurement in way they class dac/amp products in High fi or "top of the line". Some of them outright told me it's because of price. So more expensive the better it is.

That's a good question. Shouldn't products reproduce audio. A lot of companies do color the sound intentionally whether it's done in amp or not. It's definitely not accurate but more for a specific experience

You would assume that SQ would be the top priority for any and every manufacture. That's the goal with expensive audiophile gear, right? If your a manufacture you own a business, you do have a list of priorities. When asked where the priority of sound quality was one owner failed to list where the priority placed. So obviously getting the best sound possible may not be at the top of the list. Turning a profit would most likely stay at the top of the list. And the hobby is filled with ideas from patrons of guaranteed sound quality insurance if the price is high enough.

And obviously there are manufactures who charge a lot for their equipment due to the fact that it is hard and expensive to make and sound quality is the end goal. I just learned about a company making $300,000 turntables. Now maybe there is a lot of overkill in the design, but it seemed like the owner/inventor really wanted to test the limits of what was possible in the world of vinyl playback. He has sold orders for three turntables and at this point in time two of his turntables exist in the world.

 

[bigshot]

I actually do and only go by my subjective listening tests.

So you do trust listening tests, just not *controlled* ones. If I'm understanding you correctly, the problem isn't listening, it's that you think your subjective impression of what you're hearing is more important than what you are objectively hearing with your ears. Is that correct?

If you wanted to improve your subjective impression of the sound quality of your sound system, how would you go about doing that? Would you shop for equipment with beautiful illuminated faceplates like McIntosh, or softly glowing tube amps, or perhaps headphones made of polished burl wood? Would you throw a bunch of velvet and satin pillows on your leather couch, light some patchouli incense, pop the cork on a bottle of wine, and lower the lights in your listening room? Would you select music that is beautiful and serene to listen to, like Mozart or Satie? It seems to me that any of those things would significantly improve the subjective impression of recorded sound. If I wasn't particularly interested in objective sound quality with my equipment, those are probably the areas I'd focus on to improve my appreciation of recorded music. I certainly wouldn't focus on bitrates or technical issues at all. I'd just buy pretty equipment and create a comfortable space to listen in. I can totally see the value of discussing ways to improve subjective impressions. Perhaps there should be a forum called "Sound Solipsism". I'd actually enjoy participating in that forum myself.

By the way, I just want to let you know that I'm not always reading your entire reply. You inject a lot of stuff that isn't related to the point you are trying to make and I have trouble focusing on what you're actually trying to say. I'm not asking questions to try to trick you. I'm just trying to figure out what you are trying to put across. The way you say it may be clear to you, but it isn't always clear to me.

 

[amirm]

Listening tests are untrustworthy.

What I mean when describing personal equipment reviews, I myself need to spend a long time with equipment till I learn it's true nature. I have done reviews calling headphones as sounding natural, when 1/2 a year later it became apparent that the headphones were the farthest from natural sounding. So it never ceases to amaze me how undependable our hearing can be as far as testing equipment.

You just gave us an example of an untrustworthy listening test.

To wit, we don't know if either your initial impression of said headphones or the latter one are correct. There is no way to verify either. Not for us. And not for you. But I bet you ran with the latter conclusion. Why?

With controlled listening tests, we have ways of determining likelihood of such results being correct. For example, we can take a headphone and on purpose increase some frequencies by a few db and then do an AB with and without that boost. Then we test a few listeners. Those who can't tell such a boost has been applied are then dismissed from further testing. The key here is that *we know the right answer*. That is, we knew in advance that we had messed up with the sound and so only one outcome from listeners would be correct. Harman's software training for their expert listeners is based on this type of testing.

The problem facing audiophiles is that they rarely know what the right answer is in their listening tests. By dismissing objective measurements and audio science, they are left with only their own vote. It is like taking a test at school and grading your own exam! They listen to DAC A and DAC B and declare the former to have better sound. Who says they are right? Get half a dozen such audiophiles together and you can't get a consensus to save your life.

This is why to get trustworthy conclusions you need to consider three things: listening tests, objective measurements and how science and engineering. Once you have all three you can arrive at powerful conclusions. Not 100% guarantee but close to it.

Just run with casual tests and the outcome is surely wrong when differences get small. In the case of such test, we are in violent agreement that such tests results need to be thrown out as untrustworthy.

 

[bigshot]

we don't know if either your initial impression of said headphones or the latter one are correct. There is no way to verify either. Not for us. And not for you. But I bet you ran with the latter conclusion. Why?

If the goal isn't necessarily achieving optimal audio fidelity, then the reason to decide on one over the other would be related to comfort... for instance, being happy with what you've got is preferable to taking a loss on what you spent on it and having to go out and buy a replacement. Or after your ears have acclimated to the particular coloration of the headphones, they would have to re-acclimate if he changed to another and that would be going back to square one. I can see the value of taking the attitude of "if it ain't broke, don't fix it." The law of diminishing returns is a very good reason to decide to be comfortable with "good enough". At some point all the "improvements" become inaudible and theoretical anyway. This isn't to say that objective testing isn't valuable. It's just that he bought the headphones he bought based on his needs and they do the job for him. That makes sense to me.

Now, if I was shopping for headphones, I wouldn't be much interested in his subjective review of them. While everyone's objective hearing ability is usually pretty much equivalent, everyone's subjective reality definitely isn't. When I'm shopping, it doesn't mean much if someone says they are subjectively happy with their choice. I think most people want to be happy with their choice, so they just are. Likewise, people who aren't subjectively satisfied with anything churn through equipment in search of faults to focus on to justify the next replacement. Those reviews don't help me much at all either.

I can't put my head into someone else's head. All I can do is is parse the review for objective facts that let me know whether the particular thing they're talking about would do the job for me. I think that's the difference between a good review and a lousy one. Well written reviews are simple and to the point. They give the objective facts about sound quality and features to help a buyer decide if something will work for them. A well written review doesn't try to put across irrelevant subjective impressions, and it doesn't focus on hyper critical details that really don't add up to a hill of beans.

But then I'm a Producer and that means I'm a very practical person interested in getting things done right. If I had the heart of a poet I might look for different sorts of things.

 

[Redcarmoose]

You just gave us an example of an untrustworthy listening test.

To wit, we don't know if either your initial impression of said headphones or the latter one are correct. There is no way to verify either. Not for us. And not for you. But I bet you ran with the latter conclusion. Why?

With controlled listening tests, we have ways of determining likelihood of such results being correct. For example, we can take a headphone and on purpose increase some frequencies by a few db and then do an AB with and without that boost. Then we test a few listeners. Those who can't tell such a boost has been applied are then dismissed from further testing. The key here is that *we know the right answer*. That is, we knew in advance that we had messed up with the sound and so only one outcome from listeners would be correct. Harman's software training for their expert listeners is based on this type of testing.

The problem facing audiophiles is that they rarely know what the right answer is in their listening tests. By dismissing objective measurements and audio science, they are left with only their own vote. It is like taking a test at school and grading your own exam! They listen to DAC A and DAC B and declare the former to have better sound. Who says they are right? Get half a dozen such audiophiles together and you can't get a consensus to save your life.

This is why to get trustworthy conclusions you need to consider three things: listening tests, objective measurements and how science and engineering. Once you have all three you can arrive at powerful conclusions. Not 100% guarantee but close to it.

Just run with casual tests and the outcome is surely wrong when differences get small. In the case of such test, we are in violent agreement that such tests results need to be thrown out as untrustworthy.

Great read!

Is the headphone modified from the original sound using EQ or physically modified to increase the frequencies?

I agree with your post and to answer your question, I ran with the final view of the headphone sound signature only due to a complete attitude about the signature response after repeated listening. Now if that's expectation bias, yes it could very well be. And yes the grading your own exam is the lost boat many reviewers are on as their personal understanding with no other feedback takes them out to sea.


This could be the right answer.
listening tests, objective measurements and how science and engineering.

I'm a hobbyist who has loved audio starting in the mid 1970s. I worked out some personal systems of understanding equipment when I started to be able to afford better equipment in the late 1990s. And if I refer to myself I apologize as I'm my only reference here. Shortly I will start a new buying phase of equipment. My approach may be a little like many audiophiles and maybe in some ways not. I'm not really sure upon purchase if I know the product that well. I will purchase anyway simply due to factors of name recognition and reading reviews, many times a summation of single posts here at Head-Fi. But due to my past understood testing methodology, I'm not hoping to get a complete and finite understanding when in the demo phase. It would probably be better in my situation to have stores change out and trade in products like some stores do with high end sources, amps and speakers (where you pay to play), but from where I live it's not an option. So I move forward, and again as a consumer part of the curiosity and part of the fun IS the mystery of not feeling like your going to know the product till after burn-in and after interfacing the product with a wide range of equipment and listening runs. The new toy phenomena is real and is exponentially changed by the price you payed relative to your income.

 

[Redcarmoose]

You just gave us an example of an untrustworthy listening test.

To wit, we don't know if either your initial impression of said headphones or the latter one are correct. There is no way to verify either. Not for us. And not for you. But I bet you ran with the latter conclusion. Why?

With controlled listening tests, we have ways of determining likelihood of such results being correct. For example, we can take a headphone and on purpose increase some frequencies by a few db and then do an AB with and without that boost. Then we test a few listeners. Those who can't tell such a boost has been applied are then dismissed from further testing. The key here is that *we know the right answer*. That is, we knew in advance that we had messed up with the sound and so only one outcome from listeners would be correct. Harman's software training for their expert listeners is based on this type of testing.

The problem facing audiophiles is that they rarely know what the right answer is in their listening tests. By dismissing objective measurements and audio science, they are left with only their own vote. It is like taking a test at school and grading your own exam! They listen to DAC A and DAC B and declare the former to have better sound. Who says they are right? Get half a dozen such audiophiles together and you can't get a consensus to save your life.

This is why to get trustworthy conclusions you need to consider three things: listening tests, objective measurements and how science and engineering. Once you have all three you can arrive at powerful conclusions. Not 100% guarantee but close to it.

Just run with casual tests and the outcome is surely wrong when differences get small. In the case of such test, we are in violent agreement that such tests results need to be thrown out as untrustworthy.

Also I'm pretty convinced that there are many more variables for testing one headphone. That said, all headphones have sounded different to me behind whatever equipment I've used them with. So in many ways we can say a particular headphone has a general signature and tried and true response character which was noted by the group (just like testing soda).

But in the end this hobby is very much about the fine details and the synergy which can be arrived at by critical mixing and matching. So yes, a group can be used to test a headphone with a fairly well rounded amp and source, still true flat is a response that means one thing to some and one thing to another. And of course folks respond to color or other headphone attributes which could be interpreted as artifacts in nature.

That said successful headphone tuning is probably better attempted at a bell curve. It's the response from the middle 80% which is important and leaving out the bottom 10% and the top 10% of opinions are going to get you the middle ground to shoot for, as far as frequency response goes.

 

[bigshot]

Most of us here consider the "mix and match" approach a random way to achieve random results. I know I'd like to have a target in mind to aim at. I can't afford to buy stuff in a million colors and randomly mix and match to make a blend of colors I happen to like.

 

[Redcarmoose]

Most of us here consider the "mix and match" approach a random way to achieve random results. I know I'd like to have a target in mind to aim at. I can't afford to buy stuff in a million colors and randomly mix and match to make a blend of colors I happen to like.

The old-school standard approach has always been start with the source as the 1st thing. As if you try and keep correcting the source down the line and your source is messed up you become confused. But starting with a correct source many audiophiles slowly try and match warm products with colder products over time to try and reach a sonic balance or sorts.

If it's slowly done and over the course of years it's not that expensive. You'll always notice many of the audiophiles you read about will hold onto old gear that they like. As in comparison to high end speaker audio, headphone audio is relatively cheap, allowing owners to collect more gear.

The mystery is the target can at times be unknown. It's only after trial and error do we find the breakthroughs from random happenstance. That's the fun of it all.

 

[bigshot]

Start with the source made sense in the era of turntables and reel to reels, not now. Even cheap sources are capable of sounding good. Today it's best to start with the transducers. They're what actually make the sound. If you get good headphones, you can find an amp capable of properly powering them. Once you get that, the rest is easy.

I identify a problem and then work on a solution. I find it's a waste of time to work on solutions to problems that probably don't exist. I know I'm an exception in this hobby when it comes to that though.

 

[Redcarmoose]

Start with the source made sense in the era of turntables and reel to reels, not now. Even cheap sources are capable of sounding good. Today it's best to start with the transducers. They're what actually make the sound. If you get good headphones, you can find an amp capable of properly powering them. Once you get that, the rest is easy.

I identify a problem and then work on a solution. I find it's a waste of time to work on solutions to problems that probably don't exist. I know I'm an exception in this hobby when it comes to that though.

I don't think your the exception thinking that low cost digital sources are so good that their not even a concern anymore. That's a standard viewpoint and one widely held in SS. I just happen to be fascinated by the slight irrelevant differences too small to seriously matter to most. It's mostly like a hobby parallel to playing solitaire.