[KeithEmo]

In principle, your "test and validation methodology" is very sound... and quite sensible...

However, it only has two flaws, which are quite closely related, and which are to a degree unavoidable.
Conditions change, and your ability to test a particular device under a wide variety of conditions is usually limited.

What if you hear a difference today that you didn't hear yesterday BECAUSE YOU'RE USING EQUIPMENT TODAY THAT YOU DIDN'T HAVE YESTERDAY?
What if you hear differences today ON A RECORDING THAT YOU DIDN'T HAVE YESTERDAY?

And, at the risk of being contentious, I'm only going to accept the claim that "you can't possibly hear something different with a different set of headphones or pair of speakers" if you're prepared to claim that you currently own an "audibly PERFECT" set of either, and so have ruled out the possibility of ever owning a set that will be "better" and so reveal things that your current set have failed to... or if you have a "perfect quality and perfectly comprehensive test file", and so have ruled out the possibility of ever having one that is better, and so requires something additional in the playback signal chain to reproduce it "audibly perfectly". (I've certainly heard differences that seemed obvious on one set of speakers, but totally inaudible on another, or obvious on headphones, but totally inaudible with speakers, or obvious on one piece of music, and not noticeable with another.)

If they can hear it clearly in a controlled test and care about it, that's fine. But that doesn't include inaudible stuff that's a result of sloppy comparison tests or expectation bias. No one can really hear stuff that only exists on paper or in their head, and no one should waste their time caring about meaningless or imaginary stuff.

I don't believe in "improvements in hearing acuity". Your ears hear what they hear and that's it. You can be paying more attention now than you used to, or perhaps you have expectation bias to make you think that straining to hear better actually will make you hear better. But if you do actually hear something you didn't hear before, that just means that your earlier impressions were due to sloppy comparison tests and something got by you. When I get a new piece of equipment I make the effort to do a careful comparison test right away. If it passes the test, I don't worry about it any more. So far, I haven't run across any amp, DAC or player that doesn't pass the test.

 

[castleofargh]

Looking back at my post #7666 above, I believe I wasn't clear enough about what I was saying, and I can see why the post caused some irritation. I should have noted at the outset that I wasn't 100% confident that the difference I perceived is real, that I couldn't be 100% confident based on the evidence available, and that I was about 90% subjectively confident based on the evidence available to me. A confidence level of 90% reflects high confidence, but also leaves room for doubt.

I've ordered equipment for blind testing and will make a go of it. If the blind testing provides strong evidence that I misperceived, that will show that I was overconfident, despite trying to recognize and compensate for biases. OTOH, if the blind testing provides strong evidence that the difference is real, that will be an interesting outcome too.

In general, I think we need to be clear about what people are 'claiming'. Claiming to subjectively perceive differences is common and such claims aren't strong claims. Claims that perceived differences are objectively real are strong claims which need strong evidence. IMO, anecdotal claims of perceived differences can add up to meaningful evidence if they have large scale and consistency, but I certainly agree with others that testing which eliminates potential effects of biases is needed to produce strong evidence. That said, I still have concerns with the quality and external validity of the blind tests which are typically performed, so IMO we still have a bit of dilemma in settling these questions.

to me 90% is already way too optimistic, but then again I haven't tried the hugo2 and maybe the differences are indeed very clear and didn't need controlled testing. you probably know better than I do how significant the difference really is, and it's very likely that an AB switch and matched levels isn't going to change much of anything(well it's always good to have and does have the effect of bringing rapid switching to you, which is often good to have).
in my case, I got troubled, not by your results, but by how you thought the way you reached that conclusion was reasonable. sighted experience never actually challenges you to discover or notice anything. it's not a test, all along you have known what was used at any time and your brain was free to apply any preconception and agree with itself in the "findings". when differences are really massive, it's all right, even mixed with placebo and distorted by expectations, big changes still tend to come out in a sighted test. but if the differences are small, then knowing what you use at all time is too much of a bias. you're always worried about hypothetical biases, well this one is very real and as old as humanity.
if the 2 products are aiming at high fidelity like those 2 are supposed to, then the sound coming out should be rather close and the differences rather subtle(unless your headphone has challenging impedance/sensi specs). else it would just mean that at least one device is not as high fidelity as it claims to be. we agree on that much, right? one original signal, only one way to perfectly reproduce it and anything with rather high fidelity should come close to the same target. that's why my assumption was on small variations and why conclusions based on sighted experience seemed unacceptable.

but in general as I've said a bunch of times, I will simply refuse to accept a sighted experience as conclusive of anything audio related. a listening test should be something done using our ears, not our eyes and all the stuff we already know about the gears. bias bias, bias bias bias, bias! ^_^

 

[Phronesis]

if the 2 products are aiming at high fidelity like those 2 are supposed to, then the sound coming out should be rather close and the differences rather subtle(unless your headphone has challenging impedance/sensi specs). else it would just mean that at least one device is not as high fidelity as it claims to be. we agree on that much, right? one original signal, only one way to perfectly reproduce it and anything with rather high fidelity should come close to the same target. that's why my assumption was on small variations and why conclusions based on sighted experience seemed unacceptable.

The magnitude of difference I've perceived has depended on the headphones, the track, the volume, and most of all how much attention I was paying. These factors have sometimes lined up to make the perceived difference seem substantial, but other times I would characterize it as relatively small and of significance only if being picky and listening criticality to sound quality more than music.

I added the Hugo 2 to the Mojo because I wanted another DAC/amp and I didn't mind paying more because I perceived some difference and it was largely a 'just in case there's really a difference' decision (I was never 100% confident there's really a difference). But when people who have a Mojo have asked for opinions on whether upgrading to the Hugo 2 is worth it, if they say they'd be stretching financially to do so, I've usually opined that the difference is incremental and they'd likely be better off spending the money on upgrading headphones.

 

[RRod]

Re binaural:
Got this disc recently. Has three versions of the music, which is written to come out of 10 channels:
1) An HRTF version where each channel has had a binaural filter applied
2) A dummy-head version, where they played the channels here and recorded
3) A 5.0 mix

Shows kind of the weird mishmash you can get when trying to get this kind of stuff. So here we get no stereo speaker mix but 2 headphone versions. I ordered a trio of other albums recently with a stereo and a binaural mix but no multichannel mix.

 

[KeithEmo]

Here's an interesting thought to consider....

It's already been mentioned that, when we are at a real live event, part of what we experience is determined by the way in which our BRAIN focuses its attention on various details. For example, we are able to focus on the lead performers and exclude or reduce the impact of audience noise and room acoustics. This occurs by the "processing" that goes on in our brains, which is almost certainly based on visual cues, like seeing the person playing the instrument we're focusing our attention on. There also seems to be a significant element of "time" and "knowledge" to the process. The fact that we "know" who we're listening to, even if we're not looking at them, seems to enable our brains to focus on the sound, and exclude the extraneous noises, more easily. It's sort of the same way we're able to focus on listening to what one person is saying in a group (the famous "cocktail party effect").

All of this suggests an interesting thing.... perhaps the "missing element" of why a recording of a concert almost always fails to seem "convincingly realistic" is that the other elements, especially the visual ones, are missing. Perhaps, when the human brain is involved, the "fidelity" of the recording actually DEPENDS on being able to see the performance and, without the visual aspect of the performance, it's just not ever going to "sound like you're really there".

Also:



I am still trying to process your arguments.

I will wait until I am able to test binaural recordings made in my own ears, but played back with head tracking and xtc.

Preliminarily, I have to agree that several times that I play back binaural that I recorded I think “that noise was not there while I was recording”. But it was and my brain was effectively paying attention to what I was considering important at that moment.

At the time of my response to your first post, I was thinking how the perception of blind people work. Now I am not so sure.

With those premises, I assent that it makes sense to reinforce acoustically at sole audio playback something that would be naturally reinforced by all our senses and thus by our complete perception in a real event and that is not fully functioning at restricted music playback.

I don’t have a 3D 360 degrees camcorder to test binaural recording with VR and won’t have in the short or mid term... life sucks...

Edit: very hard to put those ideas into words...

 

[Phronesis]

Here's an interesting thought to consider....

It's already been mentioned that, when we are at a real live event, part of what we experience is determined by the way in which our BRAIN focuses its attention on various details. For example, we are able to focus on the lead performers and exclude or reduce the impact of audience noise and room acoustics. This occurs by the "processing" that goes on in our brains, which is almost certainly based on visual cues, like seeing the person playing the instrument we're focusing our attention on. There also seems to be a significant element of "time" and "knowledge" to the process. The fact that we "know" who we're listening to, even if we're not looking at them, seems to enable our brains to focus on the sound, and exclude the extraneous noises, more easily. It's sort of the same way we're able to focus on listening to what one person is saying in a group (the famous "cocktail party effect").

All of this suggests an interesting thing.... perhaps the "missing element" of why a recording of a concert almost always fails to seem "convincingly realistic" is that the other elements, especially the visual ones, are missing. Perhaps, when the human brain is involved, the "fidelity" of the recording actually DEPENDS on being able to see the performance and, without the visual aspect of the performance, it's just not ever going to "sound like you're really there".

I think that hypothesis is likely to be true. A live concert experience is processed based on all of our senses, even things like the discomfort of standing too long, smells, enthusiasm (or lack thereof) of people around us, and light shows. I don't see how sound alone can simulate that kind of experience, unless our brains do a lot of imaginative work to actively try to fill in the sensory blanks.

With my home sound system, I've noticed that my perception of the sound/music is affected by whether I look at the speakers, close my eyes, etc. I've also had the experience of hearing a track, not initially remembering who the artist is, forming an impression of the music, then noticing that my perception changed when I recalled who the artist is and heard the music in that different context.

When we get into the cognitive psychology/neuroscience of perception, things get really interesting - and complex! And I think we need to try to understand and address these aspects if we want to optimize our experience of music, rather than just optimizing 'sound quality'.

 

[bigshot]

However, specifically in regards to headphone amps, the manufacturers often FAIL to publish those specifications... or standards simply don't exist for stating them. For example, very few headphone amps say things like: The output impedance of this amplifier is 0.1 Ohms or lower from 20 Hz to 20 kHz.. Therefore, you should expect to see a frequency response flat to +/-0.1 dB if you use it with headphones whose impedance is 32 Ohms or higher. But, if you use it with headphones whose impedance dips below 32 Ohms at any frequency, then the frequency response variation may be wider.

That isn't a fault of the amp. The amp is audibly transparent by a fair margin. It's your fault for using the wrong headphones with it. I can try to screw in a phillips head screw with a slot screw driver and it won't work well. But that doesn't mean it doesn't work for its intended purpose- screwing in phillips head screws. (I repeat this over and over and no one listens.)

Very good points, which illustrate the difference between doing science (not a main interest of audiophiles) vs making practical purchasing decisions which account for costs, benefits, uncertainties, risks, perception, etc.

You totally missed the second part of what he said, where he admitted that he made buying decisions based on changes in his system that made him "comfortable". That is neither scientific nor practical. It doesn't even succeed at making him comfortable, because OCD isn't cured by giving in to it. If you do, it just comes up with another hoop to jump through to be comfortable again.

Conditions change, and your ability to test a particular device under a wide variety of conditions is usually limited.

I have absolutely no doubt that my system functions perfectly fine for the purposes I bought it for. My system is audibly perfect for my purposes. I have nothing on my "wish list" right now. No need to upgrade anything. But if there is an option for a totally new format, I make that decision when something burns out. For instance, when my stereo amp bit the dust, I considered the possibility of going 5.1. That option didn't exist when I bought my stereo amp. It did by the time the stereo amp died. That's the best time to change horses. I may go Atmos at some point. Not ruling that out. Just not right now.

When you're at the point where improvements are largely not noticeable, that's the time to start investing in music instead of equipment.

 

[castleofargh]

Here's an interesting thought to consider....

It's already been mentioned that, when we are at a real live event, part of what we experience is determined by the way in which our BRAIN focuses its attention on various details. For example, we are able to focus on the lead performers and exclude or reduce the impact of audience noise and room acoustics. This occurs by the "processing" that goes on in our brains, which is almost certainly based on visual cues, like seeing the person playing the instrument we're focusing our attention on. There also seems to be a significant element of "time" and "knowledge" to the process. The fact that we "know" who we're listening to, even if we're not looking at them, seems to enable our brains to focus on the sound, and exclude the extraneous noises, more easily. It's sort of the same way we're able to focus on listening to what one person is saying in a group (the famous "cocktail party effect").

All of this suggests an interesting thing.... perhaps the "missing element" of why a recording of a concert almost always fails to seem "convincingly realistic" is that the other elements, especially the visual ones, are missing. Perhaps, when the human brain is involved, the "fidelity" of the recording actually DEPENDS on being able to see the performance and, without the visual aspect of the performance, it's just not ever going to "sound like you're really there".

there is no doubt about that. vision is clearly the dominating sense and one we will rely on instead of our ears when the information from both senses are conflicting. even worst, we will in the long run recalibrate our hearing to fit what we see. not the other way around.
and to make sure that it's an absolute mess, it seems like not everybody has the same overwhelming sight domination over other senses. it's the leading sens for all, but so when some people start to be fooled by correct audio cues, a few others will still not feel like the sound is right so long as their visual cues don't also confirm the original of the sound.
personally, just watching a video clip on TV with the headphones clearly improves my impressions of distance after maybe 15mn. even more so when I've had months placed at the same distance with the same configuration. I learned to feel right under those circumstances over time. in my bedroom, I have a speaker on each side of my computer screens, and they clearly help feeling a better placement when I use crossfeed. but of course I did set up the crossfeed so that full panned sounds would be in the direction of the speaker in the first place, so I manufactured a relation to help being fooled.
I tried removing the speakers while listening to this crossfeed with the headphone and I honestly feel like it's not as good.

my most extreme example of learning and recalibrating, was a few years ago with my laptop. I was using a secondary keyboard and another screen, while the laptop was on the side. and for a few months, I still used the laptop's integrated tweeters for sound when not using headphones. it was limited to watching utube videos and maybe a few animes from time to time. in the beginning having the sound on my right was really weird and annoying, so I relied on headphones a lot. but after a few months, when I was watching someone talking on the screen, if I didn't think about it, I was hearing the voice coming from the screen. my brain had recalibrated for that use. as soon as I put on a headphone or went to watch TV(or experience any real life sounds), center was real center again. this didn't offset my senses in general. just for that specific situation clearly identified by my brain.

so yes to everything. training, plasticity, mixing of senses, sight>hearing for us. maybe like hypnosis, having the will to be fooled/influenced may help, but for many things, knowing we may be fooled doesn't stop us from being fooled. in some cases it's the opposite. and a few people will just never get into it because of who they are, how they think...
you can see this with people who really grew up on headphones. a few of them actually prefer their albums on headphones/IEMs instead of listening to those albums on speakers. we're pattern recognition machines, what experience was fed to us for a long time will have the ability to impact how we feel and what we prefer. that's intuitive enough IMO.

 

[bigshot]

The magnitude of difference I've perceived has depended on the headphones, the track, the volume, and most of all how much attention I was paying.

With my home sound system, I've noticed that my perception of the sound/music is affected by whether I look at the speakers, close my eyes, etc. I've also had the experience of hearing a track, not initially remembering who the artist is, forming an impression of the music, then noticing that my perception changed when I recalled who the artist is and heard the music in that different context.

You just threw in a whole bunch of variables there that have absolutely nothing to do with the sound the DACs are producing. I'm becoming >90% certain that your two DACs sound identical and the way your are comparing them is skewing the results. This is EXACTLY why we apply some controls to our test to eliminate subjective bias before we come to a conclusion.

there is no doubt about that. vision is clearly the dominating sense

I'm playing around with the perspective sound with head tracking in my Oculus Go, and I'm coming to the conclusion that the ability to perceive directionality in sound is much more complicated than the general theories would indicate. I don't think it's one or two or several things that affect how we perceive sound direction. I think it's a very complex blend of lots and lots of things. I suspect that it may be one of those things that is just so individual, there is no one-size-fits-all solution. I used to be interested in the Smyth Realizer. I'm beginning to be skeptical if it will even work.

 

[Phronesis]

You totally missed the second part of what he said, where he admitted that he made buying decisions based on changes in his system that made him "comfortable". That is neither scientific nor practical. It doesn't even succeed at making him comfortable, because OCD isn't cured by giving in to it. If you do, it just comes up with another hoop to jump through to be comfortable again.

I interpret "I DO have a bit of OCD about how my system sounds...." as more a figure of speech about being picky rather than a diagnosis of mental illness. If something generally makes someone more comfortable, I'd say it generally has practical value. It doesn't help to read too much into this and split hairs.

 

[sonitus mirus]

Personally, I'm not so sure that I specifically agree that "the end game isn't to find the ultimate tweak".....

I meant that there were clearly many other things that would greatly impact the sound quality over the DAC change in this scenario. The DAC was probably the best piece of gear in the system, and probably would be the last device to swap out in any pursuit to improve the sound quality.

 

[Phronesis]

You just threw in a whole bunch of variables there that have absolutely nothing to do with the sound the DACs are producing.

DACs alone don't produce sound, they produce a signal which goes down a chain and becomes the sound reaching our eardrums. The interaction of those other variables with the DACs is certainly relevant to the magnitude of perceived audible sound differences, just as a given headphone may sound too bright with some tracks but not others.

 

[bigshot]

I interpret "I DO have a bit of OCD about how my system sounds...." as more a figure of speech about being picky rather than a diagnosis of mental illness. If something generally makes someone more comfortable, I'd say it generally has practical value. It doesn't help to read too much into this and split hairs.

I'm not diagnosing illness. I'm pointing to a decision making process that has no relation to practical reality. Everyone is irrational to one degree or another. But some of us resist those irrational impulses by applying logic to our decision making processes.

One person chooses his system according to science and practicality. Another person with the exact same criteria for sound quality chooses his system according to science, practicality and things that make him feel confident. A third person comes in and listens to both systems and decides that they sound basically the same. The only difference is, the "feel good" guy spent thousands of dollars on silver cables and jitterless electronics that didn't improve the audible sound quality one iota.

I think we should all be open to opportunity for improvement, but we shouldn't let ourselves waste energy doing things "just to be on the safe side". That sort of thing is never ending. Once you do it once, you're tempted to do it again to be a little *more* safe... and a little *more* safe... and on and on. That is what causes some people here in Head-Fi to churn through equipment for no purpose. If a tool does the job, there's no reason to chuck it and get a more expensive tool that does the job exactly the same. That makes no sense.

It's most obvious with compressed audio. Ask someone what file format they use. Odds are they'll say FLAC. Ask them if they can tell the difference between high bitrate lossy and lossless. If they have actually done a careful comparison test, they'll say no, they sound the same. Then ask them why they use FLAC instead of high bitrate lossy... They'll tell you that they use lossless because they are worried about losing potential sound that they can't hear. They sleep better at night knowing they have every bit and binary digit of the music. That has no relation at all to sound quality nor practicality. It's a pure subjective preference based on doubt created by OCD.

If I told you that I was buying a car that gets 15 miles per gallon even though there is a model of car that is exactly the same (performance, price, appearance) that gets 35 miles per gallon, you would wonder why I don't just buy the more efficient car, and you'd be right. But when it comes to music, people's emotions overtake them and they don't know how to separate rational and irrational biases.

I apologize to Pinnahertz for the car analogy.

DACs alone don't produce sound, they produce a signal which goes down a chain and becomes the sound reaching our eardrums. The interaction of those other variables with the DACs is certainly relevant to the magnitude of perceived audible sound differences, just as a given headphone may sound too bright with some tracks but not others.

You are wrong there. If you are going to include your own personal subjective bias as part of the chain between source and ears, then you're in the wrong forum. Your bias has not impact on my impression of the sound produced by the equipment. If you're comparing DACs using different volume levels, different headphones and different situations like that, you haven't established a baseline for comparison. You've rolled out a red carpet for bias to make its entrance. I couldn't tell one way or the other with that many variables at play. I don't expect you to have any way of knowing either. However you are >90% confident you can. That is a red flag right there.

If some tracks sound bright with a set of headphones and others sound dull, the problem isn't the headphones... it's the engineering of the music. Music should be calibrated for a flat response. I calibrate my headphones with EQ to produce a balanced flat response. If music sounds imbalanced, it's because the music is imbalanced. Not because of my cans.

 

[RRod]

I'm playing around with the perspective sound with head tracking in my Oculus Go, and I'm coming to the conclusion that the ability to perceive directionality in sound is much more complicated than the general theories would indicate. I don't think it's one or two or several things that affect how we perceive sound direction. I think it's a very complex blend of lots and lots of things. I suspect that it may be one of those things that is just so individual, there is no one-size-fits-all solution. I used to be interested in the Smyth Realizer. I'm beginning to be skeptical if it will even work.

Well the Realiser requires you do the exact opposite of a 'one-size-fits-all' solution to get the best effect...

 

[Phronesis]

I'm not diagnosing illness. I'm pointing to a decision making process that has no relation to practical reality. Everyone is irrational to one degree or another. But some of us resist those irrational impulses by applying logic to our decision making processes.

One person chooses his system according to science and practicality. Another person with the exact same criteria for sound quality chooses his system according to science, practicality and things that make him feel confident. A third person comes in and listens to both systems and decides that they sound basically the same. The only difference is, the "feel good" guy spent thousands of dollars on silver cables and jitterless electronics that didn't improve the audible sound quality one iota.

I think we should all be open to opportunity for improvement, but we shouldn't let ourselves waste energy doing things "just to be on the safe side". That sort of thing is never ending. Once you do it once, you're tempted to do it again to be a little *more* safe... and a little *more* safe... and on and on. That is what causes some people here in Head-Fi to churn through equipment for no purpose. If a tool does the job, there's no reason to chuck it and get a more expensive tool that does the job exactly the same. That makes no sense.

I agree in general, but am reluctant to prescribe a universal definition of what's 'practical' with audio stuff. If someone is X% confident that something sounds better, spending $Y more for it isn't a big deal for them, and it makes them more comfortable to 'play it safe' and spend the money, I can't fault that.