[Brooko]

Hi Dan
 
I know Amos already answered you - but I wanted to reply personally.  I hope you don't mind.
 
I originally started volume matching with everything I was reviewing a little over a year ago. I did this after watching Ethan Winer's Audio Myths video (and one or two others), and realising that my reviewing style (while becoming popular) wasn't as accurate as it could be.  In short - my contributions weren't doing justice to anyone - the manufacturers, the readers, or myself.
 
So I initially started - simply using an iPhone app.  it was better than using nothing at all - but as you found out recently, it was still not accurate enough.  So I then bought an SPL meter (according to the documentation it has been properly calibrated), and it will measure to within 0.1 dB both A & C weighted.  I now use it on all my reviews when making comparisons. I probably should make more of a point of mentioning the methodology I use when making comparisons. Thanks for pointing it out.
 
Quote:

  Take a look at the reviews for the X5 and X3II or any of the headphones on head fi. How many are volume matched? That's right. None. 

 
Actually - you'd be surprised.  There are a few of us now.  Some are still using apps (smartphones) - others like myself, earfonia, and baycode all use SPL meters.  The more people who notice it (and who want their own reviews to become more meaningful), the more eventually will adopt it.  It is time consuming, takes a lot of checking and rechecking - but I do think it is worth it.
 

  I should add that even a volume matched comparison is not valid.

Brooko openly admits that he EXPECTED the X5Ii to sound better even before testing.
 
So, even with volume matching, the testing must be double blind with neither the test admistrator nor listener knowing which source was which during the testing itself. 
 
So, Brooko could very well have been influenced by confirmation bias, which he readily acknowledges is a distinct possibility.
 
The bottom line is that there is no such thing as a valid comparison here on head fi. Only volume matched double blind testing results are valid, and while I can't say they don't exist on head fi, I can assume they are very rare. 

 
Yep - I told you about that. And I also told you to treat it as a subjective review - all of mine are. I try to remove as much bias as I can - and I try to inform the reader in advance of any known bias I have. It is all very simply to make sure that I can give a reader of my reviews a base line in which to interpret what I've written.
 
I know you have very strong views on the X3ii.  That's OK - but you also need to take into account that:
[a] You did not volume match as accurate as possible
Your tests were also sighted
[c] You also have expectation bias - we all do - we can't escape it.
 
Now I'm not saying my views are any more relevant than yours.  It is my subjective opinion.  You talk about a wide difference between X5 and X3ii and you seem (to me anyway) to be saying that if anyone disagrees with you, they must be wrong. Just as a thought - how about considering that we may have different preferences?  I agree that there is a difference between the X3ii and X5. My tests showed that comparatively it was pretty close (to me).  X5 was flatter, smoother, blacker background. Jazz actually prefers the X3ii to the X5. Others say different again.  This is where preference comes into play.  There is no right or wrong. You seem to be on a crusade saying you an't trust any reviewers because our opinions don't exactly match yours.  I've got news for you my friend - most reviews won't - because we have different subjective preferences 

 
Anyway - hopefully you find a reviewer you can follow.  I always follow the same two or three when I need advice on audio equipment - they have similar tastes to me and I generally feel their reviews match my preferences - that is the secret I think.
 
As to my methodology.

1. I always use an SPL meter now - both for comparisons, and also making my own frequency response charts (crude but seem to be effective)
2. I match volume using set tones - usually @ 1 kHz.  Some have recommended pink noise - but I usually choose similar headphones when comparing - so I find my current method pretty effective.
3. I have a two way switch when comparing DAPs.  I volume match before they go on the switch and again during and after.  All EQ is disabled. I use exactly the same files on both DAPs and always the same headphones.  Each headphone used is volume matched. I then set-up the playing so that both DAPs are playing the same playlist virtually simultaneously.  I can then flick backwards and forwards.
4. Where possible I get my wife or daughter to assist with the switching, and I always close my eyes while they are doing it.  If They aren't around (or I can't coax them into it), I just use the switch myself, eyes closed, and very often I completely lose track of which DAP is which until I open my eyes.  The pun here is that often this has been an eye opener.  You should actually try this at some stage.  You might find those current "night and day"  differences fade to more of a twilight
   
   .

 
I hope this helps a little.  My method is not perfect.  My reviews will ultimately always have subjective bias.  But I'm actually pretty content on what I put out there nowadays.  Judging from a lot of the reactions from others - they're happy enough with the content.  But I'm always open to suggestions within reason.
 
Last thing to note - I review because I enjoy doing it.  I don't do it for free gear (I offer to return all the gear I'm reviewing - ask any manufacturer I deal with), and other times I buy the gear.  I actually don't even do it for the public here. I ultimately do it for me - because it is a hobby I love, and the review process helps me learn about my own preferences.
 
Sorry for the long post.

 

[Brooko]

  Okay, guys, regardless of all this discussion, I intend on eventually buying some of the most expensive DACs out there and doing all the possible tests on them, from casual listening to measurements and DBT/ABX. As far as I know, no one else would dream of doing something like this, so I'll do a solid for the community. Just gotta get rich first.

 
M-A - I really hope you get the chance one day.
 
I know RRod left you the link to Ethan's video - which you said you wouldn't have time for.  I'll post it again, and I really hope you take the time to watch it all.  It opened my eyes to a lot of things - and for someone who talks about doing a proper comparison of DACs at some stage in the future - both it, and the second one I've linked should be compulsory viewing.  If you are serious about this - please take the time. I can promise you it will be beneficial.
 

 
 

 

[CanadianMaestro]

   
   
I strongly disagree with your comment about the overall neutral statement. Neutral is not a single FR response line. There is no single universal gold standard target FR curve that shows "neutral." Even ideal headphone target responses or otherwise derived FR curves from speaker set-ups with various compensation models are derived from aggregate data. We have a lot of different models for neutral, but even the measurements may not be reflective of what each individual hears due to physical characteristics like ear shape or personal expectations of what neutral should sound like.
 
Neutral is refers to the overall balance of the frequency response to a listener, but can manifest as quite a few different FR curves. There are many FR curves that fall within the realm of being quite well-balanced & for all intents and purposes gives very neutral-reproduction of the source material. So when someone calls something "neutral overall," they simply mean that there are not any prominent colorations in the FR curve. There can still be other subtle variations in the sound signature, but still a quite balanced overall presentation of each region of the frequency response. 
 
There is no pair of headphones that measure "perfectly neutral" but there are many well-balanced headphones that fall into that realm of being "neutral." They do have subtle differences in variations of their frequency response, but I do think it is perfectly reasonable to talk about certain gear as being overall neutral and then delve more specifically on some subtle relative differences in FR. You can find many examples of overall well-balanced headphones with very subtle sound signature differences but can easily be accurately described as neutral in almost every price bracket.
 
 
 

 
Lucid and valid points, money.
 
I like to think of relative neutrality, not absolute neutrality. For me, transparency is more valued, in gear, than even relative neutrality, which is impossible to objectively verify. In the end, I don't obsess about the "N" word.

 

[castleofargh]

  Lucid and valid points, money.
 
I like to think of relative neutrality, not absolute neutrality. For me, transparency is more valued, in gear, than even relative neutrality, which is impossible to objectively verify.

once again, the only thing impossible to verify is your own perception and interpretation of sound. there are no objective mysteries as far as the sound itself is concerned. none!
you seem to think differently and even decide to have a signature that half false, but if something is heard and cannot be measured, you invented it. I know of no other explanation.
it's the human brain and the interactions between senses, focus, habits and experiences that are impossible to objectively analyze. not the actual music.
 
 
also neutral or "transparent" aren't impossible to get, it's just not very interesting in the actual audio industry as the albums are mastered and EQed for many reasons that usually involve balance and pleasure much more than neutrality or transparency. but it could be done with relatively good precision if the engineers worked toward that purpose only(not sure it would sell so much). as for our ears, all we really need is to get our HRTF response measured and find a headphone+DSP that would get close enough. also not simple or practical or free, but certainly not impossible.
 
there is actual neutrality, as in "the frequency response didn't change from the original(whatever we pick as being original). and there is the stuff we feel to be neutral, that compensates for our own defectsbody

to make us hear something closer to what was the actual sound. it's just a matter of defining in what system we're trying to achieve neutrality.
and of course the day we all get to try that with our own custom made signal, it could be the end of hifi as a hobby, or we might just as well realize that all our talk about striving for transparency and neutrality was bollocks, and that what we really like is anything reminding us of the old sound system that daddy played when we were young.

 

[CanadianMaestro]

  once again, the only thing impossible to verify is your own perception and interpretation of sound. there are no objective mysteries as far as the sound itself is concerned. none!
you seem to think differently and even decide to have a signature that half false, but if something is heard and cannot be measured, you invented it. I know of no other explanation.
it's the human brain and the interactions between senses, focus, habits and experiences that are impossible to objectively analyze. not the actual music.
 
 

Exactly. Hearing is perceptual, which is not easily measured.
hence my signature statement.

 

[Music Alchemist]

  M-A - I really hope you get the chance one day.
 
I know RRod left you the link to Ethan's video - which you said you wouldn't have time for.  I'll post it again, and I really hope you take the time to watch it all.  It opened my eyes to a lot of things - and for someone who talks about doing a proper comparison of DACs at some stage in the future - both it, and the second one I've linked should be compulsory viewing.  If you are serious about this - please take the time. I can promise you it will be beneficial.

Thanks for the links.

 
I'll watch them sometime. I didn't mean to imply that they weren't worth watching; I just didn't want to watch through them at the time of that post, since I was looking for specific answers, and videos tend to require watching through quite a bit of irrelevant stuff to get to what you are looking for. That's why I asked for articles covering that specific topic.
 

  Lucid and valid points, money.
 
I like to think of relative neutrality, not absolute neutrality. For me, transparency is more valued, in gear, than even relative neutrality, which is impossible to objectively verify. In the end, I don't obsess about the "N" word.

 
My personal gauge for audio gear is how realistic it sounds to me, but that's just the subjective side. I do want to get a grasp of the objective side as well.
 

  once again, the only thing impossible to verify is your own perception and interpretation of sound. there are no objective mysteries as far as the sound itself is concerned. none!
you seem to think differently and even decide to have a signature that half false, but if something is heard and cannot be measured, you invented it. I know of no other explanation.
it's the human brain and the interactions between senses, focus, habits and experiences that are impossible to objectively analyze. not the actual music.
 
also neutral or "transparent" aren't impossible to get, it's just not very interesting in the actual audio industry as the albums are mastered and EQed for many reasons that usually involve balance and pleasure much more than neutrality or transparency. but it could be done with relatively good precision if the engineers worked toward that purpose only(not sure it would sell so much). as for our ears, all we really need is to get our HRTF response measured and find a headphone+DSP that would get close enough. also not simple or practical or free, but certainly not impossible.
 
there is actual neutrality, as in "the frequency response didn't change from the original(whatever we pick as being original). and there is the stuff we feel to be neutral, that compensates for our own defectsbody

to make us hear something closer to what was the actual sound. it's just a matter of defining in what system we're trying to achieve neutrality.
and of course the day we all get to try that with our own custom made signal, it could be the end of hifi as a hobby, or we might just as well realize that all our talk about striving for transparency and neutrality was bollocks, and that what we really like is anything reminding us of the old sound system that daddy played when we were young.

 
Do you know how to measure personal HRTF, by the way?
 
There is much debate on which compensation curve (Harman curve, diffuse-field flat speaker curve, and many others) for headphones is neutral. But I'm assuming a personal HRTF would bypass that, since it depends on our own hearing and things like frequency sweeps, right?
 
http://www.head-fi.org/t/413900/how-to-equalize-your-headphones-a-tutorial
http://www.head-fi.org/t/587703/how-to-equalize-your-headphones-a-tutorial-part-2
http://www.head-fi.org/t/615417/how-to-equalize-your-headphones-advanced-tutorial-in-progress
 
I've been meaning to follow these guides to get all my headphones as close to neutral as possible, but I've been lazy. Seems like it would take hours of effort.

 

[castleofargh]

  Do you know how to measure personal HRTF, by the way?
 
There is much debate on which compensation curve (Harman curve, diffuse-field flat speaker curve, and many others) for headphones is neutral. But I'm assuming a personal HRTF would bypass that, since it depends on our own hearing and things like frequency sweeps, right?
 
http://www.head-fi.org/t/413900/how-to-equalize-your-headphones-a-tutorial
http://www.head-fi.org/t/587703/how-to-equalize-your-headphones-a-tutorial-part-2
http://www.head-fi.org/t/615417/how-to-equalize-your-headphones-advanced-tutorial-in-progress
 
I've been meaning to follow these guides to get all my headphones as close to neutral as possible, but I've been lazy. Seems like it would take hours of effort.

with some nifty little microphones inside your ears you can get a response that accounts for both the source of the sound and the shape of your ears/body(depending is it's from speakers or headphones). that way you can get something not perfect but relatively close to neutral at your eardrum. then will your eardrum and everything inside your head find neutral to sound neutral? I have no idea. but given how we work and adapt in real life, I suspect that result to be the closest to actual neutral for people.
that's how the smyth realiser measures stuff in whatever room or whatever headphone so that you can recreate another room. the purpose isn't the same but the way to do it is. and that way you get all changes, not just FR.
 
the tutorials you link are another matter as you end up trying to get flat(or like another source) by listening. so it's the same as calibrating your monitor by looking at it instead of using an automated calibration solution. you can prefer to do it yourself trusting your eyes and be it that it's right or wrong, you may like the result. but no professional would ever do that because they need to know what neutral really is to work with others and have a common reference. so I guess it really depends on what you're looking for.
  the main problem is that you first need an actual reference for flat, and we can't really expect to have that at home, sadly. I did use those methods(mostly Joe bloggs stuff) and still do when I get a new IEM, I actually made an unclear and long post thinking it would make things simple for people(me making things simple... what was I thinking?

) http://www.head-fi.org/t/615417/how-to-equalize-your-headphones-advanced-tutorial-in-progress/105#post_10740023
but the purpose is more to take something you like(let's say your speakers sound), and try to reproduce a similar signature(FR only in this case) onto some headphone.
so it's clearly not as complete or precise as trying to measure with microphones inside our ears and turn yourself into a dummy head. also it takes a lot of time, but it's cheap ^_^.

 

[prot]

Way to go Brooko .. your latest reviews are a pleasure to read and quite a big step forward.

I would say that any "impressions" post that doesnt at least follow *all* your rules doesnt deserve to be named Review... it's just some sort of personal anecdote about sound.

There are still way too few people who do it with so much care & attention but hopefully more will follow your way.

Thank you and good luck!

 

[miceblue]

Spoiler: spoiler

So you basically are saying that there is nothing that can be defined as neutral.....

There is coloration inherent to the recording process as soon as the sound is captured by a microphone. So basically yes, you can just say that nothing that is recorded is "neutral." The playback gear used in evaluating and tuning in a studio can be different than the gear the listener is using, so yes it is entirely possible we are not hearing the exact same thing as heard in the studio setting.

Worrying about what something is meant to sound ("as the artist/audio engineers intended") or sounds in the actual recording studio (the live performance) is a bit silly. Considering the post processing and mastering effects done, the final recorded song often sounds different from the live performance (often on purpose). Different master versions of the same recording can also sound vastly different. There are whole genres out there made up of synthesized sounds that have no live counterpart. The majority of the human population is not present during the actual recording event of a song, so you are right hardly anyone is qualified to make judgements on that particular aspect, but I do think everyone can judge whether an instrument they had experience playing or listening to live sounds realistic or not. People can judge whether one particular region of the frequency response is more/less emphasized than the other region without any comparative gear or specialized equipment or special training.

I think the perception that some audiophiles have of the idea of neutral as some sort of mythical standard of completely lacking in any uncoloration is just unrealistic. When thinking about neutral as transparency, it makes sense to think about it in terms of relative degrees. After a certain point, you reach a level of sonic reproduction where coloration is beyond the limits of human audio perception.

I hear your point about lacking a reference point for amp/dacs, but there are many possible routes for obtaining a comparison reference point. it's the exact same situation as headphone judgements. you either compare against alternative gear or judge based on previou how realistic the instruments are portrayed to your ears. We can even run sine sweeps or test tones. Also on the measurement side of things, we can look at noise, frequency response, distortion, and time-based errors. With all that information, I think we do have the capacity to judge whether headphones and external components are indeed 'neutral' as long as we have realistic expectations what that word means.

Just as different people have different sensitivities to dB changes in the frequency response, different people have different sensitivities in detecting reproduction errors such as jitter, wow, flutter, IMD, and clipping that affect external components. There can be measured coloration that are below our ability to detect or quite difficult to readily identify unless you have a lot of experience testing for those things as an professional audio engineer. I would say most likely difficult to detect even as a professional at certain degrees.

Perhaps the term audibly transparent would be more to your liking over the word neutral, but they essentially are synonyms that express the same idea. Perceived sound is well-balanced and realistic when using well-balanced, realistic source files.

Otherwise, instead of using commonly accepted & understandable terms like well-balanced/neutral when comparing the HD800 against the Sony MDR-XB400, we  would have to throw up our hands & just say since we actually have no reference point for uncolored sound so... idunno, whatever, can't make any judgements on which one sounds more realistic.

I already define transparent as just listening to music, not the gear. I have yet to hear a rig that sounds like that outside of the e-stats though (for headphones; I dunno about speakers yet). Not even an HE1000 on a R-2R DAC+Ragnarok can do it for me.

If neutral means not coloured, then why do people say "X sounds neutral BUT blah blah blah explanation of how it's not neutral"? I've read too many times where people say X amp is neutral. Then people often say other amps are neutral. If so many things are neutral, then why do peoples' opinions sometimes wildly differ?

The Objective 2 was built to be "a wire with gain," which in theory would be transparent if I understand you correctly. I don't think many people will agree it's "transparent," especially those who have heard other high-end gear. So is the O2 no longer "neutral" and the other gear is?

 

[Music Alchemist]

I already define transparent as just listening to music, not the gear. I have yet to hear a rig that sounds like that outside of the e-stats though (for headphones; I dunno about speakers yet). Not even an HE1000 on a R-2R DAC+Ragnarok can do it for me.

 
Agreed on all points quoted, though I haven't heard that DAC or the Ragnarok.

 

[maverickronin]

The Objective 2 was built to be "a wire with gain," which in theory would be transparent if I understand you correctly. I don't think many people will agree it's "transparent," especially those who have heard other high-end gear. So is the O2 no longer "neutral" and the other gear is?

 
I've heard plenty of other "high end" gear and I'd agree that the O2 is transparent.  I feel absolutely no need for a "better" amp for my dynamics.  Some "high end" gear, especially tube amps do sound different, because of well understood things like extra output impedance and higher distortion.  No one's ever demonstrated that amps or DACs which measure suitably well actually sound different from each other.  It's not impossible that they could sound different but such uncontrolled listening reports are far better explained by psychology than electrical engineering.  You can never prove a negative, but with the evidence we already have you might as well ask me to take just this one bigfoot or UFO sighting seriously.
 
Neutrality in amps and DACs is already well defined by measurements.  The output should be true to the input.  That's all there is to it.  Maybe we're missing an important measurement that accounts for the discrepant reports.  Maybe the the Bush family are really shapeshifting reptilian aliens.  I see no reason to believe either.
 
Speakers are pretty much the same as well.
 
Headphones are a whole different can of worms though.  There can be no such thing as neutrality in headphones.  The primary reason is that they can't provide tactile bass to you entire body the way RL or speakers can.  Different people will have different preferences between what sounds correct and what feels correct.  unless you add a crossover and subwoofer to your headphone rig it's impossible to fix.  It's always a tradeoff.
 
The second reason it's impossible is that 99.999+% of all music isrecorded/mixed/masted for playback over speakers and not headphones.  Unless you correct it with a customized DSP the imaging of a stereo recording played over headphones will never be correct.  No headphone, amp, or DAC can correct that by itself.

 

[danmlr]

Fantastic posts, brooko. I don't have anything to add, except to say that I appreciate your comments and reviews.
 
I agree wholeheartedly that a consensus with certain components is unlikely. Some variables that come into play:
 
1. the reviewer's hearing acuity. How does their hearing measure objectively?
 
2. the reviewer's subjective preferences: dark, bright, neutral?
 
There are a bunch of other small factors: shape of the ears, narrowness of ear canals (yes, they are not exactly the same!), how the pads rest on the ears, angle of the pads on the ears, subjective sensitivity/awareness, etc. 
 
And as I said before, prestige associated with price and brand, confirmation bias, etc. are all factors.
 
I think it's long been understood that hi fi is a fun hobby, and that there is a great deal of wishful thinking and snake oil involved still. 
 
With video equipment, there are STILL brands which claim superior video quality with CABLES! Impossible! A higher quality is required in SOME cases where the cable must be 2, 3 or 4X longer than a standard living room setup, but those cables in no way improve video quality. 
 
--
 
As far as reviews aside from the X3II, I had read several times that the grado's are great for lower volume listening. I listen at low volumes usually, but even then, the treble exaggeration was just too much for me in the vast majority of cases with the 60e and 80e. 

 

[money4me247]

I already define transparent as just listening to music, not the gear. I have yet to hear a rig that sounds like that outside of the e-stats though (for headphones; I dunno about speakers yet). Not even an HE1000 on a R-2R DAC+Ragnarok can do it for me.


If neutral means not coloured, then why do people say "X sounds neutral BUT blah blah blah explanation of how it's not neutral"? I've read too many times where people say X amp is neutral. Then people often say other amps are neutral. If so many things are neutral, then why do peoples' opinions sometimes wildly differ?


The Objective 2 was built to be "a wire with gain," which in theory would be transparent if I understand you correctly. I don't think many people will agree it's "transparent," especially those who have heard other high-end gear. So is the O2 no longer "neutral" and the other gear is?

There is no such thing as an amplifier that is just a wire with gain. It is just a theoretical ideal. A lot of amplifier designers say that their amplifier is built to mimic just a "wire with gain," but that is really just a marketing term. Just like the idea of a complete lack of colouration is just a theoretical ideal.

Practically, if you want to talk objectively, we can talk about is the parameters that affect sound quality and whether those measured parameters fall within our ability to perceive. That is the idea of audio transparency. You define a level where relative differences in measured values for all pertinent sonic parameters cannot be further distinguished between one another as well as where the further improvements in those measured values are no longer perceivable. If there is a large degree of separation between two measured values but they both fall below the threshold of audibly, it is irrelevant, but a smaller degree of separation between two measured values can have a impact if they are both above the threshold of audibility.

Those are two different ideas that fall under audio transparency that will need to be tested for. The O2's measurement in a certain parameter can be close enough in degree to the measured value of another component that makes it impossible to distinguish between the those two specific components. However, it can be possible that the shared measurement still is above the bottom threshold of what we can perceive, so it is possible for that the measured value to be improved to a degree that can be noticeable and distinguishable on another different component.

It is also possible that there are two components that both measure very well and quite close to to the lower threshold of what we can perceive, but the combination of their measured parameters are different enough to produce variability in audible characteristics. That would be akin to what people would be describing with a statement "both neutral but..." If you do not think that possibility is possible, then yes, "both overall neutral but with subtle differences" does not mean anything to you. That would be the analogy of two windows, X with a smudge in the left corner and Y with the smudge in the right corner... which one is more transparent/more neutral? They are both overall 'transparent' but there are relative differences. If you think that both smudges are beyond the resolution of what the human eye can pick up, then there is no point in talking about the relative differences between the smudges.

If you start with the idea that the O2 is already just "a wire with gain" & its measurements are all already below the threshold of where improvements in measured values can be perceived by human ears, then your conclusion is already pre-established based on definitions. The natural conclusion is already pre-defined in the premise. Basically, all gear with those measurements or better will sound the exactly same, and another piece of gear that sounds relatively different would have to inherently measure "relatively less well." Clap your hands & call it a day. There is nothing more to explore.

If you start off without any assumptions that a specific gear is closer to the ideal of the "wire with gain" and just test two pieces of well-measuring gear that can both be described as neutral and notice a subtle difference between the two. Then you say, they both sound neutral, but there is a XYZ difference between them.

Ideally, all external component manufacturers would be required to run the same series of measurement tests using the same protocol and publish all their data. However, there is variability on how manufacturers achieve their measured values and also the values they choose to publish. Even established values like THD+N which is a meaningful value to measure, is the aggregate of data containing numerous possible separate issues contributing to noise. The variability in the frequency and level tested at along with the number of harmonics measured, whether the full frequency range was tested, and gain setting of the device all impact in resulting measured value. How many manufacturers actually report all that information? The art of publishing specs can be used as a marketing technique rather than an objective comparative benchmark by not adding qualifiers. There are also a lot of specifications that can be published that have no effect on sound quality.

Therefore, the easiest and most common way to judge equipment with a lack of an universal and comprehensive standardized measuring protocol that all companies must follow is just using an ear test. If you cannot personally tell the difference between two components in a controlled setting, you have established audio transparency for yourself. However, your ear test may not match someone else's. The source tracks used to test and the sonic attributes that your brain is focusing on may be different which can highlight different characteristics. Another potential difficulty causing discrepancy is the headphones being used to test. Since transducer coloration is almost always greater than external component coloration, it can often be quite difficult to hear differences as the headphone coloration will mask any other audible differences.

Realistically from my experience, I have heard differences between well-reviewed components in blinded volume-matched testing, but more important than whether there is a difference, is the scale of the difference. 1) Some direct comparisons are relatively easy to correctly identify which component is which. 2) Other direct comparisons is much harder. 3) Some direct comparisons are basically impossible to distinguish accurately at all. For the first example, I pick which one sounds more pleasing and realistic to my ears and described based on that subjective standard. For the second example, I would use the descriptor "they both sound quite close to 'neutral' but there is XYZ perceived relative differences between them after extensive testing." For the third example, I would say that they are both neutral with no real appreciable differences to my ears. Another person can run the same series of tests and arrive at completely different results as the measurement tool of subjective reviews is our own ears. The possibility of Type I or Type II errors are impossible to avoid in ABx testing as a direct comparison of a 5-8 second test track may not always include the sonic characteristics or frequency response regions that may be variable. 

I wish that wasn't the case and there is just an easy table or fact sheet that gives you at a chance all the information that you need to evaluate the sonic characteristics of external components, but the industry as a whole does not practice in-depth comprehensive measurements. Agreement of what is neutral and transparent objectively is impossible without comprehensive measurements, but the high-end audio industry does not always prioritize releasing all relevant measurements conducted in a standardized testing protocol.

 

[CanadianMaestro]

   
 
 
Neutrality in amps and DACs is already well defined by measurements.  The output should be true to the input.  That's all there is to it. 

Hit nail. On head.  O/I ratio approaching 1.0 

 

[uchihaitachi]

There is no such thing as an amplifier that is just a wire with gain. It is just a theoretical ideal. A lot of amplifier designers say that their amplifier is built to mimic just a "wire with gain," but that is really just a marketing term. Just like the idea of a complete lack of colouration is just a theoretical ideal.

 

Benchmark Media Systems pretty much succeeds in the 'wire with gain.' Their Dac2 and in built hpa 2 amplifier have comprehensive measurements. Suffice to say, the measurements are extraordinary.