1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.

    Dismiss Notice

Testing audiophile claims and myths

Discussion in 'Sound Science' started by prog rock man, May 3, 2010.
871 872 873 874 875 876 877 878 879 880
882 883 884 885 886 887 888 889 890 891
  1. sander99
    Note I retracted my proposal anyway, and also gregorio gave me more good reasons to do that, but for the record:
    I don't know. But re-reading my original post I see I was a bit unclear about one thing: I wrote "Then do a well set up double blind ABX test using those specific audio fragments" but I didn't mean to use only those, I just meant to include them in the set of audio fragments to be used in the ABX test. So the whole intended purpose of the sighted listening was just to create an additional chance to find "suspected differences". And again I stress the point that that was the only intended purpose of the sighted listening, and that by no means any final conclusions should be drawn from the sighted listening itself. Without confirmation in the controlled blind test the "suspected differences" can be thrown out of the window so to speak.
  2. castleofargh Contributor
    oh sure. in here we often have people who tell us with total confidence that they can perceive stuff that I can't seem to perceive, and sometimes that I believe an average human shouldn't be able to notice. of course I would always invite those people to test the specific samples where they believe they heard a difference in a blind test. because then that test's entire purpose is to demonstrate that they indeed were able to noticed something in there for whatever reason. the blind test if passed is evidence supporting their claim. which is great even if it would be better to have people claim stuff only after they have evidence of it. but that's yet another issue.
    WoodyLuvr likes this.
  3. bigshot
    Not on purpose, but rather because of subconscious bias. That isn't intentional. As I said, we're all subject to bias affecting the way we make decisions. Sorry if I offended you. I don't think less of you at all because you exhibit bias. I do too. That's why I apply controls to my listening tests. And challenging bias is how you get closer to the truth.
    Last edited: Jul 8, 2019
    WoodyLuvr and AudioThief like this.
  4. castleofargh Contributor
    well that's not my position. I'm just in favor of blind testing to demonstrate audibility because I don't believe casual listening to satisfy the concept of "listening test". one of the first things I said to @AudioThief was that controlled tests were really important for small differences. and in the very post you had quoted, I present a list of developments that help science experiments and research in general, get it right more often than not. I start each item of that list with "ideally". I think that makes it clear enough that I don't expect things to often work that way in practice, and that I do expect flaws and various issues. but hopefully we can discuss those and fix some or agree that a test is inconclusive because those issues weren't fixed(which is my reason to find inconclusive almost all sighted impression ^_^).
  5. bigshot
    Headphones are the most difficult to test because of a few reasons... it's hard to do an ABX because the feel of the headphone gives it away, you can't do direct A/B switched because you have to physically take them off and put them on, and different people have different ears- both in the shape of the ear canal and in the preference for the sort of sound they prefer. Controlled testing is easier with other aspects because the purpose of amps and players is to reproduce with maximum fidelity. A transducer is turning signals into physical sound. There are more variables there. Usually, you consider the transducers to be the "wild card". Everything else in the chain must perform to spec, and you adjust your calibration to suit the transducer.

    In general, the goal for a good headphone is a reasonably balanced response curve (some would say the Harman curve is the best), distortion below the threshold of audibility, and comfort. Your Stax would fit the bill. Comfort and low distortion is fairly common. The trick is the response curve. However any decent midrange headphones should be able to be EQed to conform to your desired curve.

    There really is no perfection with headphones though, because commercially recorded music is designed to be played on speakers. Headphones are by definition a compromise for the sake of convenience.


    By the way, I believe that without blind testing, it is very difficult to prevent bias from affecting the results. And blind testing is one of the most effective ways of reducing the influence of bias on tests. But bias is only part of it. Perceptual error needs to be addressed too. Stating things in absolute terms doesn't change that. It just adds irrelevant semantics to the argument.
    Last edited: Jul 8, 2019
    WoodyLuvr likes this.
  6. KeithEmo
    I would like to add something to that....
    There is another legitimate purpose to "sighted listening"....
    It is one of the important ways in which we learn (in this case "learn how to listen".
    (And that may refer to "learning how to listen critically in general" and to "learning what to listen for in terms of differences between those two specific samples".)

    I have repeatedly heard how, "since a statistically significant percentage of listeners can't reliably tell the difference between a WAV file and a 256k AAC file - they are audibly identical".
    The presumption, which makes sense at some level, is that " if a real difference existed then a significant number of people would notice it".
    However I question whether this "commonsense assumption" is actually true.

    Every so often a museum acquires a painting of somewhat dubious provenance... it's "probably a real Rembrandt" - but they aren't sure.
    Wouldn't it make sense to invite a sample of the public in, invite them to see the painting, and see what percentage statistically believe that it's authentic?
    (After all, if a "statistically significant number of viewers" fail to find anything wrong with it", then shouldn't we assume it's authentic?)
    Oddly, however, they usually instead rely on the opinions of a few actual experts, who probably have a lot of experience with many real Rembrandts, as well as a variety of fakes.

    Among other things, we humans "learn how to notice things" (we learn to focus our attention on specific details).
    One obvious example is that tiny bug on the windshield that, even though you failed to notice it for days, cannot be ignored once you do notice it.
    I suggest that the same is often true for minor flaws and differences in audio gear or content.
    You may not notice that "clinker" - but, once you do notice it, or someone points it out to you, it becomes obvious - and annoying.
    (And, even if you don't notice that one odd-sounding note until someone points it out to you, it may in fact be consistently audible once you do notice it.)

    I'm not an art expert - and it's quite possible that, if I were to see both today, I might not recognize a real Rembrandt from even an inexpert forgery.
    HOWEVER, if I were to spend a week with an expert, I'm sure he or she could teach me a lot about how to tell the difference, and drastically improve my ability to do so.
    And, even more specifically, an expert on Rembrandt forgeries could almost certainly teach me the specific flaws to look for in products by various forgers.
    And, even more specifically than that, when comparing a specific forgery to the original, I suspect there are many differences I could LEARN to notice, once someone pointed them out to me.
    (I might never notice a brush stroke going in the wrong direction until someone pointed it out to me - but, since I can see it readily at that point, you cannot claim that it isn't visible to me.)

    For exactly the same reason it makes perfect sense to allow an unlimited number of sighted comparisons BEFORE a blind test.
    So what if I have to listen to the lossless and lossy copies of that song a dozen times before I notice a few small differences...?
    And so what if I only notice those differences after looking at the waveforms on an oscilloscope trace...?
    And so what if I only notice it after someone says "listen carefully to how that note fades away... just there...."?
    By doing so I have in essence "become an expert in the differences between those two particular files".
    Then, once I have reached the optimum level of my ability to detect that difference, would be the appropriate time to perform our test.
    (After all, if there really is no audible difference, then I still won't be able to detect a difference that really doesn't exist... right?)
    If, after I reach that point, I STILL fail to be able to reliably detect a difference with a blindfold on, THEN we can reasonably claim to have "tried as hard as we can to rule out the possibility that a difference actually exists".

    (And, of course, if you're like most people, you may be perfectly satisfied to have a "really good forgery" of a Rembrandt hanging in your living room.)

  7. KeithEmo
    I think we're in absolute agreement.

    And, as I mentioned once before, I had hoped that this forum would be a place to discuss some of the limitations in previous tests, and how they could be improved upon....

  8. bigshot
    As someone who has actually authenticated art professionally, I can say that authentication isn't just having a trained eye. It is an objective process of testing materials, researching and validating provenance, and comparing technique to see if it is consistent with known samples. It isn't so much discernment as it is having a body of research to refer to. People make the mistake of thinking that art is purely subjective. It isn't. It can be analyzed objectively with science just like anything else. In any case, art isn't authenticated by eye. There have to be strong supporting arguments to prove authenticity.
    Last edited: Jul 8, 2019
  9. KeithEmo
    So, if you had a good forgery, and an original, and wanted to determine which was which, you WOULDN'T ask 100 people, statistically analyze their responses, and claim that the results proved anything, right?

  10. bigshot
    I would perform controlled tests to determine whether the material was consistent with the place and time, and I would interview people and research the record to establish provenance. I wouldn't just ask people how they subjectively feel about the painting, no.

    If I wanted to establish thresholds of human perception, I would probably test a wide sampling of people and determine a range.

    With one thing, you are researching an object, with the other you are testing the ability to perceive. Apples and oranges.
  11. KeithEmo
    If you look at the performance measurements for various headphones it's no surprise at all that they sound quite different.
    (Compared to amplifiers and DACs, many headphones, even supposedly "high end" models, measure very poorly.)

    I ran across this really thorough table of test measurements for a long list of headphones.
    RTINGS seems to have a long list of remarkably thorough headphone reviews.

    Check out both the listed values and the graphs for your favorite models.....

    For example.....

    Sennheiser HD600:
    THD @ 20 Hz @ 90 dB .... over 5%
    THD @ 20 Hz @ 10 dB .... over 10%

    Interestingly, the models that are widely considered to be "best sounding" and "most accurate", don't necessarily measure the best at all.
    (I also note that, with the excellent variety of measurements presented, I

    They are all within a range such that it would be a matter of opinion, IMHO, along with many other headphones.

    The MDR-7506s have been found to be the highest fidelity headphones full stop on more than one occasion. Read the Wikipedia article, There are beauties and clunkers throughout the price spectrum. But a number of people just plain don’t like them. : )[/QUOTE]
  12. AudioThief
    Which would mean that the Sony MDR-7506 (one of the well measuring headphones) as a matter of fact sound better than say a Stax SR-009, Lambda, HD800 or Focal Utopia - and that any perceived or real difference would be a matter of taste, not in fidelity - and that taste would be a tossup or 50/50 assuming that people in general prefer higher fidelity / flat FR. So if we took a big sample size, and only looking at sound quality (i.e ignoring comfort, mobility or what have you) most would prefer the Sony MDR-7506 because it measures better and thus as a matter of fact sounds better?


    And, of course, the sole reason as to why you will in general hear people prefering the 009 or Utopias to the MDR-7506 is not because they sound any better, or even that they prefer the sound, but rather they are heavily biased by looking at the price tag or being mass suggested by the internet.
    Last edited: Jul 8, 2019
  13. bigshot
    Headphones generally are weakest in the sub bass. You really need speakers for that.

    As I said before, different people have different shaped ear canals and different preferences for response. Headphones involve variables that make it difficult to define what is "perfect". You can't judge headphones until you define your ideal response curve. That is different for different people. It isn't a matter of asking, "Does this one sound different than that one?" We know that they sound different. Every headphone sounds different. In fact, there's usually an audible sample variation between the same make and model. That's why you apply sound processing (i.e.: EQ) to taste.
    Last edited: Jul 8, 2019
  14. castleofargh Contributor
    the issue here is that the objective best fidelity headphone will massively depend on the frequency response. think about distortions, let's say we have 8% THD right in the midrange where it going to be impossible to miss. that will still mean extra unwanted signal some 20 something dB below signal. subjectively it's a lot more objectionable than a signature with fluctuations here and there within +/- 3dB of some predefined flat in the audible range. but objectively, a change in frequency response affects even the loudest signal while that horrendous THD remained 20dB or less below said loudest signal. so from a purely quantitative point of view some might argue that the FR variation results in the lowest fidelity. it's super weird to look at it that way because the very purpose of a headphone is listening, but I think we could consider a bunch of similar situations with other variables once we have settled on exactly how to quantify fidelity. if we assume that fidelity is getting the same output signal as the entry signal, then we might be tempted to make a null test and whatever comes out with the highest amplitude is the worst fidelity. in such a case a phase shift(which is under most conditions, going to go unnoticed for a listener), would qualify as some of the worst signal degradation possible.
    but even with something as simple as a null between input and output, we will have problems to solve. for example, should we do that with a recording device that is electrically neutral? or should we record the sound as it would be when reaching an eardrum? and if so who's eardrum? different people will get different changes in sound caused by their outer ear and ear canal length.

    so as you can see, we have a few things to determine before we go and claim headphone XXX to have the highest fidelity. some of which might seem more arbitrary than others despite how they can massively affect the results. like once we have determine most things, do we run our fidelity quantification formula over 100Hz to 15kHz? 20Hz-20kHz? or as much as we can because we're looking for objective fidelity without care for human hearing? in the end I'm afraid that such results would not actually lead to anybody's favorite headphone. and if we proceed toward a more rational approach where we include the typical listener's preferences, hearing thresholds, and how some noises or distortions can feel low fidelity even when they're a lot quieter than some others we might even find euphonic, then the estimation becomes even more complicated and I'm pretty sure we couldn't find a consensus on the definitive list of criteria to use. @Steve999 mentioned rtings.com, they try to implement a few sets of objective and subjective values to try and guess when a headphone is going to please the average human. it's very interesting and they certainly had a few good ideas when starting this, but it's just one way to approach a very complicated system of measurement, feelings, how they might interact, and how close you yourself are from an average human(it might seem like a joke said like that, but I'm very serious).

    or we could take a completely non audiophile and yet pretty objective approach, and determine that pretty much any headphone with head tracking is objectively superior to almost any headphone without it. all it take is to shift the reference of input sound. if we assume that albums were mastered for speaker playback, then failing to try and approach that reproduction of sound is a very serious fidelity fail.

    so I don't have a definitive answer because I don't know how things should be defined or what should take priority. sorry but if you have your own priorities, then perhaps we can reach a result. like some will find very important to have a headphone that extends far in the ultrasounds. someone else will swear by the measured THD. when someone is obsessed about one specific variable(sometimes for the wrong reasons), finding his ideal product becomes much easier.
    AudioThief likes this.
  15. bigshot
    When it comes to headphones, there's enough sound you can actually hear to deal with! Super audible frequencies and microscopic THD readings aren't the best path to perfect sound.
871 872 873 874 875 876 877 878 879 880
882 883 884 885 886 887 888 889 890 891

Share This Page