1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.

    Dismiss Notice

Testing audiophile claims and myths

Discussion in 'Sound Science' started by prog rock man, May 3, 2010.
770 771 772 773 774 775 776 777 778 779
781 782 783 784 785 786 787 788 789 790
  1. Phronesis
    Maybe the best way to test this is to manipulate files in various ways to various degrees, and then test with listeners using different time frames. It’s more a test of perception than gear. My hypothesis is still that being insignificant in the short term generally means insignificant in the long term, but that’s just a hypothesis. I don’t know of any good testing that’s been done for this.
  2. KeithEmo
    You clearly didn't actually READ what I wrote.

    I really try to be civil - but I give up.

    When you analyze a group of samples, looking for something of statistcial significance, you can either find it - or not. YOU CANNOT STATISTICALLY PROVE THE NEGATIVE. All you can prove is that you made a reasonable attempt to find something and failed to do so. From this you may INFER that what you were looking for wasn't there BUT YOU HAVE NOT PROVEN IT. If you examine a thousand samples of dirt from my back yard, looking for diamonds, and find none, YOU HAVE NOT PROVEN THAT THERE ARE NO DIAMONDS IN MY BACK YARD. You have simpy failed to find any proof that there are.

    The M&M test failed to show statistically that any significant number of test subjects could reliably hear a difference beyond what would be expected from random chance. However, several subjects as individuals scored very wel, AND NOBODY BOTHERED TO FOLLOW UP TO DETERMINE IF THEIR RESULTS WERE DUE TO RANDOM CHANCE OR NOT. (From their results there is no way to even guess whether the guy who got 8/10 right could do so every time, or only that once, because they failed to follow up on his results.)

    I simply stated that "many people do not find CDs to sound audibly perfect". I SAID NOTHING ABOUT A TEST. If you read a few, or a few thousand, reviews of CDs, you will find that a massive number of people find them to sound "not audibly perfect". In fact, I have yet to find an issue of Stereophile where at least a few people haven't commented about how bad this or that DCD sounds, or that the vinyl version sounds better. Therefore, it would be ridiculous to suggest that "everyone thinks CDs sound perfect".

  3. Phronesis
    The thread is getting weird and pointlessly contentious lately.

    IMO, the bottom line is that if we want to draw general conclusions about what gear sounds different (or not) to what degree, we need an extensive program of good testing. Without such testing, all we have to go by is a lot of non-rigorous controlled testing and a lot of casual sighted listening, and the two categories give conflicting results.

    Most people who've done both believe that differences in gear other than transducers are very small, and have firsthand experience with their own expectation bias. Many people who've only done the latter believe that there are significant differences in gear other than transducers, and are largely unaware of their own expectation bias. Two different sets of experiences, and two resulting different opinions.

    The matter won't be conclusively settled until we have an extensive program of good testing, and that may never happen. So what else is there to discuss? Are people just arguing because they're bored or like to argue?
  4. KeithEmo
    That's a good idea. The reality is that quite often expensive DACs don't perform any better than low cost ones, and some even perform notably worse, but many people seem not to realize this... and it would make a good point.

    I would also suggest specifically comparing a unit that used an ESS Sabre DAC to one that uses another brand - a Texas Instruments Burr Brown or a Wolfson. From my experience, and a huge number of anecdotal reports, Sabre DACs tend to sound different from other brands - and to have a sort of "house sound" - even though their basic measurements are very good.

  5. Killcomic
    Well, you know, audio equipment for audiophiles is a penis extension, hence why so many middle aged guys here. I've noticed there's this almost desperate need to demonstrate that they have the magical ability to hear sounds beyond normal human hearing. It is what they hope to set them apart from the common rabble. That's why there's so much utterly meaningless language like describing something as "musical" and sounding "organic".
    To actually admit that you've been buying very expensive equipment that yields no real benefits, it's a bit embarrassing. It is counterproductive to the myth they would like to surround themselves with, because you know, penis extension.
    Could be worse, they could buy a sport car.
    taffy2207 likes this.
  6. gregorio
    1. Not just lately but for quite a while!

    2. Well that's the issue; drawing a "general conclusion". There is not and cannot be any absolute proof because we cannot test everyone alive, nor everyone who was alive or will be alive. We therefore have to draw a "general conclusion" on the balance of probabilities and this is where it "gets weird" because we have an overwhelming probability on one side, supported with lots of reliable evidence and a hypothetical possibility on the other, supported by no reliable evidence at all.

    2a. We've had lots of "good" testing, going back many years. Some of it is published as scientific papers and much of it isn't, such as that carried out by international bodies, audio professionals and university departments/educators. Therefore it's extremely unlikely there will be another program of extensive testing because as far as the professional and scientific communities are concerned the matter is effectively "put to bed". To get it out of bed again would require some serious evidence, but audiophile marketing and audiophile testimonials do not qualify as "serious evidence". So we have the impasse of a segment of the audiophile community vs pretty much everyone else and as that segment appears immune to logical/rational arguments, the scientific and professional communities simply leave them to it.

  7. KeithEmo
    That makes sense.....

    If you would like to suggest a particular test, I'll be glad to try to point out at least the obvious flaws.
    I'm going to start with the Meyers and Moran study.

    One of the most common errors I see is the MISAPPLICATION of statistics to situations where they don't apply.

    The asserted purpose of the Meyers and Moran test was "to determine whether people could reliably determine an audible difference when a "CD loop" was inserted into the audio signal chain". Most of us here interpret that to specifically mean "whether any human being can reliably determine the difference". Assuming that to be the case, then here is the first (MAJOR) error. Because of the nature of the testing and reporting involved, THE PERFORMANCE OF EACH INDIVIDUAL MUST BE ANALYZED STATISTICALLY. The only way to tell whether an individual can reliably tell the difference is to run a bunch of test trials and see whether their responses score significantly better than would be expected by random chance.

    However, if you apply a bit of logic, it will be obvious that there is no reason to apply statistics to the OVERALL results, and doing so would be improper in light of the results desired. If the purpose of our test is to determine whether ANY HUMANS can reliably detect a difference, then, as soon as we find a single human who can do so, we have our answer - which is yes. We have a human who can do so, therefore "some humans can do so". That is a simple YES/NO question and NOT a statistical question. (It doesn't matter how many can do it; once we find one who can , we're done, and the answer is known: yes. We might go ahead and see how many can do it, and probably would do so, but that is outside the scope of the original test.)

    If Meyers and Moran were REALLY trying to determine "whether most people would hear a difference" or "how many people would hear a difference", then statistically analyzing the overall results was a perfectly valid way to do so. I Strongly suspect that this was what they were trying to do - since this is what most of the public, as well as most CD vendors, would really be interested in. Therefore, attempting to claim that their test "showed that nobody could do it" is simply a misinterpretation of their results. Their results simply showed that "they were unable to suggest that a statistically significant percentage of the people they tested could reliably determine a difference a statistically significant percentage of the time".

    (It is also absolutely true that "a statistically significant percentage of the human population is not able to run a mile in four minutes". However, that is FAR different from "proving that nobody can do it", along several different axes.)

    Here is the proper way to run a test "to determine whether ANYONE can reliably detect a difference".

    You conduct the first level of the test exactly as they did, using statistical analysis ON EACH INDIVIDUAL'S RESULTS, to determine whether each individual test subject scored above what might reasonably be expected by random chance. However, you treat the first level of the test as a screening round. You assume that all the subjects whose results fell very close to random are "uninteresting" and send then home. HOWEVER, if any subjects produced individual results that appeared significant, you conduct further test runs with them to determine whether their results were random or not. Typically you would set some sort of threshold level for "subjects to be passed on to round two". This would typically be either a direct threshold like "everyone who scores above 7/10" or a fraction like "the best scoring 10%". (And, if nobody scored above that threshold, then you would either declare a null result, or find a new set of test subjects and try again.)

    At this point you should also note and report two obvious potential causes of error:
    1) Your sample didn't contain all humans - so you could simply have missed one who could do it.
    2) It's possible that one of your subjects could do so - and his or her failure this time was the anomaly.

    So far that constitutes either an error of "performing a test that is inappropriate to the desired results" or "attempting to mis-interpret their results to prove something the test was never intended to measure".

    The test itself also contains numerous errors of methodology and procedure.

    We all know that, whether it's audible or not, the CD signal chain fails to exactly reproduce the original audio in many measurable ways. It adds some noise, some distortion, and some errors in frequency response and impulse response, as well as possibly other unknown differences. They should have provided detailed analyses of their test content, both "before and after". They were attempting to determine whether their sample listeners could detect differences - YET THEY FAILED TO PROVIDE DETAILED INFORMATION ABOUT THE DIFFERENCES PRESENT IN SAMPLES IN THE TEST. For all we know they used low quality original material, the CD signal loop made no measurable difference to it, and there were no measurable differences between them. They should have provided both detailed data, like product disc numbers and specifications, as well as test results like spectrum analyses, showing what was present in each sample, and what the measured differences were. In fact, since the "item of interest" was the differences, they probably should have provided an actual difference file containing those differences for easy analysis and confirmation. They should also have included measurements that showed that the differences involved were being accurately presented to their listeners - and not being obscured by any of the components used for the test. Then we would know precisely what differences the subjects were or were not able to hear.

    I should note, in their defense, that they DID note the anomalous results obtained from several test subjects.

    It was always my impression that Meyers and Moran simply set out to "provide a simple test to demonstrate that, when used to reproduce typical consumer-quality content, most consumers didn't notice any audible degradation caused by CDs".

  8. KeithEmo
    Quite true.

    However, in this case, we are assuming that all of the products we're discussing already have exceptionally low THD, exceptionally flat frequency response, and very low levels of noise - by "commonly accepted standards". The specs on most DACs, even most cheap ones, are quite good... although we may always question whether they actually meet those specs when tested. If there was a simple and audible flaw, like too much bass or treble, that would be obvious on a basic frequency response measurement. Likewise, excessive THD would show up on a distortion analyzer.

  9. KeithEmo
    Excellent point....

    Which is why, if you're choosing equipment to purchase for yourself, then what YOU are able to discern is the most important factor. The one exception is that I would always "aim a little higher" as a sort of safety margin.

    For one thing, you don't want to purchase a whole bunch of lossy files, based on the fact that you can't hear the difference with the headphones you are using today, only to find that they sound obviously worse on the higher quality headphones or other equipment you purchase tomorrow.... (It's true that you would be wasting money to purchase lossless files if you don't ever hear a difference. However, it's also true that some people who purchased 128k AAC files on iTunes, which sounded "just fine" on their cheap headphones, ended up buying them over again as lossless files when their flaws became obvious on better equipment. It's a bit of a slippery slope - in both directions.)

    I should also point out that both our hearing and our perception change with time. Physically, our hearing becomes less acute as we get older. However, many of us "learn to listen more critically" over time, and actually do notice things that we may have ignored or failed to notice previously.

    Harman International has an excellent little (free) app for "training your hearing":


  10. KeithEmo
    The two "most highly regarded" commercial programs for converting between DSD and PCM are probably
    Weiss Saracon and Korg Audiogate.

    Saracon has consistently remained very expensive...
    However Audiogate has, at various times, been available at low cost, or even free...

    (For a while Korg was offering it for free - but only if you would allow it to Tweet "I'm using Korg Audiogate" whenever you started it up... yeah... really. I haven't checked the situation lately...

  11. Phronesis
    If people could provide links to info on good testing available to the public, that would helpful. If there's better testing out there than what's in the first post of this thread, I'd certainly like to read about it.
  12. Phronesis
    Probably all true to some extent, but I wouldn't want to generalize it too much.

    I see a lot of people admitting that they're saving up to buy gear, so clearly not trying to project an image of affluence. And audio expenditures are peanuts for anyone with a decent amount of money, so even a $10K DAC isn't really a big deal (though perhaps a waste of money) in the scheme of things, still a lot cheaper than a car or college education. Personally, I don't mind saying that maybe it turns out that my Hugo 2 doesn't sound any better than my Mojo despite costing 5x more and my buying it because I thought it sounded better (I do like the looks, amp power, and portability), though I might feel peeved if I had bought a Dave and had the same result. Generally, I try to find truth, even if I don't like the truth.

    As far as magical hearing, I would again go back to the point that people perceive clear differences (even if they're not really there), so when other people say they don't hear them, it's natural to conclude that their own hearing ability is better (even if it's not). And of course, there is some variation in hearing ability, and I'm pretty sure that my own hearing ability has improved with effort and experience.

    Regarding sports cars, a lot of us buy them because we like driving them (I've actually tracked every car in my garage, including the SUV and sedan). Status display can be a factor, but a lot of us don't like the attention that sports cars garner.
  13. bigshot
    There's a psychology to internet forums that is stacked up on top of that too... There are people who feel inadequate in real life who try to construct online identities that are designed to make people think they are experts. It's a lot easier to be a "bigshot" in an internet forum than it is to be one in real life. Often these people will park themselves in a single thread or forum and comment on each and every post that comes by either talking about themselves and their personal thoughts and opinions, or cut and paste information from google searches. They rarely actually interact with anyone, choosing instead to look at other people's posts as springboards for launching into their own extended rambling monologues. These types feel threatened by people who actually do have experience and knowledge on the subject and attack them to avoid being shown up as not being what they're pretending to be. We've seen a particularly extreme example of that in this forum where multiple armchair forum "experts" gang up on Gregorio because he clearly is better informed and has more experience on the subject than most of us here.

    When you add the "internet expert" persona to the "it's either your cheap stereo or you're deaf" one, you end up with a particularly obnoxious combination.

    I and others here are offering to provide independent verification of a clear audible difference in a DAC. All we need is a clearly different sounding DAC to check.... crickets. It's one thing to spend hours and days and weeks and months spewing out paragraph after paragraph of unsubstantiated opinions, but backing them up seems to be too much effort for some people. All they want to do is tell other people how they should back up their opinions!
    Last edited: Dec 19, 2018
    Killcomic likes this.
  14. analogsurviver
    I am well aware of these two - and it is precisely because of this I wrote what I have written.

    Since I own 5 Korg DSD recorders ( 3x MR-1 & 2x MR-1000 ), I have never had any problems with obtaining and authorizing ( the High Quality version with all the functionality gets authorized to the computer to which you want to install Audiogate by connecting any of the Korg DSD capable recorders or DACs - free is only Lite Load version, which has MUCH worse sound quality/conversion and next to none but the most basic functionality ) of the latest Korg Audiogate software.

    Saracon has consistently remained very expensive. For me, it does THE capital sin - its low pass is set way too low, it filters out everything by ( 30 ? 40 ? kHz ) . I would have to re-check the comparisons with spectrum analysis etc , published a few years ago on https://www.whatsbestforum.com/ - when it was being decided which hardware and software should be used for DSD by now commercially available transfers of the better>best analogue recordings, etc. That excessive filtering above 20 kHz plus $$$ has been more than deterrant enough not to mess with Saracon.

    Korg Audiogate went from Version 2.0 trough many updates to Ver.3.0 ( here somewhere it caught up in SQ with the then latest jRiver ) and finished off by Version 4.0 and very few updates, which basically added recording both in DSD and PCM, limited to DSD128 and PCM192.

    It has one very hard to swallow limitation - it works in DSD - but ONLY using Korg devices. Everything else, even if native DSD capable, will work only by conversion to PCM.

    As none of the other DSD player soft can "see" markers made during recording with Korg recorders, I did consider adding a Korg DSD DAC just because of this - sound quality of the DSD file over the PCM takes a MAJOR hit in this case ... - insisting on native DSD playback using non Korg DAC and finding a particular spot in a "take" of one song ( actually, MANY takes of the same song in one single file ) with just a slider in style of F2K or jRiver is, for all practical purposes, hopeless.

    As I write this, I am fiddling with one of the latest additions to the DSD players/editors - TASCAM Hi Res Editor that I just downloaded and installed. It allows for processing/conversion DSD up to DSD256 and PCM up to 384kHz.
    It is FAR less robust than either Korg Audiogate, F2K , jRiver - it will stutter where all of the three mentioned will work flawlessly. Will have to see if it can be adjusted etc a bit better - but, very few controls over anything ... - BUT IT DOES warn you, at each launch, to set your laptop to High Performance ( high battery consumprtion )...

    THE problem I am having with Korg Audiogate ( any version... ) is the fact that it introduces phase lag between the channels - which I have already described some half or so year ago. Square wave recorded to DSD ( which, by default, can not introduce such an error ... ) that gets perfectly recporded and reproduced on Korg recorder(s) , with total synchronicity between the two channels, gets lost when converted using Audiogate. It is ALWAYS the precise delay between the two channels equal to the rise time in any given sampling frequency of PCM - from 44.1 to 192 kHz. In other words - one channel is still silent when the other has reached its full output - or vice versa... I forgot which channel is leading and which lagging - but it always remains the same.

    I will convert the approx 3 kHz square wave test files using TASCAM Hi Res Editor - AGAIN - and, hopefully, that unwanted delay between the channels will go away....
    Will report what happened tomorrow.
  15. Killcomic
    Maybe it's a cultural thing, but I never intended to indicate that people here were trying to project an image of affluence. That's not much of a thing where I live (there are always the isolated exceptions though). However, I am utterly aghast at the level of superstition and elitism found in what is, let's face it, a rather trivial hobby.
    Don't get me wrong, I've been subject to delusion and expectation bias myself, but to attach a sense of self importance to it is utterly baffling. If anything, I questioned what the value of what I was getting. When a lot of the "evidence" of the benefits you'd be getting from an expensive piece of equipment is based on marketing and pseudoscience, you really have to question what's going on in there.
    And as far as sports car go, sure, you do have car lovers, that's true and all respect to them because they appreciate the beauty of design and engineering. But when someone puts on racing gloves, aviator glasses and a polo shirt with a popped collar and a sweater tied around the neck, there's not mistaking that person is a wanker.
    Last edited: Dec 19, 2018
770 771 772 773 774 775 776 777 778 779
781 782 783 784 785 786 787 788 789 790

Share This Page