Well, at the risk of having it trampled on by un-thoughtful people, I might post a few more thoughts on the philosophy of testing. Although I have no control over who posts on this thread and you are all free to do as you wish, let me make a request up front that I only wish a serious discussion of the points I raise, which are related to the experience of music and the epistemology of audio performance ("epistemology" is the study of how we "know" things). If your only wish is to talk down to the "other side", I would humbly request you don't post on this thread. But I have no control, so be it.

Let me compare two experiences.

(1) Driving down an empty road in the middle of nowhere.
(2) Watching a guy in a monkey suit walk through a basketball court.

When you are driving, you are interested in not hitting anything. So you use your eyes to scan for obstacles. Healthy eyes and brains do this easily... if something unusual is moving on the road or near the road, you notice it right away. It just "comes to your attention." This is true for objects in your peripheral vision, also (in fact if it weren't, you wouldn't be safe to drive).

So you have this feeling of "taking it all in" and you have confidence that anything important or significant would come to your attention automatically.

Now let's talk about guys in monkey suits.

There was an experiment done a while back. The scientists filmed a short video of a basketball game, and at one point a guy in a monkey suit walks through the court. The test subjects were divided into two groups. Group A was told to watch the video and report what they saw. Group B was asked to watch the video and count the number of times the basketball was passed.

Afterwards, each person was asked if they noticed the monkey guy. Nearly everyone in group A said, "Yes, how could we miss it?" Nearly everyone in group B said, "What are you talking about? There was no guy in a monkey suit." They continued to vehemently deny it until shown the video again.

People in group B felt that an anomaly as large as a monkey guy walking through a basketball game would automatically come to their attention, just as an obstacle on the road would to a driver. But the experiment showed that sometimes it doesn't. Depending on how you are using your attention, there are whole areas of information that can be blocked.

The basketball game is a very rich and complex visual experience, with many things happening and too much to notice everything. Music is like that, too. The hardest thing I've found about controlling a listening test is controlling how I use my attention. In other words, if two cables sound different, is it because they really sound different, or because I noticed different things on each listen?

Getting back to "driving at night" and this idea that "you take it all in, you see every anomaly"... most DBTs I'm aware of use short bits or quick switching, and generally have the test subjects listen to "sound as sound" rather than "sound as music." (You can't experience short bits as music in a valid way, because music is not meant to be listened to that way.) Putting two short bits of sound "next to each other in time", in my experience, creates this feeling equivalent to scanning the road... it seems to lay everything in front of you for observation. However, I believe that is an illusion. Music is more like a basketball game... too rich to take in everything at once, and there is no way of "laying it all out in front of you." It's my opinion that short snippets actually block access to information; namely, musical information.

I will close with an observation of what I believe is a common fallacy. It's this: "Anything you can be conscious of while listening to sound as music, will also come to consciousness while listening to sound as sound."

I see this assumption any time someone says, "In a listening test I disregard the music and just listen to qualities of sound like sibilance." Or when someone says, "Short snippets work and it doesn't matter if they can't be heard properly as music."

Listening to sound as music is a different way of using your attention than listening to sound as sound. There is no reason to assume you can be conscious of the same distinctions between systems in each "mode." If two systems A and B create a different experience of music, there is no reason to assume they also create a different experience of "sound as sound."

Okay, maybe I scared away everyone by asking for no "political" posts. Well, to simplify things, I would be interested for starters if you are interested in the monkey-suit-basketball-game experiment. Secondly if you think that listening to music resembles watching a complex visual activity like the basketball game.

I watched the Video. I am not in the particularly surprised that so many of watchers did not perceive the gorilla suit man who is approximately the same color as the dark clothed players and pretty much the same size and shape, also the watchers in group B were instructed to ignore the players in dark clothes and only watch those in white, it is not a basketball court it is an office building and the players are in front of three elevators.

Also you said nearly everyone in group A saw the gorilla suit, but some did not. Overall this example is somewhat weak as an analogy for auditory perception techniques, they are differerent sensory systems and you have rigged the game by directing the watchers to behave in a specific way.

I once took part in an identity parade that was part of a murder inquiry, I am 6' 2" and my two pals who also took part were 6' 3" and 6' 4" ( A Black Maria picked us up in Richmond after the 1979 European Ultimate Frisbee championships) also in the line up were several other reasonably tall gentlemen all neatly dressed and all with no cuts and bruises, then the police brought the suspect out, a dishevelled much shorter individual with obvious cuts and bruises and placed him bang smack in the middle of the line of tall persons, next to me in fact, he was identfied instantly.

I think what you're getting at is that you can't listen or test different audio equipment using small bits, and must listen to a song in its entirety am I correct?

If I am, then what's stopping you from requiring an entire album? There are several albums in my collection that carry one song onto the next and should I be compelled to listen to the entire album so that I can detect for differences in how I perceived the sound?

Additionally, music can influence our emotions and thoughts, while one piece of music may make one cry, it can make another laugh depending on their moods (it can even be the same person who laughs and cries).

Sound as sound is, in my opinion, is meaningless. You hear what you hear, and suffice it to say, cutting out emotions and cutting out extraneous variables by using small 5 second samples is your best bet to really listen for changes in the system due to cables/amps/dacs etc...

By cutting out these extras you allow yourself to cut away from the bs and really detect what is going on. Once you perceive the difference (if any) then write it down as a legitimate change in sound. Listening to an entire piece in my opinion leads to null results.

Quote:



I'm saying that some differences may only be detectable when listening to music as music. It is purely an assumption that all differences are detectable as sound. Now, this leads to a difficult problem: how do you set up conditions in which you listen to music "as music"? How do you control these conditions? I don't think anyone has good answers.

I will say that because I care about the music, I only compare systems by listening to enough duration that I can get into the music. Not necessarily a whole song. But these are uncontrolled tests. I don't know how to control the conditions perfectly, and I don't think anyone knows.


Quote:



I think this is purely an assumption. There is no evidence for it. It seems like you are saying, "Cut out the stuff that's hard to control purely because it is hard to control." It may be important, but you would never know.

Quote:



Just because they are different sensory systems does not make it a poor analogy. The analogy is that music is too rich to notice everything at one glance, and that how you direct your attention has an influence on what you notice. All controlled tests "rig the game." Listening to short samples and quick switching is a powerful "rig," so to speak.

Quote:



On several levels it does. With listening you do not rely on saccadic eye movements , you use different parts of the cortex and a massively different range of EM frequencies. As comparsions these are very different.

Quote:



The point of the experiment was to deliberately prevent subjects from noticing things by hamstringing them with a particular strategy, to say look they did not notice the gorilla because there was too much to take in is wholly misleading.

Quote:



As mentioned before numerous times nobody forces you to do short sample switching in DBT, I personally use it as it works well for me, but I can listen to 5 or 10 or 60 minutes at a time if I choose.

As for controlled tests, the only absolutely necessary condition of a DBT is that you do not know the identity of the item you are listening to. A sensible extra condition is that the items are level matched but beyond that you can run it anyway you like.

Quote:



The real debate here is over this statement:

Music is too rich to notice everything at one glance, and that how you direct your attention has an influence on what you notice.

This is what I find to be true by introspecting on my process of listening to music, and many musicians and audiophiles are in agreement. In the original post I contrasted two uses of the attention: driving, and watching a basketball game. If this analogy doesn't "do it for you," then just drop it. But the statement above it critical to the issue. If you think it's a false statement, then address it directly.


Quote:



No, that's not the answer to my statement. It's an answer that "objectivists" like to give, but ironically it is never on point. I stated that all controlled tests "rig the game." If you want to dispute that, then answer it.

Quote:



If you really want a controlled test, you have to control what's going on in the subject's heads. Telling the subjects "listen anyway you like" does not control what's in their heads. I think this is completely obvious.

Excellent post!

First of all you draw a very important aspect of sound. The testing of short clips followed by switching of cables, headphones, amps is difficult and I truly believe that listening to whole albums or at least entire songs before switching gives you a more fluid understanding and not just a changing variable as music over time changes and the tones you hear from 20-30 seconds in are not the same as 31-50.

Thus even if you rewind back a little bit and than switch again. You still lose the ability to get into it and just listen and not focus on clarity or impact alone.

The point about noticing different things upon each listen is VERY possible for all conceived impressions from cables especially. If you assume or know copper vs silver conduct differently you will listen for different aspects of the audio spectrum thus blinding focusing on that which would "tune" down the other sounds making the focal point more pronounced. I believe to truly get an understanding you must listen for a minimum of at least 15 minutes to forget so much of immediate flaws and actually try to listen and assimilate everything that you hear. And not just "ooh bass, ooh treble, etc"

This is all my opinion so take it for what it's worth and always TEST before believing those who seem to be "PRO's or GURU's" because everyone hears differently.

Thoughtful post. However, I think the examples point directly at how inherently unreliable the human brain is as a test instrument. There are many things humans can do well, and one of those is to recognize our own limitations.

I generally know how much a pound weighs by feel. However, I serously doubt if I could tell the difference between 2 lbs. and 2.05 lbs. I'd switch hands and begin to doubt myself. I think most people would do the same. But a scale would easily tell the difference between two weights.

If you wanted to apply this analogy to cables, "believers" would make lengthy arguments about the problems of scale calibration, etc., all the while being unable to tell the difference by hand. Unless some sort of consensus developed along the lines that blue colored weights always weigh more than red ones. You could ge people to agree that was true whether or not it actually is. Throw in a rabid distate for any attempt to use a scale and indignant refusals to develop a better scale, and this is where we are.

You also need to take the analysis a step or two further and look at various implications.

Assuming that there is no scientific way to analyze cables, then there is no way to develop them in the first place. It means that every manufacturer is just casting about in the dark, not knowing what they're doing. At best, this is alchemy. At worst, it is fraud. The most awful part is that there is no way to tell the difference between someone who luckily stumbled across something good and someone who just makes up a bunch of marketing and sells something that is a bunch of nothing.

Going further, do you think that every cable manufacturer is entirely above board? How many of them are putting out fraudulent cables to make a buck? None? One or two? What if all of them are? How do you know? How do you tell a "real" cable from a garden hose filled with sandblasting medium and cheap residential wire from Home Depot?

You can't tell the difference. No one can. What we have is people with horribly imprecise, proven unreliable, test equipment trying to form consensus over whether red or blue weights weigh more. Or maybe the white one is heaviest of all. I know because I've picked up lots of weights over the years and I just know. Believers are so hung up on the minutiae of scales they resolutely rule them out, while insisting that their unquestionably faulty test equipment knows better.

But no one really knows anything. Any means of learning is thrown out as false, while the most unreliable method is insisted upon. Meanwhile, the fraudsters and the true believers mix it up for sales volume.

Quote:



Can you describe a test or procedure that could be used to demonstrate this proposition?
(as this is the science basement of Head-fi maybe a procedure that could possibly refute this proposition would be a better queston)

Wow Eric that was the best put argument Ive seen about cables and their theory in general. FANTASTIC ANALOGY. It really shows that someone such as yourself in the more elite is willing to say this. Makes me feel alot better and even if cables make a small difference I don't think it is worth the money period. Better to get a good surge protector power conditioner which most of us have for our plasma/lcd tvs.

Uncle Erik,
Thanks for the post, and you make some good points. However, this statement:

Quote:



is the kind of inflammatory statement I hope to avoid in this thread. However, I do like some of the points you make here.

As your quote says, you believe that "If you can't measure it, you can't improve it." This gets into the classic subjectivist/objectivist debate, and I directly disagree. I think the ultimate test of the performance of any audio gear is the subjective impression it creates. Not just cables, but amplifiers, headphones, speakers, etc. If I'm designing something, and I improve the measurements but the device sounds worse (*), then it's no "improvement" at all.

I agree that a scale is better than me at distinguishing between 2 lbs and 2.05 lbs. But I'm not evaluating audio gear according to linear, quantitative values.

Quote:



As I've said, I think that cables are no different than any other component regarding their relationship to measurements. You do bring up a good point, however. I am curious how cable manufacturers evaluate their own products, and how often they attempt to control the conditions. I might be pretty alarmed if I saw them casually discussing differences with no attempt to control bias effects. At the very least, they should be listening to their cables blind. But it is also possible they have refined their perception to a high degree.

Regarding telling the difference between good and bad cable manufacturers, well the way I would do that, primarily, is to listen. And to share experience with other audiophiles. It is distressing to think there are no hard-and-fast rules for finding the good, honest manufacturers, but a lot of life is like that. For example, restaurants and food quality. There is probably no correlation between the advertising and marketing a restaurant does, and the real quality. You have to go there and find out for yourself.

And just because Coco's serves awful "steak and lobster tail" doesn't mean that all "steak and lobster tail" is junk. In some restaurants it's heavenly.


Quote:



Again, it seems you are implying here that listening to components is primarily about sensing small quantitative, linear, uni-dimensional differences. To me, it's about comparing aesthetic impressions. It's like you are looking at two squares and asking, "Which color is more red?", while I'm looking at two whole paintings and asking what the difference in emotional tone is.


(*) What do I mean by "sound worse"? This could mean a lot of things, but one of the most interesting things would be carrying out a subjective comparison of a device, say a speaker, to real life. This could be done by playing instruments in one room, and piping a live microphone feed to another room with the speakers, then moving between the rooms. So "sounds worse" means "sounds less like the original." There are many other ways of defining "sounds worse."

Quote:



It's not a proposition. An example of a proposition is this:

If two systems A and B sound different in any way, under any conditions, then they sound different while listening to "sound as sound."

This proposition is assumed to be true, with no evidence, by many DBT advocates. I think the burden of proving it lies with those who assume it.

Quote:



Always ?, it is reported that Mozart could reproduce exactly a piece of music on one hearing, but even if true, so what ? how does that help devise a listening test that is more accurate than current listening tests and how would you prove it to be more so ?. You surely are not going to suggest sighted listening tests ?

Quote:



So , does this make them right, some audiophiles say Silver sounds different from Copper, do we have ANY verifiable evidence for this assertion ?. Even highly respected and supposedly knowledgeable people say daft things.

Quote:



Okay, it is a not at all helpful statement.


Quote:



I cannot discuss this properly unless you give me some concrete examples, from the field of audio preferably but without such examples I can answer this in 3 ways

1) Of course, this is blindingly obvious , so what ?

2) No, it is of course possible to rig any test i.e to set it up in such a way as to deliberately bias the outcomes but that is not the purpose of a controlled test.

3) Would you rather go back to the uncontrolled sighted listening tests ?


Quote:



The point of a controlled test is to control *some* variables, the variables that you are interested in and others which might get in the way. When you design a test you decide which variables you want to explore, some you control to keep conditions consistent, so you might have all subjects using the same CD player and so on. Other variables may be seen as unimportant or important, perhaps you only want to test tall people, your choice, or accountants, or just those with a certain level of formal education, your choice and if you do so your results must tempered by "with this test group under these conditions". Speculating about the internal workings of a subject is unhelpful and unnecessary since you can only infer some of them and not with any real degree of accuracy.

Quote:



So how is that a problem ?

I seldom have anything nice to say about Skinner but sometimes a black box approach is entirely appropriate, I do not discount that something internal is going on but in the context of blind listening tests it is just not interesting. Your subjects can either detect a difference or they cannot, let them use whatever strategy they prefer.