[Pio2001]

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif So we did the first two trials on Sunday. More trials will follow. We may change the protocol as we learn what seems to be more sensitive. Strictly speaking, we should only count trials that were done with the same protocol, but as this is a learning experience we may have to change it several times. Our accuracy may improve with time.

This situation makes very complex statistics, and the risk of "picking results" is big. In the negative case, if all impressions are in the head instead of the cable, you are going to pick the test conditions during which the score was maximum, and assume that the success comes from the test conditions.
If you cross-test many parameters, like duration, pause lenght, musical choice, headphones, etc, the probability that, among a lot of random answers, you can pick a sequence with a good score and associate it with a given condition, is not negligible at all.

In order to avoid this, I think that it is better to dismiss any result as long as the most relevant test conditions are not fixed.
Then, you can evaluate your reliability. For example, one mistake out of five trials.
Then, you can set the number of real trials for the real test, given the confidence that you aim to acheive, and the ratio of mistake you expect to do. I can calculate it for you, if you need.

This method is very robust, and allows you to avoid the statistical bias that occurs when many tests are done in order to find one confirmation.

On the other hand, if you get good results during the trainings, and bad ones during the real test, the final result is all the same a negative.

 

[moonboy403]

I was the one who helped Mike with the blind tests. We only did two trials because it's REALLY time consuming and tiring. Everytime sub-trail takes about 5 mins then after each subtrail, I need to switch all the cables in addition to giving 5 mins break for Mike to rest his ears. So by the time we complete one trial say ABAB, we've already spent a minimum of 35 mins.

 

[mike1127]

Quote:

Originally Posted by Pio2001 /img/forum/go_quote.gif This situation makes very complex statistics, and the risk of "picking results" is big. In the negative case, if all impressions are in the head instead of the cable, you are going to pick the test conditions during which the score was maximum, and assume that the success comes from the test conditions.

I am aware of the danger of cherry-picking. Let me explain. There are really three levels of knowledge here:

Level 1 is self-knowledge: basically, what happens when I listen to a cable blind? It's been a long time since I did so. Do my sighted impressions all disappear into a cloud of confusion? Or do I have some clear impressions? Do they resemble sighted impressions at all? (I learned that I do have clear impressions.)

Level 2 is evidence for myself that cables matter for their performance, not just as a placebo. For instance, can I give the right answer five out of six times? Four out of six? What's my level of clarity? Do I feel like I'm guessing or do I feel like I have some kind of handle on the situation?

Level 3 is real statistical knowledge, the kind that would survive a peer review.

In all honesty we probably won't get to level 3. Although I will try to arrange things so that we don't cherry-pick, I don't find it realistic that we will have the time to reach level 3.

There is a further problem. Suppose we do gather some good statistical evidence for the pro-cable side. Well, most of the "anti-cablers" would, justifiably, want to verify our protocol. After all, we did the test in private and we are reporting our results with no general verification. So they would want to repeat the test when they, or agreed neutral parties, are present and able to examine the test configuration to rule out cheating.

I really don't care enough to go that far. Well, it would be neat, but I have a full-time day job, and I just like to listen to music in my free time.

However, I respect your knowledge of scientific statistics, and I will consult you as necessary.

-Mike

 

[DanielCox]

There are some flaws and it would have to be repeated a lot to be of significance but at least you're doing it.

Save for any impedance difference or better shielding (both of which should not cost $700) I can't see any reason why the cable would be different.

 

[Bullseye]

Quote:

Originally Posted by moonboy403 /img/forum/go_quote.gif I was the one who helped Mike with the blind tests. We only did two trials because it's REALLY time consuming and tiring. Everytime sub-trail takes about 5 mins then after each subtrail, I need to switch all the cables in addition to giving 5 mins break for Mike to rest his ears. So by the time we complete one trial say ABAB, we've already spent a minimum of 35 mins.

Do you know that in those 5 minutes he is "resting his ears" he has just forgotten about how the test sounded? That is why an ABX switch is needed, because our ears are capable of making a distinction over a change of 1-2 seconds of duration.

The test taken that way makes it more of a "take your chances" game than a test.

Oh and if you want to keep doing the test that way, I am interested in the results anyways

, but bare in mind you might actually have the record of the longest tests in the audio tests

 

[moonboy403]

Quote:

Originally Posted by Bullseye /img/forum/go_quote.gif Do you know that in those 5 minutes he is "resting his ears" he has just forgotten about how the test sounded? That is why an ABX switch is needed, because our ears are capable of making a distinction over a change of 1-2 seconds of duration. The test taken that way makes it more of a "take your chances" game than a test. Oh and if you want to keep doing the test that way, I am interested in the results anyways , but bare in mind you might actually have the record of the longest tests in the audio tests

I don't know why you're making jokes out of my explaination of why Mike only did two trails. Moreover, I'm not here to prove or disprove anything but I'll definitely keep an open mind.

 

[Pio2001]

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif In all honesty we probably won't get to level 3. Although I will try to arrange things so that we don't cherry-pick, I don't find it realistic that we will have the time to reach level 3.

It all depends on how you design your trials. You talked about a total of 20 trials. If you spend 7 trials to look for the best configuration, 3 trials in order to improve your reliability, you may reach a score of 10/10 in the next trials, considered as "the real test". That would be completely significant. And if you make one mistake and get 9/10, it will still be quite significant for many readers.

By the way, if there is a significant difference in frequency response, you just have to measure it, and if it is big enough, it will prove that the the cables sound different. That would save you the need of doing blind tests.

This chart gives the audibility thresholds for frequency response deviations : ABX Amplitude vs. Frequency Matching Criteria

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif There is a further problem. Suppose we do gather some good statistical evidence for the pro-cable side. Well, most of the "anti-cablers" would, justifiably, want to verify our protocol. After all, we did the test in private and we are reporting our results with no general verification. So they would want to repeat the test when they, or agreed neutral parties, are present and able to examine the test configuration to rule out cheating.

Hey ! If they want to check, they take some cables, and spend 20 of their week-ends doing the test

!

 

[mike1127]

Quote:

Originally Posted by Bullseye /img/forum/go_quote.gif Do you know that in those 5 minutes he is "resting his ears" he has just forgotten about how the test sounded? That is why an ABX switch is needed, because our ears are capable of making a distinction over a change of 1-2 seconds of duration.

Hey, please address your comments to me and not Moonboy. He's doing me a big favor and is not on the hook to explain anything about this test.

I simply think you are wrong. I do not forget how something sounded after 5 minutes.

Explanation here:

http://www.head-fi.org/forums/f133/r...-sound-430023/

 

[XXII]

FWIW, I think that the Cardas high-end cables often sound worse than other cables on short listens because of a lack of "wow factor". It's only after extended listening that you appreciate the improvements.

 

[johnwmclean]

Quote:

Originally Posted by XXII /img/forum/go_quote.gif FWIW, I think that the Cardas high-end cables often sound worse than other cables on short listens because of a lack of "wow factor". It's only after extended listening that you appreciate the improvements.

I personally cannot A/B cables over a very short period of time. For me the best way is to leave a given cable in my system for up to 2 weeks, then do a change out.
This usually tells me a lot more about what I'm hearing, enabling me to use a much larger volume of musical genres for accessing.

 

[XXII]

Quote:

Originally Posted by johnwmclean /img/forum/go_quote.gif I personally cannot A/B cables over a very short period of time. For me the best way is to leave a given cable in my system for up to 2 weeks, then do a change out. This usually tells me a lot more about what I'm hearing, enabling me to use a much larger volume of musical genres for accessing.

I completely agree. Countless times I've swapped out my Cardas golden reference IC for something new only to go back to the Cardas after a few weeks. I've now promised myself not to bother trying anything new from now on

.

 

[Todd R]

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif Now, please go grind your axe somewhere else. You are getting flint and sparks all over my floor.

LOL

 

[Todd R]

Quote:

Originally Posted by XXII /img/forum/go_quote.gif I completely agree. Countless times I've swapped out my Cardas golden reference IC for something new only to go back to the Cardas after a few weeks. I've now promised myself not to bother trying anything new from now on .

Gee, that sounds familiar. I've done the same a few times and always return to Golden Reference.

There are some models in the Cardas line that have a mellow, softer sound Like the Cross & Quadlink for example, but I am a bit puzzled by Mike's experience with the Neutral reference.

I used Neutral Reference in the past and it is a very clean and open sounding cable, not at all rolled off or soft IMO. I wonder if there is something else going on?

In any case, I find that after using a cable for a while, you start to notice what it does for your system. Quick swaps sometimes don't tell the whole story.

Recently I was having a hard time telling the difference between an expensive digital cable from an el cheapo when switching quickly. Once I relaxed, listened to a few albums then changed it, I discovered what was different between them. (yes the expensive one was better)

 

[Bullseye]

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif Hey, please address your comments to me and not Moonboy. He's doing me a big favor and is not on the hook to explain anything about this test. I simply think you are wrong. I do not forget how something sounded after 5 minutes. Explanation here: http://www.head-fi.org/forums/f133/r...-sound-430023/

Well, let me rebate your answer with something you wrote here:

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif And because I'm listening to subtle musical and emotional effects in very rich music, I have a basic problem: I can't control in any strict sense what part of the music my attention goes to. Things "call to me." Sometimes I put my attention on something deliberately, sometimes not. [...] I acknowledge that a test track can sound different on each listen. I note that, and continue to listen.

If you can't control it, how can you remember it then? Next time you listen to the same song you will be putting your attention on other things, instead of getting back to what you "remember".

 

[mike1127]

We did two more blind trials with the ABAB/ABBA protocol. I only got one right, so my current score is 3/4 right.

I noticed a few things.

The first trial consisted of 4 sub-trials. A and B were randomly assigned to "good" or "bad" cables, and my helper randomly chose the sub-trial ordering as either ABAB or ABBA.

So the first sub-trial was A. (The first two sub-trials are always A and B, but I don't know which is the good or bad cable.) I felt the sound was dry and grainy. Then I listened to B. I was struck by a detail I hadn't noticed before, which was a slight pause in the piano... it was remarkably expressive. This detail somehow sailed right past me in A, but I really noticed it in B. My theory is that it's details like this which reveal the cable quality, and they are particularly noticeable in first impressions.

So I was guessing at this point that A was the Rat Shack and B was the Cardas. Then I listened to the third sub-trial. What was interesting was that I couldn't help noticing this detail in the piano. I wanted to be objective and try to observe it with the same "fresh mind" I used the first couple of times I heard this music, but it felt impossible to do so. I did end up feeling the third sub-trial was not a good sound, so I guessed ABAB and that B was the Cardas. In this case I was right.

I had a similar experience in the second trial... by the time I got to the third sub-trial (the third time I had heard the test track) I had lost all "freshness of mind." In the second trial I guessed the ordering wrong, but I still was right about the identity of A and B... in other words, in the first two sub-trials I guessed correctly which was the bad cable and which was the good... I just couldn't prove to detect the third and fourth sub-trials correctly.

At this point, I see "freshness of mind" as crucial. I want to change my procedure. I won't be doing an official test any more---just collecting information. One each trial, my helper will hook up A or B. I will listen to a unqiue test track. Different music every time. I will take notes about what I hear. Then at the end of a day of doing this, my helper will reveal the cable used in each trial. I will see if I consistently find good properties in the good cable and bad properties in the bad cable. Because I'm using different music with each trial, I'm no longer really comparing the cables under identical conditions. This may be a very bad way to run a test. Yet it will let me have "fresh ears" every time.

I will see what I learn.