Blind cable testing: initial report
Jun 22, 2009 at 11:37 AM Post #16 of 128
Sad that Bullseye feels the need to crap this interesting thread. I suggest to people that they just ignore and not reply to thread-crappers.

I've read about Cardas cables resulting in a soft sound when used, so the comments are interesting.
 
Jun 22, 2009 at 11:46 AM Post #17 of 128
Quote:

Originally Posted by Pio2001 /img/forum/go_quote.gif
Since Mike designed his protocol, here : http://www.head-fi.org/forums/f133/c...tocols-429868/


Then the tests becomes invalid again. With only two options, 50% chance you can always be lucky enough to get both correct.

Quote:

That's just rigorous testing. When you test a hypothesis, you should not assume it to be true or false before running the test.
A switch is actually a bit of interconnect. Assuming it to be transparent is assuming that the hypothesis under test is true.


Did you check the link I posted about that ABX switch?

Quote:

Since we must assume that interconnect cable behaviour is unknown, we can't assume it to be cumulative. The effects of shielding, for example, are not cumulative.


Read up. If the switch is well desgined there is no need to assume anything.
 
Jun 22, 2009 at 11:52 AM Post #18 of 128
Quote:

Originally Posted by Currawong /img/forum/go_quote.gif
I've read about Cardas cables resulting in a soft sound when used, so the comments are interesting.


Some people say Cardas have a gentle warm glow to them. But never mind the Rat Shack--the Neutral Ref is rolled-off compared to its little brother, the Twinlink. My best theory is that it's not broken-in. I got it from Audiogon, and that guy got it used from a Cardas dealer and barely used it, so no one can say how much use it has seen.

-Mike
 
Jun 22, 2009 at 11:55 AM Post #19 of 128
Quote:

Originally Posted by Bullseye /img/forum/go_quote.gif
Then the tests becomes invalid again. With only two options, 50% chance you can always be lucky enough to get both correct.



If you had read the OP, you would see that it was presented as initial and statistically insignificant. By the way, the number you seek is 25%. Now, please go grind your axe somewhere else. You are getting flint and sparks all over my floor.
 
Jun 22, 2009 at 12:07 PM Post #20 of 128
Quote:

Originally Posted by Pio2001 /img/forum/go_quote.gif
Thank you very much for the report. I hope that you will have the possibility to go on.

Since you already know if your answers are right or wrong, you must have decided in advance the total number of trials. How much are you going to do



Nothing has been decided. I need some experience with the protocol in order to have more confidence I've found the right one. One choice to consider in the future is the choice of music. In the two trials today, I used very different kinds of music---exciting brass music in trial one, and chamber/vocal music in trial two. While this did not affect my ability to distinguish the cables, it did affect which one I preferred. I am considering that the chamber/vocal music may be superior for future tests, because "getting it right" requires a certain delicacy and resolution. In contrast, the brass music was quite exciting no matter what cable was used.

And I have to be realistic---I don't know how many trials I can ultimately do. Moonboy has the summer off, so we have a few months to try to get something accomplished---but I work full-time and dont' want to do more than a couple trials per day, at least until I'm more sure how my critical faculties function under these conditions.

I already checked my answers because, given the trouble this is, I want to make sure I'm not wasting my time before going too far down this road.

So my first hope is to do enough trials to convince myself that I'm hearing something. For example, if I get 5/6 correct and feel pretty confident I knew what I was hearing, that will be good news for me. After that, then the hope would be to do enough trials to get a statistically significant result, at least 20.
 
Jun 22, 2009 at 12:08 PM Post #21 of 128
I have to agree with Bullseye on this one. The fact that he had only two options means that the test really is invalid. I mean, the OP admitted that his data was "statistically insignificant" but I think you could just call it insignificant overall. I'm not debating whether or not there is any difference in audio quality between the two cable setups: rather, I'm saying that we can't honestly conclude, from this test, whether or not there is any audible difference at all.
Quote:

I've read about Cardas cables resulting in a soft sound when used,


Thats great that you've read that but really unless you have pernsonal experience with the cable under discussion, I have to question the usefulness of such a comment. Assume, for example, that one source made the statement that Cardas cables result in a soft sound: if everyone took that one source's word for it, then there would be a widespread belief that Cordas cables result in a soft sound, when really that's only one persons opinion.

Please keep in mind that I do not intend to "thread-crap", as you so colorfully put it, but I'm questioning the amount of weight that you guys seem to have put in this test. It's great that the OP did this test and decided to share it with us, but I think we should all accept the fact that the data obtained really is irrelevant to anyone other than himself.
 
Jun 22, 2009 at 12:20 PM Post #22 of 128
Quote:

Originally Posted by ezzieyguywuf /img/forum/go_quote.gif
I have to agree with Bullseye on this one. The fact that he had only two options means that the test really is invalid. I mean, the OP admitted that his data was "statistically insignificant" but I think you could just call it insignificant overall.


Are you aware that each trial of an ABX test also "has only two options"? Does that make ABX testing invalid?

Quote:

Please keep in mind that I do not intend to "thread-crap", as you so colorfully put it, but I'm questioning the amount of weight that you guys seem to have put in this test. It's great that the OP did this test and decided to share it with us, but I think we should all accept the fact that the data obtained really is irrelevant to anyone other than himself.


Look, on this forum typically we have a divide---people who think quick-switch is the only valid test, and we have people who don't give a crap about testing at all. I think what has people interested is that I'm bridging the gap... I'm actually doing the freaking test. And if I keep it up, it may become statistically significant. I don't think a single person on this post has trumpeted this as proof of anything, but a few of us are glad that an experimental non-quick-switched test is underway.
 
Jun 22, 2009 at 12:28 PM Post #23 of 128
Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif
If you had read the OP, you would see that it was presented as initial and statistically insignificant. By the way, the number you seek is 25%. Now, please go grind your axe somewhere else. You are getting flint and sparks all over my floor.


That is incorrect, because unless I understood it wrong, you mentioned that the last answer you had to give to the tester was either ABAB or ABBA, no?

If that is the case you could always be guessing, and even more if the test was only done twice.

Then if you mention it was "statistically insignificant" why even write it? I mean, being that the case, you are in the end giving us another subjective opinion, nothing else. The conclusion you write or what you "learned" is really over nothing, you see where I am going?

That is why I said this write up is INVALID from the point of view of a test.
 
Jun 22, 2009 at 12:39 PM Post #24 of 128
Quote:

Originally Posted by Bullseye /img/forum/go_quote.gif
That is incorrect, because unless I understood it wrong, you mentioned that the last answer you had to give to the tester was either ABAB or ABBA, no?

If that is the case you could always be guessing, and even more if the test was only done twice.

Then if you mention it was "statistically insignificant" why even write it? I mean, being that the case, you are in the end giving us another subjective opinion, nothing else. The conclusion you write or what you "learned" is really over nothing, you see where I am going?

That is why I said this write up is INVALID from the point of view of a test.



You guys from the opposite camp, don't be so upset by the «positive» result. As mentioned by the OP, it's very provisional and not significant at this stage.

In contrast to you I think such a test should be configured as convenient to the test person as possible and in a way to create the highest likelihood to generate a positive result.

It is absolutely valid. Go on, Mike!
.
 
Jun 22, 2009 at 12:57 PM Post #25 of 128
Well, I believe, Jazz, that if you are going to do something, do it good.

Of course the test should be configured as convenient to the test person as possible, but following some schemes that will make it valid.
 
Jun 22, 2009 at 1:00 PM Post #26 of 128
Quote:

Originally Posted by Bullseye /img/forum/go_quote.gif
...following some schemes that will make it valid.


I agree.
smile.gif

.
 
Jun 22, 2009 at 4:14 PM Post #27 of 128
Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif
Before we continue, we may need to consider the possibility of some relatively large FR difference between these cables.


This is pretty unlikely from my empirical testing experience.

Can I humbly suggest that you measure the frequency reponses for the two cables yourself. I can lend you one of my USB recording devices if you wish, just shoot me a PM.

I have tested several cables of differing types from 77c to $139 of massively different designs and materials, the big FR differences you hypothesize about I have just not found. I would be genuinely interested to see if you do find any real differences.

Best approach is white noise as each frequency is more or less equally represented and you can easily see if there is an appreciable lift or drop in any range,
 
Jun 22, 2009 at 4:17 PM Post #28 of 128
Quote:

Are you aware that each trial of an ABX test also "has only two options"? Does that make ABX testing invalid?


In your case you were given two configuration possibilities and asked to choose one. You could have said ABBA without even having listened to anything and still had a 50% chance of getting it right. In order for that test to be valid (and I know this would have been time consuming, and hence you didnt't do it), you would have had to do the test, say, 10 times and each time had the option between ABAB and ABBA. Now we can see a percentage of times that you were able to tell them apart and we have some good data. That is how ABX works, to the best of my knowledge. You are posed with the question of "Which one is it" multiple times, so that instead of it being a question of whether or not you can guess the right one out of a possible two, its a question of how consistently you can do it. The consistency it what lets you say "Yes, he was actually, statistically able to differentiate between the two audibly. It was not a fluke that he was able to tell them apart."
 
Jun 22, 2009 at 4:23 PM Post #29 of 128
Quote:

Originally Posted by Bullseye /img/forum/go_quote.gif
That is incorrect, because unless I understood it wrong, you mentioned that the last answer you had to give to the tester was either ABAB or ABBA, no?


You wrote that my chance of getting both results right by chance was 50%. Actually each test had a 50% chance of guessing, so the chance of getting both right by guessing is 25%.

Quote:

Then if you mention it was "statistically insignificant" why even write it?

I mean, being that the case, you are in the end giving us another subjective opinion, nothing else. The conclusion you write or what you "learned" is really over nothing, you see where I am going?

That is why I said this write up is INVALID from the point of view of a test.


I wrote it because this particular experience moves me (and anyone who wants to build on this) closer to a valid, casual-listening test protocol. As I have said many times, I don't believe that it is valid to treat listeners as "black boxes." Rather, we have to know what's going on in their heads and have some controls on the use of their attention. So I was experimenting with the use of my attention. With the type of music and number of different tracks. With the length of the test.

It's even interesting (except to a confirmed cable disbeliever) what I experienced when I listened to a cable blinded in a casual (non quick-switched) setting because even that simple thing does not happen often, or at least is not reported often.
 
Jun 22, 2009 at 4:29 PM Post #30 of 128
Quote:

Originally Posted by ezzieyguywuf /img/forum/go_quote.gif
In your case you were given two configuration possibilities and asked to choose one. You could have said ABBA without even having listened to anything and still had a 50% chance of getting it right. In order for that test to be valid (and I know this would have been time consuming, and hence you didnt't do it), you would have had to do the test, say, 10 times and each time had the option between ABAB and ABBA. Now we can see a percentage of times that you were able to tell them apart and we have some good data. That is how ABX works, to the best of my knowledge. You are posed with the question of "Which one is it" multiple times, so that instead of it being a question of whether or not you can guess the right one out of a possible two, its a question of how consistently you can do it. The consistency it what lets you say "Yes, he was actually, statistically able to differentiate between the two audibly. It was not a fluke that he was able to tell them apart."


You are basically right but you seemed to be saying something else in your first post. You seemed to be talking about a single trial when you said there was a 50% chance of guessing right. In an ABX test single trial, you also have a 50% chance of guessing right. The number of trials is not a concept inherent to ABX testing, but to DBT in a more general sense. You can calculate a level of significance. For example if I do 16 trials and get 12 right, there is a level of significance of 5%... that is, only a 5% chance I was just guessing the whole time.

So we did the first two trials on Sunday. More trials will follow. We may change the protocol as we learn what seems to be more sensitive. Strictly speaking, we should only count trials that were done with the same protocol, but as this is a learning experience we may have to change it several times. Our accuracy may improve with time.

All this means that the chance of getting a statistically valid result by the end of the summer is small, but I still think it's worth trying.

-Mike
 

Users who are viewing this thread

Back
Top