bangraman
Headphoneus Supremus
- Joined
- Oct 3, 2002
- Posts
- 10,305
- Likes
- 65
A while ago I realised my opinions of sources changed depending on how I was listening to them. This came to the fore especially in meets when I listened to things, liked X but on buying it discovered something different. Obviously this made my opinions of any source rather unreliable. Portables are more of a bone of contention as far as opinions go because more people are more likely to talk about a similar group of machines. And I think too often we engage in pointless debates about which is better without having any real basis of comparison. Given all that, I set out to make my own opinions a bit more consistent.
Then it occurred to me a relatively short time ago that indeed we do spend an awful lot of time arguing about which is better than which, but we spend very little time arguing about how we decide which is better than which. I'm hoping this thread brings up some of those arguments.
Written below is what I’ve been doing since around three years ago every time I say "X is better than Y". It's very interesting how a direct, totally comparable test de-hypes sources for me. On the other hand, this method especially with multiple sources is very labour-intensive. You can probably understand how I get fed up of doing tests. But to me, I've come to the conclusion that if I don’t do it something like this the test is not worth doing, as any opinion I’ve arrived at otherwise is largely worthless because it’s uncontrolled. So I'm outlining how I do comparative listening of portables. I may have put it in elsewhere in more detail a while back, this is a shortened one.
I'm interested in how others do listening tests with any degree of accountability or control, as well as if you can pick any methodology holes in my comparison method. The important thing to me to establish here was that even before you get to the 'human / personal preference' portion of the comparison which really can't be controlled, there are many other external, perfectly controllable factors which can definitively influence results. The point of the testing methodology as I see it is to reduce down to the most controlled environment possible given personal tastes.
List your testing methods and why you arrived at them here, and perhaps we can arrive at an easily reproducible way to allow people to compare sources they have to give consistency to the results.
================================================== ========
Bits needed
To reproduce my comparison setup using the most to-hand parts, you'll need (apart from the players of course):
- Test songs. Need not be songs you like, but needs to be varied.
- Audio switch. A decent one, it needn't be expensive. 2:1, 3:1, etc. Whatever works for you.
- A connector / converter to allow the headphones you’ll be using to be plugged into the audio switch’s output.
- A two-way headphone splitter. Make sure it's properly wired.
- two or more interconnect cables which suit your audio switch, with a 3.5mm plug at the other end. They should be identical products purchased at the same time from the same supplier, and if you're extra-paranoid like me you'll measure them to make sure they're identical. Candidates if you have an RCA-plugged audio switch would be something like the Zu Pivot. Also ideally, they’ll be a reasonable length – 1m to 2m. Reason for this later.
- A headphone. Ideally a variety to highlight behaviour under various loads but the optimal would be two - a quality higher-end, easily-driven phone of middling impedance as well as a fairly low-cost, low-impedance earphone for judging the 'consumer' comparison.
- Rightmark Audio Analyzer, Soundcard + PC + cable to go from headphone splitter to sound card.
Setting the reference volume level
On one player, select one of your test songs that represents generally the highest volume level that you'll experience in your tests with each phone you'll be using. Using each phone you’ll be using, set the player to a comfortable listening level using the test song. Then turn it up one notch above that, and make a note of the volume.
Now - if you intend to do multiple tests at a later date, you'll do this on a player you intend to hang on for a long time. For the reason of the vague volume control, I’d recommend that this player NOT be an iPod.
Equalising volumes
This is the most crucial part of the prep. Why? I did a test a while back where I asked someone to listen to two different sources, unseen. I asked him which was better, he preferred one by a definitive margin. I revealed the sources as two identical players, one set with a higher volume which he preferred. I can't think of a better way to illustrate why it's vital to set volumes the same across all players being listened to.
First, start up RMAA and select the option to generate test tones. Take ONLY the calibration tone, and load it onto each of the players. I’d suggest you enable track repeat on each. Connect the player to the soundcard by using the headphone splitter and having the phones in one end, and the other cable going into the soundcard. This loads the circuit between the soundcard and the player with the impedance of the phone you're using, and serves as a check to make sure that we're equalising the volumes on a headphone-loaded system. This makes work slightly harder for the soundcard but it is a better representation of real life.
Now take your player with the reference volume set, and start playing the calibration tone into RMAA. Run the recording test. For this exercise, we’re not actually running the test – all you need is the level calibration stage. If (probably) necessary, adjust the soundcard gain to bring the level up to somewhere around the green level. Make a note of the level you have showing on RMAA. Do not, under any circumstances, change the volume on the player. Disconnect the player and attach the next one. On this one, adjust the player volume until it is as close to the level of the reference player. The matched level should definitely be less than 1db. Depending on the level you decided on originally, a >1db mismatch is possible due to some players having volume steps in 2db, while others have 1db increments. If you can’t get such a match, you may need to adjust the reference portable volume slightly until you can achieve a <1db match with all the portables being tested and repeat the above. This can be very labour intensive but if you’re aiming for dependable results you have to do it. Then you’ll repeat the above with any other headphones you may be using. The last bit’s not compulsory, but I do it anyway. Note down all the player volumes as you start matching them.
Playback test
Take all your players, and make sure you have the same test songs cued. Make sure the volume is at the above-matched levels. Wire up the players and headphones to the audio switch. I’d suggest no more than three at once, even if your audio switch has room for more and you’re testing more players. All the better if you can do this blind by feel. You can always adjust left/right if you’re using RCA at one end later. Double-check volumes on each player. And start playback.
I’d suggest not switching more than three times in the course of an entire track between two players, and not more than five times if comparing three. Really listen and get a feel of the overall sound, then switch. If necessary, you can compare specific bands / instruments, etc later. If you need to rewind the track, I would suggest disconnecting the headphone so you don’t immediately find out which is which.
Write up your notes before tracing out the cables. I said before the cables should be 1-2m long, and this is so that to create a bit of a tangle around the audio switch and players to confuse any casual attempts to trace out the cables by eye. Basically, anything to avoid ready identification of which source goes into which input on the audio switch.
Really, that’s pretty much it. The key points I place the most emphasis on would be:
+ Match volumes on the players that are being compared as closely as possible. This is not possible by ear unless you’re very good – and if you think you are, you probably aren’t.
+ Use a switch to allow quick – if not necessarily immediate – switching between sources. Plugging and unplugging, especially when a recalcitrant connector surfaces and delays reconnection time, does cloud the audio memory.
Edit: I had a feeling this will sink straight down the pages
Edit2: A PM raised the very valid point, why don't I set the volume with white noise instead of a tone as used by the RMAA calibration sequence? The white noise would distribute the levels across the entire frequency range and not just to a specific tone, eliminating any potential discrepancy if players offer notably different frequency responses across the audible range.
The answer is that yes in theory it is a more valid method, but I've found that the vast majority of modern players have variability in the audible frequency range of less than the volume stepping that the player offers, so you could use either method and it would be just as valid.
Then it occurred to me a relatively short time ago that indeed we do spend an awful lot of time arguing about which is better than which, but we spend very little time arguing about how we decide which is better than which. I'm hoping this thread brings up some of those arguments.
Written below is what I’ve been doing since around three years ago every time I say "X is better than Y". It's very interesting how a direct, totally comparable test de-hypes sources for me. On the other hand, this method especially with multiple sources is very labour-intensive. You can probably understand how I get fed up of doing tests. But to me, I've come to the conclusion that if I don’t do it something like this the test is not worth doing, as any opinion I’ve arrived at otherwise is largely worthless because it’s uncontrolled. So I'm outlining how I do comparative listening of portables. I may have put it in elsewhere in more detail a while back, this is a shortened one.
I'm interested in how others do listening tests with any degree of accountability or control, as well as if you can pick any methodology holes in my comparison method. The important thing to me to establish here was that even before you get to the 'human / personal preference' portion of the comparison which really can't be controlled, there are many other external, perfectly controllable factors which can definitively influence results. The point of the testing methodology as I see it is to reduce down to the most controlled environment possible given personal tastes.
List your testing methods and why you arrived at them here, and perhaps we can arrive at an easily reproducible way to allow people to compare sources they have to give consistency to the results.
================================================== ========
Bits needed
To reproduce my comparison setup using the most to-hand parts, you'll need (apart from the players of course):
- Test songs. Need not be songs you like, but needs to be varied.
- Audio switch. A decent one, it needn't be expensive. 2:1, 3:1, etc. Whatever works for you.
- A connector / converter to allow the headphones you’ll be using to be plugged into the audio switch’s output.
- A two-way headphone splitter. Make sure it's properly wired.
- two or more interconnect cables which suit your audio switch, with a 3.5mm plug at the other end. They should be identical products purchased at the same time from the same supplier, and if you're extra-paranoid like me you'll measure them to make sure they're identical. Candidates if you have an RCA-plugged audio switch would be something like the Zu Pivot. Also ideally, they’ll be a reasonable length – 1m to 2m. Reason for this later.
- A headphone. Ideally a variety to highlight behaviour under various loads but the optimal would be two - a quality higher-end, easily-driven phone of middling impedance as well as a fairly low-cost, low-impedance earphone for judging the 'consumer' comparison.
- Rightmark Audio Analyzer, Soundcard + PC + cable to go from headphone splitter to sound card.
Setting the reference volume level
On one player, select one of your test songs that represents generally the highest volume level that you'll experience in your tests with each phone you'll be using. Using each phone you’ll be using, set the player to a comfortable listening level using the test song. Then turn it up one notch above that, and make a note of the volume.
Now - if you intend to do multiple tests at a later date, you'll do this on a player you intend to hang on for a long time. For the reason of the vague volume control, I’d recommend that this player NOT be an iPod.
Equalising volumes
This is the most crucial part of the prep. Why? I did a test a while back where I asked someone to listen to two different sources, unseen. I asked him which was better, he preferred one by a definitive margin. I revealed the sources as two identical players, one set with a higher volume which he preferred. I can't think of a better way to illustrate why it's vital to set volumes the same across all players being listened to.
First, start up RMAA and select the option to generate test tones. Take ONLY the calibration tone, and load it onto each of the players. I’d suggest you enable track repeat on each. Connect the player to the soundcard by using the headphone splitter and having the phones in one end, and the other cable going into the soundcard. This loads the circuit between the soundcard and the player with the impedance of the phone you're using, and serves as a check to make sure that we're equalising the volumes on a headphone-loaded system. This makes work slightly harder for the soundcard but it is a better representation of real life.
Now take your player with the reference volume set, and start playing the calibration tone into RMAA. Run the recording test. For this exercise, we’re not actually running the test – all you need is the level calibration stage. If (probably) necessary, adjust the soundcard gain to bring the level up to somewhere around the green level. Make a note of the level you have showing on RMAA. Do not, under any circumstances, change the volume on the player. Disconnect the player and attach the next one. On this one, adjust the player volume until it is as close to the level of the reference player. The matched level should definitely be less than 1db. Depending on the level you decided on originally, a >1db mismatch is possible due to some players having volume steps in 2db, while others have 1db increments. If you can’t get such a match, you may need to adjust the reference portable volume slightly until you can achieve a <1db match with all the portables being tested and repeat the above. This can be very labour intensive but if you’re aiming for dependable results you have to do it. Then you’ll repeat the above with any other headphones you may be using. The last bit’s not compulsory, but I do it anyway. Note down all the player volumes as you start matching them.
Playback test
Take all your players, and make sure you have the same test songs cued. Make sure the volume is at the above-matched levels. Wire up the players and headphones to the audio switch. I’d suggest no more than three at once, even if your audio switch has room for more and you’re testing more players. All the better if you can do this blind by feel. You can always adjust left/right if you’re using RCA at one end later. Double-check volumes on each player. And start playback.
I’d suggest not switching more than three times in the course of an entire track between two players, and not more than five times if comparing three. Really listen and get a feel of the overall sound, then switch. If necessary, you can compare specific bands / instruments, etc later. If you need to rewind the track, I would suggest disconnecting the headphone so you don’t immediately find out which is which.
Write up your notes before tracing out the cables. I said before the cables should be 1-2m long, and this is so that to create a bit of a tangle around the audio switch and players to confuse any casual attempts to trace out the cables by eye. Basically, anything to avoid ready identification of which source goes into which input on the audio switch.
Really, that’s pretty much it. The key points I place the most emphasis on would be:
+ Match volumes on the players that are being compared as closely as possible. This is not possible by ear unless you’re very good – and if you think you are, you probably aren’t.
+ Use a switch to allow quick – if not necessarily immediate – switching between sources. Plugging and unplugging, especially when a recalcitrant connector surfaces and delays reconnection time, does cloud the audio memory.
Edit: I had a feeling this will sink straight down the pages
Edit2: A PM raised the very valid point, why don't I set the volume with white noise instead of a tone as used by the RMAA calibration sequence? The white noise would distribute the levels across the entire frequency range and not just to a specific tone, eliminating any potential discrepancy if players offer notably different frequency responses across the audible range.
The answer is that yes in theory it is a more valid method, but I've found that the vast majority of modern players have variability in the audible frequency range of less than the volume stepping that the player offers, so you could use either method and it would be just as valid.
