John,
Actually, I do both.
I actually have a couple of CDs and MDs that I have made specifically for the purpose of comparing headphones.
Assuming the phones are broken in, my procedure is usually something like this:
I listen to the new phones I want to compare for quite some time, so that I can eliminate the “new sound” phenomenon that can sometimes cloud my judgment either positively or negatively. Furthermore, I tend to believe in “brain training,” that is, I believe that the brain itself gets remodeled by experience in general, and by listening to sound reproduction devices (i.e. headphones) with a particular sound signature. After I’ve listened for at least a good week or two, I sit down to do the comparison.
The order of these steps below is totally interchangeable:
1. I listen to a song or two in a particular genre, then listen again with the other phones. I go through all the major genres with each phone. I also tend to reverse the order in which I switch between phones. I find that it can sometimes give a false sense of preference if I always go to a particular one second, so I try to vary them.
2. Then I go back and listen to specific passages between phones.
3. I try to listen in this way, comparing songs, bits, sounds, etc., for a few days to a few weeks before I write anything down, getting an overall sense of what differences I really hear, since sometimes the differences I hear at the beginning of a comparison don’t seem as present at the end. Also, I tend to do many sessions to see if the difference I heard on session 1 is still present on session 5.
4. I tend to listen to whole songs or whole CDs when I’m trying to decide about the overall sound characteristics.
Then, finally, I sit down to write, put on the songs that I think are best representative of the differences between them, and start writing.
So, that’s the basics of my method.
