There's a lot of "mysticism" in the audiophile world. You could say that "audiophile" refers to the lover of the best, the purest sound, and that audiophiles, in being so focused on fidelity in music reproduction, love music even more than normal people with "unrefined" ears. Or you could say that "audiophile" refers to a fetishized love of headphone technology itself, where music perhaps takes second stage.
Here's a simple test: do you spend more time thinking about, enjoying, obsessing over musical elements (like a riff, a key change, an orchestration choice, a rhythmic shift, a vocal timbre), or more time thinking about, enjoying, obsessing over headphones themselves (can x, amp y, how you're not sure if you should keep can a or can b, etc.). Which set of thoughts and obsessions tends to orient you? Mostly likely it's a mix. Certainly, there is a point (like diminishing returns) where being involved in the second type of thoughts becomes excessive and even maybe retards further advancement in music itself, whether as a musician or a lover of music. I'm sure there are plenty of people (but I don't mean all audiphiles) whose love of music has transformed into a love of headphones and audio technology, and who 1) would not admit this, and 2) would tell any potential detractors or critics that they are simply unrefined aurally, aesthetically, morally, or however.
For me, I only get into headphone-thinking when I'm forced to, like when a pair breaks or I lose them. Inevitably, while I'm searching for a new go-to pair, I get into all the excessive detail. I go from this set of cans to another, unsure for a month or two. And I've noticed that during these times I tend to enjoy music less. I become so worried about something not being right in my set-up (which most of the time is just an ipod and headphones), and I forget what the original point was. I also play music less, think about music itself less. But then, once I get saturated with it and find equipment I like, I go back to the music and don't think about headphones again--usually for a couple years or more, until I need to get more for some reason. At that point, I'm back into listening to music, thinking about modulations, key changes, rhythmic figures, riffs, and racking my brain about this wonderful structure called music. And I'm happier there, much happier.