This is quite the opposite. Audiophilia is about the music, not about the sound quality. Yeah, I prefer FLAC over 192 kbps MP3, but if I only have a 192 kbps MP3 of a song I'll still listen to it.
Additionally, while I certainly prefer my headphones with an amp, not being able to use my amp won't make me stop listening. My LOD cable broke (yeah ino, shouldn't have gotten the cheap one) so I'm SOL when it comes to using my E7 with my iPhone.