I wish it were otherwise, but the fact is that I spend more time reading about hifi than listening to it. Its not how it ought to be but thats how it has become. Upgrading has taught me one thing, that the higher you go the more you know about whats wrong with what you have. Its like the nature of knowledge in general, the more you understand a subject, the more you realize how much your understanding is subjective and biased. I know a lot more of whats missing in my kit now than when I began 25 odd years ago. Listening to music-with my humbly-rumbly HMV record player- was then an intoxicating pleasure, now it is an anlytical excercise mostly. Its not simply a matter of being jaded, but in the nature of the game itself, the foreplay takes centrality over the act. I am not even sure if foreplay is the right word for that at least involves a relationship, headphones especially, in someways, eptomize the individuals isolation from music as a shared pleasure.
So to answer your question, Head-fi over Hi-fi -any day!