Well, I am still learning about head-fi, but one thing I've learned so far - it's a different animal from 2-channel speaker systems.
I was in a similar place, years ago, and had an interesting e-mail exchange with Jason Lim from NuForce on the same topic (and even discussing a Yamaha receiver). Receivers and AVRs are just fine, and they are competent and very enjoyable with headphones (even HD 650s); but in my experience, even an entry level dedicated headphone amp will be ear-opening for someone coming from a receiver/AVR for headphone listening. That was very much the case for me and started me on the journey that is Head-Fi. For starters, the AVR headphone outs are noisy.
And don't kid yourself: it *is* a journey. You listen to different systems, maybe attend meets and hear a variety of rigs, and start to acclimate to sound characteristics that work for you.
I'm 50 years old and have been a stereo fiend since I was a kid (my Dad and step-father were both heavy audiophiles with expensive systems), but the head-fi experience has been one lesson after another, speaking just about myself.
And, yes, I have a 2-channel system (but head-fi is where I do my critical listening these days):