Totally agree. My 2-channel listening room probably cost me around $12-15K in gear and a few hundred hours in DIY tweaks and adjustments and experimentation to get to that "you're there" experience. This takes a lot of time, gear matching, room treatments, etc. to get there, but once it's dialed in it can be true magic. However, head-fi is so much easier (and relatively less expensive) and easier to take with you and/or use without bothering everyone else. Can't have kids around a mono-block tube amps that have 12 burning-hot tubes each side and as much as some headphones leak they don't rattle the house like a high-powered speaker and/or subwoofer can.
And while head-fi will get you very close to top 2-channel systems in terms of certain elements of SQ, it will never IMO give you the same immersive experience with the depth and width and height of true sound-stage. Head-fi in my experience plays in your head, and if you're lucky expands beyond your ears, while a well set up 2-channel system can transcend the walls of the listening room. In my system the image is such that the back of an orchestra extends well beyond the front wall and you can tell which violinist is sitting where in the recording hall. It truly is something to experience if you have never had the pleasure. If you go to something like RMAF be sure to venture beyond CanJam and into the listening rooms of the 2-channel crew (just be ready for some of them to look down their nose at you as headphone geeks).
Given all of the above trade-offs, it is telling that I've recently disassembled my 2-channel room and now listen exclusively to headphones. Maybe someday I'll go back and build a new listening room in my next house (for me building it was a good deal of the fun), but for now the headphone system is my audio-addiction.
Cheers