Sadly, I think you have to spend a lot more for all non-source components related to a speaker setup to get results comparable to what you get with a headphone setup.
Given the "right" listening room and audio setup / speakers, many (most?) people would prefer the speakers. Given real world limitations of room, family / neighbors / room-mates, and the cost of a speaker based setup, it's easier to build and enjoy a nice headphone setup.
As far as listening fatigue, I think that part of the fatigue with headphones is the "wrongness" of sounds that seem to be coming from inside of your head even with a high quality set of open headphones and say a cross-feed circuit in your headphone amp. Those of us who've used and loved headphones for ears have had our brains compensate somewhat but it's still hard to say that the right speaker setup is more fatiguing than headphones. I'm not even thinking about the physical stress on your head and your actual outer ear from the clamping action of headphones.
I think for a basic stereo setup that would compare to a really good headphone setup you'd have to spend at least $5000-$12000 new (some will argue about where that 'elbow' is where increased spending gives you rapidly diminishing returns,) whereas it would be reasonable to build a $1000-$2000 headphone setup that is sorta comparable, better in someways worse in others. Even at $500 you could have a simple but pretty killer rig if you really disciplined yourself on a single set of everything. I'm sure most of us in actuality have multiple sets of headphones, multiple sources etc. I know I've sunk a LOT more into learning about this hobby over the years than any individual one of my components would indicate. It's just easier to learn with headphone price components than high-end hifi components. ($5000 pair of Kharma speakers? Don't like em! Doh!)
With all that said I love my headphone setup and listen to it about 50% of the time, but I expect (hope) that will go down as I build my speaker based setup.
