Yeah of course I wouldn't use them in public - I just like the idea of being able to move around the house with them - I can sit on the computer or out on the balcony or lay in bed or on the couch or whatever. I don't have a listening room or dedicated spot where I listen to music.
You're right - there are really a lot of factors to consider in terms of why you're hearing what you're hearing.
Personally, though, I believe most of the difference (provided you use a suitable amp pairing) is due to the headphones rather than anything else. I think I can safely assume if I had a $2k dac, a $2k amp, and a $2k pair of phones and upgraded just the DAC to a $6k DAC, I might hear a little bit of difference. If I upgraded just the amp to a $6k amp, I would hear quite a good improvement. If I upgraded just the headphones to a $6k pair of headphones I would hear a huge improvement. Obviously it's nice to have ALL great gear, but personally, I believe the biggest amount of difference can be attributed to the cans on the end. That's why I'm mostly considering a really expensive headphone over a really expensive dac or amp. But that's just me.