Impossible to answer this question without a huge number of caveats and considerations.
Given the financial impossibility of me duplicating the quality of sound I get from my modest (and increasingly "compromised" by my increasing standards) SR80s with my regular speaker setup, I do most of my serious listening with phones. And prefer it.
Another consideration, and one that is just as (if not more) important, is the relative disposition of my lovely, beautiful wife, who will brutally castigate me for serious listening on the speakers but who is mysteriously silent when my headphones are on.
Ditto my delightful children.
That said, I would RATHER be doing serious listening on my regular speakers because delights of headphones notwithstanding, it's just easier to love the music when I don't have steel, plastic, and foam clamped to my head.