I have moved almost entirely from listening on speakers to listening with headphones. This is for my periods of attentive listening. This change from speakers to headphones happened for both practical reasons and because my musical choices changed.
I lived in London for many years and I bought a small flat ("apartment" in US) which had very little sound insulation. I really liked the other people in the building and I did not want to upset them with the sound of music coming from my flat. So I started using headphones. In fact I was using Sennheiser HD540 Ref Golds which I had in my possession. This put me into the mental habit of using headphones to "really listen" to music. Over that period in London my musical tastes changed. I think this is mostly due to age but may also have been influenced by my headphone use. I gradually stopped listening to anything from the popular genres. I had always listened to classical as well as the popular genres, but over that time I started listening to classical entirely and indeed today I only listen to classical. I also started listening to chamber music much more than big orchestral works. The late romantic composers interest me much less and now I find I am seeking out works for chamber groups and small orchestras.
Today I live in a very small detached house, so there is no need for me to worry about disturbing neighbours, however I still use headphones because they are so revealing.
I think that if you like the way in which headphones present music, then there are very good technical reasons why they will be much more revealing than conventional speakers. Headphones use single drivers so there is no crossover. Conventional speakers, of course, have crossovers most usually and it is my belief that this is often very compromising to good quality reproduction. Headphones are, obviously, much easier to drive than speakers and so for low money you can give your headphones very good amplification, fully class A operation with no negative feedback is easily obtainable. Most of all, I think, listening to conventional speakers is hugely dependant on room acoustics and the average living room is very unsuitable for listening to music compared with a good concert hall.
I do think that for using headphones for attentive listening a crossfeed is really a very useful thing. I "discovered" crossfeed when I got my Meier Audio StageDAC (which comes with it). I hadn't really thought about this before, but I have now found that the crossfeed, for me, greatly improves the listening experience. The crossfeed creates a much more realistic stereo image. It tends to move sounds away from being really in your ears instead they are now placed within the stereo image.