I used to prefer headphones, but my tastes are slowly drifting towards speakers.
The problem is that it's fairly easy to find listenable headphones, but devilishly hard to find listenable speakers in my opinion. After much experimentation, I've realized I dislike metal cone drivers (especially metal dome tweeters), ported speakers, speakers with inadequate baffle step compensation, single driver speakers, and speakers with an uneven power response. I would prefer higher efficiency speakers, but am willing to compromise on that. Price is rarely correlated with loudspeaker quality (there are many terrible $1000+ speakers), but good engineering generally is. People with weird theories (crossoverless, etc.) but no knowledge of the tradeoffs involved make it difficult for the novice to select good speakers.
Headphones are much easier. They all have a pretty uneven response, so you have to take your pick there, but as long as you avoid the known dogs, most headphones are at least moderately likeable.
Strangely, mid-fi or better integrated speaker amps are often cheaper than headphone amps in the same quality class, so I think it's fair to compare slightly more expensive speakers to less expensive headphones.