Quite a few Android phones now have good audio quality, as manufacturers realise it's something people want in the top handsets.
My Sony Xperia S sounds better than my old Creative Zen Vision:M, especially with Neutron Music Player or Poweramp. I noticed a marked improvement in detail when I started using the Xperia. In fact, the audio output drives my Hifiman RE-262 earphones, which are said to 'need' an amp, very well (there's a slight improvement with an amp, but I mean slight).
So yes, it is worth getting decent headphones for the best audio experience.