I'm going to go a little against the grain here. I use my Galaxy Note 3 with both lossless (Gonemad with FLAC) and streaming (Google Play Music All Access). It sounds good enough for me, and with the V4A module installed (and minimal amounts of hocus enabled), I am actually quite impressed. If I'm listening critically, the sound doesn't sound as good as my Leckerton Audio UHA-6S.mkii (but since I can use that as an external DAC / amp if I want to, that point is moot), but it's good enough for me to find enjoyment in the straight headphone out. Curiously, I don't even get his with my SE535s, and they're notorious for hissing on poor sources.
However, the Galaxy Note 3 sounds great with my SE535, SE215, SE846, Heir 8.A, and Noble K10 (admitting you have a problem is the first step...), enough so that I'm not looking for another dedicated DAP to complicate my life. I used to swear by my Cowon X5L and O2, but when both of those broke and I had my Note 3, I found that I didn't really miss them all that much. I certainly enjoy having a single device (with fast Internet wherever I go!), and I will take any downside that my choice may entail in order to simplify my experience. The great thing about the phone is I can just use the micro USB port and use an external DAC / amp if I so choose.
Besides, I only really use my phone when I'm out walking in the city or at the gym -- definitely not places where absolute sound quality matters. If I'm at home or at work, I'm going to be plugged into the the UHA-6S.mkii at a computer.
I think the key to all of this is just being happy with your choices. If absolutely reference quality audio is important to you while out on the go, I can definitely see the benefits of a nice DAP. But if you're like me and most of your listening is done by a computer, while the phone sees only occasional (and non-critical) use, you can certainly get by on just a phone.
To each their own, I say!