Source does affect the sound quality. I used my V-80 with my Samsung Infuse 4G both before and after installing the Voodoo Sound mod, an iPod Nano 5th Gen. and my mom's phone, the Motorola Atrix 4G.
The Infuse 4G before Voodoo Sound sounded good, but after the mod, the sound got so much richer and full. Bass extension crept lower than before and overall musicality improved dramatically. By comparison the Atrix sounded distant, almost metallic, although the built-in EQ and DSP effects helped somewhat. The iPod sounded okay, but it seemed like there was something missing in the music. Plus, I couldn't run my FLAC files on it and I was severely spoiled by Voodoo Sound.
Running them off my on-board Realtek sound card I got weak bass response, even with EQ-ing, and needed the help of a cMoyBB v2.03 with bass boost turned on. Using the cMoy with my Infuse, I leave the bass boost off.
I've never used a Cowon or a Sansa Clip, so I don't know how those compare, but my phone makes music sound so good, I really don't want to upgrade. It'll probably become my PMP if I eventually get a new phone.