Here is my impression with my portable devices and PFE-232. I do realize that the latter needs an amp to shine, so maybe comparison with PFE-1xx should be taken with a pinch of salt.
Nokia Lumia 820 (EQ off):
'thin' sounding with narrow soundstage. FiiO E5 (bass off) makes vocals come out more and soundstage wider and is a step up. JDS Labs C421 (bass off, high gain) is definately more powerfull, rich and warm sounding, intimate and makes you feel like you're in it
Cowon S9 (EQ/BBE off):
Just the player by itself is already much better SQ wise than Nokia Lumia 820 + Fiio E5. Adding E5 to the S9 is even better than Nokia Lumia 820 + C421. When paired with the C421 the S9 sounds really great: full, smooth and detailed with lots of power and clarity in vocals, soundstage much wider than all of the above.
If you are stuck with a smartphone, a relatively cheap and little device like a FiiO E5 is a step up. A good portable - but more bulky - amp like the JDS Labsdefinately can give you much more detail and warmth to music coming from a (smart)phone. A digital audio player obviously takes the sound to a higher level and makes the Phonak Audeo a real pleasure to listen to. Moreover if you prefer EQ/BBE/Bass boost then a DAP is worth looking into.
Hope this helps, just my 2 cents