i think when we spoke before i probably mentioned mainly same things as as spoken here.
amping headphone out will depend on many things but will not necessarily mean poor performance.
headphone out get's unloaded so there should be little to no bass roll off when using an amp as it is not driving phones but merelyl a line level. nearly all portables, not some portables have bass roll off and some like the d2, have massive bass roll off where it is severely limited in lower frequencies but loses response quite quickly: among higher bass.
with an amp this should all but be eliminated.
same with stereo image; it will widen. it depends on you if it will be a big difference to you.
finally: hiss. if you use an amp with no volume control, it will likely mean you have to lower the gain on the protable: meaning the hiss from the portable will exponentially grow compared to the signal that the amp is eating up.
with an amp like the fiio, there are problems i see mainly stemming from hiss from the source. with a higher priced amp with volume pot, you can make the portable go 90% or higher volume (set it to its advantage) and lower the volume on the amp, thereby securing greater signal to noise ratio.
but: i hate using amps. too big and bulky and requires extra cabling, more things to break or suddenly move. volume pots in a trouser means this: sudden volume bursts.
i prefer er4p to s adapter but it has utility only with lower ohm phones and does not seem to affect armature based phones as much as dynammic phones (at least that is how it seems to me)