I recently invested in a portable amp - the Meier Audio CORDA STEPDANCE - to pair with my UM3X, also recently acquired. My source is the iphone 4. I don't know how the headphone out of the iphone 4 compares to the DAPs you mention, but I notice a significant improvement in sound quality when hooking the STEPDANCE up to the line out of the iphone to drive the UM3X. There is much greater clarity, detail, instrument separation, neutrality and better controlled, stronger bass. Having said that, until I got the STEPDANCE, I was happy with the headphone out of the iphone 4 and it is able to drive the UM3X to a very high volume.
If you're hooking an amp up to the headphone out of the DAP - because it doesn't have a line-out - I imagine the improvement may be less, as in effect you're just amping the output from headphone amp.
I was somewhat hesitant about getting the STEPDANCE as I was somewhat sceptical about how much of an improvement a portable amp could provide, but I'm very glad I took the plunge. Whether the improvement in sound justifies the entry price is another matter and is a purely subjective question only the person concerned can answer.