A large concern with portable players with many higher-end headphones is that they are constrained by battery power, size (and cost), so the max output voltage is limited. With headphones with low sensitivity (requires more power for a given volume level) and high impedance (requires more voltage for a given power level), they may not be loud enough with such a source.
Other sources may have issues with poor distortion, particularly with headphones with low impedance and especially if you're playing them at higher volumes (requiring more power and thus more current). It's harder to drive low impedance loads more accurately because they require more current. Well, it's more complicated than that, but that's a starting point.
Another factor to consider is the damping factor, which relates to the headphone impedance and source impedance. Also, it's important to note that many headphones--and especially some IEMs like balanced armature varieties--have impedance that changes quite a lot over frequency. This can have a big impact as well, particularly if the source has significant output impedance.