I'm fairly certain that the distortion from the source / amp will be at least a whole order of magnitude, possibly even two, lower than the distortion from the headphones. I think a similar topic to this appeared in the sound science forums a short while ago, and that was the conclusion. Typical amplifiers may have a THD + noise of 0.05%, whereas most headphones will have THD between about 0.5% and 1% in the midrange.
To the OP, I suggest you have a look through the resources page over at Inner Fidelity, where Tyll has put together various measurements for a fairly large range of headphones.