As mentioned before, it isn't necessarily a matter of the amp amplifying. Rather it is an issue of impedance interactions. Because getting a low output impedance tends to be more expensive (and not just in dollars) mass market sources will tend to have higher impedances than dedicated 'amps'. The dedicated amps will provide a high input impedance to reduce the effect of the source's high output impedance as well as providing their lower output impedance to reduce the interaction with the low impedance of the actual headphone drivers.
That is the electrical science of it, Proton007. And no, I have never used a headphone amp so it is not based on some wish to believe my money was well spent. Of course, how much a particular amp improves the accuracy (if at all) depends on the exact set of source, amp, and headphones.