Most of you people are assuming there is a real gain change with the headphone amp. There is not a real hardware based gain change, only a software volume limiting change. The real gain of the headphone amp is 12db or a gain factor of 4X over the 0db gain level of the line out. The only thing that happens when you lower the gain is the driver limits to between 1-3 bits lower than the max output available. Each bit represents 6db. Dynamic range of the DAC is reduced by that amount with each "gain reduction". As such as long as you match the volume for each gain change the sound will be an exact match. If you increase the "gain" by 6db but reduce the volume control by 6db the sound will match exactly.
It is a myth that there is any sound change other than that that occures as a result of the ears sensitivity ( Fletcher-Munson curves ) with the gain changes. If the volume is matched there will be no difference in sound as there is no difference in the ears sensitivity if the volume is the same nor is there any difference in the frequency response of the amp at the difference in "gain" as there is no real difference in actual gain of the amp itself.