Since no one else has answered;
This frequency dependent phase shift has been argued to death in various forums thru the years.
To avoid this type of phase shift, you would us an amp with widest possible bandwidth, for example, -1.0 db @ 100 kHz would lead to very little high fequency phase shift in the audio bandwidth.
One thing to keep in mind: distortion, phase shift, frequency deviations,etc. are all cumulative.
For example, if you have a DAC which is 0.3 dB down at 10 kHz and an amp which is down 1.2 dB at 10 kHz then when you use them together you are 1.5 dB at 10 kHz.
Same goes for distortion, when you are listening to headphones & amp & DAC combination you are listening to the cumulative distortion of all three pieces. You could argue that the headphones are distorting the distortion from the amp which is distorting the distortion from the DAC!
Regarding the noise floor thing (I'm not referring to hiss), anyone notice their sound system sounds better at 3:00 a.m?