Sorry, I just used TT2 as a simple example. Only diff between DAC and Amp mode is fixed vs variable output. DAC mode is 4 V (or maybe 4.5). Manual specifically mentions using Amp mode when paired with equipment (amps or powered speakers) which needs higher input levels.
So, to rephrase, consider two hypothetical DACs with approximately equal quality, one of which outputs 4V and the other which outputs 12V. Will the lower output of the one DAC negatively impact sound quality, assuming output to cans for which the HPA4 still has sufficient gain.