HI, I come from a different perspective. For a long time I used the Hugo straight, because it made sense to me that additional amplification would only degrade the signal. The thought was that you could add an amp for pleasing color/distortion. However, you could not improve on the sound quality; only alter it to taste.
I think this approach may be misleading. The Hugo's output may top out for IEMs but more can be done for full-size phones. If this weren't the case, we would not have such things as small portable headphone amps. They would just be called headphone amps, built in the space required. The raft of full-size components that are employed in full-size headphone amps are doing more than providing an alternate frequency response or euphonic harmonics.
Compared to portables, including the Hugo, full-size amps improve on low-level detail, soundstage dimensions, imaging accuracy, dynamic range, etc., i.e., they improve on performance. The handling of voltage swings that translate into nimble dynamics that translate into realism of sound require certain components and design. A good full-size amp improves sound quality in ways you cannot expect from the Hugo. That is to say, there is a reason the Hugo TT exists.
I should qualify, I cannot speak for entry-level to mid-tier full-size amps. In that case the tradeoffs might very well be a wash. However, the misleading part of the Hugo-only principle is that there is no way up by addition. I'm currently using the excellent Moon Neo 430HA and it isn't just a different sound; it is better.
So, though you are losing accuracy to the Hugo DAC signal, it should be pointed out that you are gaining accuracy in regards to the source recording. I understand that adding an amp to the Hugo does not net you the Hugo TT. In some ways it will be worse, and in some ways it will be better. This is where it becomes more a matter of taste.