Usually I listen with HD-800s, LCD-XC or UERMs. Sometimes the DITAs, JH13FP or LCD-X.
I think two things are going on with the Hugo and impressions: One is what we're used to. If you're used to an old-school PCM1704 DAC then the Hugo might sound bright in comparison. The other is that I think the output stage of the Hugo has a few quirks in its design making it not always play nicely with other components. The main one is that it outputs as much as 3V, which is way over the 2V max most components expect. So if a downstream component (eg: A pre-amp or power amp) doesn't have a volume control on the input, but after the gain stage, then the gain stage will be overloaded and the sound will distort.
By the way, I did: Mac > Centrance M8 (USB) > Hugo (optical) as well and it sounds good. I'm with you on the differences being minimal. I think though that USB is a tiny bit harsher sounding and we're all probably caring too much about the fine details.
Actually, it will OP 5v RMS, but the pre-set setting is 3v RMS. 3v RMS is the OP I have been using for some time, it allows some gain for use with passive pre-amps. Also, all CD players and DAC's tend to produce more than 2v RMS, so I would be surprised if these levels will overload pre-amp inputs. My pre-amps always had the volume control first, as a input buffer always degraded transparency, so you could never in practice overload the inputs. Fortunately, you can set the volume to any level you like, and leave it that way. That said, to really enjoy Hugo, you need to remove the pre-amp and feed it directly into the power-amp.
That said, there is something else going on with Hugo - that is the output impedance or damping factor. Now the damping factor is very important with loudspeakers as it damps the motion of the cone, and reduces distortion from the loudspeaker. A loudspeaker load impedance varies with cone excursion, and this, coupled with the OP impedance of the amp, creates distortion.
A very long time ago, I had a power amp that had feedback around the loudspeaker terminals, so the loudspeaker inputs had a damping factor of 10,000. I then added very high quality resistors, so I could hear the effects of damping factor. For damping factors of 20 or smaller, the bass sounded thick and muddled, with a cozy warmth to the sound, but the bass tended to be one note, very difficult to follow the tune. With damping factors of 20 to 200, you got improvements in bass timing, and the one note quality disappeared. It sounded less fat and warm. Now above 200, it actually changed the tempo of the music, with it tending to sound faster. Bass was much more dynamic, and an awful lot tighter, with a leaner overall balance.
Now with Hugo, I designed the amp as if it was a power amp to feed loudspeakers. Indeed, I had a very enjoyable evening a few days ago, with Hugo (and only Hugo) driving Axjet horn loudspeakers outside in the garden! It was surprisingly loud... Anyhow, Hugo has a low 0.075 ohms OP impedance, much lower than regular headphone amps. Perhaps damping factor is important with headphones too? I will be doing some listening tests on this issue shortly.
Rob