I have used my Duet with AKGs, Sennheisers, Grados and Etymotics. My opinion is that it does better with the higher impedance AKGs and Sennheisers. I have had no issues driving any headphones to loud and well controlled volumes within the first third of its output capability.
I have been listening to my new Grado SR225 a lot since I got them, and of course they're easy to drive. However, what I note with both these headphones and the Etymotic ER-4S is that they become pretty bright and and somewhat thin in the bass department through the Duet. This may be the effect of an extremely detailed DAC pushing headphones with extended treble capabilities up to regions unexploited with other sources, or some other issue, but by now I have concluded that I won't use the Duet's headphone output in the long term with either the SR225 or the ER-4S.
On the other hand, the HD580 responds extremely well to the amplification that the Duet provides, and the headphones seem to be laid back enough that the treble doesn't become an issue. The bass response is noticeably more authoritative through the Duet than through either a Hornet or an Arietta, and the mids remain classic Sennheiser in the sense that the soundstage is quite broad with a nice sense of space. The Duet is neutral enough that it doesn't have the effect of bringing the HD580's mids forward at all.
I didn't buy the Duet to serve as an all-in-one device, and I won't be using it as such. I have my doubts as to whether the DAC can be beat for the price at this time, and the headphone output certainly is worth using with certain headphones. I will have a home amp added to my setup quite soon, and will then use it to drive the SR225. I'll then have the option to switch other headphones conveniently between the home amp and the Duet as appropriate. I don't think the Duet can be faulted at all for not driving every single headphone the way a person would like. Even if someone bought a standalone headphone amp, that would be too much to expect. I just happen to like the Grado sound enough (I fell instantly in love!) that I feel it worth it to invest in a separate home amp known to be particularly well suited to the SR225. If I were to stick with Sennheisers and AKGs exclusively, I definitely would not invest in a separate headphone amp.