if you use sensitive iem you may have too much volume at even 1 V peak-to-peak drive V
there are op amps designed for 3 V supply that work fine - most DAP are limited by the small size of their output DC blocking Cap due to wanting smaller player size - not really that much by the inherent quality of available low voltage op amps
but you will need smt building skills - many newer op amps are only available in surface mount packaging
DC-DC converters are electrically noisey and efficiency at low powers is hard to achieve, only a few portable headphone amps have used them to boost battery V