Yes it's possible and quite realistic, but probably only noticeable for certain headphones and with certain sources.
If your source has like 8 ohms output impedance or sucks at driving low impedance loads at higher volumes, that's mostly a non-issue if you plug that into an external amplifier, which will have much larger input impedance than headphones and thus will be much easier to drive. Then the the effective output impedance will be pretty much whatever the external amp has.
For those IEMs that vary from the single digit impedances at some frequencies to say 80 ohms at others, yeah, that will make a big difference to have a close-to-zero output impedance with your external amplifier, as opposed to 8 ohms output impedance or so.
Anyway, point still stands that the vast majority of the time you're better off just upgrading the IEMs/headphones though.