Originally Posted by jooppi
Does it really matter which phone I get if I'm just using IEMs?
It depends on the IEM - there can be some multi-BA driver IEMs that have too many drivers and their crossovers can still present a complex load to the amp circuit in the smartphone's audio chip. Also some smartphones have slightly better audio chips, or at least a slightly different response, but the "better" one isn't always the best for some IEMs. My Wolfson chip SGS1 is too warm with bass-enhanced IEMs like the MEE M6P (that I got for the mic and the fit), but it works a bit better with the SGS3; the stock SGS3 earbuds also sound better on the SGS1.
Personally I wouldn't worry about it too much as long as there isn't any feedback of below-par performance on a particular smartphone. For example, my brother had an older HTC that was too tinny on his Westone2's, even on the foam tips, compared to my SGS1 at the time (based on memory alone even my SGS3 is better) and I couldn't fix with using NetronMP's EQ. Even then, it had no issues that I would attribute to lack of power, unless that tinny sound was due to harmonic distortion and not the DAC or software.
Still, given your list, I'd say go for the Note3 since it does 24/96 audio, so that's some kind of future-proofing. Not sure if it works or gets downsampled via USB though.