If you are hearing differences, I would ask you...
1) Are you carefully line level matching?
2) Are you patching through a preamp that allows instant direct A/B comparison?
3) Are you comparing line out (not headphone out) so it isn't just a simple impedance mismatch with your headphones?
3) Are you testing blind?
If the answer to any of those questions is no, I'm betting your results aren't correct. Reasonably well designed players should be audibly transparent. If they aren't, they are probably broken.
The only way to know for yourself for sure is to do a controlled test. I've done that dozens of time with the equipment I've bought over the years. I know,
I missed that part earlier.
I agree there is always the possibility of mismatch impedance with iems/headphones and you are quite right about reminding to use the "line out" instead of "headphones out" when testing with a preamplifier. That is the only way to do it right without having two preamps in the chain and gain mismatch...
But in the case of iPhones, there is no line out (Headphones out only) = iPhone were designed to deliver sound to headphones and iems. As such, I think most people wondering about sound differences would assume the comparison was conducted with iem/headphones rather than with a preamplifier... and then maybe with an external portable amplifier...
Anyway, the IP4 sounds fairly transparent and I am very happy with it. I will not upgrade until the next iPhone sound signature combines the technicality of the iP5 with the organic signature of the iP4, that's all.