It's not ridiculous at all. The iPhone 4 and iPhone 5 use different audio chipsets. The 338S0589 IC (containing the CLI1495B0 and CS42L61) is inside the iPhone 4, and the 338S1077 IC (containing the CLI1583B0 and CS35L19) is in the iPhone 5. Different chipset, different sound.
You may hear no difference between them, and that's fine. That's you and your earphones and your ears. I don't find it difficult to tell them apart; the difference was apparent on the first listen. After an hour with a dozen familiar tracks last night, it was even more clear to me that these two don't provide the same listening experience. Another brief listen with both today confirms. The change in tone is hard to miss, but I can see how the perception of dimensionality and air would differ depending on the earphone or headphone. Lossless files on the iPhone 4 sound better than LAME -V0 on the iPhone 4, but lossless files on the iPhone 5 sound worse than LAME -V0 on the iPhone 4.
Hopefully we can see some measurements soon.