I am sure that if you have compared the X5 as the driving force fed a good digital signal (CD quality at least) to any iPhone you will agree that the difference in SQ is quite marked. The X5 is considerably better either on its own as a DAP or taking the feed from say laptop/fidelizer/jRiver.
iPhone or Android phone direct to headphones did not compete, IMO, when I made the comparison.
I have only ever tried it with earphones (ditas) and while there is a definite improvement, some extra refinement in the treble peaks mostly, i feel that the main difference is tonal. And i guess the fact that some of the top iem makers that ive spoken to have confirmed my sentiments make me somewhat more certain?
I guess it also boils down to the possibility that we look out for different things in our music. I personally value things like soundstaging and imaging a ton lot more than anything else once i get a certain tone, and the x5 doesnt quite perform there.