There is a difference. It depends on how important that difference is to you, and how much you perceive that difference.
For some guys, needing a portable iDevice DAC + Amp is essential. They know they are getting the best they can at all times. For me, that's way too much hassle, so I use earphones that are good straight out of an iPod. But I do carry portable gear with me and set up at work, coffee shop, hotel...
The difference your amp makes depends on your earphones too. Impedance matching, sensitivity and many other factors depend on how much mileage you get out of an amp. For me, the differences are not drastic enough to where I want to tote a brick on-the-go. But that is entirely up to you.