This depends on the device however. I started out amping my Palm Tungsten E and the difference between being amped and not amped with my E2Cs was huge and made it almost a neccessity.
I got my IPOD and I tried listening through the line out (via a Maxell dock line out) and it cetainly did sound a bit better than directly from the headphone jack but I decided it wasn't worth the extra bagage.
It's kind of nice to just throw the IPOD in your pocket, plug the headphones in and go instead of packing it all in the bag, hooking up the cables, turning on the amp, etc. At this point I've decided it's not woth the baggage. I already carry far too much stuff as it is.