Since I don't use the E5 all that much, this discussion has led me to hook it up to the Clip with a Cardas mini to mini and do some listening. Cassiohui, I will let you know how it sounds by end of the day. Also, LOD is line out dock, which is a cable you can buy (Lil' Knight, a head-fi'er, made mine) to use out of an iPod (or any DAP with that capability, Zunes and Clips don't have that option) that goes into you amp, rather than using the headphone out.
With the E5, you can go either way, as is the case with any amp (except some of the cheaper CMOYs and E3, which have no volume control).
Anyway, they also make LOD docks for iPod, which require just a mini to mini cable. But I have no experience with them.
Update:
I did a very quick comparo, and I really feel the E5 doesn't take away from the PFEs or degrade them (listened to bits and pieces of several genres of music). I used a good quality (Cardas) mini to mini, with the Clip (at max volume) and the E5 bass boost on, and it sounded excellent. I am using the black filters and medium silicone sleeves on the PFEs. I then went straight to headphone out, and didn't find any appreciable difference in SQ, detail. But of course, that's all my ears. YMMV.
You idea isn't dumb at all, as long as the E5 doesn't degrade the PFE sound, and it doesn't for me.