I fall into the "it all sounds better with an amp" category, but I don't own Shure E5xx phones. I find that across the board, hooking up an amp to my phones/DAPs (and they are a decent mix) enhances my experiences. Of course, the caveat is what I am going to be doing when listening. The lighter amp I have, an inexpensive Penguin, is perfect for stuffing into a pocket with my Sony A816, and my PKs or 716s. If I am not going to want an amp, I use the PK3s or the Atrios, and that's good. But on a long distance ride (train, plane, bus, etc.), I'll cart along the MiniboxE+, which is an outstanding amp for the money (there are some strong reviews for it on Head-Fi). That's to use with the 716s, usually, since they isolate sounds best (and really sparkle with an amp).
Anyway, I like what an amp brings to the experience, depending on the situation. Glad I have a couple of low-cost, pretty good quality little boxes.