I guess going from the E5 to the E6 would be an upgrade, but more in terms of functionality than sound quality. You get two more equalizer settings, and a smaller size I believe. The headphone jacks also don't hold plugs quite as well as I'd like, the plugs don't slightly click in place like with the typical headphone jack. Honestly I doubt that getting the E6 is going to do you much good, it's basically a slightly shinier version of what you already have.
I'd recommend you step up your price range to the $85 FiiO E7. It is a DAC and an amplifier, so you'll be able to use it with a computer via USB, and even charge it at the same time. It also has a digital screen with a menu, that can set things such as bass boost and a volume limiter. It is considerably bulkier than your E5, but so much more functional and useful.
Other options exist such as the ZO2 and likewise, but I find the E7 to be very useful for my own audio setup. It's attached to my cellphone most of the time.