My sources are either an ODAC+O2 or a FiiO e7.
I also have a Galaxy S1 and 32GB iPod Touch 3G as portable sources.
I can't seem to justify buying anything else, as anything else is either ridiculously expensive or inferior.
It all depends on how far you wanna go. I am for the first time satisfied with my setup. It is the HE-500 and my x-can v8p with my e17 as DAC. I don't necessarily need to go further, but I would still like to.
It seems like you are in the same situation.