Upgrading my earphones/headphones made a much bigger difference to my listening experience than source did. I spent most of my budget on earphones, I worked my way up to a set of ie800`s and every step up was noticeable; it`s not like I got to a certain level of earphone and the source capped off the quality.
For instance, my ie800`s always sound better than my cheaper gear through the same source. I recently bought a Fiio Q1 and an e17k, and they don`t make a huge difference, but a very noticeable one. Though again, most of my gear scales up similarly, my Fiio EM3 is improved just as much as my ie800`s are. But the sound produced from my $20 EM3`s and $180 DAC/AMP don`t get anywhere close to a solid $200 set of earphones. I would only invest in a better source if you have a set of really nice headphones, otherwise the gear will probably make a bigger difference.
If you`re intent on upgrading your source, I personally found a DAC to make a much larger difference than an AMP. An AMP is more useful with larger headphones and output impedance. Regarding output impedance, if you find your gear sounding drastically different from different sources, chances are one or some of them have a high output impedance and it`s altering the sound of your gear. In this case, even a cheap amp will make a big difference.