headphones make a very large OBVIOUS difference, that is, there are things about changing headphones that hit you right off the bat.
However, I'm a big believer in garbage-in-garbage-out. Changing your source will make improvements that allow the rest of your system to shine. Detail, resolution, rhythm, imaging are all things that are affected by the quality of your source AND source material.
I think amps complete the picture, sort of the pipe in the middle. You need to make sure that nothing in your system is too weak.
On the whole, the headphone makes a huge difference, and the source brings the best out of your headphone. The amp is the workhorse, but for me, it doesn't make as dramatic a difference unless you were starving your system for power.
As you get more experienced on what to listen for, the source and amp can together make a big difference provided your headphones are revealing enough.