I don't have the motivation or energy to read through all of the posts in this thread, so forgive if I repeat anything here:
1) Recording quality of the CD I am listening to.
2) Headphones.
3) Source & Amplifier (tied).
.
Large gap.
.
4) Interconnects.
.
Large gap.
.
5) Power Conditioning.
To start with, good to great recordings are paramount in determining how good your musical listening experience will be. Bad recordings and low bitrate MP3s are going to sound like crap regardless of how amazing your rig is.
The next most important component IMHO is tracking down the proper headphones that will best suit your needs, tastes, listening habits, comfort requirements, budget and preferred music genres. You need to establish what house sound suits you best, and then go from there. Otherwise you are just pissing in the wind.
The next step is to have a decent source and ample amplifier to do justice to your quality recordings and allow your preferred headphones to shine and perform to their fullest extent. Some sort of a transport/sound card with bit perfect out to an affordable, yet decent quality DAC has always been good enough for me. The important part with the amplifier is determining if a tube amp or solid state amp will work best (according to your own ears) with your preferred headphone(s), then go from there. Then purchase your amplifier of choice that fits within your budget.
Finally, decent yet affordable interconnects and power cords aren't a bad idea. I personally never saw any point in investing money in anything higher than Bluejeans cables. I would rather spend $500 on a new pair of headphones instead of an uber boutique level interconnect.
If you are bored, and want one final tweak for your rig, you can look into decent, yet affordable power conditioning/surge suppression. I went this route, and although I didn't detect any sort of perceivable change in sound quality, it is nice to have piece of mind that my equipment is protected, and that any 'dirty' electricity or spikes coming into my house are 'supposedly' cleaned up before it reaches my rig.
So in summary (and to answer the original question):
Assuming the CD I am listening to has excellent recording/mastering quality, Headphones (by far) have always had the biggest impact on the sound of my system.
