I've been switching sources and amps recently, with my LCD2s; originally I was using a Marantz CD5003 disc player, DACMagic, and Woo WA6SE. Very good sound, but I was looking for an overall quieter background, inside and outside of the headphones.
I'm now using a Mac Mini, with an AIFF iTunes library, going out to a Stage DAC and then to a Concerto. Slightly different sound signature than with the Woo, but I'm liking it a lot. Actually the difference in sound between these two amps is less than I thought it would be, but besides the music, there is now no other noise! Totally black background.
So for me, the source or the amp could be the most important; it would be whichever contributes most to the silence between the notes. I may have a Woo for sale soon . . .