Maybe I just pay attention to the littlest details, but I notice that pairing the same dac/amp with various headphones gives me a similar sound signature across the board with differences being the frequency response bump in some areas of those specific headphones. That's why I care so much about the quality of the source itself, so I can pair most of my headphones to it with little problems.
I guess it depends on what you're defining as source here. I take source to mean the actual digital file being played, and possibly the DAC. Since most DACs are largely regarded to have imperceptible differences unless they're old and/or a low quality unit, it's probably safe to say the source is file+DAC. And if that's the case, there shouldn't be any actual influence on the sound itself. After that, it's the amp and headphones that flavour the sound.
Using the same source, the most distinctive aspect of audio sound signature can usually be attributed to the headphones themselves - they're carefully tuned to provide a unique sound on the basis of driver selection, impedance, cup shape/materials, pad shape/materials, the room they're being used in, etc. There are far more factors influencing sound in headphones because there are so many physical components involved that can influence that sound.
Amps themselves are typically less variable. Some amps clean the background noise better than others - this can influence things like detail, imaging, and soundstage. But all good quality amps should have a very low noise floor. Some can certainly flavour sound, potentially adding a little warmth, but to a much lower extent than the headphones themselves. Simply standardising the volume and gain between amps will probably sort out most if not all of that difference. In standardised blind tests, most perceived differences between amps mysteriously disappear...
Take this with a grain of salt of course - experiences can differ, and mixing different equipment with their own slight sound differences might create unique sound signatures. But in general, it seems that once differences in gain/volume are controlled, differences in sound signature are largely attributable to headphones. People really tend to over-estimate their ability to actually hear differences - most of us don't have "golden ears", and without years of dedicated formal training and experience, even golden ears aren't that golden.