The transducers themselves, followed by the enclosure, followed by the damping materials used, and whether it's an open or closed design, dynamic or electrostatic, and of course how well all of this is implemented. After those 10 factors or so for the headphones themselves, the 11th and 12th most important things are the source and amp (in that order, at least in my experience).
While I can understand what people mean when they say "source first", I do think it's a bit of euphemism for saying that the source is a very important consideration. A great pair of headphones with a crappy source won't sound nearly as good as they can/should. But a great source with a crappy pair of headphones will still sound absolutely horrible no matter how great that source is.
The arguments are obviously circular because the impact of the amp can also be fairly important, and even cables can make a difference (naysayers notwithstanding). But since the sound that is delivered to your ears comes out of those headphone thingies that you're wearing, they really do have the biggest impact. They're the core, while everything else plays a supporting role.
I know, I know. Someone is going to come along and quote me, saying "I agree with everything he's saying except that it's the source that matters most and everything else is supportive," but that just doesn't play out (at all) in my experience. Here's why: Switch back and forth between sources or amps while listening to the same headphones and you'll notice some differences. At times, those differences can be quite dramatic. But change headphones while listening to the same source/amp combo and the difference (after volume matching for impedance issues and such) is almost always transforming. In other words, with different sources and amps, it will sound like the same system (usually) but with a different tone. With different headphones, it will sound like a completely different system (usually), although some headphones (by chance) do happen to sound fairly similar to each other.
The same thing is true with speaker-based systems, by the way. You can swap in and out amps, preamps, sources, cables and tubes all day long and have a lot of fun doing so, but the minute you put a different pair of speakers into a settled system, it's suddenly a whole new system (for better or for worse, depending on all of the things that depends on).
All of that being said, when it comes to putting together an A-Class headphones based system, if I were starting all over I'd put most of my time, effort, and money into the source and amp! Why? Because once I've got those things right (for my ears), then I can flip through all sorts of headphones and every last pair will benefit from my A-Class source and amp. I'm not saying that this same logic would apply to a budget system, but it definitely makes sense for a high end system.
Think of it this way: if you're going to spend $10k on your system (in total), you can get a really great source and amp for $3k each and still have $4k left over for headphones. Phones, not phone. You can get at least 4 or 5 excellent pairs of headphones for that kind of money, and maybe even more! Although the headphones themselves matter the most in terms of my listening satisfaction, I wouldn't allocate the buik of my budget to them.
It's just too much fun to play the theme and variation games for me to give up that flexibility by spending a big chunk of money on just one pair! Some people (most probably) go for that one perfect system, but not me. I value the ability to alter the sound by switching headphones in and out (the "process of becoming" or journey) more so than I value the end result ("achievement of goal" or destination). When you know you've got great electronics behind your headphones, you're free (of mind) to just kick back and enjoy each pair for what they do best, and even match them up with different genres of music to get the most out of them.
This is why people end up with something like a G08 and an SDS-XLR. That way, they can play around with all sorts of headphones all day long and they will all sound great! Can't do this with speaker rigs. There you need to focus on the speakers themselves, and then build around them.
