The amp, by far. I'm not sure if there is any DAC on the market that has distortion in the range where humans can hear it. Jitter is a solved problem, too. Like distortion, jitter rates may vary, but all of them fall below what people can hear. The only big variable left is output voltage. If one DAC has a higher output than another, the louder one tends to sound "better" since we're pretty well attuned to hear differences in volume and louder=better to our brains. Of course, if you level-match DACs, they all sound about the same. Build quality and reliability are different matters - you'll want a DAC that's well-made so it'll hold up. Also, some DACs put (unnecessary, IMHO) tubes in the output to color the sound. Whether that's "better" to you is a matter of opinion.
Amps have a wider variety of sounds. Solid state has a more focused range than tubes and tend to have flat impedance curves. Tubes usually have a lot more variation in output impedance, usually a curve. Impedance tells you how well power transfers. When the impedance curve changes over 20Hz-20kHz, it means that power delivery varies. Further, you'll notice that headphone impedance changes over a range, too. Impedance rarely stays flat throughout the range. So what this means is that power delivery varies depending on how the output impedance varies along with the headphone impedance as frequencies vary. Since we're sensitive to differences in volume, the dance of output impedance and headphone impedance gives different sound signatures. Measurable ones, too. Further, tubes introduce noticeable distortion sometimes and they have different output curves (separate from impedance), too. Further, tubes have different output curves depending on bias voltage. The same tube can sound different in different amps, depending on where the bias voltage is set. And there's more to it, but you get the idea. All this stuff is real, too. You can put them on various test gear and demonstrate the differences.
So the amp you choose introduces a lot of variables when it comes to the headphones you're driving. No two behave the same.
A DAC mostly varies by output voltage. But you can adjust that with a volume knob. This is why I don't make much over digital sources. Reliability aside, they're pretty similar. You'll pay a lot for fancy cases and new chips roll out every year or two, but I don't see much reason to upgrade.
In terms of what's important, the recording is most important. If you have a lossy file or a brickwalled track, no amount of gear will help. After that, the headphone is hugely important. Amps play a role in fine tuning, but aren't as important. The DAC isn't very important at all and ignore the snakeoil. You'll pay a lot of money for something where no difference can be detected.