Thought I'd put in my 2 yen worth. I have the VentureCraft Go-DAP 4.0, CLAS, and at one stage the HP-P1 too. My portable amps being the SR-71B & the Pico Slim.
At least to me, to answer the OPs question really depends on a variety of factors. And in my case it seems to start from the headphones. With my V-Moda M-80, it hardly needs any intermediate components between the source & it. That's not to say it doesn't make a difference but the difference is slight that (at least to me), isn't worth the $500-$800 components in between (NB: the Fostex doesn't cost $799 BTW).
On the other hand my Ultrasone Ed8LE and Unique Melody SE530x8 seems to benefit from some component in-between my source and the headphones/IEM. And at least to me, the difference is big enough to justify the cost of the component. The night 'n day difference to me is big enough that cr@ppy music that I normally wouldn't listen to would pique my interest - I find myself even enjoying listening to '80s Eurotrash!
As to -which- component matters the most, I found the amps made the biggest difference in terms of immediate deeper bass extensions, or improved clarity in the trebles, and somewhat wider soundstage. the DACs I have OTOH seem to provide a smaller but noticeable improvement in the resolution and detail (and further tightness in the bass). For some tracks, they sound "smoother" somewhat too.
So in summary, to me good portable amps have provided the immediate bang for the buck. As for the DACs, the improvements seem somewhat less of an impact. Having said that, once I've heard "better", I've almost always used an external DAC. But again, all this really depends on what headphone I'm using in the first place.
P.S. I have an old iPod Photo. It's not iModded (yet?) but sounds pretty good with just a LOD/SR-71B...not that I could attach my CLAS to it anyway.