I think we're the ones not getting what you mean.
You're using the Fiio X5 as a source and DAC, with the fiio e12 working as the amp.
You're wondering wether a desktop amp, used instead of the Fiio e12, would make the X5 sound better ? As said, that depends on the amp.
Or do you mean using the fiio x5+ the fiio e12+ a desktop amp ? That would just be reamping the signal and would result in distortion, and generally worse results.
For what it's worth:
I use a fiio X5 and fiio e12 at the office, and the same fiio x5 plugged into a schiit bitfrost dac and an asgard 2 amp. I have never tried using it plugged directly to the asgard only, but the sound is definitely better in all fronts.
This means, of course, that i'm not using the e12 at home, but instead the asgard2.