It helps that someone on this thread agrees with me, I was feeling a little alone on this. Thank you for sharing your point of view.
I've decided to give the Fiio X5 the benefit of the doubt, and will not return it (though I can see myself swithering on this point daily until the firmware improves). It is a very good USB DAC, and on that basis alone I'm content to keep it. The quality of output from a DTS HD-MA-mastered Blu-Ray through this (obviously down-mixed to stereo by the PC's media player first) makes me feel as though I'm in the cinema, with so much detail and ambience. I'm even planning on swapping my 32GB cards for 64s, which is a big deal seeing as my fallback DAP, the Nokia N8, doesn't support 64GB cards.
So, I'm planting my feet. Hopefully patience will pay off.