James, have you tried re-encoding some of your files with a different encoder just to be sure it isn't a codec issue? I have an occasion where one of my FLAC file is also audibly distorted, but re-encoding fixes the problem and turns out to be the issue of the encoder.
Just run a RMAA test recently on both wav, flac, mp3 and acc using X3. Don't see any problem on wav and flac but mp3 and acc are both noticeably more distorted for sure, though in different ways. From the surface, they do look more like compression artifact. Then again, there is no easy way to be sure from my end.
You may have missed my earlier posts and the samples I provided. The links in my quoted post below are still active if you want to check it out for yourself.
The fact that the background noise is gone after you transcode the mp3 to flac should be sufficient proof that it's indeed an issue with the X3 decoder. Any mp3 compression artifact would definitely still be present in the transcoded flac file.
Those shares have expired, but Fiio already listened to my test files and confirmed this to be a bug two months ago. Anyway, I just made new shares for the mp3 and the flac that I transcoded from the mp3, in case you're interested.
Like I said in my post back then, it's a lame encoded mp3 (by myself, iirc) that exhibits horrible background noise on the X3 if you crank up the volume a bit. These are not mp3 compression artifacts, because the same file plays fine on my other DAPs. And moreover, if you transcode the very same mp3 file to flac (second link), it plays without that background noise on the X3.
As a side note, these artifacts sound very similar to those produced by PlayerPro on my SGS3 Android phone if I activate the PlayerPro's DSPPack. The moment I deactivate DSPPack, the background noise is gone. I've documented this bug on the PlayerPro forum a year ago, but it never got fixed.