After listening to the Cayin C5 and Fiio X5 combo some more, I can say with certainty that it is a much better sounding combo than the X5 + stock E12 to my ears. I was actually disappointed with the X5/stock E12 combo, because I felt that the E12 actually degraded the listening experience in most cases compared to the X5s headphone out. I did note that the E12 provided better separation and more control to the sound, especially with something pretty hard to drive like my HE-500. Overall however, I felt that the built in amp of the X5 provided a smoother, warmer, more musical sound with better soundstage depth. The E12 was technically nice, but somewhat cold and synthetic sounding. The timbre is somehow off on the stock E12 IMO.
Pairing X5 with C5 is a completely different story. I clearly prefer the sound of the C5 to that of the X5s built in amp and by a good margin. There are still instances where X5s headphone out can sound a little more fun due to its warmer coloration with more prominent bass and low mids, but those are far and few in between. Gone is the somewhat cold, unnatural sound that the stock E12 introduced and C5 sounds much cleaner than the X5s headphone out as well - much cleaner. I think the difference in separation and clarity is more pronounced in case of stock E12 vs. X5s headphone out - E12 sounded cleaner, but seemingly not by such a great amount. Switching from the C5 to X5s internal amp makes the latter sound positively muddy in comparison. C5 also offers much better separation, a much wider soundstage and a lot more depth to music detail. C5 sounds far snappier, more dynamic and effortless vs X5s built in amp with all my headphones, even those that are easy to drive. There's a much blacker background with the C5 too - each sound occupies its own space, and sounds can leap out with striking realism. Switching the X5s internal amp reveals a much flatter musical landscape with far less dynamics and articulation. I am really surprised how poorly X5s headphone out fares vs. the line out to C5 and I wonder if Fiio engineers could've improved the internal amp quality any further, considering the unit size limitations. Probably not, or not by any significant amount. Ok, I am not saying that the X5s internal amp sounds bad, because it doesn't, but it does not do justice to the DAC at all IMO, not being able to reveal a lot of its potential. The C5 is able to show much more of what the PCM1792 can do IMO. Now, I also wonder if higher end amps can improve the sound of the X5 even more...