Yes, this might get trickier than you had expected.
I say that because I recognize my own path a bit in what you are describing: Having something one is quite content with, upgrading and finding the sound to have improved overall - but - there is something missing or emphasised as well.
I don't know you particular system and setup but it seems to me you have just embarked on the Head-Fi journey.
[size=x-small]
Welcome to Head-Fi.org, we're sorry about your wallet.[/size]
Seriously, if you add a quality item to your system, chances are that it will expose weaknesses in the rest of the system. The question is then, how to go about the issue; where to begin the troubleshooting? Depending on the severness of your Hi-Fi perfectionism and the size of your wallet, there are different approaches I think.
IronLion had a down to earth and sane recommendation IMO. Really, if you found the sound to be better without the X-DAC, by all means remove it! On the other hand, you could ask your self: why was there a difference, and actually an improvement. You have been warned. Persuing this might break your budget.
If you ask me, I'd say, the source is paramount in the sound, provided you have - not necessarily top high-end - but at least "good" headphones. Amplification then soon gets important. In a sub $1000 headphone system, I would say that this is the order of importance.
1. Headphones
2. Source
3. Amplification (on this rank because of the major influence Solid State or Tube circuits can have on the sound)
4. Ancillaries (cables, racks, tweaks, etc)
Happy listening.