The issue is that the Dharma may respond in unfavorable ways to the higher impedance output. I assume most of us want to make reasonable spending choices and $ spent on something I may not use doesn't make a lot of sense to me.
Apologies, I thought you were planning to buy an aftermarket cable either way and you were trying to decide which to buy. If that were the case, buying balanced and getting an adapter for SE is always the way to go.
Since that's not the case and you won't be able to test both ports, I do want to comment on your question. From looking at the specs on your unit, the XLR output seems to be fully balanced (as opposed to something like the Liquid Crimson where the XLR port is SE, and it's just there for convenience). As such, it will be outputting double the power of the SE output. I believe this is going to make a much bigger difference than the impedance. It will give you much better driver control on the dynamic driver and that should directly translate into better sound.
With all other things equal, I'd agree that going for the lower impedance would be the right move. But all other things are not equal and the power of the balanced output will certainly offset any losses in quality from the impedance difference. Also, just to throw this out there, the "rule of 8" for impedance matching is controversial and not everyone agrees that it's such a big deal.
With a balanced DAC and balanced amp, I think it's a bit of a waste to only use the SE output. You're losing a lot of power, plus the other benefits of balanced like noise rejection. Hope this helps a bit.