Yes there is impedence mismatch on the headphone out ports if you use lower impedence headphones. But are you unhappy with the sound? Do you hear distortions to the point where it actually sounds terrible? Trust me it wont be better with the line outs, I will tell you that. Also you wont get headphone surround through the line outs, so there goes one of the biggest features of the card itself.
It is recommended to connect the amp to the line outs because of some people are scared of double amping. But I would test the amp on both ports and see what it sounds like to you with YOUR headphones.
Lastly if you are adding an amp to the ZXR, then output impedence is irrelevant as you only have to worry about the output impedence of the last source device in the audio chain's impedence. This case you amp, not the ZXR.