Most companies offer a fixed level line output on their DAC because it can bypass the subsequent buffer/gain stage and volume knob needed for the headphone output (especially the analog volume), therefore improving the S/N and L/R crosstalk.
The line output is designed for high impedance input on an amp (typically 10-50kOhms) so it doesn't have enough power to drive any headphones.
X3 is an unusual case where the headphone output (in Low Gain setting) set up to maximum volume (60) achieves almost identical performance to the line output, when connected to a high impedance amp, so I don't see much benefit of using the line output.
The only time a problem occurs is when you use the High Gain setting on the headphone out, in which case the signal starts distorting around Volume 54. As long as you don't max out the volume in High Gain setting, the sound quality of X3 should be very similar regardless of whether you use line out or headphone out to hook up to the amp.