If amp has its protection, it's fine. Nowdays most balanced headphone amplifiers comes with protection for something that "might" damage amplifier. Does it sound weird? Not for me but I admit SE sounds easier but also dimmer.
But well, I guess this kind of discussion won't go smoother than this anymore so let's just end it.
The reason why SE sounds dimmer is because SE outputs on a balanced amp will have HALF the gain. I strongly recommend you not drive your amp the way you are, even if it has protection circuitry it is not good. You spent how much on the amp, yet will not spend the small bit more to mod your phones? If the phones have 2 cables to a Y then it really should be no more than buying 2 XLR connectors and spending 5 minutes to solder them on. You can then use an adapter to convert your XLR terminated headphones to TRS termination for use with non-balanced amps.