Are you sure you're output impedance is 32 ohm? that is a very unusual value to have. I think the more likely scenario is the output is rated for use with 32 ohm headphones, not the amp actually having a 32 ohm output impedance.
If you do actually have a 32 ohm output impedance and want to reduce this to less than 5 ohms, a 4.7ohm resistor from output to ground will achieve this, but it will attenuate the output about 18dB in the process, which would likely make it an impractical approach. The amp might not like such a low resistance hooked up to it either, could degrade performance or maybe even damage it.