The speaker amps have too much gain because speakers arn't all that sensitive as headphones.
Headphones, particularly dynamic drivers are tiny and easy to move, but speakers are huge so they require a lot more power to move.
So, if there is too much gain, there a few issues. Since, headphones are easy drive compared to speakers, that high gain from the speaker amp can cause noise due to headphone's high sensitivity. Too much gain would put out too much power that the volume knob wouldn't have as much fine adjustments for headphone, early channel imbalance from the volume knob pot, and even the minute volume change can amp too much, too fast, too loud. Also, can blow up the headphone driver since they are weak.
Above is if you drive it out of the speaker taps, but the headphone out jack has protective resistors so you don't blow out your headphones. This resistor raises output impedance, and reduces damping factor for low ohmage headphones (applies to just dynamic drivers, not planars), and may not perform well for low ohmage headphones. The high ohmage ones maybe ok, like Sennheiser 300 ohms or Beyer 600 ohms (higher the best). Also planar magnetic drivers despite low ohmage since, it's also like a resistor. Dynamic drivers are like inductors so output impedance matters if the ohms are lowish.
Headphones are typically about close to 100dB sensitivity, but there are one very low in sensitivity like the HE6 with 83dB, a planar magnetic type (this is an odd rare situation for headphones). For the HE6 people try speaker amp taps since it's significantly less sensitive than typical headphones. You can still blow it out as HE6 is in an odd position (not sensitive like the others and avg headphone amps can't drive it well).
