I'm not a physicist either but I've taken several college level physics courses and I understand things like conservation of energy.
You're not using that power. It's not "in reserve" for your bass notes. It doesn't contribute to impact or dynamics. If you have a 500 mW amp and a 5 W amp driving headphones at the same volume, there's no inherent reason to believe the 5 W amp will sound better. It could just by the nature of different amps having different coloration, transient response, etc. But it's just as likely the 500 mW amp will be the better sounding one.
I've driven my Arya with a JDS Atom Amp+ (1 W max), NITSCH x Schiit Magni Piety (1 W), and Questyle M15 (probably around 215 mW). And as far as impact goes, they're all fairly close, but the M15 sounds better than the JDS and the Piety sounds better than both of them. But the Piety sounding better than the Atom Amp+ is obviously not due to power since it has the same amount, and same with the M15 sounding better than the Atom Amp+ since it has less. So the Piety sounding better than the M15 is, at least partially, not due to power, even though it has almost 5 times the amount (which is really only a 6.7 dB increase in max volume).