excessive DC Offset is the main danger. this is what causes the 'thump' when an amp is turned on or off.
if a circuit is prone to wide swings, it will be recommended to keep headphones unplugged when powering the amp on and off.
protection circuits, or relays, are often used in such situations. these are designed to basically keep the headphones disconnected for a few seconds after the amp is powered up, even though plugged in.
it's not really a big concern for the vast majority. i've only ever heard of this being an issue with some DIY amps, and even then mostly in the prototyping stage.