depends on what you call normal headphone output. from a sensitive IEM to a not so sensitive Headphone, the power needs for the same volume output can be one or 2 orders of magnitude apart. and you can see that with how different headphone amps will have significantly different maxed power specs into a given load.
so can I damage my IEM or portable headphone with my old sony DAP maxed out near 0.5V with music? I'm going to be pretty safe with that. but can I destroy something with some high gain headphone amp and full scale test tones? the probability is certainly getting higher.
a given driver will be tested by the manufacturer and they'll obtain some known typical limit. it's often expressed as power handling or some other spec like a maximum volume output, etc. I'm not always clear on what exactly where the testing conditions TBH. but the basic idea is always the same, they send a signal into the driver and at some point the driver ... let's say it is unhappy about the situation. ^_^
the type of damage or limit will certainly depend on the driver. maybe some glued part will come off, maybe the coil will start hitting areas it shouldn't touch, maybe the coil will overheat and melt... I don't really know the all variety of stuff that usually happen, as I don't destroy gears as a hobby(at least never on purpose).
so yes it will depend on the transducer. and no there is usually nothing protecting a headphone or an IEM. usually they're going to be super loud and people using them will simply reduce the volume, making things safe for the headphone and hopefully for the user's ears.