Underpowering a speaker including headphones is worse than overpowering them. If the amp "clips" that means the amp runs out of clean power to drive the speaker. When this happens it flattens the waveform of the incoming sine wave. That turns the signal into heat instead of sound. That can instantly fry a voice coil.
You'll know pretty obviously if you've fried a coil. The bass especially will sound like flabby, popping crap. If you try different songs & they all sound "crackled" then you might have done permanent damage. If they sound good at moderate to moderately loud volume then you're probably ok. Never turn an amp volume to 100% ever IMHO. If you need more volume get a more powerful amp. You risk the amp running out of power which is bad. Running an amp that is rated at TWICE the power your headphones are rated for is better than an amp with <half the recommended power because the amp won't run out of juice & "clip". Instead, the volume will get way too loud for you to enjoy or you will hear distortion (of the speaker) long before you do permanent/instant damage.
You will most likely damage your ear drums before damaging the speaker.
I hope this info helps bud.