It depends on the amp.
In pentode amps very interesting things happen. Part of the way a pentode amp sets its gain is by the load impedance. Indeed, a pentode loaded with an infinite load has gain of (close enough to) infinity. Normally this gain is kept in check by the load which is created by the output transformer. Now the problem comes from the fact that the output transformer dosnt have an impedance of its own, but just an inductance, coupled to another inductance... This magnifies the impedance of the load back to the drivers. When no load is connected the magnification of (infinity=no load) gets infinity * whatever the stepup is (its actually limited to some real numbers, but its close enough). This can cause issues because the pentode which normally has a gain of (lets say) 10 as the last stage of the amp now has gain of infinity. Any input signal (or noise) into the pentode causes the pentode to swing from 0V to 2*B+ which is not really a good thing when you look at how much power that uses, it can be several hundred watts! chaos ensues. The normal failure is for the plate of the output tube to melt, which then deforms and shorts B+ to ground through the glowing mess in the bottle. This blows out the output transformer.
There are other issues too! In amps with feedback the load may be necessary to dampen certain ringing on the output. Its quite a delicate balancing act. Without the load the feedback loop is required to dampen this ringing, but as you probably guessed it cant! This causes any number of things to overload and go into an uncontrolled oscillation. The end result is similar to above, The output tubes melt and the amp self destructs.
Triode amps, amps without global feedback, and OTL amps in general are generally less sensitive to load impedance.
I should note now that I have given the doom and gloom of a catastrophic tube failure that it is exceptionally unlikely in a headphone amp. Considering the liberties people take swapping headphones hot, if it was going to happen it would have already, and it hasnt.
If you think you are any better off with an SS amp think again! The vast majority of SS amps are DC coupled. Any DC offset on the output is right there on the headphones. When an output device fails, your headphones go. Despite my best efforts to provide an entertaining description of a fireworks show the failure of a transformer coupled tube amp does not always (or often) destroy the speaker.
