I don't think it was necessarily due to the physical requirements of the driver diaphragm size, but it does seem that the accepted driver size is 10mm or larger for sub-bass to mids performance in a hybrid configuration and 8mm or smaller for mids to treble performance in a multi-DD configuration. High excursion drivers are challenging to maintain any significant agility in, as it can be difficult to design a magnet structure and voice coil in such a small package that can generate a strong enough flux to net you that excursion when you're using a driver with significant mechanical spring force, so you'd be looking to use a driver with a more flexible surround with less mechanical spring force and rely primarily on electrical spring force to pull the driver back to its nominal state. You tend to lose agility and "texture" when you go this route, but it can result in incredibly deep bass with a very "warm", "smooth", "honeyed" quality to it. When you opt for a driver with comparatively low mechanical spring force, then the role that the enclosure design plays is more pronounced, but still not as noticeable as in actual loudspeaker designs, since a lot of the theories and calculations don't scale accurately down to that form factor.Also, in comparison to headphones there is relatively less need for oversized drivers/diaphragms as the sound is not captured by the outer ear (incoming reflections being captured at different angles etc). Different driver/diaphragm designs experience break up at different levels of torture too. However, I believe there is still a general minimal requirement about its size due to the constraints of physics; unsure of where it lies though (unlike in microphones where it has been explained and agreed upon)
Basically, if you want to use an incredibly stiff driver to give you strong bass performance, you want it to be as large as you can fit in the shell. It gets more complicated, of course, since the larger driver can typically produce higher volume output in the higher frequency range than you ideally want, so you've got to tune it well or cross it over with other complementary drivers.