One of the biggest things is induction: If you want to change the charge on a coil it takes a certain amount of time (called induction or induction rate). In a speaker you are constantly varying the voltage to cause it to vibrate. Voltage is an electromotive force, so the higher you have the faster you can make this change. Of course you can't just charge a speaker at a super high voltage and not expect it to be destroyed, so you have to do something to limit the total current being applied. Total work being done has to remain the same as if the force (voltage) was the same as with a low resistance system.
If you crank up the impedance, you can use a higher electromotive force, and therefore increase the rate of induction. This gives you a faster response time to the signal being provided. This is wasteful (energy use wise), which is why it's harder to drive higher impedance phones.
<-- did a lot of CNC equipment, and electric motors are just the same (a coil is a coil). You can get them to change phases faster by building a current limiting circuit (although in our case we could get the efficiency back by using some specialty chopper circuits to achieve specific levels at very high voltages (IE providing 70-90 volts to a motor rated for 5 volts so it could switch phases with close to maximum torque at high speeds). In cheaper systems they just stack some big resistors at the powering section and crank the voltage, but then the power requirements go up significantly.