Impedance can be a bit misleading.
In absolute terms a low impedance driver would present most difficulty to an amplifier, as it is with speakers.
Low impedance demands more current from an amp. For speakers that can be an issue, as the current demands can be very high indeed (in the order of tens of amps).
With high impedance drivers, the current demands are low, but the voltage demands are higher. So a high impedance driver needs more voltage to reach a certain amount of volume. Voltage is typically easier to provide by amplifiers, so in absolute terms higher impedance tends to mean easier loads.
Now... when it comes to headphones, the absolute terms don't apply that well.
For headphones efficiency is often times more important than impedance.
For example... a low efficiency, low impedance headphone would be hell to any amplifier, as it will demand both high current and high voltage.
A low efficiency, high impedance headphone would be tough to drive too, but the demands will be mostly voltage oriented.
With high efficiency headphones, the demands on both (current and voltage) aren't that great, so it doesn't matter that much what the impedance is.
As far as IEMs go, having a higher impedance model usually means you'll just need more voltage from your amp than your typical 32 ohm IEM. For most portable sources that can be a problem.
On the other hand very low impedance IEMs can become a nightmare if the source you're using has a high impedance output and will struggle to drive the headphones correctly (due to insufficient damping factor and the inherent distortions introduced in the sound).
Sorry for the long rant, hope it's not too difficult to follow (just got off from work)