Impedance is only part of the puzzle of amplification.
There are four factors at work:
1. Headphone impedance
2. Output impedance of the amp
3. The amp's output power
4. The sensitivity of the headphones
The closer the headphones' impedance matches the output impedance of the amp, the better the power transfer from the amp to the headphones. There is a formula that will show you the power loss from a impedance mismatch.
The second part is just how much power transfers from the amp to the headphones. If the amp puts out 250mW, consider how much of that actually goes to the headphones based on the impedance match. If there's a 50mW loss from an impedance mismatch (for example) then a maximum of 200mW is going to the headphones.
Next, you have to consider the power reaching the headphones in light of the headphones' sensitivity. You'll see this listed in terms of Decibels (dB) on the package. Given a certain amount of power, the headphones will get so loud. Take the amount of power coming from the amp (in light of loss from impedance mismatch) and you can see how loud the headphones will get given the amount of power that gets through.
There are a few complicating factors. First, the power output of the amp is almost never the listed amount. A 500mW amp only puts out 500mW cranked all the way up with the source cranked all the way up. If you have the volume at 9 o'clock and the source is playing a quiet passage, you might be putting only 80mW (or whatever) through the amp into your headphones. Second, amps and headphones have impedance curves - impedance doesn't stay the same all the time. If impedance changes, it affects how much power transfers.
This is where it starts getting complicated. But I hope you see how the relationships interplay here.