You guys only have half of the equation. The output impedance of your amp plays a big part, as does the power output of the amp.
If your headphones have an impedance of 32 Ohms, they will be driven very, very differently from a solid state amp with an output impedance of 2 Ohms and a tube OTL amp with an output impedance of 150 Ohms. Unless you know the output impedance of the amp you're using (even soundcards and iPods have amps) then headphone impedance is almost meaningless.
Once you figure out the impedance relationship between the amp and the headphones, then you can calculate how much power gets through to the headphones. For that, the power output of the amp figures along with the sensitivity of the headphones.
If you want to get slightly more complex, the ratio of the headphone impedance to the output impedance is referred to as the Damping Factor. The lower the output impedance is to the headphone impedance, the higher the damping factor and the more control the amp has over the headphones.
Also, keep in mind that both the headphone impedance and output impedance are not static numbers. Both vary depending on frequency and are plotted on a curve. It gets complicated, but this means that power delivered to the headphones will vary depending on frequency. This is where a lot of the sound signature of an amp comes from and it determines synergy with particular headphones.
So, to the OP, the real answer is it depends. How easy something is to drive depends on what you're driving it with. This is why I recommend picking the headphones you want, then finding a proper amp for them. Not all headphones and amps sound good together, but it's more important to have the headphones you want than it is to get the amp you want and not have it match your headphones.