Actually, there are faults in this kind of reasoning, both with your idea of using "high quality passive resistors" and your comparison between headphones and speakers.
The answer is the same for both: load impedance.
Speaker amps have circuitry designed to deliver optimum power at around 8 ohms (or 4 or 16 ohms, depending on design - even 2 ohms if it's in a car).
Headphones typically vary between 32 and 300 ohms, with outliers at 16 and 600.
If you apply resistors on the output of an amp, they need to be significant in comparison to the headphone load impedance, to have any noticeable affect in gain.
When you do that, you are altering the output impedance of the amplifier and its corresponding damping factor. The result is that you have altered the frequency response of the headphone.
You can use input resistors. We recommended that on the Starving Student, for instance, because of the high inherent amplification factor of the tubes used. That simply throws the usable volume control range into a higher position on the pot. It doesn't really alter gain. Taken to extreme, it means less control range of the volume as well.
Do a bit of research around here where questions are asked about the best receivers to use for headphones, and you'll find that the worst simply apply a dropping resistor between the receiver circuitry and the headphone connection, resulting in altered frequency response of the headphone. The best receivers include separate circuitry altogether for the headphone connection.
Similarly, and as cause for the above, the circuit used for speaker amplification is optimized for a different load impedance and power goals. Usually - most things being equal - the best headphone listening is with an amplifier designed specifically for headphone loads, not speakers. Yes, you can have some crossover where speaker amps are used on headphones and headphone amps are used on speakers, but the systems are far from optimum. Headphone circuits are also designed for much more intricate detail response because power demands are not that great; speaker amps are usually more power focused, but again, the impedance mismatch causes all sorts of issues with headphones