This has been debated many times. The bottom line is there are good, mediocre, and bad sounding integrated amplifiers and receivers (both tube and solid state) and there are good, mediocre, and bad sounding headphone amplifiers out there.
When you find, through trial and error or word of mouth, that a particular headphone and amplifier are a good match, if the amplifier or receiver happens to be an older but very high quality solid state unit such as a Marantz unit, you will be saving hundreds of dollars over purchasing a dedicated headphone amplifier and still have very fine sound.
At one time, receivers and integrated amplifiers did not have headphone jacks as headphones were not in common usage. When headphones came on the market, if your amplifier did not have a headphone jack you could purchase a coupler box that was installed between your speaker terminals on your amplifier and the wires running to the speakers.
These headphone coupler boxes would have a cutoff switch for the speakers and one or two 1/4" headphone jacks. These coupler boxes also had a pair of variable resistors (potentiometers or pots) to enable the user to more closely match the headphone level to that of the speakers so switching from one to the other did not blast the listener.
As headphone jacks started to appear on receivers and integrated amplifiers, these circuits included fixed value resistors that would provide a reasonable match in acoustic level between listening with speakers and switching to the headphones. Early headphones were rated as low as 4 ohms and as high as 250 ohms.
As receivers became less expensive to manufacture and sell, features were scaled down or left out entirely. At some point it became cheaper to install an inexpensive op-amp chip to power the headphone jack than to wire a resistor network from the main power amplifier. If you have high efficiency headphones this may provide adequate volume but only mediocre sound quality. If your headphones require even a bit more power than the op-amp circuit can provide, you need to look at getting a dedicated amplifier. Keep in mind that all dynamic headphones, with one notable exception (the AKG K1000 Earspeakers) require far less than 1 watt for maximum output.
As with single ended triode tube amplifiers, the first watt is the most important watt of power. This is especially true of headphone amplifiers. It does not matter if the amplifier is capable of 75 watts per channel at 8 ohms, maximum power output into a headphone impedance of 300 ohms will be at most just a couple of watts. What matters is how well the amplifier performs when connected to a high impedance load in a power range of perhaps 1 milliwatt (1/1000 watt) up to perhaps 300 milliwatt (1/3 watt). Compared to the latest crop of portable MP3 players that can provide from 15 - 30 milliwatts of power, 300 milliwatts is 10X more power. This tenfold increase in power will be perceived as being just 2X as loud as when being driven by 30 milliwatts. Hard to believe so much fuss is being made over so little amplifier power, but then people pay thousands of dollars for triode tube amplifiers capable of just a couple of watts of peak power.
From personal experience, the Marantz receivers and integrated amplifiers from the 70s work very well with the high impedance phones from Sennheiser, Beyer Dynamics, and AKG. If you like Grado headphones, look elsewhere as the 32 ohm impedance of Grado phones is not a good match for these amplifiers. Less than $100 in vintage Marantz amplifier or receiver will purchase about $300- $400 worth of amplifier in terms of sound quality. The Marantz units also have many more inputs, controls, and outputs than a dedicated headphone amplifier. To be sure, you can buy a better sounding amplifier, but it will cost several times what the Marantz will cost.