I agree that many headphone amplifiers are too expensive. It is not that they are overpriced, it is just that these are low volume products, largely designed, assembled, and soldered by small companies of perhaps a handful of employees. There is no way these can be manufactured cheaply, except maybe as a kit for the do it yourself type of audiophile.
There is an alternative. Search the amplifier forum for threads on vintage amps. There are some very good values in vintage solid state amps and receivers from the 1970s. Receivers and integrated amplifiers that cost $400 and $500 in 1975 would cost $1800-$2000 to built today, and largely sound as good or better as any current amp up to the kilobuck range.
Your money will go alot further if you buy vintage gear. A plus is that while many of the higher wattage receivers and amps can cost several hundred dollars today, the low powered models from the same period are often undervalued and can be purchased for well under $100. For headphone listening, the power rating of the main amplifier section of a receiver is not relevant, as the headphone output typically had its own amplifier circuit, and these were standard circuits throughout the product line. I can't imagine that Marantz, Sansui, Yamaha, Kenwood, Technics, and Pioneer spent money manufacturing more than one headphone driver circuit.