If you're talking about amps only, then you'd have to consider which headphones you're trying to drive. If you've got a pair of Denons, based on my experience, they won't sound that much different with a $300 amp or a $2500 amp. They'll be improvements, but not so much as it would be worth it if you were buying an amp for those headphones alone. True for other headphones as well. Yet some headphones don't sound nearly as good as they could with most less expensive amps (though they can sound okay), but scale, sometimes dramatically so, with the better, and usually more expensive amps, taking into account individual amp, synergy, etc. (HD650, K701, R10, K1000, the higher end Grados: these I know).
If we're talking sq only, I think the point of diminishing returns doesn't really begin enter until you're in the $2000-$3000 range for amp alone. If you include lifestyle and budget, etc. it becomes highly individual. What's worth it to you may be very different than for someone else, and since there are a lot of very good sounding budget rigs, I wouldn't worry about finding something that's pleasing unless you don't research what you're buying well (e.g., putting a 701 with a portable amp).