I agree with you mostly, but I don't believe that there are many "poor DACs" out there. If you research and shop wisely, you can find a number of DACs that decode accurately and "do no harm" that are very affordable. Same with amps - well engineered and sufficiently powerful amps dont have to cost a lot.
I think the law of diminishing returns applies to upstream gear to a much greater degree than it does to headphones. That's my point in a nutshell.
I agree with you on the diminished returns with newer DACs. I find that after about the $1000 ish range things start to plateau pretty quickly.