Diminishing returns kicks in hard once you get past $1,000-$2,000 in raw parts cost. Not the retail price, but what the parts in the component actually cost. You can go over that with über-exotic SET amps that have separate filament transformers, chokes, interstage transformers, and the rest of those goodies. But asking $6,000 for everyday household tubes (at least in the 1950s) on a $50 PCB is not money well spent.
When you get past that point, you also forego serious investment money that should be *earning* you money, not depreciating. $6,000 on audio gear isn't all that far from a down payment on another rental unit in Arizona. The rental will generate maybe $600 a month in rent, give me an armload of deductions and should appreciate in the long term. If you look at pure dollars, the rental will bring in maybe $75,000 over 10 years, while the amp will be "dated" and "not as good as the new ones" then, and might get $1,000 on Audiogon. I don't mind dropping a few hundred on a pair of cans or parts for a DIY amp. That's fun. Anything more could have a huge impact on an early retirement - something I value a little more than the last 2% of sound quality that may or may not exist.