The trouble is that there's not much quantitative "worth" with this hobby- it is mostly subjective. I feel like there should be a tipping point around the $300-400 range like mentioned before, but my opinion is evolving as I put more hours on my LCD-X.
I paid around $350 or so for my HD650's a couple years ago, and just paid $1700 for my LCD-X. Are they really 4-5 times better? Are the 650's 18 times better than the Apple Earpods? I'm not sure how to answer that. I would say both examples are significantly better in every regard. I feel both were entirely worth the investment. That's good enough for me, but I don't think it answers the question. Yes, there is a bigger change from the 19 dollar earpods to the 350 dollar HD650's than there is from the 350 dollar HD650's to the 1700 dollar LCD-X's.
Some hobbies may be a bit curvier than others. From personal experience going from consumer-grade gear to professional/hi-fi grade gear in both hobbies, I'd say photography is a fair bit more linear than headphones. Though unlike headphones, you can very easily measure many quantitative differences between different camera bodies and lenses.
In the end, I think if you're passionate about listening to music, any amount you spend on it is "worth it". I've said it before- I can't put a price on my sanity.