For the OP's question, yes, I think it is worth getting high-end headphones if you only use 320 kbps audio files. In my experience, headphones continue to be the most important part of my setup, with both lossless and 320 kbps lossy files. All of the DBT that I have personally done indicates that I can't tell a difference with an overwhelming number of songs. I have not seen any reliable proof to suggest otherwise. If anyone claims to easily be able to identify a difference, they have not provided any confirmation that is accessible to the masses and this premise remains anecdotal to me, so I will continue to remain skeptical for now.
I would, again, strongly suggest that any listener attempt their own ABX test with their current equipment. It's the only way to be sure. Any DBT can only prove that someone can hear a difference. It cannot be used to scientifically prove two files sound identical. You can only prove this to yourself. Those claiming the differences are obvious and simple to detect without providing any proof are suspect, as this can easily be shown by posting the results of an ABX test.