Indeed, listen, and then listen some more. However, will one headphone (even if it's as high end as they come) fulfill all your needs? Some excel in other ways than other, and maybe it's worth thinking about opting for two headphones with different strengths. For example, a Sennheiser HD800 for its incredible sound stage and imaging and then an Audeze LCD-3 for it's more romantic and warmer sound.
Probably can get both of these with plenty money to spend on a good amp/DAC combination such as a Schiit Gungnir Multibit and Mjolnir 2 combination.
Used, these should easily land you under $4,500, and you could invest in some upgraded cables and tubes.
If you really want to go as high end as possible, I'd try to find an Audeze LCD-4 used. But, will it be worth the expense over an LCD-3, with the option to switch to another headphone if you so desire?
Good luck