I can't take issue with your reasoning, but I would suggest that you make the effort to listen for yourself. If you hear no difference, you've just saved a pile of money with no regrets. If you do hear a difference and like it, then even though it will cost you money, you'll have better sound for it.
I think of it like a waterslide - you can slide down with absolutely clean, distilled water for almost the whole length, but if there's even one part of it that's dirty then you won't be totally clean coming out the other end. It's probably just the scientist on me trying to see things from a technical point of view, and I can't say what difference it'll make to others.
Maybe when I've built up a rig that can only be improved by things like interconnects then I'll consider some better wires, but as of now I don't see the point in spending as much on a pair of headphones again on a cable that makes a subtle or non-existent difference. The thing is that so many accepted traits of audiophilia are mystical anyway (burn-in?) that science has also shown makes little or no difference are well-supported by the majority of people here.
I don't think buying quality ICs is bad if you've got the cash to spare, but I do have a problem with people being scammed into thinking that a $700 cable will be vastly better than a $20 cable. Then again, some people are happy with $250 headphones and some spent $5,000, with also perhaps not as much difference as you'd hope in between.
...This hobby is unfathomable, but I think that's why it's so interesting. I still call for a blind cable test