Yep, same experience here.
For a lot of recordings there is no way anyone can hear a difference - then there are these beautiful exceptions
So you're extrapolating from your own inability to hear a difference; thus you say "for a lot of recordings there's no way
anyone can hear a difference".
That's hundreds of millions of people you're talking about - some, just some, might have more sensitive ears than you do.
it's still good not to mistake our wishes to be special kids, for actual facts. I wish I could fly, but know better than to tell people that I can.
mp3@320 vs lossless, that's just like with great wines, you can find some that have pretty noticeable differences, yet you will find a lot more that are different but it becomes hard to tell which is the "officially better one" or what year it was even though one year is famous and the bottle costs 5 times as much as the other year.
and on the people side, you have most people who don't know anything but still talk about it a lot, you have sommeliers who might not all be great at wine tasting, but at least they have most of the facts right. and only a handful of guys who can tell you what is what at all times in a blind test. yes they exist, but chances are we're not one of them is what I'm saying.
people are spitting on mp3@320 not because the sound is horrendous, it's really isn't. not even because they can tell it's mp3 by ear(who is talking against mp3@320 after passing 20 out of 20 abx from 2 files they converted themselves to make sure it comes from the same master at same loudness? it's a rhetorical question, I know the answer). most people spit on mp3@320 only because they are snobs, and that's sad.
there is nothing wrong with deciding to use lossless files, nothing wrong with wishing to have the best. mp3 is for those who prioritize storage and not everybody has to.
but please people, look down on mp3 only after you made sure you can easily tell it apart from lossless. you guys want lossless audio, how about lossless claims instead of wishful thinking?
yes some small passages will sound different in mp3, even in mp3@320. most people don't notice because it was the main purpose of mp3 and they did some stuff right. those who notice often can't tell which was the mp3 file, they find a difference but it's so small they don't know for sure which one is the "better" track. that's what actual tests seem to reveal. and people who can really tell mp3@320 apart with any kind of song, well I'm not sure even one exists.
if you're the one, as I always say for cd vs highres, contact the Guinness book, there is money and fame waiting for you!
now for the OP question, mp3@320 has signal fidelity down to about -60db or something like that where the data starts to be manipulated following masking profiles to save space. if we look at headphones distortions levels -60db that's 0.1% distortion(at the measured loudness) so it's not too hard to find headphones that have signal fidelity worst than mp3. be it for that, for balance of frequency response where even great headphones are expected to have 1db or more differences in the trebles. also mp3@320 doesn't roll off the trebles anywhere as much as most headphones do. so to me from an objective point of view, yes it can be worth it getting a better headphone to listen to mp3.
just keep in mind that famous and expensive doesn't always mean high fidelity for headphones and even more so for IEMs.