Yes, absolute polarity (not phase) should be inaudible. When a difference is heard, it can be traced to non-lineararity in the loudspeakers or headphones. The driver cone behaves differently pushing out versus pulling in.
It sounds like you play the trumpet? Can you record yourself playing whatever you believe illustrates a difference in polarity, and post the file? I'd like to hear an example of a file that sounds different forward versus backwards, and analyze it to see what's going on.