@Ghoostknight, how do you even know when phase is getting better with your little local adjustment(I'm guessing with EQ option)? What told you that phase was off in the first place? The phase graph from a FR in REW? The phase graph from the impedance measurement (not the same and the one you should actually look at)? Just a guess from tweaking things by ear?
What
@gregorio discussed is phase, but as he said, it's also something else entirely as his job is to deal with delays between any 2 things. And I agree with what he said. We are indeed sensitive to phase at some point (like with anything, put enough of it and even us humans will notice something isn't right), but our sensitivity manifests in different ways and at very different magnitudes. What we are very good at noticing even in small amounts is interaural shift. Whatever your views on evolution, the end result is a species that makes use of that a lot to locate sound sources. The rest is often more about the impact it will have on the amplitude of the signal when 2 out of phase signals change how loud something ends up being. The delay has to be huge for us to notice more than the change in amplitude. I would argue that even 180 isn't a big deal for most frequencies.
The one aspect where you might create your own problem/impact is with EQ. Depending on the type of filter, the frequency response and the amount applied (would need to be kind of massive in most cases), you can end up with a sound that's different from the "same" EQ done with basic (and often the best choice) IIR on both channels. Personally, I at most notice some impression of placement that's different when I fool around with the various toys on my EQ. It's still subtle and any other change in more typical EQ variables (amplitude, frequency, Q) will have more influence for me.
Absolute phase/polarity on your transducers is something peculiar, and I don't think it validates or invalidates anything else. It isn't really changing much IMO and while it can turn out audible for a bunch of reasons in some playback system, I really doubt that someone is going to listen to a setup and spontaneously go "this sounds inverted".
I check for that out of habit from when I fooled around with IEM measurements a lot. Looking at the impulse response in REW, and double-checking that it was not me inverting the contacts, I've seen a few IEMs being "inverted". It's been the all thing and both channels showing the same, with absolutely nobody complaining about it or describing weird sound. But of course, once someone mentioned it, there is always some feedback about how some dude did notice something or knew it all along. Hindsight hearing is its own can of worm that's best ignored in favor of actual listening tests.
I only saw one case from Earsonics(can't remember the model) where the bass driver on just one channel was inverted. I don't think I saw measurements of other pairs, so I don't even know if it was a trick to make the bass sound weird and even more "everywhere", or if that one pair had the technician F up when soldering the driver.
In the end, we have the entire speaker family adopting crossover designs to argue that the phase you seem to be talking about isn't an issue for listeners. And on top of that, a giant portion of IEMs (more so with flagship models) also heavily relying on crossover and often messing up with phase well beyond 180° or even 360°. Can something be noticed sometimes, perhaps. Is it at large something of concern? I vote for a loud no (again, for that peculiar phase thing we see pop up with impedance measurements online. Phase as initially discussed by greg is super-duper important and used everywhere in recording and prod).