The difference is not discernible to some people, while identifiable with a fair amount of accuracy to others... You can check this thread:
http://www.hydrogenaud.io/forums/index.php?showtopic=28925&hl=
There was also a thread here I read last year in sound science and can't find it anymore. I met a local guy who went through the trouble of building a physical phase inverting box, but you may as well just download a plugin and try inverting each channel separately... then you'll hear it for sure, in real time. It is quite dramatic if only one channel is out of phase (it sounds like something is definitely wrong). Then you can go on to inverting both at the same time with the Yggy and perhaps you'll be able to notice it easier... Apparently some recordings are presented 180 degrees out of phase, and I suppose that's why the switch is available. I've also heard individual instruments/mics/tracks could be out of phase, but then you're pretty much screwed. Hopefully that isn't the case with professionally recorded music. I've found as I mentioned before that drums probably are impacted the most.
Edit: I actually have those two test wav files from that headfi thread on my laptop still. If anyone wants to hear them. Just PM and I'll email them.