This effect is the strongest on natural, minimalist recordings.
On multi-tracked pop/rock recordings, polarity of individual tracks in the mix may be inverted, for example inverting the polarity of the singers voice is a common trick to make the voice better "cut through" a dense mix. But if you fix the singers vocals polarity, drums and guitars may now be inverted. So simply select what sounds "right" which may depend if you focus on drums, guitars or vocals.
To test this the way you suggest, take any simple mike'd direct recording you like (regardless of recording polarity) and use any audio editor to invert polarity (for example Audacity can do this).
That way you have two identical pieces with opposite polarity to listen to. and you can experiment.
While no necessarily in agreement with all points raised, we can recommend the following article as additional reading on this whole polarity issue:
http://www.positive-feedback.com/Issue1/cjwoodeffect.htm