Whether or not incorrect phase will be perceptible probably depends on the resolution of your playback system and whether or not the recording you are listening to made any attempt to preserve phase initially (I think a lot of recordings now may actually be a mixture of normal and inverted phase signals, so neither normal or inverted phase will be totally correct).
Assuming a recording that preserves natural phase and a playback system resolving enough to reproduce it, inverted phase would present itself like a singer seeming to inhale while singing something that is naturally produced by exhalation (your headphones / speakers produce a rarefraction where a compression of the sound wave is technically correct).
