Regarding absolute phase/polarity (both terms are used), the ear is a single ended device. It is sensitive differently to compression and rarefaction components of a sound wave. It is actually less responsive to compression. Some speculate that this is an evolutionary adaptation. Anyway, when you switch absolute phase, you are reversing these compression/rarefaction wave fronts to rarefaction/compression.
I use my preamp's remote control to select the correct phase for any recording. Fortunately, this can be done from the listening position. There is an easily heard difference between the two settings. What I identify as the correct setting (again, for each specific recording) has deeper bass, a more balanced upper midrange & treble, and a more solid center image. What I identify as a wrong setting for that recording lacks the deepest bass extension and a degree of bass impact. The upper midrange/treble takes on a degree of hardness. Instrumental timbre is not as realistic and natural. That frequency range can take on a slightly nasal quality.
I emphasize that the identification of the correct absolute phase cannot just be done once for any specific system. It doesn't matter if the system itself inverts phase or not since the non-inverting/inverting of the live performance varies from one recording to another, so you just have to adjust for phase by ear. I've heard different releases of the same recorded performance having opposite absolute phase.
It's important that the polarity control paths for the non-invert/invert selections are the same. That is, it's a bad idea for the "invert" setting to cause the signal path to go through an additional switch compared with the "non-invert" path. The path with the extra switch will sound inferior, because that's what switches do. Ideally, the phase/polarity control does not add an additional switch to the signal path at all.