This does not sound normal! From my experience, when I switch from one source to another with some volume gain, at the very most I hear a "tick" sound. I have been told this is normal because different sources will have varying voltages. It also depends on the preamp circuitry.
Even though the volume is zero, unless you have a hard mute, there could still be residual sound coming from the source selector.