I did some further troubleshooting today and now I am even more perplexed. I disconnected the #2 input and then switched to it. There was a loud buzzing (similar to a 60hz hum but dirtier) with no input connected at all. I had thought this was just a simple switching noise.
I use the tape loop for a Grant Fidelity tube buffer, and when I leave that off the noise disappears. The unit switches with no pops. What puzzles me is that the buffer is absolutely silent when switched in with both inputs connected (aside from the initial thump). I don't change inputs all that frequently when the amp is powered up, so this was a big surprise.
I only use 2 inputs and the tape loop, just to check I switched it to 3 and four and was greeted with the same loud buzz. Weird. The obvious workaround would be to turn the tape loop off while switching, but I don't understand why this noise happens at all.