I have used two source switches to do exactly as you describe. The only issue I encountered was a ground loop due to one switch having a common ground amongst all the inputs. The fix was to eliminate the ground loop at the source. It wasn't really an issue with the switch at all.
As for SQ, if you buy a quality switch, I doubt you would hear any degradation. I know I didn't.
One of the fancier switches is the Manley Skipjack. It's a very complicated piece of work and I have heard it's very transparent. If it can be made to be transparent, I am sure a simple input switch can be made transparent as well.