For low level signals, such as interconnects in and around your source components, gold may indeed be the better choice due to it's intrinsically better oxidation immunity and, perhaps even lower contact resistance since the softer golds conform better and yield a better surface contact area. Actually, both Gold and Silver had excellent conductivity and, any realizable difference between the two is more likely due to the specific alloy of gold (it's impurity content) or, the base metal it is applied to or, the method of deposition (flash, spot weld or plated, diffusion, etc), or the actual contact area under a given force.
So, there is not a cut and dry answer to which material is better in actual use. I would say for speaker connections it is moot. For interconnects, I prefer gold. In the end, the electrical and/or design characteristics of the cable itself will have a greater impact on it's sonic characterisitic than whether it has a silver or gold termination.
