Greg,
A 50K works fine, but a 20K gives a more constant impedance. You have to go to other configurations, such as a T pad, to get constant impedance regardless of volume setting, but in practice all this is a bit academic.
If you reduce the pot to around one tenth of the target impedance, that is a 4K7 pot, then the variation in impedance going into an AKSA is very small. However, driving a 4K7 load is pretty hard going, and certainly many tube preamps would have trouble with this. The average opamp output stage on a CD/DVD player has a source impedance around 100R, so this is a walk in the part, nevertheless the jury is out on whether one can hear differences between a 5K load and a 50K load.
There are other issues too. A 50K load will be far more susceptible to hum intrusion, by a factor of ten times over a 5K load. If shielding is good, however, this is usually not a consideration, though noise issues do intrude if the output is sub-10mV, like a Moving Magnet phono.
You have chanced upon a morass of compromises!
Cheers,
Hugh