As long as your source can drive a 5k impedance (worst case input impedance scenario with a 20k pot), the concern with using a different pot value is the input capacitance of the 19J6 and whether it will create high frequency roll-off.
Let's do the math!
Cmiller = Cg-k + Cg-p * (stage gain + 1)
From the 19J6 datasheet, Cg-k is 2.6pf and Cg-p is 1.5pf. Let's call stage gain Mu (though it's much lower in reality). So Cmiller is about 60pf. But we have two 19J6s in parallel for each channel so double this to 120pf.
f = 1 / (2 * pi * Cmiller * Z)
We'll use the full 20k impedance from the pot as Z and remember to express Cmiller in farads, not micromicrofarads (uuf or pf).
1 / (2 * pi * 20,000 * 120 x 10^-12) = a little over 60khz
The 50k pot PM spec'd gives a roll off at about 26khz using the same Cmiller as above. You'll actually have a little more high frequency bandwidth. I don't think we have to worry about oscillation here (60khz is still pretty low in the grand scheme of things), but you could add a 300-1k grid stopper if it worries you.