It could easily be dirt, and in switches as well as the volume pot. In one of these oldsters, the signal has to get through many switch contacts as well as the pots, even if that part of the receiver isn't being used. Check loudness buttons and tape monitor switches as well as the speaker selector. Could be dirty/pitted contacts in the speaker muting relay too.
Wualta (who knows much more about this stuff than I do) has it exactly right. Ross, you really need to unplug the unit, open the top cover, and get into the switches from there. Spray in the deoxit, and then operate the switch or control fully about 10 times or so.