If you're using a volume-controlled source into a volume-controlled amp, you're likely "double amping" (unless you have a passive preamp somewhere). It's unavoidable with some portable sources, but undesirable in home equipment.
Every volume attenuator degrades the signal in some way. Most degrade the signal the least when they're wide open, i.e. full volume/0 attenuation.
More complex than that is dealing with your amp's input sensitivity, which might be mismatched to your source's output voltage. Tough to tell.
The general rule is to leave every volume control but 1 at 100%, unless doing so causes something to clip. If you can hear clipping distortion, find it and turn that component's volume down until the distortion is gone. That's just a general rule, though -- feel free to experiment if you already own all the gear.