I have had similar experiences to yours with USB audio output. I have a laptop with 3 USB ports-- one sounds fantastic and the others are terrible outputing to my DAC.
The best explanation for this is that the USB hubs in your computer are configured in a hierarchical fashion. Most likely the 'good' USB port is not shared with anything generating
alot of data activity. The 'bad' USB port is probably shared with your wifi circuit, keyboard, bluetooth or something else that is generating traffic. The result is that the port you are using for audio doesnt ge sufficient CPU and bus resources to keep the audio bits flowing consistently. This results in jitter or dropouts. Alot has been written about how this sounds-- but it is harsh and unpleasant to be sure.