Have you tried increasing the buffer in your player? Have you tried other players? Have you tried Wasapi event mode? Where is your source (external hard drive or internal?) Is it solid state or a spinner?
I set Foobar for Wasapi event mode with a 200ms buffer or so, and it never has any issues playing off my laptop or desktop drive. Externals will occasionally pause during playback for a few seconds because Windowsneeds to read the directory or something silly like that. This only happens when I have left it idle for a while and I assume it could be solved via power settings. Haven't looked into it yet.
I think most USB issues are not actually jitter, just related to buffering.
Just some ideas and places to investigate. Hope you can eliminate the issues