As I implied before, when you take a lossless signal and make it losey through the use of the volume control your optical output is dealing with that differently than your usb connection. Jriver has the ability to use more digital volume attenuation than foobar before changing a lossless signal to losey. At minus 20db you have likely done just that with foobar. The difference is just more obvious to you over optical. Try your experiment with both set at 0db and see if you get the same results.
I am assuming you are using wasapi event with both programs? If not, and you are using basic wasapi then you are also going through the Windows mixer or perhaps you have that set correctly on one program and not the other?
yes i'm using wasapi event on both. K i tried 0 db on both, both program sound the same now.