This sounds like a classic example of a buffer under-run. This stuttering issue can be caused by many things. Sharing USB with other devices, bad cables, hub issues, background process. If you set J.River buffering of 500ms with large buffers. Tried setting IFI ASIO buffers to the max and extra safe. Nothing is sharing the USB bus that the IFI Nano is on. Try memory playback setting in J.River. These are just few of the things you can try. A native 4xDSD file of 3 mins is about a bit more than 500megs and that far amount of data to transfer from the hard drive to the USB pipeline without stuttering. It is like streaming a bit faster than Blue-ray video on USB. You doing the same thing when you converting to 4xDSD in realtime. The slowest J.Mark speed I have ever seen for DSD playback is 2500 reliably but it sounds like you are converting to 4xDSD in realtime. A JMark of 3000 or better is what the developers of J.River like to see for DSD playback. The J.Mark score is just a guide.
If all fails you can try to use J.River offline conversion tool found under advance tools section of J.River. To see if you get better results.
Yes with JRiver most of the audio processing is done with using just one thread. I like that because you will run into issues with micro stuttering like on my Ethernet port for DLNA which is set for load balancing and off loading for server needs but not good for streaming on my previous setup that was with a slower 4 core CPU. Sometimes you may run into issues with CPU parking and unparking in some cases. With the 4xDSD you ran into the max of your CPU can handle in your case. Have you tried to turn off speed step in the BIOS if you have it? Sounds like you still converting PCM to DSD. Have you tried using DSD files in bit streaming mode?
With my old setup I was running into 31% load on the CPU but it was the spikes I was concerned about. Not a problem with copying files but near the edge of what my old setup can do with continuous data streaming.