Ok, so I decided to do some experimentation to try and help you out on this. I put in my really low impedance and all BA set (Audiosense T800 @9 Ohms). I have tested on both ports for these results and both OSes also. Also keep in mind I am using the DX300 so YMMV a bit with the DX320. Almost blew my ears out though, because anything above 35 is actually really loud on these...
In Mango OS the only noise I ever get is the 2 second hum when switching ports, otherwise the background is absolutely black. This was done with low, medium and high gain in both ports.
On to android, in Mango android, it is much the same as with Mango OS except the hum lasts for about 5 seconds for me.
As for streaming, this is where I DO get some noise. I am using Amazon HD. When first turning on WiFi, I DO get some pretty audible pops and cracks for about 5 seconds then it goes away until I turn off WiFi and turn it back on. I also get this same sound when opening the app. When pressing play I notice a much less audible popping in all three gain modes. Pressing pause it lasts for about 5 seconds then goes away. One thing of note is that the popping heard during playback isn't audible all the time, as it comes and goes. We'll have to see if this changes with burn-in. If listening to a longer song it seems as though those pops go away altogether and there is no audible noise (to my ears), that is until switching to the next song, and then the popping comes back (off/on) until it goes away. This is what makes me think that these really need to settle before making judgement calls on this card. I could be wrong though.
Has @ Paul - iBasso made any comment on this as of yet?
Edit: Actually after listening for a bit, the noise seems to be happening only when the song is buffering (WiFi is accessing the internet). After that, I hear black. So, if the song takes a long time to buffer, you might hear it longer.