I haven't had any problems just using one output or the other, with nothing else plugged in. Sounds like maybe a ground loop issue that's being temporarily solved when you plug in to the other device.
Try taking your DA8 to a different power outlet somewhere else in the house. Try not to have much else plugged in at the same time. Run the DA8 that way and see what happens. I could be wrong, but this has ground loop written all over it. It's either that, or maybe defective RCA cables, or else your DA8 is flawed.
It seems to be an issue with the way the Audio-GD handles the XLR ins and outs. I plugged the outputs of the DA8 into my Mjolnir, and the SA-31SE sounds normal even with the Mjolnir off and set to and that works fine with the Mjolnir off. If flip the switch on the Mjolnir to single ended while its connected even if its off, same problem on the SA-31SE. I had a similar issue with connecting the XLR outs of the NFB-28 to the Mjolnir while simultaneously connecting another amp to ACSS. It turned out the solution to that was the opposite, ie. to flip the Mjolnir to single ended mode to get ACSS to work properly. As Kingwa explained to me in that case, the ACSS output could have issues when connected simultaneously with the XLR, which flipping the switch on the back of the Mjolnir seems to solve. I hope thus makes sense, I'm not a complete expert on the subject, but it seems that on some equipment the XLRs are always active even when the equipment is shut off, and on some the XLR are shut off when not in use, even when connected. Something similar seems to be happening in this case, so I think in my particular situation I'm better off using either one output or the other connected at one time. Fortunately its not a ground loop or defective equipment :) Oh and I also tried several different RCA and XLR cables for good measure, result is the same.