It'll work fine. In theory you'll lose 3dB of signal level, which in theory might lead to a tad more noise in make-up amplification, but in the real world... don't worry and enjoy your music. I've done this and similar things many times over the years.
If you don't like this idea, you could easily build a switch box. Use an aluminum enclosure, get a double-pole, double-throw switch. Mount the switch in the top of the box. On one side mount four RCA jacks. Put two on the other. That's the input side. Solder the hot (center) left and right leads to the center contacts on the switch. The outer leads go to the output jacks. Grounds all get tied together, and they should be isolated from the cabinet except at one point. That's what the insulating washers that come with the jacks are for. You connect your DAC to the input side, your two devices to the other side, and throw the switch to the one you want.
