Xitel's "Hi-Fi Link" series use Phillips Semiconductor USB receivers. I can't remember exactly which one, so I can't look up the datasheet online, but I wouldn't be surprised if it could only handle 48kHz. (These receivers are really old parts by computer standards and Phillips has since discontinued all of them except one.) When your clock is running at a multiple of 48kHz, you need a hardware PLL to be able to handle 44.1kHz without resampling, and the receiver would have to support this.
(BTW, Linux gets serious praise from me for being honest about the capabilities of the hardware, rather than transparently resampling to 48kHz as Windows is prone to do.)
Your best bet is to find a USB gadget that uses one of the PCM27xx or PCM29xx series chips manufactured by Burr-Brown/TI. These are supported by the Linux kernel and can handle 44.1kHz properly. Unfortunately I'm not aware of any commercial products using these chips that have a digital output, but they are probably out there.