I am not a big fan of upsampling either. The first time I used it is when i realized the usb32 module used a single-clock. So I figured it performed better with the 48k-derived rates (12Mz master clock). As a matter of fact, this improved the bass response.
But then, finding out i could improve the upsampling quality in audirvana, i tried that feature and found it worked great. The point is i normally avoid doing upsampling with software since it tends to smooth out the sound too much. But again i believe audirvana does it so well that it is pretty much like playing a native 24-bit signal. And the dac does not seems to oversmooth that 24 bit signal. So it is much like upgrading all you recordings to 24 bits. In theory, proper upsampling is a matter of using good software and applying enough processing power. And in theory, it is supposed to work. And that is what I hear.