That's because as a DAC, the Fiio D3 isn't really anything to write home about. It can get better with some modifications, but the stock version is truly just a basic optical->RCA converter. Plus it needs an active micro USB connection as well to draw power since it has no battery or anything like that.
So yeah, you're right. Concerns about quality drown those gears into oblivion here even though they may or may not be that bad.
Edit:
More info on D3:
The original Fiio D3 had a Wolfson DAC chip, and it definitely sounds way above its price segment. But then it was discontinued in favor of a different version (D03K) that used a Cirrus Logic DAC chip instead. I haven't heard that one, so I have no idea how it sounds like personally. But impressions say it doesn't measure up to the original. More confusing is that the original D3 (that uses the Wolfson DAC) has 2 different versions that have slightly different components. I also have no idea how they may differ. But I have the original (silver-colored plugs) D3, and It's truly an awesome DAC.
Perhaps it's the confusion over the various versions that ultimately drowned the DAC in a sea of other DACs. But personally, the Fiio D3 is my favorite midrange-centric DAC. To date, I haven't heard anything else that renders midrange quite like it does.
It still needs to be coupled with an amp, and I had to custom-order a RCA interconnect cable that's made specifically for it, but once those things are over with, I think it is truly a good value.