Sure most of my USB DACs working out-of-the-box with my Samsung S4 on Android 4.4.4 (5.0.2 don't change anything). But the micro iDSD uses XMOS which needs proper driver support (you need to install on Windows drivers too, it's not plug & play - the same with Anroid). USB Host mode only mean basic need for USB audio, but dosn't mean all DACs will run. The drivers are the problem. My Sony Z1 needs UAPP as well, like my Samsung S4. For example with the Sony Z3 or the Samsung Note 4 the vendors put some OEM driver into this devices for USB audio, so e.g. a Note 4 with Android 4.4.x is able to play trough all apps to the micro iDSD without problems an no need for UAPP.
Actually it has nothing to do with XMOS at all - the reason for driver is because Microsoft refuses to include USB Audio Class 2 driver in their OS, which is required for all USB DAC that runs beyond 96kHz. For example, Linux and Mac both have included the driver from very early on (around mid-2010, and USB Audio Class 2 was officially released around 2009 ) into their OS and thus you don't need to install it separately. Again, it has nothing to do with XMOS, as any USB DAC that runs over 96kHz needs USB Audio Class 2 driver, regardless of whether it is based on XMOS or not. The only difference is Microsoft has consistently refused to add it to their OS, for no good reason no less.
Now you might wonder - since Android is based on Linux, why doesn't it supports USB DAC natively? Well, it is because at the beginning, Google never thought anyone would need to run an USB DAC on a smartphone, so they stripped the driver away to slim down Android (*also, don't forget early Android hardware is pretty weak and probably won't run an USB DAC anyway). In fact, it is not till Android 3.1 (around 2012) that Google started to include USB Host mode (aka OTG) , but only for hosting external memory and nothing much else (*USB audio is mentioned in 3.1, but Google left it blank for smartphone maker to figure it out on their own). It is not until now (Android 5 Lolipop) that a basic USB Audio Class 1 driver is officially included. Long story short - Samsung actually has included a custom USB Class 1 driver as early as S3 and Note 2 since 2012. IIRC, HTC is the second smartphone maker to do so, around 2013 on their M7. Sony on the other hand only introduces their USB Audio driver since Z3 (but they also make it available for Z2, after a firmware update in September 2014), though it is a USB Class 2 driver instead of Class 1 driver and therefore it natively supports iDSD without a problem. All other previous Sony only has host mode for memory, not audio.
Now, to put the long story short - each implementation of Android is different, as intended by Google to allow for greater freedom for the smartphone maker to implement whichever feature they want.That's why the same version of Andorid works with USB DAC on some models but not on others. For example, Xperia Z2 on 4.4.3 after the Sept 2014 update supports USB DAC natively, yet your S4 on 4.4.4. doesn't. But as you have said, some of the basic requirement must be met for it to work, and host mode for audio is one of such requirement. My point is that, without audio host mode, there is little to no chance that USB DAC will work, regardless of what firmware iDSD is running. Even with audio host mode, there is still no guarantee that it will work. But at least it might work with simpler, USB Audio Class 1 DAC, such as those based on PCM270x chips.