Two parts answer:
1) Yes, it is a firmware bug and you are not supposed to be able to use 12dB in USB-in (I assume your "16dB" is a typo). It should have automatically switch to 6dB gain with USB-in. However, because FiiO didn't implement the firmware tight enough, you are able to fool the system and select the 12dB gain.
2) The reason FiiO doesn't want the user to use 12dB gain is because that is too high a gain for USB mode. It will just clip the amp section and result in distortion.
A very late Thank You for your detailed answer! Though I don't hear the E17 clip when set to +12db (it wasn't typo earlier it's really me thinking wrong since I don't have my E17 with me when I asked the question) using USB mode. I guess I'll have to listen again very carefully for any possible clippings..