Yep - that's it in a nutshell.
It doesn't stop people claiming high gain is better - but its amazing how the differences disappear when you put an SPL meter on it
Not everything is measurable especially using SPL meter. If we are depending on measurement, there will be very little to no different between Fiio X1 to Fiio X5ii, or even AK240. Our ears, though often subjective, hears more than what common measurement tools can show. Depending too much on measurement tools is not being honest to our ears.
It is known in amplifier design, that amplifier behaves differently in different gain setting. For example, in different gain, the maximum bandwidth is different. Some amplifier not stable in low gain, and some stable in low gain. Theoretically gain does affect amplifier behavior. Amplifier is analog circuit, all parameters affecting the performance.
With Fiio X3ii, I observe very little different between low gain and high gain sound quality. Of course at same volume.
With my DX90, the difference is more audible. At low gain the detail is not as good as in high gain. My friend who loan my DX90 for a few days also told me the same, that he prefers high gain setting. At high gain the mids and bass are tighter, with better drive. Therefore for DX90 I keep the gain setting at high.
Please take note, it is not the same for all players, so it is not a common rules.