I have tried the impedance switch (D-EJ2000 --> Headphile Blackcoral mini-mini --> SM --> AKG501), and I would say, yes, the volume does go down with the impedance switch enabled, but equally in both ears for me, using seven different recordings.
I would email Xin and ask.
Nice rig Rellik! The D-EJ2000 works well through the line out with the SM don't you think?