I do not think the adapter alone, is responsible for the ER-4S's performance. Typically higher impedance in headphones is due to heavier magnets and voice coils for more damping and control.
Increasing the resistance of a 32 ohm can will help control any low level noise in the background, and will help attenuate gain that is too high from an unruly amp, but that is about it. There should not be any qualitative difference in the sound quality since you are not changing the voice characteristics of the can itself (e.g. you are not really converting it from a 32 ohm to a 100 ohm headphone, you are just artificially increasing the load required)