[Edit] It's a 3.5mm male o 3.5mm male then to stereo female to mono male then to stereo (both sides) coupler then to mono mail to stereo female with is where the headphones plug into. A couple of problems. It always seems like the 3.5mm jack has to be moved into just the right place to work and overall, I think there may be a grounding issue. Once in a while there is distortion.[Edit]
The fiddly-ness of the adapters are because they are probably not made to the same spec. I have a cheap HOSA 1/4" to 1/8" that just does not seem to work with about 20% of my 1/4" connectors. You have 3 adapters, making the odds pretty good that something will be out of alignment.
Not sure what you mean by distortion, but partial contact between the adapters could cause all sorts of shenanigans from hiss/static to volume and perceived tone issues.
After researching this a bit more, the distortion might be caused by impedance mismatch or worse, a temporary short. Every way I have thought about trying to do this with just wires will create a short where the amp's output drivers can try drive each other instead of your headphones. Once the Left and Right wires touch each other, either in the middle or by connecting them to the same pin/ring, the signal can bypass the headphone drivers depending on resistance/impedance. So, you have to first sum the signal and then split signal without a low-resistance path between left and right. I did find a note from Rane that addresses the summing part (
http://www.rane.com/note109.html). Unfortunately, the Rane note refers to summing circuits for subwoofers after a beefy active crossover network, so the diagrams may not apply to headphones and full-range sound.
https://electronics.stackexchange.com/questions/71903/how-could-i-build-a-headphone-stereo-to-mono-converter
I cannot vouch for the above circuit as my basic understanding of LR circuits only goes as far as isolated circuits, not combined between an amplifier and transducers. We have beefy resistors, Electrolytic (I assume) Capacitors, and bleeder resistors for those caps...Gonna make a pretty ugly cable!
I wonder if beefy diodes may work or not. Again, I know what diodes do, but not how they might affect a circuit or sound.
Your problem does not appear to have a simple solution...