Do you remove the cable a lot? If you do, you may have loosened the female jack in the left driver.
The cable is detachable, yes, but generally I think that the detachability is just for ease of replacement/repair rather than for you to be able to remove it on a frequent basis. The smaller pin on each plug is the signal pin, try pushing that pin in more. You might have to add a little hot glue like I suggested above.
I would suggest tightening the female jacks' tabs for better contact, but when I tried taking mine apart to inspect, I noticed that the voice coil wires go over the jacks' housings before going into them, so I do not want to mess with them for fear of damaging the wires. I've loosened the female jack in the right side on mine a bit, but I just have to push the signal pin in a bit more to maintain the sound's balance.