Yeah, that's a known problem for creating the Ultrasone sound signature on Sonys. Congratulations, you have yourself an ULTRASONY now.
Warning: Spoiler! (Click to show)
It really shouldn't affect the sound, at all. The only thing I can think of is it causing some crackling or cut out sound due to it not doing its connecting job as a connector, but that sort of noise would be pretty apparent as sound would be cut out from one channel. In other words: don't worry, if there's a faulty connector, the sound problems wouldn't creep up, but hit you in the face saying... nothing, literally.
Assuming I understood your situation correctly.
Edited by Coq de Combat - 1/12/14 at 10:44pm