Your real risk of yanking cables around is breaking the cable, so it won't "sound worse," but you might eventually start getting intermittent sound, or the sound might go out completely. Electrical failures like this are pretty much "all or nothing" affairs.
For the most part, you only have to worry about two things on the main run of the cable. You'll either pull it apart (but in your case I doubt the tension is great enough for that), or you might be creating a rubbing condition for your cable, which could eventually tear away the shielding. How quickly the shielding degrades depends on what it's rubbing on. Sharp metal means (relatively) fast damage, but a smoother edge would take longer to get through. This should be pretty easy to spot and prevent, though.
The real thing you should be worried about is the terminals. That is, the places where the copper connects to something else are easier to break. The pulling you describe could be (essentially) pulling the wire out of your headphones. It will be fine for a while, but it would be weaking that point, and continued stress could break the connection. The same thing could happen at the other end (at the jack), but it will be a little more resilient since you are expected to pull (at least a little) on that end when you disconnect the headphones.
Are these headphones always in the same place? If so, you might consider re-routing the cable with some sort of straps so you limit how much of the cable is free to move. Even without clipping points, you might be able to route your cable so it doesn't get caught in anything.