The CD has two levels of error correction: C1 and C2. C1 errors happen all the time, even with brand spanking new discs, and are 100% correctable. They're just part of the normal data reading strategy. C2 errors occur when the data can not be recovered perfectly. In this case the player interpolates between the previous good audio and the next good audio it can get. This is usually short and rather transparent, but it is none-the-less an "audible" error. Once the gap gets too big for interpolation, you will hear distortion like it's a scratched CD. The distortion can also be due to the optics inability to focus on the CD correctly, again due to smudges and scratches.
So, yes, eventhough you may not hear skips, it is quite possible that a scratched CD is putting out errors in the audio. It is not guaranteed though--the scratch has to be big enough to cause a C2 error and then on top of that the error has to be big enough for a "theoretically audible" distortion to turn into an "obviously audible" one. A quick way to gauge is to rip the CD to your computer using Exact Audio Copy and see if the C2 error indicator bar lights up at all.
By the way, due to the way the data is packed onto a disc, a radial (parallel) scratch on the disc is much nastier than a perpendicular one. Thats why you shoud always clean your CDs in a straight line from the inside out to the edge as opposed to a circular direction.
