From Steve Nugent (a leading authority of digital audio). It's a little dated, but there's some fantastic info here. I "bolded" what I found to be the most significant parts.
Jitter Correlation to Audibility
The correlation of jitter measurements to audibility is in its infancy IME. The problems start with the characterization of jitter. Generally, manufacturers of crystal oscillators specify jitter in terms of RMS jitter amplitude. The problem is that they often neglect to state that this is specified at 10kHz and higher. There is also no spectral or frequency content information specified. This makes it very difficult to tell which oscillators will have audible jitter or objectionable jitter.
For instance, Empirical Audio uses two oscillators that are both specified at 2psec RMS jitter. The two oscillators sound radically different to me when used in a re-clocker in a resolving audio system. This leads me to believe that the spectrum, or frequency content of the jitter is as important or maybe even more important than the amplitude. I also believe that correlated jitter or jitter with a relationship to the data pattern or audio signal is also more audible than random jitter. This seems to be the consensus in a number of AES papers.
Studies by the AES (analysis, not human testing) conclude that these are the thresholds of audibility:
 120psec P-P jitter audibility threshold for 16-bit DAC and 8psec P-P jitter audibility threshold for 20-bit DAC
 20psec P-P of data-correlated jitter audibility threshold at certain frequencies and "A simple model of jitter error audibility has shown that white jitter noise of up to 180psec P-P can be tolerated in a DAC, but that even lower levels of sinusoidal jitter may be audible"
Since many measurements (that don't specify any particular frequency content) performed by Stereophile in  are above 150psec or close to this, I do not believe that we have reached the limits of jitter audibility yet. I suspect that P-P jitter needs to be almost an order of magnitude smaller, or around 15psec to be inaudible in all systems.
I believe the ability of the human ear/brain, particularly the trained ear, to hear minute differences, particularly data-correlated jitter, is grossly underestimated. The live listening AB/X studies published to date (that I have read) are inconclusive IMO. The systems used were not resolving enough IMO, the recording quality was not good enough and the test signals were random and not correlated and therefore inadequate to properly test for jitter audibility. I tend to believe the numbers arrived at by the AES analytical studies rather than the A/BX listening tests.
There are a series of double-blind tests being performed by many audiophiles using synthetic jitter tracks provided by HDTracks. These may shed some new light on true audibility. Again, the effectiveness of these experiments is only as good as the quality of the tracks provided, the jitter that was synthesized and the audio systems that are used for testing. The results from the first set of jitter tracks shows just how unresolving most audiophile systems are. There are couple that could pick out the majority of the tracks by increasing jitter, but the majority could not hear any difference between the tracks, even though the jitter ranged from 0 ns to 1000ns I believe.
Another interesting thing about audibility of jitter is it's ability to mask other sibilance in a system. Sometimes, when the jitter is reduced in a system, other component sibilance is now obvious and even more objectionable than the original jitter was. Removing the jitter is the right thing to do however, and then replace the objectionable component. The end result will be much more enjoyable.
Jitter can even be euphonic in nature if it has the right frequency content. Some audiophiles like the effect of even-order harmonics in tubes, and like tubes, jitter distortion can in some systems "smooth" vocals. Again, the right thing to do is reduce the jitter and replace the objectionable components. It is fairly easy to become convinced that reducing jitter is not necessarily a positive step, however this is definitely going down the garden path and will ultimately limit your achievement of audio nirvana.
Sibilance in a system caused by preamp, amps and other components and cables can also be so high that changes in jitter are not very audible. This is why there is such contention on the web forums about jitter and its importance. What matters in the end is if you are happy with the sound of your system, and whether or not you can hear this distortion.
Edited by robertsong - 6/19/13 at 2:46pm