The famous AES paper on the audibility of jitter:
http://www.nanophon.com/audio/jitter92.pdf
"At 20kHz the peak to peak sampling jitter must be less than 20ps, increasing at 6dB per octave for lower frequencies until approximately 500 Hz where the limit is 1 ns. Below 200 Hz, the jitter may up to 500 ns in amplitude before the sidebands could become audible.
For oversampled systems the sampling jitter sensitivity may be worse. As the sampled signal could have a higher frequency than the Nyquist worst case of 24 kHz the sensitivity increases further. For a delta-sigma DAC any jitter at 150 kHz, for example, may modulate with the shaped modulator noise at around that frequency creating modulation products falling in the most critical parts of the audio spectrum"
Many DACs oversample and a large majority are delta-sigma. If you look at how OS and delta-sigma work, it's no surprise that they're more susceptible to jitter. So that 20 ps number.. it's actually a conservative number.
This is probably one of the most technical and informative posts about jitter by an RF engineer with years of experience working with spdif (and jitter):
http://www.audiocircle.com/index.php?topic=90220.0