Archimago has a nice write-up on this, with reference to an AES paper Benjamin & Gannon “Theoretical and Audible Effects of Jitter on Digital Audio Quality”.
But take a look at this paper from 2005, particularly the last sentence of the conclusion: “The threshold values [for listener perception of audible change] seem to be sufficiently larger than the jitter actually observed in various consumer products.”