My thought, and what I have heard from some others, is that has more to do with the filtering involved rather than how high the sampling. Higher sampling rates get the advantage of being able to use less of sloped filter adding less phasing and combing artifacts at lower frequencies.
The same idea can be said of better A/D converters. Better ones have better, more accurate filtering with fewer effects. As well as better beter input sections and power supplies and overall components. Just saying that filtering is a big part of a good A/D filter vs great ones.
Isn't it also legend of Rupert Neve making mic preamp's sounding better by removing the 20k filter cap. Not because it was restricting anything, but because of the artifacts it was creating at lower frequencies and overall quality of sound.
I am not trying to say this explains everything, but something I have not seen discussed in this thread. I know I can't hear anything past 15k. Some people like to talk about psychoacoustics and perception, like "your eardrum hears it, but the rest of your ear can't." Or, "eventhough you can't hear above 15k, you CAN perceive it." I don't subscribe to this, but scientifically there is more evidence in filtering artifacts. This might explain at least some of it.