HRTF variance certainly exists regardless of fit, as even measured eardrum responses have shown to be different among individuals. According to Rin's blog, the
best available DF response curve is based on measured eardrum responses of 47 subjects. Note that not all of these subjects have been tested under the same conditions at the same time, but data was gathered from various studies and averaged into one curve. To put things into perspective, the world population consists of roughly 7,100,000,000 (7.1 billion) people.
Having said that, HRTF variance is actually taken into account in Rin's graphs in form of a
target corridor with blue and red dotted min/max lines. This corridor is about 4db wide in the range where our ears are most sensitive and for which ABX tests have shown that
differences of 1db or more are definitely audible. Which, in my understanding, means that HRTFs of different individuals can differ quite audibly and may still be considered as within tolerance of the same DF target corridor.
Jm2c, from a skeptical layman's point of view.