I imagine you are right with your last sentence.
Re: diodes the operating speed, voltage and current must be a factor, the faster the speed the better the rectification. If it is struggling to keep up with the input to the diode before breakdown, that must affect the sound in a way. I don't think diodes are 100% efficient so this could be a factor, also when breakdown and reverse polarity occurs I think the efficiency of recovery to the correct current flow would have an effect. I don't know how that would affect sound but it could do. Obviously the better the diode switches on and off the better it will be. I have an amp, the APPJ, that has a diode upgrade which I haven't done on it, but I assume that the reason it is effective is because it is faster.
Just my twopenny worth.