I understand "speed" to be the ability of a driver to start producing a tone, stop producing a tone, or both simultaneously, which you should be able to hear as instruments' notes coming and going faster, not actually at a higher tempo, but more clearly defined. Drums are most easily used to evaluate speed.
also, for the purpose of this post, tone=given frequency
I am by no means an expert, or even close, but my limited understanding of measurements and subjective analysis lead me to believe that the decay would have just as much effect on the speed of a driver as the attack. We hear the waveform as a tone, not a moving signal, if that makes any sense. The characteristics of the waveform our ears are subjected to defines the characteristics of the tone we hear, not the characteristics of the changing of the tone, which is what I understand "speed" to be. Assuming all of this is correct, that means that the nature of the driver's squarewave and impulse response is of no use in determining the "speed," however, it does help in determining the subjectively perceived tone, which can have an effect on the subjective "weight" of notes, which may lead to a differently perceived "speed," doesn't actually change the speed as defined at the beginning of the post. Decay and attack, on the other hand, are the name of the game when it comes the speed. CSD plots measure how long it takes for the driver to stop producing a frequency. I'm not aware of any measurements that measure how long it takes for the driver to start producing a frequency, but I would be interested in seeing some of those. Measurements of the time transitioning from one frequency to another would also be interesting.