There's a difference between speed and speed.
High speed of the driver means great transient response so it can reproduce high frequencies better. If you see an early roll-off the driver is "slower".
Perceived speed is a more complex matter. For example, boosted sub bass may give the impression of "slow" and "lagging behind" bass.
Or if a headphone reproduces the faster decaying overtones of an instrument louder, for example because of a treble boost, it's probably gonna be perceived as "faster".
The speed and timing relationship didn't really change though.
I could also think of high amounts of distortion mudding up the whole spectrum which could cause the decay of distinct tones to be perceived as faster (they would disappear earlier in all the added distortion products).
I think that's why I think headphones sound "fast" or "slow." :/
i.e. LCD-2 sounds "slow" to me because its bass is really prominent; the STAX SRS-2170 sounds "fast" to me since its treble seems to be reproduced without any obstruction of other sounds.