This is incorrect, and can be seen as such through real world experience without even understanding the science behind it. By this reasoning, IEMs with their micro drivers would always be better than full sized headphones, and transducers in speakers would all be abysmally slow.
What you are talking about here is the transient response, and in speakers, it's not the general ability to "change it's motion," but more precisely, the decay after the sound has been generated--the ability to stop the motion--that differentiates between what people often describe as "fast" or "slow" when it comes to this audio attribute.
Even so, "fast" and "slow" is a bit of a misnomer in reference to what happens physically with transducers. The faster a driver can move, the more SPL it can output.