There's a 60 page thread on another forum I will not mention that discuss speaker break-in at length with measurements and citation to research papers.
TL DR, break-in can happen, it can be measurable, if it happens it does not take much time (minutes to a few hours); it's unlikely you can hear a change because any measurable difference is so minor.
But here's another possible reason why: it's all in your head.
Harmon Kardon published research about sighted bias and beliefs about a product on the perception of sound: "The results are very clear. Figure 4 shows that, in subjective ratings of four loudspeakers, the differences in ratings caused by knowledge of the products is as large or larger than those attributable to the differences in sound alone...In other words, knowledge of the product identity was at least as important a factor in the tests as the principal acoustical factors. Incidentally, many of these listeners were very experienced and, some of them thought, able to ignore the visually-stimulated biases."
https://www.harman.com/documents/AudioScience_0.pdf