I agree with you in principle, however, I would say not quite back to square one. After I adjust to a particular headphone, and then switch among my harem of phones, I'll find the re-adaptation process gets much faster. So much so, that I could probably switch between any two and adjust within a few minutes.
Of course re-adaptation is much faster. You already acquired a memory of how a headphone sound like. Just like any skill you learned and stopped doing it for a long time. You'll get rusty, but it will come back to you sooner than learning it the first time.