I almost hate to reply to this so many days after the original post, because i'm betting a lot of people have shared what I'm about to. But just incase that isn't true,
@Dregur it might depend on what phone you're using these with. Even on older phones with QC SOCs on the 800 series prior to 845, as well as other 6 and 7 series, I've found that if I have a True Wireless model that is otherwise capable of being used in mono, i can pair each individually, and use one at a time. When both are out, the handoff is generally seamless. The phone stays paired with the TWE that was out first, and worst case scenario I've had to resume the audio. Can't speak to how this might work with an IOS Device or on Android phones with mediaTek SOCs.
Regarding the WF XM4,
@DigDub both TWEs pair to the phone at the same time. If one is placed in the case then the audio usually and I'd say 90-95% of the time, doesn't stop playing from the other TWE. I've been using these pretty consistently for 12 days now, and have switched to mono with each often, and never had the audio stop. But after reading your post, I wanted to see what would happen if I kept swapping them wrapidly. only then did the audio stop and I had to resume it FWIW.
I switched them in and out probably 20 times, and the audio stopped twice. But in more normal use cases, as I mentioned, switching from one to the other while keeping one in has been smooth sailing.
Also, the WF XM4 remember the state they're in, so if I leave them in Ambient, put them both away and pull them out later, they're in Ambient. if I place the left TWE in the case while in ANC then the right TWE seems to switch from trying to use ANC to passive. i can hear it cycling, but it doesn't go in to Ambient mode. But if I've previously had Ambient mode on while both were out, then ambient remains active with either TWE being used in mono.