Hey, thanks for the reply. I tried positioning it differently and adjusting the swivel a ton, unfortunately without much avail.
I will share what adjustment I think fixed the problem for me yesterday (please excuse the quick MS Paint drawing haha):
I noticed that, atleast for my headband, the shape itself didn't contour well on my head out of the box in my preferred adjustment setting. I applied force with the headphone on my head until the headband comformed more like in the right diagram here. I now seem to get closer to the ideal comfort that was more hit or miss previously. The clamp now feels more solid and I'm not getting those feelings of instability that caused me to keep adjusting all day sometimes. I feel like the headband making contact in as many places along the head as possible is the most important factor here.
Just thought I'd share in case it works for anyone else, even if this seems fairly obvious now in retrospect.