Well, I spent an hour last night stealing everyone's phones (and having to explain to each one What a headphone amp was lol). It was affected by 3/6 phones in the house.
Then I noticed I could make the issue worse by touching various parts of the chassis, which sounds a lot like a ground issue. So I searched this thread for "ground" and found a bunch of posts talking about grounding issues with these. For example:
I opened mine up and sure enough, I had 1/3 ground points scraped.
I scraped them, unfortunately this didn't stop the interference, but maybe that solves some problem that I didn't notice yet.
Also checked the opamp. It was secure, but thank you for the suggestion.
So for now I guess I'll live with it, and plan to get the Bluetooth add-on. Honestly this thing is so thicc that even if I didn't have this issue, I probably would have never strapped it to my phone (the Note 5 big enough as is)...but that grass looks greener.