How difficult would it be to add a switch so you can choose whether to use the headphone jack or speaker taps? Even having headphone jack take precedence would be fine, as only the speaker taps are more of a hassle to plug/unplug every time you want to switch to speakers.
It's called a speaker switch Get a 2 way speaker switch and connect speakers to one output and headphones to the other.
To do this inside an interface box you'd need a switch built for speaker gauge size wire. Since many of us are into our music we tend to use 14 to 8 gauge speaker wire and that really would be difficult to work with inside the box. It's doable but it wouldn't be easy.