The basic switching
The basic switching between inputs is most probably the easiest part:

B1L and B1R is balanced input 1
S1L and S1R is single ended input 1
B2L and B2R is balanced input 2
S2L and S2R is single ended input 2
OBL & OBR is the balanced output.
CIS is the single ended input of the single ended to balanced converter
COBL and COBR are the outputs of the single ended converters
With the relays K1-K4 you can switch between the two inputs (you need a lot of relays though)
With the relays K5 & K6 you can choose if the input is going through the Single ended to balanced converter.
It is of course easy (and will consume a massive amount of relays to enhance this schematics to 4 input channels.
Next is the control circuit. Since this will be the most interesting part.