OK, let's talk high-end.
If you want the best, no-compromise, RCA switch, 'ya gotta talk to Dr. P! (Lloyd Peppard). I have no idea if he his still in business, but try him!
You want to have an LR1 built for you. Typically it is for switching two or more sources into one amp, but your use case is fine too.
Welcome to Mapletree Audio Design
Or, do what I did, and go the patch panel route. Here you need one with audiophile grade connectors, wire, and solder. The best in the world -- One Visit Media. I cannot say enough good things about these guys.
Actually, you want a "feed thru" enclosure, not a patch panel. Get a half-deep unit, "normaled" -- that means front-to-back straight-thru wiring. Specify top-of-the-line wire, solder, and RCAs -- get 8 RCAs on it, front-and-back, all female.
Wire your stero pair sources (this allows two sources, even though you are using only one now) to the first four RCAs on the back, and wire your two amps to the last four.
Now you never touch the back again -- push it up against the wall -- and you make whatever connections you want on the front. You will need 16 short M-M RCA cables -- go to BlueJeans Cable for these. With the half-depth unit it is easy to steady these by hand if you don't rack 'em, as you make/break connections.
One Visit Media Custom Patch Panels
A switch is easier for sure, but a wired enclosure (which the world views as a patch panel) is way kewl!
See two of mine (I have four, each with 16 patch points: 2 XLR and 2 RCA. I use them top-and-bottom for left/right, but you can just use adjacent pairs for L/R) -- top right in the photo:
http://lh5.ggpht.com/_240MzL2EA8w/Sr...with%20IMP.jpg