That's how to do it. Set your PS3 to Dolby Digital 5.1, and uncheck everything else.
However, if you have more than the optical cable connected (for example, using the HDMI cable...obviously), you can turn the Multi-Audio Out setting on, and it will send a 2 channel PCM signal to every other device that isn't connected via the optical (i.e. HDMI). That way, when you're not using the Mixamp, you can still get audio from the TV speakers (through HDMI), AND whatever is connected to the TV's audio out without having to switch your settings on the PS3. Since most TV's only do 2 channel audio out, the TV wouldn't benefit from getting a 5.1 signal, so the PS3's multi-audio out is perfectly fine.
Keep in mind that the PS3 defaults to Multi Audio Out: OFF if you tweak the audio settings, so always make sure it's on after you tweak any audio settings.
My setup:
PS3: Dolby Digital 5.1 through the optical cable > Mixamp
Multi Audio Out: On
HDMi to TV
TV's audio out: (optical out on my TV) > Fiio D3 DAC > Vizio Soundbar's RCA in (my optical input is broken on the soundbar). Had my optical worked on my soundbar, I wouldn't need the Fiio D3. Most TVs tend to have RCA audio out instead of optical, so use the proper cable for your setup.