Different compression, different technology. Dolby Digital is the required digital audio film and DVD spec. It's printed inbetween the holes for gears on the actual film. It's a 5.1 system that can have a number of bitrates but is almost always 448k/sec for movies. DTS is an alternate spec and isn't manditory. It's not on the actual film itself, but comes off of CDs that are sync'd to the projector. On a DVD it's just anohter audio track. It can do 5.1 or 6.1 discreet channels, can can also do 24-bit 96kHz in it's 24/96 variety. It can be either 768k/sec or 1.5m/sec, but on DVDs is generally 768k.
The net effect is that, largely due to the higher bitrates, DTS sounds better. It tends to have better detail, better highs, etc. Since the X-Fi decodes both, if you have a DVD with DTS, use it.
I would use CMMS with this, espically if you are listening on headphones as it'll do HRTF to try and give you a real surround image through just your headphones.
I'm not sure what you mean by listening through S/PDIF. Do you have an external DAC, or do you have a reciever?