The S/PDIF cable (or connection) matters a lot; the USB cable and connection should not matter at all (within reason).
The whole point of an asynch USB adapter (like the Audiophilleo or the HiFace) is that the adapter does the clocking instead of the computer. Since the clock is now generated by the adapter, any timing issues that occur before the adapter no longer matter (unless they are so bad that they actually change the numbers). The computer and cable now supply perfect numbers, the adapter supplies a very good clock, and the timing of the stuff coming in through the USB cable should no longer matter. (This means that, if you can hear a difference between USB cables with this type of adapter, it's only because the adapter is failing to entirely filter it out).
The S/PDIF connection now becomes more important because, since it is AFTER the adapter, it has the opportunity to introduce new jitter into the process - and, since it's after the adapter, the adapter really has no control over it. Some adapters (like the AP) offer options about the shape of the data waveform at the output to try and improve this, and the quality of both the output driver in the adapter and the input receiver in the DAC matter a lot, but what really helps is either a direct connection or a really good (and really short) S/PDIF cable. Any of these will limit the overall signal quality, and can quite possibly completely negate the benefit of a fancy USB interface adapter if they're bad enough. A direct connection (like the AP uses) seems like the best choice there.
In addition, any DAC that uses a sample rate converter CORRECTLY to eliminate jitter, or some other method of entirely reclocking the signal, should entirely obviate ANY audible differences of ANY upstream components since it, like an asynch adapter, uses the original data but entirely replaces the clock with its own (so the jitter should now be TOTALLY dependent on the jitter in the DACs clock and nothing else). Assuming this is done correctly, this should be the best solution, and should eliminate ANY audible (or measurable) differences between cables or interfaces.
Anything like an upsampler, which creates entirely new numbers according to some algorithm, basically becomes an entirely new signal source - so how good IT sounds depends entirely in how good a job it does... and what's between it and the DAC.