I think foam tips are much more affected by sizing than silicone tips are. Obviously, either foam or silicone tips, if too small for a user's ear, will not form a seal and the sound will suffer.
However, when you get into larger sizes, silicone is much more forgiving than foam, in my opinion. For instance, if a medium sized silicone tip is enough to form a perfect seal in a particular ear, the sound will generally not degrade if a larger size is used (assuming there is still a seal)... It may just cause the user discomfort.
Conversely, going too large with a foam tip often results in the end remaining pinched, effectively narrowing or closing off the port... Resulting in muffled, loose, or otherwise bad sound.
I think many people forget that a properly sized foam tip will not seal immediately. It should feel too small when first inserted, and then gradually seal over roughly a thirty to sixty second period as the foam expands and fills the ear canal. Assuming the sizing is right, the port should remain fully open and sound quality will be unaffected.
In my experience, it is far easier to achieve great sound with a silicone tip, but taking the time to find a properly sized foam tip can result in identical sound, with the added benefit of superior isolation. Personally, I only use foam tips... Assuming the correct fit is possible. And I think the Shure Olives are some of the best on the market, along with the Westone True-Fit.
But like everything in this hobby, YRMV...