I leaned very heavily into trying to answer this question for myself (what became the Opto DX review linked in my sig below). With Hugo mScaler doing the heavy lifting to upsample, living with 192k and below really wasn't a big deal (although it was annoying to be limited to DSD64...I have a bunch of DSD128 and DSD256 content). Until very recently, my preferred reference setup was a NUC (USB) to Uptone ISO Regen (USB) to Matrix Audio XSPDIF2 to TOSLINK to HMS to OptoDX (optical) to DAVE. No fuss, no muss.
I still recommend the Chromecast Audio as a Roon end point to TOSLINK to anyone who'll listen. Best bang for the buck in audio, with minimal complexity/hassle factor.
Alas, if you want to get more SQ, you have to deal with the complexity/hassle factor.
For me, the Matrix Audio is a very clear step up from the CCA (I attribute this to a better clock, no WiFi, better power and ground management, etc). The ISO Regen to clean up the USB input to the Matrix Audio is also a clear step up (vs no ISO Regen). Powering the ISO Regen with a great power supply (in my case, the Uptone LPS 1.2) also made a big difference. The better power to the ISO Regen also has the side benefit that I could get rid of the power supply to the Matrix Audio, and have the MA powered by clean power from the ISO Regen (over USB).
If you're OK with a modest amount of hassle/complexity, an optical chain with the Matrix Audio is great intermediate stopping point.
However even with this chain (USB regeneration, high quality mains isolated power, two levels of optical isolation, etc), changes in software and configurations running on the NUC are still VERY clearly audible (maddening but true...signal integrity seems to really matter too)
All that being said, USB can (and for me, it now does) exceed what I was able to get from from the optical chain. It has taken a lot more focus on cables and power and all that icky stuff (and regrettably it will drag me kicking and screaming into the world of master clocks at some point), but I'm now back to USB end to end and incrementally tweaking things up again. Alas, things are also creeping back to lots of hassle/complexity.
If you're a Roon user, my advice is to start with the CCA ($30 used), remembering that it sounds like absolute ass if you're not using Roon. If you like what you hear, invest in better power to the CCA. If you like what you hear, track down a used Matrix Audio XSPDIF 2. If you really like what you hear, get a ISO Regen to put in front of it, with a solid power supply (LPS 1.2 is a great choice). Everything after that is 5x more hassle and cost to get the last couple percent of improvement (beware...here Dragons dwell
If you don't hear any differences or you get to the point of diminishing returns, happily stop and enjoy a fine beverage while listening to Anouar Brahem channel the angels on "The Astounding Eyes of Rita". If you do hear differences and want more, there is always more to be found down the rabbit hole.
Net net: noise matters and optical does an amazing job of cleaning it up, but alas, everything else STILL matters, even with optical everywhere...we haven't yet figured out how to build the mythical "moat" that makes everything upstream of it irrelevant, but an optical chain is in the sweet spot of the 80/20 rule.