Visanj,
I'm thinking you must have meant that you enabled High Dynamic Range, to get the best sound quality.
As I have said in earlier posts, I am a big proponent of using the Chromecast Audio as an all-in-one streamer/DAC/headphone amp - with headphones plugged directly into its 3.5mm output. Used that way, the sound quality is very near as good as my $1500 desktop setup, which includes a standalone streamer and a combo DAC/amp. The only place it falls down a notch is on very loud orchestral music and aggressively-played solo piano, where there could be some distortion and smearing of the sound. Typical recordings of jazz, pop, acoustic, chamber music, etc are all reproduced with no issues.
I have usually powered my CCA with a battery powerbank, feeling that it made for a slightly cleaner sound. Being the CCA came with a 1 amp "wall wart", I always used a 1 amp powerbank. Recently I had occasion to use a powerbank rated to provide 2.4 amps and I was somewhat shocked to find out the loud-passage distortion was eliminated. Evidently the CCA can draw more than 1 amp when called upon to deal with difficult passages. I listened to nothing else for a full week, with a wide variety of music, and never heard a problem. When I went back to my desktop system I did not notice any great improvement. The CCA really shines when more amperage is available.
Now, I realize that synergy between amp and headphone can have a big effect on SQ, but I find the CCA does work well with the half-dozen headphones I have used with it, varying from 16 to 300 Ohms. So YMMV. On the other hand, I find the audio output of the CCA exhibits no synergy with any of the amps I have used it with. Sound quality, while listenable, is unremarkable. I suspect it has to do with an impedance mismatch or somesuch. As many others have pointed out, the SQ when using the optical output with an external DAC is great.
I honestly feel that, SQ-wise, the CCA could suffice as my only "system". What it lacks is the ability to do gapless playback. That's a dealbreaker for me.
Happy listening