Not sure what the root cause really was but I had to enable Airtime Fairness in my router (Asus RT-AC88U) settings. The Chromecast Audio does not work in my network when this is disabled. But after enabling it, I had problems with many devices: with my smart plugs, my TV, even with phones and tablets sometimes.
There is an external thread dedicated to this problem:
https://www.reddit.com/r/Chromecast/comments/780xs4/big_issues_with_asus_router/
So in my case:
Enable Airtime Fairness > Chromecast Audio works but other wifi devices are often randomly disconnected and it is really annoying.
Disable Airtime Fairness > Chromecast Audio does not work but my wifi network is fine: 10-20 devices connected without any problems...