I had the same problem with my Chromecast Audio. Dropouts with 96 kHz. Then I mounted the UGREEN Ethernet Adapter for Chromecast, since then the reaction of the Chromecast is much faster and the problem with 96 kHz does not exist anymore.
UGREEN Ethernet Adapter for Chromecast
Signal path with 96 kHz 24 bit
Signal path with 192 kHz 24 bit
