Welcome to the Roon community, @Alex_Kozlenkov.
DSD512 requires a PCM sample rate well in excess 768 kHz, and your kit may not support this.