We don’t support DSD256 over DoP yet–Roon only does DSD256 output in native mode (meaning, on Windows using an ASIO driver).
The main barrier is: we haven’t got a piece of hardware in our test lab that supports that configuration. DSD256-over-PCM requires a DAC that supports 768khz PCM input–these are pretty rare.
Once we get our hands on some hardware, we can validate/test this configuration and then enable it in Roon.