It likely works on Windows and possibly on Linux too. But it requires possibility to send raw DSD to the DAC without DoP. However, CoreAudio of macOS doesn’t support such possibility. So for DSD256 to work over DoP the DAC would need to support 705.6/768k PCM because DoP in these cases has 50% overhead for sending DSD data.
So it is limitation of the combination of macOS and the DAC. Neither one alone have this limitation, but combined limitations of both cause this to be the case. Resulting capabilities are intersection of the capabilities of both.
If you wanted to you could load HQP and the Audiolab Audio driver onto a Windows machine, send the output from Roon on your MBP over your network to HQP and connect the Windows machine to your DAC.
That way you could continue to use Roon on your MBP Core without changing anything, continue to control HQP on the MBP using RDP for Mac (or HQP Client as noted by Jussi below) and be able to send DSD 256 to your DAC.
If you do decide to try this out then check the Roon KB about connecting to HQP on another computer. There is a Network Control icon in HQP that you have to toggle on.
