That option is no longer available in 1.2 and I can only set max sampling rates in PCM. When I attempt to play DSD files there is no sound.
Just to clear something up: Roon has never had a “DSD Playback Strategy” setting for RoonReady devices like the Aries. That setting is for direct-connected outputs only–RoonReady devices manage their own DSD playback details (this is done in Lightning DS for the Aries).
One detail I’m not 100% clear on is whether or not your AV8802 supports DoP-encoded DSD on its coaxial input or not. I’m also not clear on what exact settings you’ve chosen in Lighning DS or Roon, so I’m guessing a bit here. If I’m off the mark, help me understand some of those aspects and we’ll dig deeper.
Anyways, I just pulled an Aries down off the shelf, set it up to use the Coax, and ran through some test cases.
I plugged it into a DAC that I know can do DSD64 in DoP mode over Coax.
First I enabled “DSD over PCM (DoP V1.1)” in Lightning DS. In that configuration, DSD playback works fine, and the DAC recognizes the stream as DoP and plays it properly with this signal path:

Then I disabled the DSD Over PCM setting in lightning DS and played the same file. Now, Roon converts the DSD to PCM on the server, as I would expect:

Unfortunately, I hear silence…Maybe this is the thing you’re running into, not 100% sure.
Anyways, there’s a good answer to why I’m hearing silence: The Aries is trying to play 352.8k audio out its Coaxial S/PDIF port, and that’s too high of a rate for S/PDIF.
This is happening because the Aries tells Roon that it can support that format. I’m not sure if this is because Auralic knows of situations where that rate is useful, or if it’s just a minor oversight in their RoonReady implementation.
I got things working again by switching the “Max Sample Rate (PCM)” setting to 192k. That made Roon do this instead:

And that makes DSD64 playback work for me again, since 176.4k over S/PDIF is ok.