Since it is based on ES9039, if you have enough processing power, I would recommend to try running it at DSD512, with the default filters in HQPlayer for starters at least, and you could try ASDM5EC-ul modulator. If this works, you could also try -light and -super (and x512 fs) versions of those modulators. This takes quite a bit of processing power, so depends on how powerful your PC is, and whether it has an Nvidia GPU for offloading some of the processing. If DSD512 doesn’t work, I would try with DSD256.
Once this works, you can also switch to ASDM7EC version of the same modulator (like ASDM7EC-ul) for example and compare. Once you know which modulator you prefer, you can look into the oversampling filters. In the PDF manual, there’s a filter table and some rough suggestions (by no means definitive) which ones may be good fit for your source material.
Please also remember not to turn HQPlayer volume higher than -3 dB, this leaves (usually) enough headroom for inter-sample overs that commonly happen with modern source material. Otherwise there is risk of triggering the limiter, which is best avoided. If limiter is triggered, Limited-counter increments and the volume knob turns red.