I see two things here.
First–our Convolution engine is probably not up to the task of running anything but a very tiny filter at DSD sampling rates. This is the same use case that pushes the HQPlayer people to spend thousands on dedicated upsampling machines with expensive GPUs on board. There is some room for optimization in our Convolution engine, too–but I can’t guarantee that optimizing the code will get you there by itself (or have a sense you what hardware you might need) until that work is done. So using “Enable Native DSD Processing” and Convolution together might not be practical. I recommend the Native DSD stuff off for now if you’re having trouble combining it with other processing.
Once you do, convolution will only be happening at 352.8kHz and 384kHz–which seems like it should be very practicable.
Except in your data, it isn’t happening.
There’s another performance pitfall that you might be running into, too. Maybe you’re having Roon up-sample the convolution filters and they are getting very large?
For example, if you provide us a 131k tap filter at 88.2kHz, then attempt to convolve at 352.8kHz, it would seem like 4x as much work is being done to generate 4x as many samples. The problem is: it wouldn’t be correct to use your original 131k tap filter at 352.8kHz since it would change the meaning of the filter. Instead, we need to up-sample it, just like the signal.
So that 131k tap filter is resampled to a gigantic 512k tap filter. So Roon is not doing 4x the work–it’s actually doing over 16x the work. That’s a big multiplier–enough to take a situation that would have been stable before and make it unstable.
Most software generates the same filter order regardless of the sampling rate–REW and Acourate both fall in this category. So they’ll make a 65k or 131k tap filter for both 44.1kHz and 352.8kHz. They can do this because they understand the meaning of the filter that’s being generated–whereas Roon must blindly resample because we don’t know the internal details of the filter.
I’m not sure if this is happening with you–but if you were to generate convolution filters at each sample rate where convolution might be taking place, and put them all in a zip file it may help.
This is a signal path that I run very comfortably on a NUC5i5. Your sonicTransporter is probably a faster machine. In this setup, the convolution always happens at 352.8kHz, and the filter was generated by Acourate. I think it should be possible to get to someplace like here on your transporter.