This discussion is starting to occur about once every other week.
The solution to getting the best out of bit-perfect playback is to follow the Roon guidelines by separating your Core and Endpoint into two separate boxes and using RAAT over the network. If you want the absolute best in upsampling / DSP then you bounce through HQP which is already supported by Roon.
The fact that your Mac sounds terrible as the endpoint is not a Roon problem. Yes, plenty of software packages do a better job of optimizing the Mac for playback but Roon has never indicated they intend to “fix” the Mac; its a poor endpoint. The reference architecture is to use a Roon Ready dedicated endpoint that is optimized for that role within your Roon set-up.
In order to make the Mac sound better Roon would need a Mac optimized Bridge doing the same tricks that Pure Music, Audirvana, etc. do on the Mac to make it sound good. That’s not their goal. They want to keep the heavy CPU optimizations (like DSP) on the core and keep Bridge doing as little as possible.
Checkout the thread from last week helping this individual get better quality by adding a Raspberry Pi. Please investigate doing the same. Ok, Don't Flame Me. Settings to sound like Audirvana?