can you try with:
DOTNET_GCRetainVM=1
DOTNET_gcServer=0
?
I think you got this right already, but I want to try to answer anyway.
When making changes using /etc/environment, you’ll need to reboot, then attempt to use Roon “normally”, and see if physical memory usage seems to grow to consume all available memory on the machine.