Log out from the desktop and roon bridge will run smoothly (controlled w/ a remote or roon-tui from the console.
You could also install a pulseaudio-free desktop like Xfce (e.g. xubuntu, purge pulseaudio), but you can expect some issues (firefox e.g. does not run without pa).