I am using a similar vintage MacMini. +1 on disabling Time Machine. +1 on turning off anti-virus and similar programs. You want nothing else to be running at the same time Roon is running. +1 on setting Roon to Exclusive Mode. These measures give Roon better control of your core without interruptions or other demands on the computer’s resources. I back-up with a separate program set to run at 3 am.
But I also note you have your music files on the same SSD as MacOS. I assume you do the same on your MacBook Pro. Have you tried putting music files on a separate external drive? That is what Roon recommends. Since your problem occurs with streaming music as well, this may not solve your problem, but is a best practice for using Roon.
Also if your music database is corrupted, that too will cause problems even if you run Roon on another Mac. See this thread Database repair?
Do you have problems with downloads or streaming when running any other programs on either Mac? If not I would discount network issues.
Good luck!