I’ve got Roon Core installed on a virtualized Ubuntu 22.04 host.
The playback to multiple devices such as a HiFi Berry DAC+ equipped RaspberryPi and a Naim ND5 XS2 are running perfectly fine, usually with minimal CPU usage.
Additional to these devices, I have set up two AirPlay Receivers in a Control4 System and when I try to play music to these using Roon the CPU Usage instantly spikes to the point where the playback to the other devices isn’t even running smoothly anymore. This happens, no matter how many CPU cores I assign to the VM. It’s currently running 6 out of 12 Cores on the following CPU: 12 x Intel(R) Core™ i7-10710U CPU @ 1.10GHz
As soon as the AirPlay Playback is stopped, the CPU usage drops to usually below 1% and as soon as it’s restarted it goes back up to between 80 and 95% and starts to drop audio on all devices again.
It seems to be trying to send the AirPlay signal with all the available Bandwith.
Everything is fully updated and not showing any helpful error messages. Rebooting the system does not affect this behavior at all.
I might try to move the VM to a more powerful host to figure out the limit of how much CPU cores it would need to run this in a stable manner, but as the CPU usage of the roon process changes by a factor of around 1000 so far this seems to not be a power-related issue.
Any logs or additional system information that would be useful for troubleshooting this?
Could you give a few more details about the virtualization software and host OS you’re using and the vm networking details please? I suspect no amount of cores are going to solve the issue…
The VM is running on Proxmox 7.3-4 and has a virtualized network card that is passed through a linux network bridge inside the host OS. I could maybe try going through a dedicated USB Network Adapter. As it’s just a NUC running Proxmox on this Host, I won’t be able to properly add another network card.
Interestingly, the issue only occurs when streaming to the Control 4 system over AirPlay. If I use the Naim as an AirPlay destination, the CPU usage just increases slightly from the base level and the network throughput is a lot lower.
All the network traffic is definitely reaching the network, as I can also see big spikes in switch and AP CPU utilization that directly aligns with when music is being played from Roon over AirPlay and when not.
Even more interesting, so it’s not simply airplay that’s the issue. I’ve not gor any proxmox experience. What are the hot processes on the VM? I suspect that the issue is related to the processes handling the virtual networking. I have seen userland-proxy processes cause havoc on Docker boxes in response to heavy network traffic.
Edit: I’m thinking bad Airplay implementation causing network spike. That leads to a proliferation of processes to handle the traffic. All just guesswork though.
Also AirPlay clients will end up being downsamples a lot for any 24bit content so extra cpu process there. Also Roon only uses one core for standard use so doesn’t matter how many cores you throw at the VM it won’t help.
Likely the environment it’s running in, Roon isn’t optimised for a VM it’s a very busy client even when not in a VM. Perhaps something else is going on causing that high though as it seems excessive. It’s never gone very high when I have used my Ubuntu install when my Rocks out of action.
The CPU usage only spikes when the AirPlay Output to the Control 4 system is actively playing. A few seconds after playback stops, the CPU usage and network traffic both drop down again.
I’m currently restoring the VM on a different host to see if direct PCIe access to the network device or just a bunch more cores might help in this situation.
So scratch my theory, or the network process part of it at least. Given it’s the Roon process alone that’s running away. Interesting, but not in a good way for you…
Edit: Roon’s a pretty verbose logger, have you looked at the last 1k lines while this is happening?
The following 4 lines of log messages appear over and over:
Sooloos.MulticastSocketReceiver._ReadLoop
Sooloos.MulticastSocketReceiver+<>c__DisplayClass8_0.<_ReadLoop>b__0
System.Threading.Tasks.TaskToApm+TaskAsyncResult…ctor
System.Net.Sockets.Socket.BeginReceiveFrom
The following message including AirPlay occures often:
[airplay/discovery] Sent mDNS discovery list response
The following messages occur regularly as well but not as often:
System.Threading.ExecutionContext.RunInternal
System.Threading.Tasks.AwaitTaskContinuation.RunCallback
System.Threading.Tasks.Task.RunContinuations
System.Threading.Tasks.Task1.TrySetResult System.Threading.Tasks.ValueTask1+ValueTaskSourceAsTask+<>c.<.cctor>b__4_0
System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation
System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted
System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute
System.Threading.ThreadPoolWorkQueue.Dispatch
System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart
01/07 16:19:33 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
It logs so much that all the available 21 past logs get filled up in, as far as I could figure out, 1 to 2 seconds (There’s not a single of the regular stats log messages present in all 20 available history files). As soon as the AirPlay is stopped, the Log Rate drops drastically.
Do you have any suggestions what to specifically look/search for in the logfiles?
I suspect this line says it all, the RoonServer thread pool has reached it’s limit, so one process but 100 threads strong. That’d explain the CPU activity.
I suspect here’s where it’s spawning the thread.
This then chokes the thread pool and it’s game over. Have you tried this outside of a VM? If so I presume it behaves normally?
Installed Roon directly onto the Host and it somehow made the issue even worse. It now has spikes to over 1500% CPU usage. Error messages seem to have stayed the same. This is a bit hard to check again as all of them get overwritten each second.
This is a brand new Ubuntu 22.04 install that has just the dependencies, Roon Server and nmon for checking the per core usage installed.
OK, then it’s not virtualization is the issue at all. It appears it’ll consume all available resources. You’re at the stage where support’s required. The Roon team might be interested in the behaviour but on balance of probability I’d guess it’s a Control4 issue. I’m tagging @support here to raise it.
Testing wise, you could try a different Linux flavour or two. I’d be interested to see how it behaves on other OSs like Windows or Mac.
I’d agree although they’re all pretty similar distros, Ubuntu is basically Debian testing at the time. Ubuntu 18 rules out something like a recent kernel issue so you’re probably right in thinking that Linux versions are a dead end. I’m not trying to make work for you but there is always ROCK
Windows is probably the most sensible next move BTW.
I didn’t get ROCK to install just now under Proxmox for some reason (couldn’t even get the installer to launch). Might give that one another try too. Just for testing it out, I’d like to keep at least the Linux Distros as VMs because it’s just so much more convenient to set up.
This also is the first issue I ran into running roon as a VM under Proxmox. I migrated it from bare metal to Proxmox VM about a year ago and it has been running perfectly fine ever since and even handled a dist-upgrade without any issue.
Currently working on the windows clean install for testing there.
