RoonBridge armv8 memory leak

[MusicPlayer_Silabs]

Dear support,

I have recently installed RoonBridge in a fresh Raspberry OS bullseye arm64 image. This is a recent upgrade of the traditional Raspberry OS for the 64 bit architecture. My problem is that that the client exhibits a memory leak, just by staying activated on the background.

I have followed your instructions to install the client:
$ curl -O http://download.roonlabs.com/builds/roonbridge-installer-linuxarmv8.sh
$ chmod +x roonbridge-installer-linuxarmv8.sh
$ sudo ./roonbridge-installer-linuxarmv8.sh

The client version is:
$ cat /opt/RoonBridge/Bridge/VERSION
100800880
1.8 (build 880) stable

The culprit process is RoonBridgeHelper. Here are a few snapshot of its memory usage:

root 4619 36.8 2.0 1156484 40640 ? Sl 15:11 0:03 RoonBridgeHelper --debug --gc=sgen --server RoonBridgeHelper.exe
root 4619 0.8 2.7 1701324 52940 ? Sl 15:11 0:04 RoonBridgeHelper --debug --gc=sgen --server RoonBridgeHelper.exe
root 4619 0.4 5.2 2222868 102960 ? Sl 15:11 0:09 RoonBridgeHelper --debug --gc=sgen --server RoonBridgeHelper.exe

So we can see this process starts at about 40Mbytes, then increases steadily. Again the Roon client is doing nothing, the rest of the system is also idle.

The memory leak rate is about 1.6 MBytes per minute.

Thank you for having a look at this issue, as it can kill a headless always-on system. Also please ask for additional details as you need.

Best regards

[MusicPlayer_Silabs]

Sorry I made a mistake in my computation, the memory leak rate is about 1.6 kBytes per minute.

For the record, today RoonBridgeHelper has reached 524Mbytes:
root 4619 0.3 26.9 2528448 524456 ? Sl Feb25 10:13 RoonBridgeHelper --debug --gc=sgen --server RoonBridgeHelper.exe

Best regards

[MusicPlayer_Silabs]

Dear support,

Here are additional info about the memory leak I tried to gather with an utility called memleax. This is brute trace and I hope it can be helpful as a hint for you:

== Begin monitoring process 4619…
CallStack[1]: memory expires with 112 bytes, backtrace:
0x0000007faa7817b0 libc-2.31.so calloc()+0
0x0000007fa71cba50 libMonoPosixHelper.so CreateZStream()+56
0x0000007fa493bd10 ??
0x0000007fa493bd10 ??
CallStack[2]: memory expires with 7160 bytes, backtrace:
0x0000007faa780aa0 libc-2.31.so malloc()+0
0x0000007fa71f4d98 libMonoPosixHelper.so zcalloc()+40
0x0000007fa71f51c8 libMonoPosixHelper.so inflateInit2_()+204
0x0000007fa71cbae0 libMonoPosixHelper.so CreateZStream()+200
0x0000007fa493bd10 ??
0x0000007fa493bd10 ??
CallStack[3]: memory expires with 40 bytes, backtrace:
0x0000007faa7817b0 libc-2.31.so calloc()+0
0x0000007fa71cbb2c libMonoPosixHelper.so CreateZStream()+276
0x0000007fa493bd10 ??
0x0000007fa493bd10 ??
CallStack[4]: memory expires with 4096 bytes, backtrace:
0x0000007faa780aa0 libc-2.31.so malloc()+0
0x0000007fa71cbb6c libMonoPosixHelper.so CreateZStream()+340
0x0000007fa493bd10 ??
0x0000007fa493bd10 ??
CallStack[5]: memory expires with 24 bytes, backtrace:
0x0000007faa7817b0 libc-2.31.so calloc()+0
0x000000000089dad4 mono-sgen monoeg_g_calloc()+60
0x000000000089dad4 mono-sgen monoeg_g_calloc()+60

Best regards

[MusicPlayer_Silabs]

Dear support,

Also please note that the process RAATServer also has a memory leak, but somewhat slower than RoonBridgeHelper.

Just after roonbridge service startup:

root      127430 10.9  2.4 1361252 47120 ?       Sl   12:54   0:04 RoonBridgeHelper --debug --gc=sgen --server RoonBridgeHelper.exe
root      127441 10.0  2.3 1173440 45204 ?       Sl   12:54   0:03 RAATServer --debug --gc=sgen --server RAATServer.exe

After some time:

root      127430  0.3 16.2 2387352 316312 ?      Sl   Feb27   3:46 RoonBridgeHelper --debug --gc=sgen --server RoonBridgeHelper.exe
root      127441  0.1  6.3 1371488 124384 ?      Sl   Feb27   1:50 RAATServer --debug --gc=sgen --server RAATServer.exe

Best regards

[MusicPlayer_Silabs]

Hello,

For the record, now that several days have elapsed, RoonBridgeHelper is now 772MBytes. It seems that it never stops, unfortunately.

root      127430  0.4 39.7 2685044 772808 ?      Sl   Feb27  18:41 RoonBridgeHelper --debug --gc=sgen --server RoonBridgeHelper.exe
root      127441  0.1  6.0 1462824 118264 ?      Sl   Feb27   6:29 RAATServer --debug --gc=sgen --server RAATServer.exe

Does support had a look to this issue ?

Best regards

[wizardofoz (Mr Fix It aka The Wiz)]

@support what gives here guys?

[jamie]

Hey @MusicPlayer_Silabs,

Welcome to our Roon Community. We’re sorry to see that it was a spot of trouble that prompted your visit. Can you please install the latest version of Roon Bridge to that device and see if it helps? We’ll be watching for your update. Thanks!

Roon 1.8 (Build 913) is Live

Hi everyone, A few hours after the release of Build 911 our advanced crash analytics system began reporting a significant increase in the number of non-fatal crashes being experienced by our user base. As a non-fatal error this issue was not presenting any outward symptoms to users, but was potentially impacting performance and stability of Roon systems. The root cause was determined to be a change in the system which handles messaging between our cloud services and Roon Cores and Remotes. The…

[MusicPlayer_Silabs]

Hi @jamie,

If I understand well the release notes, the RoonBridge hotfix has been delayed at the last minute. Indeed I can see my PC Core Roon has been updated to build 913, but the Roon Bridge is still build 880, and is labeled as the latest version. Then I will wait for the next release.

Best regards

[Yaohan]

Hi there,

yep cleaning up some last minute thing and it should be out soon than late.

[MusicPlayer_Silabs]

Hello,

Following release of the latest 918 build, I did a quick test and unfortunately the memory leak is still there.

Here is the version:

cat /opt/RoonBridge/Bridge/VERSION
100800918
1.8 (build 918) stable

Here is the memory usage after a couple of hours:

root         592  0.2  8.3 1644576 162400 ?      Sl   10:51   0:18 RoonBridgeHelper --debug --gc=sgen --server RoonBridgeHelper.exe
root         607  0.1  3.0 1248180 59880 ?       Sl   10:51   0:17 RAATServer --debug --gc=sgen --server RAATServer.exe

Unfortunately it is not possible to leave this that way. As as said earlier, in the end the memory leak will kill a headless always-on music client. BTW I did the test at home, after several days memory is full, music playback starts to stutter, and the system has to be powered cycled.

When do you think you can solve this issue ?

Best regards

[MusicPlayer_Silabs]

Dear support,

In order to help you reproduce the issue, I checked on my side that the memory leak is easily reproducible on a vanilla Raspberry Pi OS. I did the following: install latest 64 bits Raspberry Pi OS on a MicroSD card, update the distribution, and install RoonBridge. The memory leak starts immediately. Nothing more is added/removed from the Linux system, and there is no need to start a Roon Core.

Here are more details. Download and install latest Raspberry Pi OS Lite in 64 bits from here:

Operating system images – Raspberry Pi

I used this one:
https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2022-01-28/2022-01-28-raspios-bullseye-arm64.zip

Boot the Linux system, connect in SSH and update everything:

sudo apt update
sudo apt upgrade
sudo reboot

Then install the RoonBridge client:

curl -O https://download.roonlabs.net/builds/roonbridge-installer-linuxarmv8.sh
chmod +x roonbridge-installer-linuxarmv8.sh
sudo ./roonbridge-installer-linuxarmv8.sh

You can start observing the memory leak immediately with this command:

ps aux | grep -E 'RoonBridgeHelper|RAATServer'

Best regards

[MusicPlayer_Silabs]

Hello,

Do you have some news about the memory leak issue ?

Best regards

[jamie]

Hey @MusicPlayer_Silabs,

The memory leak issues were remedied for Roon all-in-one and Server in the newest build, let me check with the team about those problems in Roon Bridge. A new bridge build wasn’t included in the newest update.

[MusicPlayer_Silabs]

Hi @jamie,

Indeed the standalone RoonBridge component has not been updated yet. Do you have any news of when an update will be available ? Thanks.

Best regards

[jamie]

Hi @MusicPlayer_Silabs,

Thank you for following up. Your report is currently with our QA team, they’ve been trying to replicate it without success. Have you noticed any specific steps that lead to this behavior that they can use for testing?

[MusicPlayer_Silabs]

Hi @jamie,

No unfortunately, no specific steps. As I said earlier, immediately after installation of the RoonBridge client on a vanilla Bullseye RapsberryPi OS (64bits), the memory leak starts immediately. Maybe it is something specific to my network environment, but I have no idea what it could be. If you want we can try a local debug session.

Best regards

[MusicPlayer_Silabs]

Hi @jamie,

Could you reproduce the issue now ? Do you want that we do a test on my local network ? Can I enable some sort of debug log files and send them to you ?

Best regards

[MusicPlayer_Silabs]

Hello @jamie,

Do you have any news? Could you reproduce the issue in-house?

Best regards

[MusicPlayer_Silabs]

Hi @jamie,

Do you have some news about this issue? Thank you.

Best regards

[ivan]

Hi, @MusicPlayer_Silabs. Sorry for a late reply. I tried to reproduce this issue with the RPi device that I have locally, but this issue does not affect my setup. May I kindly ask you to follow the instructions described in this article and provide logs from your RoonBridge?

1. Go to /var/roon/RoonBridge folder on your RPi device
2. Convert the folder called Logs into a .zip archive
3. Share this archive with us.

Sorry for troubles.

–
Ivan