Roon Server installed 1.7 (Roon Version 1.7 (build 710) stable) on Ubuntu 20.04.1 LTS.
Roon Server installed on hardware with i5 -9500T with 16GiB DDR4 SODIMM and Western Digital SN720 NVMe SSD.
95000+ tracks stored on external USB 3.0 (connected to core) Western Digital EasyStore HDD formatted as EXT4 (10TiB).
Network Details (Including networking gear model/manufacturer and if on WiFi/Ethernet)
Core and endpoint are hardwired to the network (and are on the same subnet), both devices are plugged into a Netgear GS105 (unmanaged switch).
Router is a pfSense device.
Audio Devices (Specify what device you’re using and its connection type - USB/HDMI/etc.)
Raspberry Pi 4B with RoPieee 2.596 STABLE installed. (Roon Version 1.7 (Build 571) stable)
Chord Qutest connected to RoPieee via USB.
Remote Devices
iPhone 11 ProMax with iOS 14.2 (Roon Version 1.7 (Build 610) stable)
MacBook Pro i7 16-inch 2019 with macOS Catalina 10.15.7 (Roon Version 1.7 (Build 710) stable)
Description of Issue
I am experiencing slowdowns that are apparent on both macOS and iOS remotes, this gets worse over a period of time (7 days in this latest instance) and the only way to restore performance is restart the roonserver process on the core. In addition to the obvious slowdown Roon presents me with a “Adding this album is taking longer than normal. Roon will keep trying in the background.” message.
In the most recent instance of this slowdown, cockpit was reporting that the core was utilizing over 6GiB of memory before the roonserver process was restarted, after a restart of roonserver memory utilization dropped to about 2.5GiB. Top also shows a similar story regarding memory utilization.
This manifests itself into very poor performance with the remotes; searching for albums and adding to the library are very slow. Music does continue uninterrupted.
Video 1 was created after the roonserver process had been running for 7 days.
Video 2 was created just after the roonserver process had been restarted.
Troubleshooting Performed So Far
Data gathering
Restart of roonserver process relieves the situation for a number of days.
My core is running on Ubuntu 20.04.01 LTS without any problems, but i will test if i can reproduce the issue.
So that i can test a little i have installed cockpit as well.
Is your machine solely running Roon Core, or do you have other software running that potentially could cause an issue?
@PeterD the only job of this computer is to run Roon, there is nothing else on it that supports another task. It doesn’t even have a desktop environment or web admin component.
In the videos linked above I have included top displaying both before and after the roonserver process is restarted. There is only about 5 minutes between the videos and nothing else changed, all I did was issue a systemctl restart roonserver command. I also have a copy of the log files, all times are in pacific time zone.
I also run Ubuntu Server 20.04.1 + Cockpit (+ Syncthing + MinimServer). 2k track library on internal SATA SSD. Virtual and real size are like after your reboot. I’ve never noticed them getting much bigger, or at least I’ve not noticed slowdowns that would make me investigate.
@Fernando_Pereira that’s interesting, I have a couple of questions. What kind of hardware are you running it on? How many days is the roonserver process typically up for? Keep in mind as well 2k local files is quite a bit different from 95k. The story that is shown in videos is quite clear with the top output being shown at all times. There is literally nothing else on this server running, even cockpit is operating as a bridge.
FWIW, I experience exactly the same behavior as @Robem, on similar hardware and on the same OS. I too restart regularly the Roon Server process, and even the memory usage after restart and at the point the user interaction with Roon becomes unbearable , are the same.
There was a typo on my earlier message, 20k tracks (21883 to be exact). This core is on a Zotac CI662 nano 8x Intel® Core™ i7-10510U CPU @ 1.80GHz, DDR4, 16 GB, 3200 MT/s, 4TB SanDisk SATA SSD). But I have also a NUC core (i5-7300U, NUC7I5DNHE, 8GB (2x4GB) DDR4 SODIMM 2400MHz Memory, 120GB M.2 SATA SSD, 4TB Samsung SATA SSD) elsewhere that also works comparably well. Same tracks.
Number of days up varies between a few and several weeks, depending on how often I update software or power cycle them for one reason or another.
Hello @Peter_Bruderer, yes I am familiar with the logs and tools to check the contents, in fact I have already looked at these. I have noticed about the time of the first video starting hundreds (maybe thousands) of nearly consecutive entries like the following.
The are also post entries for what looks like metadata queries for albums that I haven’t gone anywhere near in months, example below.
01/07 20:03:13 Trace: [identification] <20559919> Identifying album [James Brown - James Brown Plays Nothing But Soul (Flac)] with 6 tracks
01/07 20:03:13 Debug: [metadata http post req]: https://identifier.roonlabs.net/identifier/2/album?uid=5cd11155-9e68-4029-9f77-94998a7fe92d&lid=&token=505d9619-1031-4eb3-a893-b15538654193
01/07 20:03:13 Debug: [identification] <20559919> identifier request id: 8367ebbf
I have found this thread that appears similar as well…
Looks like Roon is analysing/indexing your library and fetching metadata.
How long do you have this server?
Did you backup/restore the library?
I have the impression, your server never finished analysing your library. During this process it allocates more RAM and uses much more CPU then normal. The problem is not related to Ubuntu.
Two weeks ago I decided to trash my Roon Core and completely build it from scratch. With the default setting using 1 core for the analysis, it took almost 24 hours to analyse all tracks and to fetch the metadata.
I see you got 95K tracks, assuming the same speed it will take your machine 6 days easy to complete this job.
The server has been in operation for about three weeks, I migrated from another less dedicated Ubuntu Desktop 20.04 because of similar symptoms. I performed a backup and restore of the database and it has been sat idle (stable) without scanning the library for a significant amount of time. The log files in December do not show the same kind of activity. Maybe I just need to trash the database and start over. That’s not cool though, there should be a way to fix it instead. I will have to put my thinking cap on and work out what I would lose if I went that route.
I have performed some additional testing this evening.
The test started with the server idle at 21:35, this was verified by watching a tail of the roonserver logs.
At 21:39 I performed a search for “Yello” using roon and added two titles from Qobuz to my library. The second title was extremely slow and I was presented with the message “Adding this album is taking longer than normal. Roon will keep trying in the background.”
Examining the tail shows that at 21:40:31 the server has decided that it needs to rebuild a query instead of re-sort item by item. At this point the server has been reduced to a crawl.
Also of note is that the local albums that are being retouched have been in my library for a long time (example of Duran Duran shown above), they are not new additions and for the most part haven’t even been played in a long time.
I have read on these forums that Roon support staff say “you don’t need to worry” about dirty items. Why did my adding a title from Qobuz to my library trigger this crazy utilization that impacted my ability to use Roon?
I have also decided to start keeping a spreadsheet with memory utilization stats in it, these will all be taken when the server is confirmed to be idling. It is my suspicion that memory utilization grows over a period of time, I believe that it isn’t until the roonserver process is restarted that it returns back down to normal level. This might be described as a memory leak by some. Here is the start of my spreadsheet.
At 8:28 this morning, I got up and thought that I would record memory usage after I had been sleeping overnight. I started the tail to validate that the server was not under any kind of load and recorded the usage in my spreadsheet.
I then browsed to an artist that I was interested in listening to (John Foxx) and all of a sudden I was presented with the Roon equivalent of an hour glass. Checking the tail output it was obvious that my activity had triggered the “[dbperf] flush” and now Roon was in the process of re-identifying a number of existing albums in my local library that I hadn’t touched in some time. This activity that was impacting my ability to browse Qobuz titles using Roon lasted until 08:39.
Thanks for all your testing. You describe my exact experience. At saving new albums from Tidal, some reindexing or metadata discovery is triggered and the user interface becomes unresponsive.
As a reference point, used memory on the server at idle has grown from 3.0 to 4.5 GiB in less than 24 hours. I have been deliberately stress testing this by importing large batches from my CD collection but I have ensured that I have allowed the processes to complete before recording the reading.
The memory usage increases significantly when importing local files into the library, this increase in memory usage does not seem to return to a previous state. A plotted graph looks like bitcoin prices, always increasing and rarely dropping, even when it does drop it never returns to where it once was unless the roonserver process is restarted.
Browsing and adding streaming platform titles to the library seems to trigger a reindex/metadata refresh of old files within my library.
@Andreas_Philipp1 has said he has similar behavior at least for observation #2. @Fernando_Pereira, how often do you perform one of the above scenarios?
In the last year: imported 247 local albums and added 108 Qobuz albums to my library. Can’t recall any extreme slowdown.
AFAIK, the Roon library is implemented as a (partially?) in-memory object-oriented database, so it would be normal to see an increased virtual memory size as the library grows.
Thank you for the clarification @Fernando_Pereira, your answer helps me understand more why you may not be seeing the same issues.
I can get the memory utilization to come down to a level where I think it should be. To do that however would mean restarting the roonserver process. This increase in memory usage (~300% in about a week) is not due to progressive database growth.
