My Roon Core has crashed due “Too many open files” exception. It’s the first time I have seen this error and probably the first full crash of Roon core server - though I have had issues with memory utilisation.
04/12 21:25:11 Info: [stats] 7520mb Virtual, 2402mb Physical, 1574mb Managed, 8189 Handles, 94 Threads
There were a range of exceptions, either out of memory or too many open files.
This is a core i5 with 8gb ram running Ubuntu 22.04
How many tracks have you got in your library?
I feel it could be a low RAM issue as you have 8gb RAM.
Also worth a read is this
I have 4300 local tracks, though most of the exceptions look to be related to HTTP connections, so not sure it’s files causing the issue. Also hasn’t done this before in a few years.
I’ll check the link. I see the default startup scripts sets unlimited to 8192 so assume that’s why it fell over, but doesn’t really explain the numbers of open handles.
Do you use Tidal or Qobuz?
Having reviewed my logs this is happening repeatedly - I don’t recall it doing this for previous version.
Looks like a handle leak to me.
Steven, I am running Roon server on Ubuntu 22.04, with currently 310 k tracks, between local files and streaming services favorites. From regularly monitoring my Roon logs I know that 8189 handles is way way out of any normal expected range…
When starting up my server, handles run about 550, and after a scheduled backup job of Roon’s database, the handle goes down to about 340. There are several regular background processes run by Roon which bring the handle count sharply up (metadata updates, Tidal and Qobuz library maintenance…), but I have never ever seen a handle count above 890-900…
The background processes in fact don’t release handles; but running a Roon backup job does…
I would recommend you reclassify your thread under the forum’s Support area, so Roon’s support team may have a look at your logs and hopefully find the reason for your problem…
I use Qobuz and this is my main source.
I will update this under Roon support as based on the logs it looks to be a new repeating issue.
I will investigate to understand what handles are being retained.
How many tracks are there in your library including Qobuz? Streaming services count towards the library count.
I have 940 qobuz albums in the library - so roughly 9000 tracks. This has only started happening recently.
The handles appear to relate to UDP sockets, based on output of lsof command.
The Roon Appliance process appears to be leaking 3 UDP sockets on a regular basis.
This issue is still happening for version 1401.
Hi @Steven_Moore,
Thank you for the post.
We’ve pulled diagnostics and can see recurring network reachability changes logged by RoonServer. We’ll investigate further with development. In the meantime, can you please elaborate on the network topology upstream of this device (routers, switches, etc.)?
You can try changing the file descriptor limit in the meantime with ulimit. Ulimit, Soft Limits and Hard Limits in Linux - GeeksforGeeks
Thank you!
Hi @connor ,
I have seen network reachability log events consistently, though I’m not aware of a network issue.
The network is a fibre broadband connection to a Draytek Vigor 2862ac, which is connected to a Netgear 24 port managed switch. The Roon Server is connected by ethernet to this switch.
In order to get ARC working, I set a specific port and manually configured the port forwarding on the Draytek router - as I have not had any luck with UPNP (on this router or a previous ISP provided one).
The network has very low ping time, and no network errors reported on the switch.
Until last month I was restarting RoonServer on a nightly basis due to an issue with .Net memory utilisation on Ubuntu 22.04, however recently I have not experienced that issue, but has been replaced by the handle count issue. I’m fairly familiar with the logs and have not noticed the handle leak until recently.
I can change the ulimit though I feel this would just delay the inevitable restart, probably due to UDP port exhaustion.
If I perform a netstat -ano I can see that the RoonAppliance process is leaving UDP listen session open - and this looks to be the main consumer of the handle.
Please let me know if there is any other information I can provide to assist.
Thanks,
Steve
Based on the questions, I checked the syslog, and noticed some errors related to ipv6, which is disabled on my Ubuntu 22.04 server.
I will re-enable ipv6 (i.e. set back to default config). This is not a recent change but will reset this to rule in or out as a cause.
Re-enabling ipv6 has cleared the syslog error related to ipv6.
I also noticed the following error is being recorded fairly regularly:
/var/log/syslog.1:Apr 13 23:56:34 smoore-Aspire-M5910 start.sh[1057]: System.Net.Sockets.SocketException (104): Connection reset by peer
/var/log/syslog.1:Apr 13 23:56:34 smoore-Aspire-M5910 start.sh[1057]: at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
/var/log/syslog.1:Apr 13 23:56:34 smoore-Aspire-M5910 start.sh[1057]: at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
/var/log/syslog.1:Apr 13 23:56:34 smoore-Aspire-M5910 start.sh[1057]: at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
/var/log/syslog.1:Apr 13 23:56:34 smoore-Aspire-M5910 start.sh[1057]: --- End of stack trace from previous location ---
/var/log/syslog.1:Apr 13 23:56:34 smoore-Aspire-M5910 start.sh[1057]: at System.Threading.Tasks.TaskToApm.End[TResult](IAsyncResult asyncResult)
/var/log/syslog.1:Apr 13 23:56:34 smoore-Aspire-M5910 start.sh[1057]: at Sooloos.RnetJsonClient.<>c__DisplayClass65_0.<_BeginRead>b__0(IAsyncResult ar)
Left the system overnight after re-enabling ipv6. Syslog much clearer.
Still same issue in RoonAppliance - currently 650 open UDP sockets after about 10 hours. So perhaps issue has slowed, but appears to still be leaking sockets.
I will investigate if there are any other reasons for network reachability resets.
Thanks,
Steve
@connor
I’ve decided to clean install Roon server on Debian 12. Looks like network reachability issues are Ubuntu specific as looks much quieter on Debian.
Suspect there is a handle / UDP socket leak in one of the events triggered by network reachability change on Ubuntu.
Will monitor for a few days.
This topic was automatically closed 36 hours after the last reply. New replies are no longer allowed.