CentOS 8 - RoonServer keeps disconnecting and reconnecting with network reachability errors [Solved - IPv6 Issue]

Jimmy_Legrand · January 27, 2022, 6:00pm

Roon Core Machine

Roon Core is installed on an Intel NUC7i5DNKE
i5-7300U CPU @ 2.60GHz
8GB
Kingston M2 disk
CentOS 8 stream
1Gb wired

Networking Gear & Setup Details

All devices involved in Roon are connected with 1Gb ethernet cables.
Except of course on my iPhone which is connected via wifi.
The wifi router is a Livebox 4 provided by my the ISP (Orange)

Connected Audio Devices

The Roon endpoint is the couple Chord 2Go/2Yu which is connected to a Chord Qutest DAC

Number of Tracks in Library

I just have 5815 tracks in my library

Description of Issue

The issue is the iPhone application is almost unusable. The application lost the connection very often and sometimes the application crashed.
It’s now a challenge to be able to put some music.

Here a video about the problem:

ashley · February 1, 2022, 4:49pm

Hey @Jimmy_Legrand thanks so much for sharing reaching out to us as soon as you experienced this type of behavior. I especially appreciate you providing the Youtube video, it’s super helpful to understand what you’re seeing.

Since this issue first occurred, have you tried to reinstall the application on the iPhone? If not, can you please try reinstalling Roon on this remote and see if the issue persists?

If the issue persists after a reinstall, can you please reproduce this behavior, provide us with the reproduction steps (if you have any to share), note the date and time of when it occurs, and use the directions found here and send over a set of logs to our File Uploader?

Thank you!

Jimmy_Legrand · February 10, 2022, 12:04pm

Sorry I didn’t see your reply.
I tried to reinstall the application, which I’ve already done before but without success.

I’ll send you the logs in a couple of days.

ashley · February 10, 2022, 10:24pm

No worries, @Jimmy_Legrand! Thanks for the update.

I look forward to hearing back about the logs - please be sure to reply back to this thread with the date and time of when the issue occurred so that we can take a look at the proper information. Unfortunately, I’m not alerted once logs are uploaded to our drive.

Jimmy_Legrand · February 11, 2022, 5:59pm

I just uploaded the log files.
The last test I did happens February 11th between 18h38 and 18h41.
To reproduce the issue, I just had to browse the music either via Qobuz or in my library. Even if I just wait I have the issues as well.

ashley · February 14, 2022, 10:56pm

Thanks @Jimmy_Legrand! I’ve alerted the technical support team so that they can take a look at the logs. Once we have more information, we’ll be reaching back out to update you!

noris · February 25, 2022, 8:39pm

Hi @Jimmy_Legrand ,

Thanks for your patience while we reviewed logs, it looks you are running out of file handles because you have too many open files:

02/10 14:01:08 Critical: scx: System.IO.IOException: Too many open files
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at Sooloos.IPAddresses.GetArpIPs()
   at Sooloos.SOOD.Connection.SendMessage(Dictionary`2 map, Action`2 response_callback, Action done_scanning_callback)
   at Sooloos.Broker.Transport.RaatServerDiscovery.Refresh(Boolean dointernetdiscovery)
   at Sooloos.SynchronizationContextThread._Dispatch(SendOrPostWrapper& ret)
02/10 14:01:08 Info: 
Local Time:            02/10/2022 14:01:08 +01:00
Device Serial Number:  423B8930-0D33-4A50-BFE4-0EFEB63FADD6
User Id:               feb33ab1-4d4c-4bd3-a445-ba9bd1018f4e
Roon Version:       1.8 (build 898) stable
OS Version:            Linux 4.18.0-358.el8.x86_64
Mono Version:          unknown

Application Domain:    RoonAppliance
Assembly Codebase:     file:///opt/RoonServer/Appliance/RoonAppliance.dll
Assembly Full Name:    RoonAppliance, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

   Exception Source:      System.Private.CoreLib
   Exception Type:        System.IO.IOException
   Exception Target Site: Environment.TryGetUserNameFromPasswd
   Exception Message:     Too many open files
   Exception Data:        none

It also looks like your network reachability is changing multiple times in short periods, which I suspect is also playing a part in the issue:

    Line 27: 02/11 18:28:29 Trace: [server] network reachability changed. Kicking off discovery cycle
	Line 46: 02/11 18:29:08 Trace: [broker/accounts] network reachability changed. refreshing
	Line 47: 02/11 18:29:08 Debug: [tidal] network reachability changed. refreshing token
	Line 53: 02/11 18:29:08 Trace: [roonapi] network reachability changed. Kicking off discovery cycle
	Line 85: 02/11 18:29:10 Trace: [server] network reachability changed. Kicking off discovery cycle
	Line 96: 02/11 18:29:32 Trace: [broker/accounts] network reachability changed. refreshing
	Line 97: 02/11 18:29:32 Debug: [tidal] network reachability changed. refreshing token
	Line 103: 02/11 18:29:32 Trace: [roonapi] network reachability changed. Kicking off discovery cycle
	Line 112: 02/11 18:29:34 Trace: [server] network reachability changed. Kicking off discovery cycle
	Line 133: 02/11 18:29:43 Trace: [broker/accounts] network reachability changed. refreshing
	Line 139: 02/11 18:29:43 Debug: [tidal] network reachability changed. refreshing token
	Line 140: 02/11 18:29:43 Trace: [roonapi] network reachability changed. Kicking off discovery cycle
	Line 169: 02/11 18:29:45 Trace: [server] network reachability changed. Kicking off discovery cycle
	Line 185: 02/11 18:30:30 Trace: [broker/accounts] network reachability changed. refreshing
	Line 191: 02/11 18:30:30 Debug: [tidal] network reachability changed. refreshing token

ISP routers are not suggested to be used in a Roon setup, so changing this might help. From our Networking Best Practices Guide:

Poor quality or underpowered routers can play a major role when it comes to networking difficulties. 802.11n was superseded by 802.11ac five years ago, so at this point, 802.11n routers should be considered obsolete. Furthermore, the performance difference between early 802.11ac routers and current models can be substantial due to differences in CPU/RAM allotments. ISP-provided routers are often underpowered and poorly behaved. Finally, we strongly recommend against using Apple network devices such as the Apple Airport Extreme and Airport Express (note: it’s fine to use the Airport Express as an Airplay device). To ensure the best performance, we recommend against using these devices.

Jimmy_Legrand · February 26, 2022, 2:36pm

Hello @noris

Thank you for your feedback.

Regarding the too many files issue, it doesn’t match when the problem occurs. I just have a dozen of occurrences in several days of logs whereas the issues can occurred 10 times in 1min.

However I increased the soft limit for the open files to 2048.

But I think there is something to do on the application side because I saw a lot of udp connection I can’t explain:

...
udp        0      0 0.0.0.0:47864           0.0.0.0:*                           0          43715323   490698/RoonApplianc 
udp        0      0 0.0.0.0:56060           0.0.0.0:*                           0          43357414   490698/RoonApplianc 
udp        0      0 0.0.0.0:56070           0.0.0.0:*                           0          43364897   490698/RoonApplianc 
udp        0      0 0.0.0.0:35594           0.0.0.0:*                           0          43134073   490698/RoonApplianc 
udp        0      0 0.0.0.0:39729           0.0.0.0:*                           0          43146340   490698/RoonApplianc 
udp        0      0 0.0.0.0:43834           0.0.0.0:*                           0          43336974   490698/RoonApplianc 
udp        0      0 0.0.0.0:60220           0.0.0.0:*                           0          43159697   490698/RoonApplianc 
udp        0      0 0.0.0.0:35663           0.0.0.0:*                           0          43642050   490698/RoonApplianc 
❯ netstat -laputen|grep RoonAppli|wc -l
480

480 udp connections for just for one Roon stream, it’s pretty weird.

Regarding my ISP wifi router, I admit it’s not the best device but as a reminder, in my case the wifi connection is just for the remote function on my iPhone. All the other Roon devices are 1Gb wired.

I do lot things with my iPhone with the same network configuration and I can’t recall any problem like that with other applications. Even HD streaming works perfectly. Or Streaming from my iPhone to my AppleTv works very well.

Jimmy

noris · March 2, 2022, 9:49pm

Hi @Jimmy_Legrand ,

I suspect this might be a band-aid to the issue but won’t prevent it. I suspect that the Network Reachability changes are attributed to the issue. Are you using multiple Network Interfaces on your Core? I would also try using a non-CentOS Roon Core to confirm that this issue only impacts your current setup or if other Cores are impacted.

Jimmy_Legrand · March 4, 2022, 9:01am

Hi,

I agree with you about the soft limit increase.
About the interface, I just have one 1Gb interface configured in IPV4 and IPV6.

About CentOS, I can’t reinstall my core just to test another operating system. So far as I know CentOS is supported by Roon. And even there is differencies between linux distribution, it’s still a linux.

I plan to do a tcpdump this week-end, maybe I could extract some new information from.

One more thing about the network, I tried with another iPhone, a 13 one, and I have the same issues.
I also tried to use my iPhone with a wired connection via usb adapter, (wierd but it works) and I had the same issues as well.
We can definitively exclude a wifi problem.

To finish with, I have a question. What does the network reachability error mean?
Because I can’t detect any network problem about this server.
And whty a Tidal error whereas I don’t have Tidal account configured.

03/04 09:19:29 Debug: [tidal] network reachability changed. refreshing token

I have a local and internet smokeping which are perfect. No problem as well when I make a internet bandwith test from this node.

Smokeping to the roon server from my NAS:

Smokeping to the internet gateway from my NAS:

Bandwidth test from the Roon server:

Jimmy

noris · March 10, 2022, 7:21pm

Hi @Jimmy_Legrand ,

Thanks for those details! I wonder if this issue could be caused by DHCP leases that are set too short, have you checked to see what the lease time is on the Core / Router? Is there any way you can increase it as a test? Also, if you haven’t tried a different Ethernet cable for the Core, this would be a good test as well.

Jimmy_Legrand · March 11, 2022, 5:55pm

I just took a look on the dhcp activity. Here the logs:

Mar 10 02:27:35 JimRoon NetworkManager[908]: <info> [1646875655.1856] **dhcp**4 (eno1): state changed new lease, address=192.168.1.34
Mar 10 14:27:35 JimRoon NetworkManager[908]: <info> [1646918855.1857] **dhcp**4 (eno1): state changed new lease, address=192.168.1.34 
Mar 11 02:27:35 JimRoon NetworkManager[908]: <info> [1646962055.1865] **dhcp**4 (eno1): state changed new lease, address=192.168.1.34 
Mar 11 14:27:35 JimRoon NetworkManager[908]: <info> [1647005255.1880] **dhcp**4 (eno1): state changed new lease, address=192.168.1.34

As you can see, the lease time is set to 12h. So I guess it’s not the issue.
Concerning the lease time increase, my ISP router doesn’t allow that but I can modify the network configuration to a static one.

Finally I can try another cable but when I see the smokeping, the bandwidth test and the interface statistics (see below) which don’t show any errors, I’m not convinced it could be the source of my problems.

RX packets 48890031  bytes 50591085170 (47.1 GiB)
        RX errors 0  dropped 857055  overruns 0  frame 0
        TX packets 43255074  bytes 49123039290 (45.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I will let you know when I will make these tests.

Jimmy

Jimmy_Legrand · March 13, 2022, 3:56pm

Hi @noris
I changed the network configuration to IP static.
I have the same issue.

And I still have the "network reachability changed " trace in the log.

I will change the cable next week. But once again I don’t think it’s the problem.

One more thing, I just realized It would seem I don’t have the issue with the Mac OS Roon client.

Jimmy

Bernd_Kurte · March 13, 2022, 4:34pm

Any VPN clients that are or were installed on the iOS device in trouble?

Jimmy_Legrand · March 13, 2022, 4:37pm

Nothing at all.
I don’t recall any usage of this kind of things.

Bernd_Kurte · March 13, 2022, 4:40pm

That’s good. I had quite a history of permanent iOS crashes and that was a question posed. Well I had such clients installed but since a couple of versions this has gone. Knocking on wood.

One brute force attempt you might try if you like to hurt yourself:) would be to factory reset your device. Do not restore anything and only test the Roon client.

Jimmy_Legrand · March 13, 2022, 4:44pm

The thing is I have thee issue on two iPhones.
I can eventually try with an iPad.

Anyway thanks to take time to help me.

Bernd_Kurte · March 13, 2022, 4:47pm

Welcome. This would rule out or confirm the problem is rooted in your iOS history and what it takes into the devices by restoring or copying setups. Maybe worth a try if stuck currently.

noris · March 22, 2022, 5:25pm

Hey @Jimmy_Legrand ,

Are you still having issues in this area?

I think a good next step here would be to try to separate the OS from the hardware, my next suggestion would be to image the NUC and try ROCK/Windows on the NUC and see if that also exhibits the issue.

Jimmy_Legrand · March 23, 2022, 10:37am

I stil have the issue.
Is there anyway to know what the “network reachability changed” are about ? More details. What was the request, the host…

Because I made a tcpdump, but it’s difficult to do the match with the log. Even with the timestamp.

Concerning Rock installation, I can try even if I think it’s maybe a little overkill.