RoonServer on Ubuntu 22.04.4 [Apple hardware]: a high rate of reachability/discovery storms and other issues [Solved?]

[Wade_Oram]

@Martin_Webster

Can’t you eliminate the power line, and use Ethernet, to see if things improve? At the best of times theses devices aren’t entirely reliable.

Agreed.

Having said that, I use a high spec power line (Devolo Magic 2 Lan - nominal 2.4Gbps) and it usually works without issues. However, I also have a 50m CAT 6e ethernet cable (cost less than £30) which I can use to temporarily replace the powerline system whenever there is anything suspect going on. More often than not, it makes no difference - but it is always best to elliminate the possibilty of the issue being powerline related before spending potentially much more time investigating other potential causes and the 50m cable allows me to do that without moving equipment around.

TL;DR

Powerline systems will introduce a much larger variation in ethernet packet delivery times which have the potential to cause issues with streaming systems (or anything else that imposes an effective deadline on packet delivery that the powerline transmission can’t reliably meet).

In my network, with the 50m ethernet cable in play, packet delivery times around the home network are less than 1ms . With the powerline in play, packet delivery times over routes that involve the powerline vary from about 5ms (best) to more than 70ms even when there are only two devices involved. When the powerline is being used by multiple pairs of devices at the same time (I have 4 powerline adaptors), it can get even worse.

For my part, whilst powerline has proved to be an effective short term solution, I am planning to elliminate it from my home network at some point over the next 6 months.

As an aside, during the last year of my professional career, I was working from home because of covid and whilst, in general, the powerline system worked well and delivered higher data rates than WiFi, it could not cope with MS Teams video meetings. WiFi was actually much more reliable for this purpose so I switched between powerline and Wifi according to what I was doing.

[Jeff_Greenberg]

@Martin_Webster

Can’t you eliminate the power line, and use Ethernet

@Martin_Webster, I think you misunderstood my reference. I’m not using a mains powerline as a transport. My adpater is connected to my Cisco switch via a short 1m cat-6 cable. The reference to “powerline noise” was my mistake – I had flipped events/sec and sec/event as @Andreas_Philipp1 pointed out. So the frequency of reachability event occurance is 1 event per minute not 60Hz as I had previously claimed. That knocked out one theory (powerline noise, which occurs at 60Hz on mains lines in the US) about the possible source of the event timing.

[Martin_Webster]

Thanks for clearing that up. However, I don’t understand the why you would even mention powerlines if you don’t use them. Moreover, they don’t operate a 60 Hz, that is the US AC line frequency. The digital communication happens in the MHz range. All audio and digital equipment, unless there is a fault, should be immune to mains hum (we hear 100 / 120 Hz.)

[grizaudio (Stephen)]

The title of this thread could be better.

Ubuntu does a fine job for most [including me], the issue here is Apple hardware.

“Apple Macbook - Ubuntu 22.04.04 & Roon server issues…”

[maximasr]

I ran different versions of Ubuntu and a couple of other Linux distros for over a decade on Intel Macs with no problems. There are a few ways to do this. Using the new Apple arm processors is more difficult but doable.
Looking at these threads, some of these errors could be from a hodgepodge network. Nobody uses powerline adapters if you want a clean network connection. The best is using a wired connection. The other reliable way is to use a mesh network using 6e backhaul if your computer is isolated from being a simple wired network, then use an Ethernet port off the mesh network router to your computer. I have 4 6e mesh routers around the house with excellent coverage.

[Martin_Webster]

@maximasr

I ran different versions of Ubuntu and a couple of other Linux distros for over a decade on Intel Macs with no problems.

It was much easier to install Linux on early Intel-based Macs. However, anything Intel after circa 2016 is much more problematic, and is certainly not recommended as a daily driver … it’s more for tinkering.

This discussion appears to be unnecessary. If you have a MacBook, run Roon Server natively on macOS. If you want to run Roon Sever on Linux, use supported hardware, e.g., an Intel NUC or equivalent.

[maximasr]

I agree if you have a newer Mac, I have been running Roon on Mac osx for years and before that audirvana, pure music, amarra on Mac’s for over a decade. But if you have a pre M1 Mac mini, any Mac Pro with intel, with enough resources and you want to run Linux, why not? Older Intel Macs are constrained with new OS releases so why not use a current Linux release? I’m thinking of getting a 2013+ trash can Mac Pro just for Roon and Homebridge using Linux.

[Martin_Webster]

@maximasr

… any Mac Pro with intel, with enough resources and you want to run Linux …

No, that’s not what I said. Intel MacBooks from around 2016 are not recommended for running Linux unless you want to tinker, and don’t mind incompatibility and stability problems.

Moreover, it seems pointless running an LTS on hardware distributions don’t provide supported a supported installation image.

[Jeff_Greenberg]

@grizaudio Stephen…I agree and will change the thread title shortly, It was a late night joke. I was paraphrasing a oh-too chipper response to issues with 22.04 with a gleeful “22.04 is fine for me!” It may not have even been on this board. In any event jokes don’t make good thread titles.

FYI I’ll also be editing the original post to add information and correct an error.

[Jeff_Greenberg]

@maximasr

I ran different versions of Ubuntu and a couple of other Linux distros for over a decade on Intel Macs with no problems.

I agree. And this is in Intel based system rather than ARM so that reduces the complexity a bit.

Looking at these threads, some of these errors could be from a hodgepodge network. Nobody uses powerline adapters if you want a clean network connection.

That’s true and I don’t use a powerline network either and I never said that I did. My use of the word “powerline” was in reference to 60Hz power line noise not to a powerline network.

The best is using a wired connection. The other reliable way is to use a mesh network using 6e backhaul if your computer is isolated from being a simple wired network, then use an Ethernet port off the mesh nework router to your computer. I have 4 6e mesh routers around the house with excellent coverage.

That is very close to my actual network except that the server is hardwired directly to the main router (Orbi RBR50). The Bluesound Node 2 is in the same room and is also connected via wired 1Gb ethernet. There is a wireless backhaul to the Orbi satellite (RBS50) and my NAD T777v3 connects to the satellite via a hardwired connection.

The only completely wireless zone is the one using WiiM Pro Plus which anchors my headphone listening station.

[Jeff_Greenberg]

This log excerpt has been redacted to hopefully eliminate sensitive information. But it shows the typical pattern observed in my RoonServer logs:

• a [push2] retry message followed by
• reachability changed message, leading to
• [push2] exception thrown then
• kicking off discovery cycle and
• accounting info sent to roonlabs
• then rebuilding the device map

That cycle repeats rather endlessly in the logs. It’s not clear what the [push2] operation is and where the exception is occurring. The network manager journal entries show no state changes on any interface while this is happening,.

My next step will be to enter a ticket for support to see if Roon itself can shed any light on what’s actually happening.

04/11 23:59:42 Debug: [easyhttp] [31942] POST to https://api.roonlabs.net/discovery/1/query returned after 161 ms, status code: 200, request body size: 74 B
04/11 23:59:57 Info: [stats] 9188mb Virtual, 2172mb Physical, 832mb Managed, 6349 Handles, 77 Threads
04/12 00:00:12 Info: [stats] 9108mb Virtual, 2172mb Physical, 833mb Managed, 6349 Handles, 71 Threads
04/12 00:00:27 Info: [stats] 9108mb Virtual, 2172mb Physical, 831mb Managed, 6349 Handles, 70 Threads
04/12 00:00:28 Trace: [broker/accounts] [heartbeat] now=2024-04-12 4:00:28 a.m. nextauthrefresh=2024-04-12 4:59:32 a.m. nextmachineallocate=2024-04-12 6:45:28 a.m.
04/12 00:00:42 Info: [stats] 9116mb Virtual, 2172mb Physical, 830mb Managed, 6349 Handles, 72 Threads
04/12 00:00:57 Info: [stats] 9116mb Virtual, 2172mb Physical, 832mb Managed, 6349 Handles, 72 Threads
04/12 00:01:13 Info: [stats] 9140mb Virtual, 2172mb Physical, 831mb Managed, 6349 Handles, 75 Threads
04/12 00:01:13 Trace: [push2] retrying connection in 0ms
04/12 00:01:13 Trace: [broker/accounts] network reachability changed. refreshing
04/12 00:01:13 Trace: [broker/accounts] [heartbeat] now=2024-04-12 4:01:13 a.m. nextauthrefresh=2024-04-12 4:59:32 a.m. nextmachineallocate=2024-04-12 6:45:28 a.m.
04/12 00:01:13 Trace:  [broker/accounts]   doing auth refresh, next=2024-04-12 4:59:32 a.m.
04/12 00:01:13 Trace: [broker/accounts] refreshing account info for email='xxxxxxxxxx@yyy.zzz' -userid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:13 Trace: [fiveaccountserver] POST https://accounts5.roonlabs.com/accounts/3/login
04/12 00:01:13 Trace: [fiveaccountserver] BODY token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc
04/12 00:01:13 Trace: [push2] exception thrown. restarting connection (The operation was canceled.)
04/12 00:01:13 Debug: [tidal] network reachability changed. refreshing token
04/12 00:01:13 Debug: [kkbox] network reachability changed. refreshing token
04/12 00:01:13 Trace: [roonapi] network reachability changed. Kicking off discovery cycle
04/12 00:01:13 Trace: [mobile] [remoteconnectivity] Port Verification started due to: network reachability changed, port verification not in progress, starting a new attempt
04/12 00:01:13 Trace: [raat] [sood] Refreshing device list
04/12 00:01:13 Trace: [raatserver] [sood] Refreshing device list
04/12 00:01:13 Debug: [easyhttp] [31943] GET to https://api.roonlabs.net/push-manager/1/connect returned after 148 ms, status code: 200, request body size: 0 B
04/12 00:01:13 Debug: [push2] push connector url received from push manager: ws://push-connector-v2-1.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:01:13 Trace: [push2] connecting to push2 connector at ws://push-connector-v2-1.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:01:13 Debug: [easyhttp] [31944] POST to https://api.roonlabs.net/accounts5/accounts/3/login returned after 159 ms, status code: 200, request body size: 42 B
04/12 00:01:13 Trace: [fiveaccountserver] GOT {"status":"Success","userid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","token":"ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc","expiration":30,"email":"xxxxx@xxx.xxx.com","groups":[]}
04/12 00:01:13 Debug: [easyhttp] [31945] POST to https://api.roonlabs.net/discovery/1/query returned after 167 ms, status code: 200, request body size: 74 B
04/12 00:01:13 Debug: [easyhttp] [31948] GET to https://api.roonlabs.net/porttest/1/myip returned after 144 ms, status code: 200, request body size: 0 B
04/12 00:01:13 Debug: [easyhttp] [31947] GET to https://api.roonlabs.net/internetradio/2/api/location?format=msgpack& returned after 223 ms, status code: 200, request body size: 0 B
04/12 00:01:13 Trace: [radio/library] got location US
04/12 00:01:13 Trace: [push2] connected to push2 connector at ws://push-connector-v2-1.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:01:13 Trace: [fiveaccountserver] GET https://accounts5.roonlabs.com/accounts/3/profileslist?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc
04/12 00:01:13 Trace: [fiveaccountserver] GET https://accounts5.roonlabs.com/accounts/3/userinfo?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc&machineid=ddd44979-fcc3-dacf-e080-361fbb8e1390
04/12 00:01:13 Trace: [broker/accounts] updated token. New expiration is 2024-05-12 12:01:13 a.m.
04/12 00:01:13 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:13 Trace: [bits] myinfo: {"pushid":"broker/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","roon_auth_token":"ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc","os":"Linux 6.6.25-t2-jammy","platform":"linuxx64","machineversion":200001392,"branch":"production","appmodifier":"","appname":"RoonServer"}
04/12 00:01:13 Debug: [easyhttp] [31946] GET to https://api.roonlabs.net/oauthcb/3/tidal/refresh?token=eyJraWQiOiJoUzFKYTdVMCIsImFsZyI6IkVTNTEyIn0.eyJ0eXBlIjoibzJfcmVmcmVzaCIsInVpZCI6NDgxMjIxNDAsInNjb3BlIjoid19zdWIgcl91c3Igd191c3IiLCJjaWQiOjI0MjUsInNWZXIiOjAsImdWZXIiOjAsImlzcyI6Imh0dHBzOi8vYXV0aC50aWRhbC5jb20vdjEifQ.AQMUwvup-XM5Up-5N1DNXTcRQu83gOMAEnNPxN5WNrssubGALfVRgcCASlsOECG1QmoTGxOXJbxwCi6Ky2yCRIJmAEoiRD9bfQdJBdy-9thXYQwj2IZr12itr6QMSHObfvahDgEub_-7tnlJef5leHduoGr7cByMIlGKf6WTCmf9HmWE returned after 380 ms, status code: 200, request body size: 0 B
04/12 00:01:13 Info: [broker/locations] updating location Tidal:Name=TIDAL:Id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:13 Debug: [easyhttp] [31950] GET to https://api.roonlabs.net/accounts5/accounts/3/profileslist?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc returned after 158 ms, status code: 200, request body size: 0 B
04/12 00:01:13 Trace: [fiveaccountserver] GOT {"status":"Success","profiles":[{"id":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","name":"XXXX","photo":"https://sooloos-profileimages.s3.amazonaws.com/five-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx--xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx.jpg","birthdate":"xxxxxxxx"}]}
04/12 00:01:13 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:13 Debug: [easyhttp] [31953] GET to https://api.tidal.com/v1/sessions/e4aaf06d-47be-403d-a947-27e1218f2fdc?countryCode=US returned after 139 ms, status code: 200, request body size: 0 B
04/12 00:01:13 Trace: [tidal/http] GET https://api.tidal.com/v1/sessions/e4aaf06d-47be-403d-a947-27e1218f2fdc?countryCode=US => Success
04/12 00:01:13 Trace: [tidal] transition loginstatus from LoginSucceeded to LoginSucceeded
04/12 00:01:14 Debug: [easyhttp] [31951] GET to https://api.roonlabs.net/accounts5/accounts/3/userinfo?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc&machineid=ddd44979-fcc3-dacf-e080-361fbb8e1390 returned after 237 ms, status code: 200, request body size: 0 B
04/12 00:01:14 Trace: [fiveaccountserver] GOT {"user":{"tfa":{"enabled":false},"userid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","firstname":"XXXX","lastname":"XXXXXXXX","email":"xxxxx@xxx.xxx.com","joinmailinglist":true,"allowpushnotifications":true,"class":"Normal","groups":[],"dncs":[],"trialallowed":false},"status":"Success"}
04/12 00:01:14 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:14 Debug: [easyhttp] [31952] POST to https://api.roonlabs.net/bits/1/q/roon.base.,roon.internet_discovery.,roon.debug.,roon.broker.,roon.dsp.,roon.sood.?roon_auth_token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc returned after 255 ms, status code: 200, request body size: 253 B
04/12 00:01:14 Trace: [bits] updated bits, in 257ms
04/12 00:01:14 Debug: [easyhttp] [31954] POST to https://api.roonlabs.net/roonmobile/1/cores/announce returned after 245 ms, status code: 200, request body size: 1 KB
04/12 00:01:14 Debug: [easyhttp] [31955] GET to https://api.roonlabs.net/profileimages/five-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx--xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx.jpg?random=d2e22461-1641-4326-b190-7341b6e91f34 returned after 237 ms, status code: 200, request body size: 0 B
04/12 00:01:14 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:14 Info: [mobile] GOT HTTP API /hello
04/12 00:01:14 Trace: [mobile] Got Hello Request body={"coreId":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"}
04/12 00:01:15 Trace: [remoting/brokerserver] network reachability changed. Kicking off discovery cycle
04/12 00:01:18 Debug: [easyhttp] [31956] POST to https://api.roonlabs.net/discovery/1/register returned after 141 ms, status code: 200, request body size: 1 KB
04/12 00:01:18 Trace: [inetdiscovery] registered 1 devices, 5 services
04/12 00:01:24 Debug: [easyhttp] [31957] POST to https://api.roonlabs.net/device-map/1/register returned after 164 ms, status code: 200, request body size: 8 KB
04/12 00:01:24 Trace: [devicemap] device map updated
04/12 00:01:28 Info: [stats] 9357mb Virtual, 2172mb Physical, 834mb Managed, 6352 Handles, 102 Threads
04/12 00:01:43 Info: [stats] 9156mb Virtual, 2172mb Physical, 824mb Managed, 6352 Handles, 74 Threads
04/12 00:01:43 Debug: [easyhttp] [31958] POST to https://api.roonlabs.net/discovery/1/query returned after 225 ms, status code: 200, request body size: 74 B
04/12 00:01:56 Trace: [push2] retrying connection in 0ms
04/12 00:01:56 Trace: [broker/accounts] network reachability changed. refreshing
04/12 00:01:56 Trace: [broker/accounts] [heartbeat] now=2024-04-12 4:01:56 a.m. nextauthrefresh=2024-04-12 5:01:13 a.m. nextmachineallocate=2024-04-12 6:45:28 a.m.
04/12 00:01:56 Trace:  [broker/accounts]   doing auth refresh, next=2024-04-12 5:01:13 a.m.
04/12 00:01:56 Trace: [broker/accounts] refreshing account info for email='xxxxxxxxxx@yyy.zzz' -userid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:56 Trace: [fiveaccountserver] POST https://accounts5.roonlabs.com/accounts/3/login
04/12 00:01:56 Trace: [fiveaccountserver] BODY token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc
04/12 00:01:56 Trace: [push2] exception thrown. restarting connection (The operation was canceled.)
04/12 00:01:56 Debug: [tidal] network reachability changed. refreshing token
04/12 00:01:56 Debug: [kkbox] network reachability changed. refreshing token
04/12 00:01:56 Trace: [roonapi] network reachability changed. Kicking off discovery cycle
04/12 00:01:56 Trace: [mobile] [remoteconnectivity] Port Verification started due to: network reachability changed, port verification not in progress, starting a new attempt
04/12 00:01:56 Trace: [raat] [sood] Refreshing device list
04/12 00:01:56 Trace: [raatserver] [sood] Refreshing device list
04/12 00:01:56 Debug: [easyhttp] [31960] POST to https://api.roonlabs.net/accounts5/accounts/3/login returned after 154 ms, status code: 200, request body size: 42 B
04/12 00:01:56 Trace: [fiveaccountserver] GOT {"status":"Success","userid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","token":"ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc","expiration":30,"email":"xxxxx@xxx.xxx.com","groups":[]}
04/12 00:01:56 Debug: [easyhttp] [31963] GET to https://api.roonlabs.net/porttest/1/myip returned after 152 ms, status code: 200, request body size: 0 B
04/12 00:01:56 Debug: [easyhttp] [31962] GET to https://api.roonlabs.net/internetradio/2/api/location?format=msgpack& returned after 210 ms, status code: 200, request body size: 0 B
04/12 00:01:56 Trace: [radio/library] got location US
04/12 00:01:56 Debug: [easyhttp] [31959] GET to https://api.roonlabs.net/push-manager/1/connect returned after 228 ms, status code: 200, request body size: 0 B
04/12 00:01:56 Debug: [push2] push connector url received from push manager: ws://push-connector-v2-1.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:01:56 Trace: [push2] connecting to push2 connector at ws://push-connector-v2-1.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:01:56 Trace: [fiveaccountserver] GET https://accounts5.roonlabs.com/accounts/3/profileslist?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc
04/12 00:01:56 Trace: [fiveaccountserver] GET https://accounts5.roonlabs.com/accounts/3/userinfo?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc&machineid=ddd44979-fcc3-dacf-e080-361fbb8e1390
04/12 00:01:56 Trace: [broker/accounts] updated token. New expiration is 2024-05-12 12:01:56 a.m.
04/12 00:01:56 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:56 Trace: [bits] myinfo: {"pushid":"broker/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","roon_auth_token":"ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc","os":"Linux 6.6.25-t2-jammy","platform":"linuxx64","machineversion":200001392,"branch":"production","appmodifier":"","appname":"RoonServer"}
04/12 00:01:56 Trace: [push2] connected to push2 connector at ws://push-connector-v2-1.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:01:56 Debug: [easyhttp] [31961] GET to https://api.roonlabs.net/oauthcb/3/tidal/refresh?token=eyJraWQiOiJoUzFKYTdVMCIsImFsZyI6IkVTNTEyIn0.eyJ0eXBlIjoibzJfcmVmcmVzaCIsInVpZCI6NDgxMjIxNDAsInNjb3BlIjoid19zdWIgcl91c3Igd191c3IiLCJjaWQiOjI0MjUsInNWZXIiOjAsImdWZXIiOjAsImlzcyI6Imh0dHBzOi8vYXV0aC50aWRhbC5jb20vdjEifQ.AQMUwvup-XM5Up-5N1DNXTcRQu83gOMAEnNPxN5WNrssubGALfVRgcCASlsOECG1QmoTGxOXJbxwCi6Ky2yCRIJmAEoiRD9bfQdJBdy-9thXYQwj2IZr12itr6QMSHObfvahDgEub_-7tnlJef5leHduoGr7cByMIlGKf6WTCmf9HmWE returned after 354 ms, status code: 200, request body size: 0 B
04/12 00:01:56 Info: [broker/locations] updating location Tidal:Name=TIDAL:Id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:57 Debug: [easyhttp] [31966] GET to https://api.roonlabs.net/accounts5/accounts/3/userinfo?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc&machineid=ddd44979-fcc3-dacf-e080-361fbb8e1390 returned after 177 ms, status code: 200, request body size: 0 B
04/12 00:01:57 Trace: [fiveaccountserver] GOT {"user":{"tfa":{"enabled":false},"userid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","firstname":"XXXX","lastname":"XXXXXXXX","email":"xxxxx@xxx.xxx.com","joinmailinglist":true,"allowpushnotifications":true,"class":"Normal","groups":[],"dncs":[],"trialallowed":false},"status":"Success"}
04/12 00:01:57 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:57 Debug: [easyhttp] [31968] GET to https://api.tidal.com/v1/sessions/e4aaf06d-47be-403d-a947-27e1218f2fdc?countryCode=US returned after 115 ms, status code: 200, request body size: 0 B
04/12 00:01:57 Trace: [tidal/http] GET https://api.tidal.com/v1/sessions/e4aaf06d-47be-403d-a947-27e1218f2fdc?countryCode=US => Success
04/12 00:01:57 Trace: [tidal] transition loginstatus from LoginSucceeded to LoginSucceeded
04/12 00:01:57 Debug: [easyhttp] [31965] GET to https://api.roonlabs.net/accounts5/accounts/3/profileslist?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc returned after 218 ms, status code: 200, request body size: 0 B
04/12 00:01:57 Trace: [fiveaccountserver] GOT {"status":"Success","profiles":[{"id":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","name":"XXXX","photo":"https://sooloos-profileimages.s3.amazonaws.com/five-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx--xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx.jpg","birthdate":"xxxxxxxx"}]}
04/12 00:01:57 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:57 Debug: [easyhttp] [31969] POST to https://api.roonlabs.net/roonmobile/1/cores/announce returned after 161 ms, status code: 200, request body size: 1 KB
04/12 00:01:57 Debug: [easyhttp] [31967] POST to https://api.roonlabs.net/bits/1/q/roon.base.,roon.internet_discovery.,roon.debug.,roon.broker.,roon.dsp.,roon.sood.?roon_auth_token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc returned after 248 ms, status code: 200, request body size: 253 B
04/12 00:01:57 Trace: [bits] updated bits, in 255ms
04/12 00:01:57 Info: [mobile] GOT HTTP API /hello
04/12 00:01:57 Trace: [mobile] Got Hello Request body={"coreId":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"}
04/12 00:01:57 Debug: [easyhttp] [31970] GET to https://api.roonlabs.net/profileimages/five-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx--xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx.jpg?random=0e859a4b-0b4d-4737-809b-8a889d602d60 returned after 436 ms, status code: 200, request body size: 0 B
04/12 00:01:57 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:01:58 Info: [stats] 9381mb Virtual, 2172mb Physical, 828mb Managed, 6355 Handles, 105 Threads
04/12 00:01:58 Trace: [remoting/brokerserver] network reachability changed. Kicking off discovery cycle
04/12 00:02:01 Debug: [easyhttp] [31971] POST to https://api.roonlabs.net/discovery/1/register returned after 138 ms, status code: 200, request body size: 1 KB
04/12 00:02:01 Trace: [inetdiscovery] registered 1 devices, 5 services
04/12 00:02:07 Debug: [easyhttp] [31972] POST to https://api.roonlabs.net/device-map/1/register returned after 317 ms, status code: 200, request body size: 8 KB
04/12 00:02:07 Trace: [devicemap] device map updated
04/12 00:02:13 Info: [stats] 9381mb Virtual, 2172mb Physical, 829mb Managed, 6355 Handles, 103 Threads
04/12 00:02:13 Debug: [easyhttp] [31973] POST to https://api.roonlabs.net/discovery/1/query returned after 212 ms, status code: 200, request body size: 74 B
04/12 00:02:28 Info: [stats] 9148mb Virtual, 2172mb Physical, 827mb Managed, 6355 Handles, 72 Threads
04/12 00:02:43 Info: [stats] 9148mb Virtual, 2172mb Physical, 826mb Managed, 6355 Handles, 76 Threads
04/12 00:02:58 Info: [stats] 9148mb Virtual, 2172mb Physical, 829mb Managed, 6355 Handles, 76 Threads
04/12 00:03:13 Info: [stats] 9084mb Virtual, 2172mb Physical, 827mb Managed, 6355 Handles, 68 Threads
04/12 00:03:28 Info: [stats] 9116mb Virtual, 2172mb Physical, 828mb Managed, 6355 Handles, 72 Threads
04/12 00:03:43 Info: [stats] 9693mb Virtual, 2172mb Physical, 828mb Managed, 6356 Handles, 144 Threads
04/12 00:03:58 Info: [stats] 9709mb Virtual, 2172mb Physical, 827mb Managed, 6356 Handles, 146 Threads
04/12 00:04:13 Info: [stats] 9838mb Virtual, 2173mb Physical, 830mb Managed, 6355 Handles, 162 Threads
04/12 00:04:13 Trace: [push2] retrying connection in 0ms
04/12 00:04:13 Trace: [broker/accounts] network reachability changed. refreshing
04/12 00:04:13 Trace: [broker/accounts] [heartbeat] now=2024-04-12 4:04:13 a.m. nextauthrefresh=2024-04-12 5:01:56 a.m. nextmachineallocate=2024-04-12 6:45:28 a.m.
04/12 00:04:13 Trace:  [broker/accounts]   doing auth refresh, next=2024-04-12 5:01:56 a.m.
04/12 00:04:13 Trace: [broker/accounts] refreshing account info for email='xxxxxxxxxx@yyy.zzz' -userid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:04:13 Trace: [fiveaccountserver] POST https://accounts5.roonlabs.com/accounts/3/login
04/12 00:04:13 Trace: [fiveaccountserver] BODY token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc
04/12 00:04:13 Trace: [push2] exception thrown. restarting connection (The operation was canceled.)
04/12 00:04:13 Debug: [tidal] network reachability changed. refreshing token
04/12 00:04:13 Debug: [kkbox] network reachability changed. refreshing token
04/12 00:04:13 Trace: [roonapi] network reachability changed. Kicking off discovery cycle
04/12 00:04:13 Trace: [mobile] [remoteconnectivity] Port Verification started due to: network reachability changed, port verification not in progress, starting a new attempt
04/12 00:04:13 Trace: [raat] [sood] Refreshing device list
04/12 00:04:13 Trace: [roondns] flushed 6 last-known-good entries
04/12 00:04:13 Trace: [raatserver] [sood] Refreshing device list
04/12 00:04:14 Debug: [easyhttp] [31978] GET to https://api.roonlabs.net/internetradio/2/api/location?format=msgpack& returned after 485 ms, status code: 200, request body size: 0 B
04/12 00:04:14 Trace: [radio/library] got location US
04/12 00:04:14 Debug: [easyhttp] [31977] GET to https://api.roonlabs.net/oauthcb/3/tidal/refresh?token=eyJraWQiOiJoUzFKYTdVMCIsImFsZyI6IkVTNTEyIn0.eyJ0eXBlIjoibzJfcmVmcmVzaCIsInVpZCI6NDgxMjIxNDAsInNjb3BlIjoid19zdWIgcl91c3Igd191c3IiLCJjaWQiOjI0MjUsInNWZXIiOjAsImdWZXIiOjAsImlzcyI6Imh0dHBzOi8vYXV0aC50aWRhbC5jb20vdjEifQ.AQMUwvup-XM5Up-5N1DNXTcRQu83gOMAEnNPxN5WNrssubGALfVRgcCASlsOECG1QmoTGxOXJbxwCi6Ky2yCRIJmAEoiRD9bfQdJBdy-9thXYQwj2IZr12itr6QMSHObfvahDgEub_-7tnlJef5leHduoGr7cByMIlGKf6WTCmf9HmWE returned after 517 ms, status code: 200, request body size: 0 B
04/12 00:04:14 Info: [broker/locations] updating location Tidal:Name=TIDAL:Id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:04:14 Debug: [easyhttp] [31976] POST to https://api.roonlabs.net/discovery/1/query returned after 554 ms, status code: 200, request body size: 74 B
04/12 00:04:14 Debug: [easyhttp] [31980] GET to https://api.tidal.com/v1/sessions/e4aaf06d-47be-403d-a947-27e1218f2fdc?countryCode=US returned after 112 ms, status code: 200, request body size: 0 B
04/12 00:04:14 Trace: [tidal/http] GET https://api.tidal.com/v1/sessions/e4aaf06d-47be-403d-a947-27e1218f2fdc?countryCode=US => Success
04/12 00:04:14 Trace: [tidal] transition loginstatus from LoginSucceeded to LoginSucceeded
04/12 00:04:14 Debug: [easyhttp] [31981] POST to https://api.roonlabs.net/roonmobile/1/cores/announce returned after 147 ms, status code: 200, request body size: 1 KB
04/12 00:04:14 Debug: [easyhttp] [31974] GET to https://api.roonlabs.net/push-manager/1/connect returned after 747 ms, status code: 200, request body size: 0 B
04/12 00:04:14 Debug: [push2] push connector url received from push manager: ws://push-connector-v2-0.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:04:14 Trace: [push2] connecting to push2 connector at ws://push-connector-v2-0.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:04:14 Trace: [push2] connected to push2 connector at ws://push-connector-v2-0.prd-roonlabs-1.prd.roonlabs.net/
04/12 00:04:14 Info: [mobile] GOT HTTP API /hello
04/12 00:04:14 Trace: [mobile] Got Hello Request body={"coreId":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"}
04/12 00:04:14 Debug: [easyhttp] [31975] POST to https://api.roonlabs.net/accounts5/accounts/3/login returned after 886 ms, status code: 200, request body size: 42 B
04/12 00:04:14 Trace: [fiveaccountserver] GOT {"status":"Success","userid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","token":"ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc","expiration":30,"email":"xxxxx@xxx.xxx.com","groups":[]}
04/12 00:04:14 Debug: [easyhttp] [31979] GET to https://api.roonlabs.net/porttest/1/myip returned after 901 ms, status code: 200, request body size: 0 B
04/12 00:04:14 Trace: [fiveaccountserver] GET https://accounts5.roonlabs.com/accounts/3/profileslist?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc
04/12 00:04:14 Trace: [fiveaccountserver] GET https://accounts5.roonlabs.com/accounts/3/userinfo?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc&machineid=ddd44979-fcc3-dacf-e080-361fbb8e1390
04/12 00:04:14 Trace: [broker/accounts] updated token. New expiration is 2024-05-12 12:04:14 a.m.
04/12 00:04:14 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:04:14 Trace: [bits] myinfo: {"pushid":"broker/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","roon_auth_token":"ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc","os":"Linux 6.6.25-t2-jammy","platform":"linuxx64","machineversion":200001392,"branch":"production","appmodifier":"","appname":"RoonServer"}
04/12 00:04:15 Debug: [easyhttp] [31984] GET to https://api.roonlabs.net/accounts5/accounts/3/userinfo?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc&machineid=ddd44979-fcc3-dacf-e080-361fbb8e1390 returned after 141 ms, status code: 200, request body size: 0 B
04/12 00:04:15 Trace: [fiveaccountserver] GOT {"user":{"tfa":{"enabled":false},"userid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","firstname":"XXXX","lastname":"XXXXXXXX","email":"xxxxx@xxx.xxx.com","joinmailinglist":true,"allowpushnotifications":true,"class":"Normal","groups":[],"dncs":[],"trialallowed":false},"status":"Success"}
04/12 00:04:15 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:04:15 Debug: [easyhttp] [31985] POST to https://api.roonlabs.net/bits/1/q/roon.base.,roon.internet_discovery.,roon.debug.,roon.broker.,roon.dsp.,roon.sood.?roon_auth_token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc returned after 168 ms, status code: 200, request body size: 253 B
04/12 00:04:15 Trace: [bits] updated bits, in 171ms
04/12 00:04:15 Debug: [easyhttp] [31983] GET to https://api.roonlabs.net/accounts5/accounts/3/profileslist?token=ee1cfdf2-284d-4a7b-a8de-7b02cba5bfcc returned after 220 ms, status code: 200, request body size: 0 B
04/12 00:04:15 Trace: [fiveaccountserver] GOT {"status":"Success","profiles":[{"id":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx","name":"XXXX","photo":"https://sooloos-profileimages.s3.amazonaws.com/five-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx--xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx.jpg","birthdate":"xxxxxxxx"}]}
04/12 00:04:15 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:04:15 Debug: [easyhttp] [31986] GET to https://api.roonlabs.net/profileimages/five-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx--xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx.jpg?random=e61ff3f8-d8b6-4487-a634-120d9d2d2660 returned after 286 ms, status code: 200, request body size: 0 B
04/12 00:04:15 Trace: [broker/accounts] Data updated. AccountStatus=LoggedIn MachineStatus=Licensed UserId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
04/12 00:04:15 Trace: [remoting/brokerserver] network reachability changed. Kicking off discovery cycle
04/12 00:04:19 Debug: [easyhttp] [31987] POST to https://api.roonlabs.net/discovery/1/register returned after 145 ms, status code: 200, request body size: 1 KB
04/12 00:04:19 Trace: [inetdiscovery] registered 1 devices, 5 services
04/12 00:04:25 Debug: [easyhttp] [31988] POST to https://api.roonlabs.net/device-map/1/register returned after 236 ms, status code: 200, request body size: 8 KB
04/12 00:04:25 Trace: [devicemap] device map updated
04/12 00:04:28 Info: [stats] 9413mb Virtual, 2173mb Physical, 833mb Managed, 6358 Handles, 105 Threads
04/12 00:04:43 Info: [stats] 9164mb Virtual, 2173mb Physical, 832mb Managed, 6358 Handles, 76 Threads
04/12 00:04:44 Debug: [easyhttp] [31989] POST to https://api.roonlabs.net/discovery/1/query returned after 217 ms, status code: 200, request body size: 74 B

[Fernando_Pereira]

Just took a look at my Roon server logs (on Ubuntu Server 20.04.4, Intel box) and I see the [push2] exception 2-3 times/day. I’m wondering if this some kind of failure to reach one of Roon’s servers…

[Jeff_Greenberg]

Full form submission

What’s happening?

· Something else

How can we help?

· I am experiencing freezes or crashes

---

Describe the issue

Running RoonServer on Ubuntu 22.04.4 results in occasional freezes/crashes. Logs reveal an unusual amount of network activity prompted by "network reachability changed" events.

Detailed descriptions of the issue including some log extracts are documented in the following thread:

RoonServer on Ubuntu 22.04.4 [Apple hardware]: a high rate of reachability/discovery storms and other issues

I’ve changed the title of this thread to be more specific about the issues I’ve encountered. I’m also correcting an error in the original post related to the rate at which the “reachability changed” messages occur (thanks to @Andreas_Philipp1 for catching that!), I’ll also be updating with actions taken since the original posting and the current state of play. In all likelihood these edits will not occur all at once as I have limited time available to post. – Jeff I’ve read the various threads …

I have been unable to resolve this problem with community help and believe that only the Roon team has the technical knowledge to find the root cause of the issues. Thanks.

[Bill_Janssen (Wigwam wool socks now on asymmetrical isolation feet!)]

This kind of “network reachability changed” storm has happened to me twice. Once it was because I’d enabled, on my router, a feature where it would dynamically reassign WiFi channels to my WiFi clients in order to optimize throughput. Every time it reassigned a wireless endpoint, Roon would freak. The second time was due to a Chromecast bug in which a phantom cast group would be created and destroyed periodically, thereby creating a network change.

[Jeff_Greenberg]

The second time was due to a Chromecast bug in which a phantom cast group would be created and destroyed periodically, thereby creating a network change.

@Bill_Janssen Bill, that’s interesting. The only device on my network that supports Chromecast (AFAK) is the WiiM Pro Plus and it’s a fairly recent addition. I had been thinking it would be a good idea to take each of my endpoints off line to see if it had any impact. I’m currently traveling but I’m going to try that experiment next time I’m at home.

[noris]

Hi @Jeff_Greenberg ,

I’ve moved your post back to your original thread, as mentioned earlier, running Ubuntu on a Mac is not an officially supported platform to run Roon on, so we can’t provide specific suggestions regarding your Mac.

I did look over your thread and I can offer a few suggestions though. When I’ve seen network reachability errors like the ones you describe, it was sometimes due to IPv6 configuration or having multiple networking interfaces active and Roon getting confused.

It looks like you have multiple adapters and you might be using IPv6 in your setup, so I would try to simplify this way and check if it helps. If you temporarily host the Roon Server on another PC and connect it in the same way as the Mac, do you also see network reachability errors that way?

If you can reproduce the network reachability errors on a system spec that we support, we can do a deeper dive into this as a supported case.

[Jeff_Greenberg]

I’ve been spending some time looking through the Network Manager logs in trace mode and finally come to the conclusion that it is Network Manager that is the source of the erroneous reachability changed messages. To test this, I disabled Network Manager completely while leaving the static IP configuration unchanged (i.e. manually configured static IP, manually configured DNS, etc.)

The result was the complete elimination of the reachability events described in my initial post!

So far, I have observed no ill effects on Roon or anything else as a result of this change. However, I have not tested long enough in this configuration to know if the occasional RoonServer crashes are also gone. And network traffic between my server and Roonlabs looks like it may be reduced by close to 99% based on the elimination of the 1 per minute re-authorization exchanges that have been happening (I haven’t directly snooped the network to verify that but it seems likely given what I’m seeing in the logs).

As for what exactly is going wrong in Network Manager I can’t say for sure. Symptomatically, NM appears to adjust the routing table entry associated with my wired interface by dramatically increasing the routing metric. Then, it changes it back to its original reasonable value moments later. But the routing table change is associated with a network change event that must be filtering down to RoonServer. It’s not clear if the metric change is a cause of the problem or just the first visible manifestation of some other change that isn’t reported in the logs. I’m not really motivated enough to dive into the Network Manager source code to track it further at this point.

[Jeff_Greenberg]

@noris Thanks for the suggestions. I had previously tried disabling ipv6 but this had no effect on the situation. I also spent time ensuring that one and only one interface was being managed by Network Manager but, again this had no effect. As indicated in my most recent post, it was not until I completely disabled Network Manager that I was able to eliminate the reachability events.

As there seem to be many downstream changes to Network Manager in Ubuntu 23+ (including integration with Netplan which is not installed on my 22.04 system) tracking this further doesn’t look like a good use of Roonlab’s time or mine. If the workaround is stable I’ll just go with it until the Ubuntu network stack settles down a bit.

[Andreas_Philipp1]

@Jeff_Greenberg

I’ve been spending some time looking through the Network Manager logs in trace mode and finally come to the conclusion that it is Network Manager that is the source of the erroneous reachability changed messages.

That’s interesting… I remember that I and some other users in the second half of 2020 under 20.04 LTS experienced Roon server crashes, which I could trace back to NetworkManager. At this point I had installed Ubuntu desktop which uses NetworkManager, and decided to reinstall the machine with a minimal Roon server setup, which per default doesn’t install nor use NetworkManager. I then, three years ago, installed Netplan just to get familiar with it… The crashes went immediately away…

Two years ago I did a fresh installation of 22.04 server, which per default installed Netplan and set my Ethernet interface up to be managed by systemd-networkd. I have set up IP reservation on the router, so it’s not a really static configuration, but never again have I seen any problem, since giving up on NetworkManager…

Even if using Ubuntu Desktop which by default uses NetworkManager, you can install NetPlan and set it easily up to delegate to networkd instead of NetworkManager which is fishy…

[Jeff_Greenberg]

@Andreas_Philipp1

@Andreas_Philipp1

installed Netplan just to get familiar with it… The crashes went immediately away…

That’s actually good new to me. I’ve been considering a Netplan install to make configuration a bit easier so it’s good to know it didn’t cause more problems when you used it.

In fact, it’s entirely possible that all of my grief is traceable to a decision to install Ubuntu Desktop rather than Ubuntu Server. I had read about both and it seemed they were nearly identical. True, but at the time I didn’t know enough about Ubuntu to understand what nearly really meant.