ROCK on Intel NUC 10 NUC10i7FNH1, Samsung SSD for OS, WD USB drive for music, fast CAT6 cable connection.
Networking Gear & Setup Details
Starlink on IPv6, direct bridge mode adapter, ASUS RT-AX3000
Connected Audio Devices
Denafrips Ares II with upgrade - USB connection from NUC10
Number of Tracks in Library
100k+
Description of Issue
Everything was working on Early Access. After seeing that you had released the early access builds for IPv6 to production I switched everything back to production from early access. I’m on 258 production for ROCK OS, 1277 production for Roon Server & Remote. I switched my Android Auto system back to the production Roon ARC as well so it’s on the latest version.
The Roon ARC test says everything is fine but Roon ARC on my Android Auto system says it can’t find the Roon Core. I’m pretty sure the issue must be with Roon ARC on Android Auto since everything is working fine on my home network and the Roon ARC test succeeds there. I know I have a strong Internet connection in my car because it installed Roon ARC in a few seconds.
Please provide some test suggestions to help me figure out why this configuration won’t work on the production builds.
Can I please get a response from support? If nothing else, can you tell me if Starlink with the bridge adapter and IPv6 which worked fine on early access is supposed to be working on the current production release or should I go back to early access?
Ben with support here, thanks for writing in! We’ll take a closer look into your core diagnostics - but things typically should be functioning normally with your system being on the new production build.
One follow-up question for you: can you clarify how Arc was installed on your pixel? Did you manually install it from the APK file, or via the Play Store? If manually installed, we’ve seen problems in the past in regards to updates.
I would uninstall Arc, power down your device, and reinstall Arc over stable wifi from the Play Store, and let me know if your issues persist.
I did this on both my phone and my Android Auto system but am getting the same result. However, I just noticed that my pre-IPv6 settings were still in my router for the IPv4 port forwarding so I removed this along with the IPv4 Port Trigger which I’m not sure if I ever needed. I then ran the Roon Arc test and it’s still working fine with only the IPv6 firewall port passthrough enabled. I checked and if I disable this the Roon Arc test immediately fails with an error showing it cannot reach the Roon Core.
I tested this and it didn’t change anything. I’m getting the same results on my phone (Pixel 7 Pro) as I am in my car on Android Auto so I guess that moves the likely cause of the problem somewhere on my home systems.
Basically I have this situation:
The Roon ARC test says everything is fine
When I use Roon at home it works perfectly
When I try to connect remotely with Roon ARC it shows me the cached environment from the setup but if I try to play anything it says poor connection, cannot reach the Roon Core or something similar. Everything else (Qobuz, Tidal, etc) works fine on my phone/car.
Lmk if you can think of any other settings I may need to add in order to make things work over IPv6 or any tests I can perform to narrow down the cause.
Unfortunately, after reviewing logs, it appears you’re encountering a known issue for which we have an urgent ticket with development. Cores that have recently migrated, restored from Backup, or switched branches (as in your case) are encountering erroneous poor connectivity errors. It’s almost certainly the case that the error doesn’t reflect your actual connectivity situation, but is in fact an authentication issue caused by the recent branch switch.
The known workaround is to delete and reinstall ARC; you can do this via the Play Store or the .apk installer file. Unfortunately, you’ll lose customization in MUSE and any locally downloaded tracks, but there’s a good chance it will resolve your issue. We should have a solution released shortly but don’t have one public as of this time.
I tried uninstalling and reinstalling Roon ARC but it’s still doing the same thing. The Roon ARC test says everything is fine when I run it on my home network but when I’m away from home Roon ARC won’t connect. After I uninstalled it the reinstall couldn’t even complete the initialization where it does the original connection to the Roon Core. I had a very fast 5G connection to my car. As I mentioned previously, the same exact thing is happening on my phone (Pixel 7 Pro) and the Android Auto system in my car.
Don’t you guys have any trouble shooting tools to figure out what is going on with this? I’m on StarLink running bridge mode with an IPv6 address. I was getting connections with the early access beta but was having various problems so I switched to your new production release and now I can’t even get a remote connection.
I’m pretty sure it’s not that issue since uninstalling and reinstalling didn’t help. However, here is one detail that may help you guys figure this out: after reinstalling Roon ARC and seeing that it wasn’t going to connect, I just switched over to Qobuz and finished my drive home. I didn’t realize I left Roon ARC trying to connect until I entered my garage and it connected to my home network. Roon ARC immediately connected and started playing where I had previously left off. I wondered if Roon ARC on my phone would also connect since I had tried your uninstall/reinstall suggestion on both but on my Pixel 7 Pro ARC locks up with the initial image on the screen and I have to manually exit and kill the app to get out of it.
This may be due to the phone having the latest version of Android and my car is Android 10. Lmk if you have any other suggestions for me to try.
To confirm, did you reinstall ARC via the Play Store or the .apk installer file? And with that, have you tried both methods? Not that this is the long-term solution, but more so an important piece of information that would be helpful to our development team.
And with that (my apologies for the redundancy) if you could recreate the issue and share the date, time, and track attempted to play that would be immesnly helpful.
I appreciate your patience as our team continues to pin this down!
I was testing with Roon today and didn’t notice your message until I was done but here is a new update for you…
I’ve never tried installing with the APK, just from the Play Store.
When I reinstalled Roon ARC in my car and it didn’t work I didn’t close it and when I pulled my car into the garage it connected to my home network and Roon ARC started playing music. This shows without a doubt that the problem is somewhere between the Roon servers and the external interfaces of my router (e.g. WAN vs. LAN/Wifi). I had assumed this because everything works fine with Roon Remote but I hadn’t thought of testing Roon ARC inside my home network.
I reverted everything to early access and unfortunately that doesn’t work either now but I left it setup with the beta code for now.
I temporarily turned off both my IPv4 and IPv6 firewalls today and as I assumed that didn’t make any difference.
Today I noticed that even though Roon ARC says it cannot connect to my Roon Core, I can play any of the albums that are on Qobuz or Tidal. I don’t have anything stored on my car system and I can see the music downloading over my connection. I guess this makes sense because both Qobuz and Tidal work fine without Roon but this also shows that the problem is the connection to my local library files on ROCK.
Regarding the information you requested, I don’t have the exact time for the failure test but I was trying to play a low bandwidth track from my music server which is probably unique enough to find in the logs. Here is the info: Today 6/30, roughly 2:30-2:45 PM PDT, Artist - “Overhead”, Album - “Telepathic Minds”, Track - “War to End All Wars”, format - MP3 44.1/320. If you can’t find this I will run a new test and log the exact time for you. In case it helps, I ran a successful test of the same artist/album/track from Qobuz on Roon ARC at about 3:05 PM PDT
Btw, I’m a computer hardware/software engineer/architect with experience in software development, networking and systems administration so if your team needs me to help with anything on my side I can do whatever they need regardless of how technical it might be.
Starlink modems don’t offer port forwarding or IPv6 pin holing, and so the Asus router you have that is most likely set to Bridge mode, is instead acting more like a filter that isn’t passing your IPv6 address properly.
If possible, can you share a screenshot of the RT-AX1800S Asus web GUI (with your IP address blurred?)
I’ve been through all of the Starlink updates posted by the Roon teams and have my systems configured exactly as the Roon engineers have specified. Here is a short overview:
My Starlink router is disabled and is connected to what Starlink calls a “LAN adapter” which is a hardware device that puts Starlink into bridge mode. This means that the Starlink router is doing nothing on my network except for providing a bridged CAT6 connection to my ASUS router just like any ISP using PPoE or similar connections would do so I don’t see how the Starlink router can be causing problems since it isn’t being used. I know something is causing an issue but as you can see below, the Roon ARC test passes fine so it’s not something obvious.
I have an IPv6 address from Starlink so I am not using CGNAT or any other IPv4 technology. All communications to my router comes in to my IPv6 address which you should be able to confirm in your logs.
My ASUS router is configured correctly for Roon ARC and the Roon ARC test confirms that it has a verified secure connection to my Roon Core so it’s not the Starlink router or CGNAT. I will post an image below which should prove this to you.
The IPv6 addresses that are truncated are for my Roon Core and everything matches between the ARC test and my router settings. I do not have any IPv4 port forwarding enabled since I am not using IPv4 for Roon. I've verified that enabling or disabling the IPv4 port forwarding makes no difference to Roon ARC. However, disabling the IPv6 firewall hole immediately breaks the Roon ARC test. However, I'm wondering if the problem with Roon ARC is that it's trying to use the IPv4 address even though the Roon ARC test says everything is fine over IPv6. Could there be some Roon ARC setting that could allow me to specifically set the IPv6 address for my Roon Core?
I have installed the latest early access of the Roon OS on my NUC which contains ROCK. However, it did not ask me to update the ROCK code (e.g. the Linux code) since my initial installation. It says I’m running Version 1.0 (build 258) of the ROCK OS and version 2.0 (build 1292) early access of the Roon Server software. Is this correct?
Thank you for your response and for your patience. We’re aware of your useful contributions to the Starlink thread @benjamin shared above and how you’ve bridged the Starlink modem.
Here’s what we can see is wrong from our end:
Diagnostics from ARC’s framework for your account show ARC rejecting an attempt at IPv6 pinholing from your external IP due to an improperly-formatted IPv6 address. Specifically, diagnostics report a hex digit violation in the 10th character position. If you’ve entered any addresses manually, it’s worth double-checking nomenclature here and reviewing any zero omissions.
Starlink offers native (dynamic) IPv6 with intermittent rollout; their prefix delegation is still in beta and users have complained about router solicitations failing. Your geographic location (the same as mine, so I’m familiar with Starlink’s idiosyncracies here) indicates that you were likely in an earlier pool for the rollout. We’ll need to try to get a full picture of your IPv6 service and check for kinks in the network.
Create a Backup of your Roon database, first, for due diligence. Then, unplug your ROCK from your network and set up a Roon Core on your Windows machine after specifying an IPv6 address for that computer. If you create a pinhole rule for that IP, are you able to replicate this issue on the second Core?
I understand that’s a laborious test, but it should help us pinpoint the obstruction here by eliminating RoonOS as a variable.
RoonOS will always advertise the IPv4 address, so it’s best to disable IPv4 entirely to avoid a default to IPv4.
That’s correct, a version mismatch doesn’t appear to underpin this problem.
Wow, thanks! It’s great to know the cause of this problem. Yes, I was very early in Starlink program but chose to wait for the new square dishes so my account date was much earlier than my hardware. I’m not sure if that makes any difference.
I did not manually set the IPv6 address. I’m running the default native settings for my ASUS router which I posted above so it would be great if you could check this. I have disabled all port forwarding via IPv4 if that is what you meant by disabling IPv4. I don’t think I can completely disable IPv4 since I’m pretty sure Starlink still uses it in a parent mode over IPv6. Lmk if there are any settings I need to make regarding this.
I went through that Reddit thread and although that could explain why things were originally working on early access then stopped working when I switched to the production build but I don’t have the same problems being reported there. I’ve had the same IPv6 address & prefix since Roon added IPv6 support (I didn’t check before that).
My ROCK is installed on a Samsung NVMe drive on a NUC10 and I have a Windows 11 machine that my previous Roon Core was installed on. I use SMB for moving music files between Windows and the USB 3.0 drive connected to my NUC that contains my music files. So to run your recommended test I would just need to resurrect the Roon Core on Windows and plug the same USB drive into the Windows machine which shouldn’t be difficult unless I’m missing something. I’m curious what would it would tell us if this test works?
Reviewing the case and your response, it appears there are two problems that might be independent or intertwined:
An IPv6 formatting issue. Since you didn’t manually assign the IP, this IP should have been dynamically assigned and prefixed by Starlink’s servers. There’s a possibility that the hex digit violation in the 10th character position is a nomenclature disagreement between the machines/servers in this chain - looking at the four-bit addresses itself, there’s a possibility that a zero omission has been treated incorrectly somewhere, possibly in RoonOS on the ROCK or in ARC. I’m not a network engineer, but this is an educated guess. Let me confirm this internally with the team today to eliminate that as a possibility if possible.
Switching to the Windows 11 machine was intended to remove RoonOS as a variable since the Windows 11 machine is IPv4/IPv6 dual-capable as well. Honestly, it’s an imprecise test that I’d suggest if you have the time to tinker. There’s probably more we can learn from escalating internally with development on our end.
That said, let’s investigate the second possibility here, which we can likely still test for:
A Core connectivity issue related to build mismatch production to earlyaccess.
We can troubleshoot this by investigating the connection between your Remotes and your Core. First off, are you able to share a screenshot of the Web GUI of your ROCK? You can find it by opening a browser on your network, steps here. This will just confirm basic diagnostic functioning and the IP address your ROCK is actually using to solicit Starlink
Also, open a Roon Remote (not ARC) on a laptop or phone on your network and sign in/out of Roon. Do you have any issues, or are there any warning messages about Core connectivity?
I reviewed your ASUS router settings and there don’t appear to be any culprits here. DHCP prefix delegation is on, your IPv6 is set to Native to accommodate for Starlink’s dynamic assignment. The only way this could break is if the IPv6 machines behind your router aren’t prefixed properly - thus my suggestion of reviewing the Starlink Reddit thread. There’s no telltale sign broken prefix delegation is the culprit here - there would be multiple other IPv6 addresses visible from your account in our logs, and we’re only seeing the IPv6 addresses assigned by Starlink and Verizon to your internet and cellular networks.
We’ll get through this, @Dean_Kimball. I know how frustrating these stop/stop networking situations are when you just want to get back to the functional app you so recently had. Thanks for your patience again.
Sorry for the double post - we’ve discussed internally with ARC development, and there’s a distinct possibility the issue here is that your ARC install is tied to the earlyaccess beta program through Google Play Store. Since we’ve introduced single-sign-on and related authentication changes recently in earlyaccess, ARC and the Core aren’t properly shaking hands.
If you haven’t tried this yet, let’s give it a shot. When you leave the earlyaccess program, it’s Google (not Roon) that has to dissociate your account from the beta track. There’s often a several-day delay, which means your phone probably still has earlyaccess installed.
I put my ROCK Roon Core back on production using the normal method with the instruction file. I then installed Roon ARC on my phone (Pixel 7 Pro) and on the Android Auto system in my car using the apk url you sent me. It fixed the problem of Roon ARC locking up on my phone and both RoonARC systems initialized and ran while I was connected to my home wifi system. However, both my phone and my car still have the same problem as soon as I drive out of wifi range and connect to the cellular network. I first get a “Poor connection” message while it’s trying to connect to my Roon Core. When I click the retry button it comes back in perhaps 15 seconds with a “Cannot connect to the Roon Core” message.
I just ran these tests at approximately 2:45 pm PDT on my phone and about 2:50 pm PDT on my Android Auto system (± 5 min). It was still on the same album - Artist: Overhead, Album: Telepathic Minds & probably the first track “War to end all wars” if you can find it in the logs.
I think it’s beyond coincidence that you found a hex digit violation in the 10th position which is in the only set containing a leading zero in my IP address so I agree with you that this is likely the cause of the connection problems. Afaik there is no way I can get Starlink to regenerate my IP address plus no reason to expect that a new address wouldn’t also have one or more leading zeros so I’m not sure we can test this theory. However, since leading zeros are common in IPv6 addresses, this issue is either fairly widespread or something else may be triggering this issue in my case. Perhaps your dev team has some thoughts on this that could lead to a fix or a temporary work-around.
I’m open to doing any other tests you recommend including the Windows test but let’s do that one as the last resort.
Thank you for the immediate and thorough response - we’ll investigate ARC and Core diagnostics at the timestamps around the Overhead track you’ve mentioned.
Around the time your ARC download would have been on the beta track, we were releasing some SSO-related changes that may have affected treatment of IPv6 format. Since you’re seeing the same issue on the public ARC build now, we’re going to investigate thoroughly to head off a potential more widespread issue if this is in fact a Starlink DHCP issue.
We’ll have an update shortly after out next sync - thanks again for your patience.
I switched back to early access so I can pick up any updates you release regarding this problem. The switch to early access on my ROCK and the Roon Remotes went fine but I ran into a problem installing the early access version of Roon ARC from the play store. It goes back to the mode of locking up on my phone (it must be killed to get out of it) and on my Android Auto system it goes into some weird loop. I made of video of this if you are interested but here is what happened on Android Auto:
First the update from production to early access failed so I had to uninstall the install the EA version.
After installing I got a very scary message from Roon that said something like “Unless you can verify that you installed this version of Roon from a verified source then exit immediately and uninstall it.” Since I had just installed it from Google Play Store I ignored that and found that I was not in Roon ARC but in my account on the Roon website.
After exiting my account, Roon ARC started then put up a big red banner at the bottom saying “Something failed” (not too helpful).
I killed that instance and restarted Roon ARC and this time it went into the loop where it says to login but instead of getting the normal ID/PW fields it tries to go to the Roon website where you see a button appear in the top right that says “Free Account” then the site reloads showing this same thing over and over until you kill it.
I then uninstalled the version of Roon ARC from my phone and Android Auto and installed the APK you gave me. This works but wants me to update which if I do this goes back into the above failure mode.
I installed the EA version of the Roon ARC APK and both my phone and Android Auto system are back to working the same way as production (e.g. it works when connected to my home network in the garage but not on the cellular network). However, I should now pick up any changes you guys provide. I don’t think it can only be me who isn’t working correctly when running the Play Store version of Roon ARC EA with IPv6 although it could be limited to Starlink IPv6. Any idea why the APK is different from the Play Store version?
Thanks for posting your experience in detail - I’ll pass this along to both our QA team and ARC dev, as we’re currently working hard to add considerable changes to ARC/Core synchronization.
Unfortunately, Google owns the implementation for the Play Store #earlyaccess track. This means there is sometimes a lag in account authorizations when you switch between the Play Store Production and EA builds, and sometimes it will take several days for an ARC account curated by Google Play Store to recognize a Core on the same track (EA vs. production) after you’ve switched.
The APK downloads don’t involve the Google authorization wrapper like the Play Store downloads, so account authorizations are more likely to instantly match.
In any case, it shouldn’t be as complex as you experienced, and we’ll hopefully make this process more smooth, considering that switching to EA is part of troubleshooting your original issue.
We’ll look into diagnostics today to see what else we can learn. I’ll follow up here. Thanks again!
