It is unfortunate that roonlabs is still ignoring this bug.
It is a time sink for me to continue responding to all of you who are trying to be helpful but are only reiterating the party line that it’s the network when all evidence points elsewhere.
It is time for roonlabs to stop ignoring the 160 reported instances of this bug, plus the who-knows-how-many unreported instances when it magically heals itself and so goes unreported. I count approx 6 reports in the last 6 weeks. Given 130 weeks since my first bug report on this and around 160 hits from the query I posted earlier, the reporting frequency is pretty consistent (1 x 130 ~= 160).
It is time for roonlabs to fix this nuisance and give users back the time they waste working around this or fiddling with their networks, hoping that some random change will trigger another edge case that temporarily “fixes” the bug.
FWIW, gentlepersons, I’ve been writing software since the 70s and architecting large systems since the 80s. I designed and built back-end systems and user-facing apps at amazon. In 1987, we put our dev team on the NSFNET (a successor to the ARPANET) before the internet existed and enabled remote developers. I’ve been maintaining my home networks since the dawn of the PC era. I built the first browser-based ATM locator mapping app on the internet (and the back end for it) for Visa for the 1996 Atlanta Summer Olympic games a decade before google gave free maps to the world. We have sufficient expertise in-house to isolate and tell the difference between network and software problems without having the code to root-cause bugs.
Again, @DaveN, I’ll answer your question even though it’s not relevant to resolving this. Yes, I’ve had the NUC on the wired part of the lan once and it was still broken. noris postulated that packet fragmentation might be the problem, and as you read, I showed there was no packet fragmentation here. I continued this discussion via PM with other support staff a couple of months later, they didn’t want any more information, said they’re still looking into it, yet I doubt they are after 2y4m since they are definitely capable of fixing this.
Thanks for taking the time to answer my question. I guess the bottom line is that this is a relatively rare bug - 1 report per week, from several hundred thousand users - and doubly difficult to pin down as it’s intermittent. There’s clearly something in your setup (by which I mean both hardware and software), and the way in which it interacts with the relevant Roon server, that triggers the problem.
Yeah, you would think they would be able to fix it, but I think it depends on what needs fixing. There are a few examples here that demonstrate that it’s not always a straightforward process, as I’m sure you’re aware.
I’d like to take a look at your core’s logs. Can you leave the core online and if possible, cite the date/time you observe the issue occurring?
I already know from reading your thread here that this question won’t make you happy but I need to know. Has your modem or router config changed? From what I see in your old post you have a modem gateway (and I know from experience that your IP scheme is Xfinity when I look at the server side of things). It also sounds like you have another router being used, is that correct?
The bug had gone into hibernation by the time I wrote this post four days ago. The logs from before then, when the bugs were active, are gone. Log backup retention seems to be set to 20.
My ISP is Charter/Spectrum. The router is still the same R6400 as when I made by first report 2 years ago. The NUC is still on the other side of the wifi bridge, same as before. Its metadata improver was paused for most of the past 2 years while running roon core under Windows 8.1. A few months ago, I installed ubuntu on the NUC and roon core has been behaving well on it. Note that there was no change in network topology, just a fresh install on a fresh/different OS.
Charter/Spectrum replaced my gateway (cable modem) in 2020 or 2021. That was almost certainly after I filed the original bug report.
I looked at one of the recent logs on the core server that this report pertains to, while roon was working correctly, and could see that it was having trouble getting to some of your google cloud hosts. The log had several “[easyhttp] … No route to host” errors though it was able to resolve the IP addresses! I don’t know how easyhttp was doing the DNS name resolution because the resolved configuration was screwed up by me while trying to troubleshoot these bugs after adding the second NIC. (Since fixed!) In spite of these errors, I was pulling metadata on demand.
08/17 12:22:37 Warn: [easyhttp] [1] Post https://accounts5.roonlabs.com/accounts/3/login web exception without response: No route to host (34.148.110.116:443) No route to host (34.148.110.116:443)
08/17 12:22:37 Warn: [easyhttp] [4] Post https://accounts5.roonlabs.com/accounts/3/login web exception without response: No route to host (34.148.110.116:443) No route to host (34.148.110.116:443)
08/17 12:22:37 Warn: [easyhttp] [2] Post https://accounts5.roonlabs.com/accounts/3/machineallocate web exception without response: No route to host (34.148.110.116:443) No route to host (34.148.110.116:443)
08/17 12:22:37 Warn: [easyhttp] [5] Get https://geoip.roonlabs.net/geoip/1/lookup web exception without response: No route to host (35.231.208.158:443) No route to host (35.231.208.158:443)
08/17 12:22:37 Warn: [easyhttp] [3] Get https://devicedb.roonlabs.net/1/devicedb-prod.zip web exception without response: No route to host (35.231.208.158:443) No route to host (35.231.208.158:443)
08/17 12:22:37 Warn: [devicedb] While refreshing, status: 999, body: System.Net.WebException: No route to host (35.231.208.158:443)
---> System.Net.Http.HttpRequestException: No route to host (35.231.208.158:443)
---> System.Net.Sockets.SocketException (113): No route to host
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at
<snip>
at Base.EasyHttp.QueryAsyncInternal(HttpMethod method, Params p, CancellationToken canceltoken, IAuthProvider auth)
Finally, the core server is usually on a 10Gbe QNAP managed switch with another machine with a fast NIC. Both it and the other machine were temporarily on an old D-Link 1Gbe 8-port unmanaged switch while troubleshooting this. They’re both back on the QNAP and working correctly.
The next time the bugs rear their heads, I’ll grab logs and get back in touch.
About a week ago, I again reported this 2+ year old bug again where roon cannot identify music when it’s added to a library and the metadata improver pauses indefinitely. It’s been closed by someone so I’m continuing it here.
I notice in the log files that roon appears to cache dns entries.
08/17 12:22:31 Info: Starting RoonServer v1.8 (build 1021) stable on linuxx64
08/17 12:22:31 Info: Local time is 8/17/2022 12:22:31 PM, UTC time is 8/17/2022 7:22:31 PM
08/17 12:22:31 Trace: [roondns] loaded 22 last-known-good entries
@Wes, does this explain how roon seems able to resolve FQDNs when it can’t route to google’s cloud as noted in my last post in the above thread?
This has been reported five more times in the last five days. Maybe some users have resolved this by following the usual advice to check their users’ lans, reboot, and so one, maybe some have been resolved by random acts of buggy behavior, maybe some are still open.
@ezman, I read your previous posts. Have you had any more instances with the metadata improver since your last post? If so, were you able to submit your logs to @Wes? If no issues, has Roon been stable otherwise for you?
You’ve definitely got a lot of data for us to peruse here. I’ve been discussing your case with research and development and we’d like you to try something for us.
Will you please try changing your core to a different machine, adding music, and seeing if you can duplicate the issue? Temporarily changing your core to a Windows or Mac machine could help us tremendously. If you duplicate the issue, please notate the date/time and provide logging for the instance.
@Wes I’d like to make a plan for this experiment with you, and perhaps with input from your R&D team, regarding the network location/configuration of another machine. I’m moving our conversation to PM for now.
@ezman - I appreciate your fighting the cause here. I am one of the five users that you highlighted a few posts ago and the issue has still not gone away. Please keep us updated with progress. Like you, I’m a software veteran, admittedly more in the mainframe side of things although my current role involves branching into many Linux-related areas.
I must admit I haven’t got around to the router / modem restart yet, but there are two reasons for that, one is that it would inconvenience other users of the network, and the second is that none of my other devices are experiencing any network or connectivity issues whatsoever, so we’re sort of living without the metadata improver right now.
But why is my networking hardware assumed to be the problem here? Every other component within Roon that requires network comms is working perfectly. I have also highlighted the anomaly between this message in the log:
08/18 18:24:19 Warn: [easyhttp] [16] Post https://identifier.roonlabs.net/identifier/2/album web exception without response: Network is unreachable (35.231.208.158:443) Network is unreachable (35.231.208.158:443)
yet the same box can ping that IP address from the command line without fault:
dloader@deb-mus-svr:~$ ping 35.231.208.158
PING 35.231.208.158 (35.231.208.158) 56(84) bytes of data.
64 bytes from 35.231.208.158: icmp_seq=1 ttl=104 time=106 ms
64 bytes from 35.231.208.158: icmp_seq=2 ttl=104 time=100 ms
can someone explain to me how my router could cause that? Either a remote IP address is reachable through a network device or it isn’t. If ping can reach it but the software can’t, then I’m sorry but the problem is with the software.
As for those 4 networking ‘best practices’ steps - I’m sorry, what? Why? Roon should damned well work with whatever DNS I have set, thank you very much. Every other piece of network-dependent software I run manages to. Disable IPv6 and not use certain bits of networking gear? I expect this kind of nonsense with open source but this is a $600 piece of proprietary software. This stuff should be taken care of within the code, not lumped onto the back of the customer to deal with.
@ezman - if you need me to provide any logs from my machine in order to make meaningful progress, please let me know.
For me, the only plausible network-related problem that could make these two functions fail while everything else network-related works is some critical latency requirement, probably related to an authentication call.
Along the way – PM? bug thread? – someone from roonlabs speculated that packet fragmentation might have been the culprit in my lan. I surmised from that that if some network call to an authentication service sends packets that fragment, and the fragments take very different routes, and the round trip latency for the call after reassembly exceeds some threshold, their code assumes that the user isn’t licensed and so they turn off these two metadata-related functions in the code.
Otherwise, yeah, if you can ping their cloud hosts, and dns is resolving, and your browser is working, and everything else in roon that uses the internet is working, then the problem with the metadata improver and batch loading metadata for new music isn’t related to the lan or wan interface.
Right - but again, if that’s true about the packet fragmentation then I would question why there are conflicting decision paths for authentication in the first place - either you are authenticated or you are not.
It’s not a great design if the main bit of the software presents to the customer as if everything is fine and dandy, but there’s this mysterious red triangle that you need a degree in computer science to get rid of.
How do you think the customer is going to rate their experience if they have to fathom out how to adjust their DNS lookup path for software that is supposed to be simple to use and for which the asking price is $600 or whatever it costs now for a lifetime license.
I have not had this problem reoccur since my previous post about 2 weeks ago. I returned the DAC I was using at the time and had it replaced but since these were noise problems it’s unlikely this particular problem was related to my DAC so perhaps Roon fixed this - at least for me.
AceRimmer
(Smoke me a kipper, I'll be back for breakfast!)
44
Just a word of caution here
Please refrain from reposting flagged and agreed/removed posts, there would have been a reason for its removal that may not be clear to you but it was a moderator/admin action.
Thank you for your patience and understanding
I hear you. There’s no discernible pattern to these two bugs that I can discern. They pop up and go away, can be around a short while or persist for months on one machine while not on another on the same lan. That’s a strong argument against it being network-related.
Their appearance is apparently random, but all bugs have some root cause.
@AceRimmer - Sure, I get that but I did not reply to those posts, I replied to the primary topic - unless this system works differently than normal forums…
roon was running fine two days ago (Thursday) on build 1021 here. The log file shows that the server downloaded build 1105. Today, I fired it up again, it’s running 1105, and the red triangle is back.
Neither network topology nor network configuration were changed between then and now.
I’ve sent @Wes log files from Thursday and today in our support PM conversation. Maybe that will help them fix it.