Timeouts when fetching images

Note: This was split from another topic as @John_Steer1 is experiencing a different issue. Customer is reporting that images are failing to load in the Roon remote. Logs show timeout errors when contacting Roon’s cloud services


@AMP Hi Andrew, I have the same error continually on my ROCK server. I reboot it and it works fine for a day or so then the artwork disappears again and the log is full of the GET errors until I reboot again. Is this change you have done on the Nucleus available for the ROCK?

@John_Steer1, go ahead and give it a shot. Please restart RoonServer on your ROCK twice using the button in the web interface.

@AMP What is the change required on the ROCK before the restarts? Just restarting only temporarily solves the problem.

Thanks.
John.

I made a settings change on your core just before I sent that message. The first restart forces your core to download updated settings. The second applies the change.

@AMP Ah… sneaky back end access :slight_smile: Restarting now. Will the change be lost with subsequent upgrades?

@AMP Worked for a day now back to missing artwork and the log full of messages like:

https://imagecache.roonlabs.net/im/1/albums/3e011ba63bf997a40e4aab8161f0ff46b955/cover/500.jpg: statuscode=998 error=Request canceled due to timeout error

I have the same problem now and then. It seems to be network related.
Restart your router, that solves the problem - at least in my case.
Maybe it will help you as well.

Guenter

Thanks @Guenter_Borgemeister ,

I have tried rebooting the router but the only temporary fix seems to be restarting the actual ROCK Nuc which is connected via cat6 cable directly to the router.

Its almost as if something in the ROCK has stopped responding to the GET requests because I can run them manually from any other pc on the network and retrieve the images in a browser with no problems. Even tried a complete reinstall of the ROCK and restore from backup. Still keeps happening. It used to work great until a couple of releases ago. Hopefully a bug will turn up and get fixed.

@John_Steer1, I made a calculated guess without looking at your logs. Indeed you are seeing a different issue that others who have image loading problems. Our team is looking into it.

@AMP Thanks Andrew. I’m away for a couple of days if you need anything from me. It would be nice to find out what the problem is.

Hi @John_Steer1 ,

We have reviewed your logs further and it looks like the issues happen in batches, when there is an image error, it looks like network requests overall start to have issues.

Can you please provide some more details as to your network setup? What is the model/manufacturer of all your networking gear? How is your ROCK connected to the network?

Do you live in an area prone to network issues, in a rural area for example? Is your router up to date on firmware versions?

Hi @noris Thanks for looking.
I have a Ubiquity Amplifi router. The ROCK intel NUC is connected directly to the router via Cat6 cable. The firmware is all up to date. The internet connection is fairly solid. I’m in a metro area and internet is via a fixed wireless connection direct to my ISP at at 120/30MB with very few dropouts.

Recently whilst trying to diagnose the problem I even reloaded the ROCK and restored the Roon backup. I rebooted it yesterday and it was fine all day but the problem is back this morning. Doesnt seem to last much longer than 24hours before the problem reappears. The issue only seemed to present around a couple of 2 or 3 releases ago around the time you thought you had solved a lot of the extraneous DNS calls?? I have tried using my ISP DNS on the ROCK server and also Google DNS and currently 1.1.1.1 all seem to roughly the same.

Interestingly just restarting the Roon software on the ROCK doesnt necessarily solve the problem but a reboot of the ROCK does. It appears that something in the ROCK OS is going astray… maybe… just my general feeling. The Get calls that are failing can still be run manually on other pc’s on the network in a browser and still work fine.

If you need anything else just let me know.

Thanks
John.

1 Like

Hello @John_Steer1 ,

Thank you for those additional details and your patience here, I’ve consulted with the team regarding your case.

It is interesting that you have to do a full reboot. Looking over your Roon logs it looks like just the images are impacted, right after the image fails, your Core is able to contact our other servers, so this looks like it’s impacting only the images. We have a few troubleshooting steps we suggest next:

  1. Try a different cable or port on the router if you haven’t yet, it would be very strange if this was the issue, but good to check nonetheless.

  2. Try to verify if temporarily hosting the Roon Core on another PC also has the same issue:

  • Create a Backup of your current database
  • Open Roon on the other PC you wish to try as the Core
  • Roon Settings → General
  • Disconnect
  • On the “Choose your Core” screen, press “Use this PC”
  • If asked to Unauthorize, you can go ahead and do so. You are limited to one active Roon Core at a time but you are free to switch between them as often as you’d like.
  • Verify if the same behavior occurs on the different PC
  1. Do you run these GET requests at the same time as ROCK is failing? If you try the GET requests after a while that ROCK fails, it would be better to test them side-by-side, in case there was a temporary network issue but then GET requests work later in the day.

  2. If you are able to let us know when the issue happens and can leave ROCK in that state for a while, QA may be able to gather more info surrounding the issue via remote diagnostics.

1 Like

Hi @noris ,

Thanks for the suggestions. I’ll work through them one by one and see how we go. I have noticed that apart from the images also the streaming from Qobuz also stops working. It can see the album but when you hit play on a track it just sits there trying and the log file gets these error messages:
5/16 23:16:58 Debug: [easyhttp] [3107] GET to https://streaming-qobuz-std.akamaized.net/file?uid=1677402&eid=148979851&fmt=7&profile=raw&app_id=188245549&cid=1192784&etsp=1652744641&hmac=4blxITceLyOLluqQg-ameqgDIfw canceled after 5043 ms
05/16 23:16:58 Debug: FTMSI-B-OE qo/4C3227A6 interrupted req 393; missing block 0
05/16 23:16:58 Debug: FTMSI-B-OE qo/4C3227A6 rid:393 request ended – first block: 0 blocks read: 0 download speed: -256kbps response time: -1ms
05/16 23:16:58 Debug: FTMSI-B-OE qo/4C3227A6 created new req 394 for block 0 p 0; active requests 1
05/16 23:17:00 Warn: FTMSI-B qo/B2986E2C: block downloader too much time locked
05/16 23:17:00 Warn: FTMSI-B qo/2446B7EA: block downloader too much time locked
05/16 23:17:03 Warn: FTMSI-B qo/AB6DB8DD: block downloader too much time locked

If I drop the GET statement into a browser on another pc on the network it streams ok. Same as the image Get statements. Interestingly Tidal continues to stream ok.

First off though I have just changed the cable and the port on the router. So its up and going at the moment so 24 hours should tell if its going to fail again. I’ll let you know how it goes.
Hosting on another PC is a bit of an issue due to the size of my local library, it takes forever to get up and running on my laptop with 122,000 local tracks.
One step at a time :slight_smile:
Regards
John.

1 Like

Hi @noris ,
Just a quick update. Problem is back this morning after running 23 hours.
Yesterday I changed the cat6 cable and also changed the port on the router and rebooted the router. I have also updated to build 943.

@Rugby also queried the specs of the NUC which havent been included in this thread so just for your interest its an Intel NUC Gen 8 i5 with 8GB RAM. Purchased on 17/7/20 and has been running ROCK seamlessly for the last 18 months.

I’ll leave it in its current state for the moment so the guys can try doing some remote diagnostics.

Thanks,
John.

John.

1 Like

Hi @John_Steer1 ,

Thanks for the update here, we were able to take a look over the system today and discovered an additional data point.

Even thought your ROCK logs are showing signs of the image timeouts, it looks like RoonOS (the operating system itself) is still able to access the image ok, the issue appears to be localized to when RoonServer is making that request.

We are going to discuss this further internally, but in the meantime it would be good to rule out if the database is related. Can you please temporarily try to use a fresh Roon database and a small subset of your usual content for however long it takes to see this issue? You can set up a fresh database with these instructions:

  • Stop RoonServer from running in ROCK’s WebUI
  • Navigate to your ROCK’s Database Location
  • Find the folder that says “RoonServer”
  • Rename the “RoonServer” folder to “RoonServer_old”
  • Restart the RoonServer in the WebUI to generate a new Roon database folder
  • On the Roon Remotes, press “Use another Core” and connect to the new database
  • Verify if issue persists

Thanks!

Hi @noris ,
Cool. Good to know that its maybe not hardware, network related. I have just set up a new database and maybe a quarter of the library and its currently importing tracks. I have added both Qobuz and Tidal as well. I’ll leave it running and see how it goes.
I’ll get back with more info as it comes to hand.

Regards,
John Steer.

Hi @noris ,
Well that didn’t take long. All seemed to be going fine whilst the tracks were importing and being analyzed. Once that finished though I noticed that on the Home page the artwork for Qobuz and Tidal where missing. Trying to play Qobuz it just hangs again but Tidal is still playing just missing artwork as before. The artwork for the local tracks hasn’t disappeared as yet. There were a couple of albums that didn’t pick up embedded album artwork, but I added them as a cover.jpg in the album directory and it picked them up ok.
There is currently 14,791 tracks so it appears that the problem isn’t the database or the size of the library. Good to know at least.
I’ll leave it in its current state so you can check the logs if you wish.
Thanks,
John Steer.