Unfortunately my Roon is still crashing on the new QTS.
I managed to catch it in a crash today. The process has gone crazy: see the attached screenshot of top running on the server. Given this is an 8 core NAS, the process has taken up an entire core and 8GB of memory, and is just chewing it over.
I’ve also discovered that if I leave it for about 5 minutes, it recovers without the need for a restart.
I looked at @Jamie_Tudors last log files. Some contain only one error message:
11/05 08:56:08 Critical: scx: in OnExit: System.NullReferenceException: Object reference not set to an instance of an object
at Sooloos.Broker.RoonApi.TransportService_2.EndpointToJson (Sooloos.Broker.Api.Endpoint ep) [0x00028] in <c3f83e578441427188bf8c55d2a55639>:0
at Sooloos.Broker.RoonApi.TransportService_2.<ZoneToJson>b__37_3 (Sooloos.Broker.Api.Endpoint x) [0x00000] in <c3f83e578441427188bf8c55d2a55639>:0
at System.Linq.Enumerable+SelectListIterator`2[TSource,TResult].MoveNext () [0x00048] in <ae162b7061064bfaa55021254699ac67>:0
at System.Collections.Generic.List`1[T]..ctor (System.Collections.Generic.IEnumerable`1[T] collection) [0x00077] in <370a0c27f4b74d1a81431037df6d75bf>:0
at Base.JList..ctor (System.Collections.Generic.IEnumerable`1[T] l) [0x00000] in <e6fb1b8f00f2413595e2298124e24ad2>:0
at Sooloos.Broker.RoonApi.TransportService_2.ZoneToJson (Sooloos.Broker.Api.Zone zone, System.Boolean for_diff) [0x00102] in <c3f83e578441427188bf8c55d2a55639>:0
at Sooloos.Broker.RoonApi.TransportService_2._ActuallyChanged (Sooloos.Broker.Api.Zone zone) [0x00000] in <c3f83e578441427188bf8c55d2a55639>:0
at Sooloos.Broker.RoonApi.TransportService_2.<_update_subscriptions>b__25_0 (Sooloos.Broker.Api.Zone x) [0x00000] in <c3f83e578441427188bf8c55d2a55639>:0
at System.Linq.Enumerable+WhereEnumerableIterator`1[TSource].ToList () [0x0001b] in <ae162b7061064bfaa55021254699ac67>:0
at System.Linq.Enumerable.ToList[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x0001f] in <ae162b7061064bfaa55021254699ac67>:0
at Sooloos.Broker.RoonApi.TransportService_2._update_subscriptions () [0x000d1] in <c3f83e578441427188bf8c55d2a55639>:0
at Sooloos.Broker.RoonApi.TransportService_2.OnThreadExit () [0x00034] in <c3f83e578441427188bf8c55d2a55639>:0
at Sooloos.Broker.RoonApi.Module.ev_exit () [0x0000b] in <c3f83e578441427188bf8c55d2a55639>:0
at Sooloos.SynchronizationContextThread.OnExit () [0x0000a] in <25d73f18cc0f460fa0e05f69a9433340>:0
As I do not see something that is specific to QNAP yet, maybe @support can have a look.
Hi @Jamie_Tudor,
Thanks for reaching out to us, I have merged your two threads so that we can keep all the information in one place.
I am looking in the logs and I do see the errors that @crieke mentioned, but I am not seeing what triggered it initially since this has been overwritten in the logs (logs automatically overwrite new info when they reach a certain file size).
I do see quite a few Roon API traces, are you by any chance using a Roon Display or streaming to a Chromecast zone right before this issue occurs? Does this behavior occur when there is nothing playing to the QNAP?
Can you provide some more information regarding your network setup? Info such as the model/manufacturer of your router, switch & access points might be helpful.
I will also inquire with the technical team to see if there is more context that they can provide with regard to the output trace.
Hi there
I am not using a Roon Display and I am not using a Chromecast zone. There are a couple of Chromecasts in the house but I have never used them as an endpoint.
The behaviour always occurs when something is playing and most often at the end of a song.
I’m using the Unifi 48 Port PoE Switch which the QNAP is plugged into. The endpoints are all also plugged directly into that switch, and so are the wireless access points (Unifi UAP Pro) which the clients connect to. It’s a flat network - no VLANs or any other esoteric configuration.
The clients are mainly iOS based, but I also use a Windows client. The crashes can happen whichever client I am using at the time.
Also - this has worked perfectly since I started using Roon in March. The crashes only started happening about a month ago. There has been no change in hardware.
I have a full log dump that I can provide for you if that helps which should give you the historic logs. It’s a large file though, so I’ll need to place it on a website and send you the URL privately: let me know how I can do that if it would help.
Cheers
Jamie
Hi @Jamie_Tudor,
Thanks for confirming the above info. No need to send me the log package, @crieke has already sent it to us.
Since you don’t have any Chromecast or Display Zones, I believe the API traces are related to the Ropieee zones you have on the network and it looks like your logs are being filled with this recurring error.
Can I please ask you to temporarily disable both Ropieee zones in Roon Settings -> Audio) and leave only your BiFrost zone active and check to see if you are able to reproduce this issue?
Oh dear - that means I’ll have no Roon for the whole house for a couple of weeks: the Ropieee devices feed 12 rooms via a wired multiroom system: not sure the family will like that too much…
I’ll give it a go but if there’s any other thoughts about how to narrow it down that would be helpful.
P.S. One thing I did do about a month ago was enable the Spotify Connect component of RopieeeXL. I can’t remember if that was around the same time the crash started happening.
I’m going to switch that off and see if that improves things any.
Hi @Jamie_Tudor,
Checking the behavior without the Ropieee zones would be useful, but another test that we can perform here is checking to see if this behavior carries over on another Roon Core.
Do you perhaps have another PC around the house which can temporarily host the Roon Core? I would be interested to see if this behavior reproduces without the QNAP in the equation.
I’ve just had another crash this morning, so this wasn’t related to the changes I made on the RopieeeXL client to enable Spotify.
I don’t have another free computer to host the Roon core, apart from a Windows/Linux VM which I could create using the Virtualisation on the QNAP. I’m not sure this results in the removal of the QNAP that you’re after?
Hi @Jamie_Tudor,
Thanks for checking the changes.
Not quite, my hopes here was to eliminate the QNAP completely from the equation.
Since checking on another Core is not possible and temporarily disabling the Ropieee zones is too inconvenient, have you by any chance reinstalled the Roon app yet? I would be curious to know if reinstalling the Roon app to a fresh state still causes this issue to occur.
Note: Please create a Backup of your Roon database before doing this step. I would suggest trying with a fresh database for however long it takes for this issue to typically appear and then try restoring the backup afterwards if things are still stable.
Hi @Jamie_Tudor,
Just one other piece of info which may be helpful - to set up a fresh Roon database, you would need to change the database location in the QNAP WebUI to a different folder. I would suggest trying a fresh database as the next step in troubleshooting this issue. Thanks!
I’ve started by re-installing the server and keeping the same database. I did that yesterday.
If that doesn’t work my next step will be to remove the database then re-install the server from a database backup.
If that doesn’t work then my last step will be to start completely from scratch with a fresh database. This isn’t a final solution even if it works because of all the modifications I have made, unless there is a clever way of retrieving them and applying them to a new database.
I will report back as I move through these steps.
OK, re-installing the application but leaving the database untouched did not work.
I have now trashed the database, re-installed the application, and restored from a database backup. Let’s see how this goes.
OK. Trashing and restoring has also not worked, I’m still seeing the same problem.
If I start fresh with a new database is there any way to re-apply all the changes I’ve made to my current database? It was days and days of work to get my database how I want it, so restarting from scratch without a way to re-apply my changes isn’t really a viable solution.
Cheers
Jamie
You have tried every software remedy you possibly can. At some point you have to consider it may be a hardware issue. The support guys have asked you to temporarily move your core to a computer. You said you don’t have one. Really? Not even a laptop? What are you typing these messages on? The core will run on a laptop. It would be interesting to install a fresh core on a laptop or desktop and see if you still see the crashes. You have to at least consider this.
I know you’ve had a hardware issue related to memory in your QNAP, but your whole QNAP was unstable, whereas with me this issue presents very differently - it’s just Roon that’s unstable. This doesn’t smell in any way as a physical hardware issue. 99% chance this is software, database corruption, or operating system. Given this problem seems unique to me, my gut feeling is database corruption. It might be an issue with QTS, but i would be shocked if this is hardware related and I have a wealth of professional experience in this.
I’m going to create a Ubuntu VM on the QNAP and run it on there. Whilst this doesn’t take the QNAP out of the equation entirely, it does take QTS and the QNAP Roon package out of the equation.
In the meantime it would be helpful to know from @noris if there is any way to re-apply database modifications to a fresh database, even a hacky way.
Hi @Jamie_Tudor,
This sounds like a good next step if you don’t have an additional PC that can run the Roon Core. The best test would be to of course try running Roon on a proper PC that eliminated the QNAP from the equation.
Unfortunately, I am not aware of any methods to extract pieces of the database, it’s a complete restore kind of process. If you believe this to be due to the backup itself, you can try sending me the complete backup and I can ask QA to take a look, but the best data point right now would be if the same issue occurs on another PC.
OK, well my new Ubuntu VM Roon Server lasted about a day before crashing in the same fashion - so that’s not solved the problem.
I’ll install it on my desktop PC now and see where I get to with that. @noris what part of the Roon Backups directory should I send to you to check? The whole backup directory is ~6GB and I can’t figure out a way to just pick out the latest backup.
Hi @Jamie_Tudor,
Thank you for checking the VM.
Can you send me the entire backup or alternatively your live database by using these instructions? If you can’t upload the entire backup/database you can use an app such as 7-zip to split the archive into multiple segments.
Hi Noris - PM sent with details where to download the database zip from.
Cheers
Jamie