Database corrupted - works for days then app fails to connect

Core Machine (Operating system/System info/Roon build number)
NUC10i7fnh1
OS: Version 1.0 (build 227) stable
Roon Server: Version 1.7 (build 667)

Network Details (Including networking gear model/manufacturer and if on WiFi/Ethernet)

Netgear GS105 - Ethernet

Audio Devices (Specify what device you’re using and its connection type - USB/HDMI/etc.)

Multiple - Bel Canto 3.7 - USB

Description Of Issue

I retired my NUC7i5BNH Rock server and migrated to a new NUC10i7fnh1 recently. I was able to restore a backup and move all music files to the new internal 4TB SSD and no longer use the external usb drive. All was working fine until I shut down (via “power off” in browser). I entered the bios to view cpu temps and exited out of the bios without saving. When the Rock came back up I could see it on the app (where it shows the Core Kit and a Connect button) and see it was running via the browser but I could not connect via the Roon app. It would not get past the spinning Roon logo. Not from a PC, not the iphone, and not the iPad app. I restarted multiple times, restart on the Roon server software, Reinstalled the operating system with no change. Finally, I restored the backup from the original Rock server and I could get into the Roon app again fine. If this was a one time thing I would think nothing of it but this has happened 4 times now!

The last time it had not been powered off at all and had been running a couple of days. This time the Rock server was working hard you could tell because it had gotten a lot warmer than usual. I shut it down to let things cool off and tried again. App would not connect so I wiped out the database again and restored from a 2 day old backup. This time I still could not connect to the database! SO I reset the database again but this time I didn’t restore. I went ahead and logged in and let Roon startup like it was new again. So now it is searching all my albums again. I had a ton of settings and album fixes and DSP settings! Is there anyway I can restore a backup and clean it up somehow so I don’t lose all my work? Having to analyze almost 45,000 files takes a while too. Setting up backups again… services… fixing split albums… combining box sets correctly… ugh… don’t make me do this again!

I have the logs saved from the last time it crashed if that would be any help.
My only consolation is that at least I am back to listening again.

Hello @Muel, could you send me those logs using dropbox so I can take a look?

I messaged the link to you.
It has been working all week with the fresh database but I just let it update to the latest build and now I cannot access via the app anymore.

I can browse to the Core via another computer and see that the Roon Server Software starts and runs for about 20-21 seconds before it says “Not Running” and says Running 0 minutes, 2 seconds… and if I refresh the screen multiple times I can see it count up to about 20 seconds before says not running again and starts counting seconds again. The NUC starts getting warm after doing this for a bit and of course, I can’t get to the Core from the Roon App on any device.

Still waiting for a reply… don’t expect any over the holiday but would like some helpful comments when available. I would really like to rebuild my original database if possible. @support @nuwriy
I’ve wiped out the database started over… it’s been stable for a week so far but I’d really like to get my database back! I was backing up to 3 locations so I have plenty of backups

Sorry for the delay here, @Muel! Our team’s queue is a bit longer than typical at the moment, but we have a ticket open with our QA team and we are hoping to get feedback from them soon, and we’ll be back to you right away with next steps. Apologies for the continued trouble!

Hello @Muel, I’d like to collect additional diagnostics from your system to add to the QA team’s investigation. Could you please reboot your core and reproduce the issue one more time so I can enable the additional diagnostic? Please let me know when you’ve had a chance to do this!

Hey again @Muel, I got an update from the QA team on this. We noticed some tracks that might be causing the issue, can you remove the following and let me know if things change?

12/16 18:55:08 Trace: [analysis] analyzing trackid=3745842 url=/roon/sys/storage/mounts/InternalStorage/FLAC/Dido/No Angel/06 - Thank You.mp3
12/16 18:55:08 Trace: [analysis] analyzing trackid=2422834 url=/roon/sys/storage/mounts/InternalStorage/PH/1/All music/01 Oh Happy Day.m4a

OK… they are removed. Were they looking at diagnostics/logs today perhaps? …or something that would cause possible interruptions? I’ve had a few pauses that were unexpected.

Hello @Muel, the team identified those files as causing an issue, and wanted me to ask you to remove them. If that helped us, they’d like to collect a copy of those files from you so we can test with them and make Roon better. Please let me know if removing the files has halted the crashes!

So it is failing again and this is with the new database that I let it build recently. The server software keeps restarting every 23 seconds or so. I cannot access ROON from any device now.
Roon Server Software
OK
Version 1.7 (build 710) stable
Running 0 minutes, 7 seconds.
If I refresh the screen I can see that it keeps restarting over and over.

I managed to capture the log files and sent the link to @nuwriy.
So the database created by a fresh install still blows up and the server software continues to restart over and over. This is not very encouraging! At least it ran for a few weeks instead of just a day or two after restoring the original database. It was working for the last few weeks still using the fresh database and was fine last night. I will have to turn off the core because it starts to get warm from restarting over and over a few times per minute. Your help is appreciated! @Bocatech

edit - So I’ve activated another Roon Rock server… I was without music and I’ve spent enough days like that! This was the old ROCK NUC that I retired back in November. I don’t have all the music on the old ROCK since the data drive is only 2 TB. I am waiting for instructions for next steps on my problem with the new ROCK server.

Hello @Muel, I wanted to confirm that with those files removed from any watched folder you’re experiencing crashes every 23 seconds?

Yes, I removed the files you listed.
Also, I ran Memtest86 with no errors on the NUC.

Hello @Muel, I have gone ahead and enabled diagnostics mode for your account and what this action will do is next time your Core is active, a set of logs will automatically be generated and uploaded to our servers for analysis. From there I’ll send the new report over to the QA team for further review.

Currently I am using my old core and not the new one where I’ve been seeing the problem. On the old core I have around 1.2TB less of my music loaded because I have less space on my data drive available. I’m not sure if diagnostics on the old will give any helpful information. I can reload ROON from scratch a 3rd time on the new core and get it running again until it crashes. If you think music files could be causing a problem then that is what I would need to do to get them all available in the database.
Should I go ahead and move to the new core again? Were there no clues from the logs I was able to capture after the server started crashing every 23 seconds a week ago… or does the diagnostics mode collect more information than the normal logs?
Thank you

Well, that didn’t take long! I guess we have verified that the problem is definitely NOT the new Core! The old core is having the same problem. On the old Core the ROON Server Software runs about 29 seconds and then restarts over and over again. I hope the diagnostics shows the problem and we can have a solution soon! This has been driving me crazy for over a month!

Hello @Muel, and thanks for testing that! I have since received diagnostics and have sent them over to the QA team for a look.

So, @Bocatech and @nuwriy, I didn’t want to be without music another night so last night I reloaded Roon server software on the new core, reset the database and restored a backup. The latest two backups failed to restore but the 3rd newest one restored OK. I was able to use the system last night for a few hours and shutdown the core when I was done. When turning on the Core just now I found that the server software is restarting over and over again. This is really frustrating! There should be plenty of opportunity to collect any diagnostic data that would be available since this is happening so quickly now. I think we have eliminated any hardware problems… 2 different NUCs running the core, new cables, new switches… your help is appreciated and needed asap!

@Bocatech and @nuwriy, Do you have any help for me?
I have 2 cores that are both crashed. I reinstall the server software, restore the database or let build from scratch and it crashes againI This is on totally separate hardware so it can only be OS or Software related. You had me remove a couple of music files you thought might be an issue but that didn’t help. I have not added any new music files for many months but there has been a lot of Tidal streaming.

This has been going on for over a month… I’m missing the music!