Ah, so you did give my suggestion some thought…the one about fixing the problem causing the need for hard power cycle…even though you insulted it when I gave it.
So, I have done a hard power cycle of a Roon Core running on linux many times and the backups were just fine. I have done the same on macOS running Roon. Never have I had issues with the database.
So why do you think this is a problem with Roon instead of Synology? Maybe Synology are doing something with the caching of written data that allows Roon to think the backups are done and data written when, in fact, that is not the case. Maybe the writes were cached by the Synology OS and never actually written to the storage media.
Look @musicjunkie917, you don’t know what is going on. And I’m going to guess you probably know less about “Synology OS” which is Linux than I do. I was literally running and hacking on the Linux kernel in the early 90’s before the 1.0 kernel was released. Not saying I’m the most knowledgeable person by any means, but I definitely have more information about what is going on than you do.
Which is to say, your suggestions to a question I did not ask is not helpful.
For the record, Synology support claimed this was caused by 3rd party RAM. So I went out and spent over $700 on the official Synology branded RAM which has the exact same specs as what I had installed. Guess what… it happened again and I’m continuing to work with them to see if they can fix the issue.
In the mean time, it would be great if Roon could be as reliable as the other software I have running on my Synology- software with various databases like PostgreSQL and SQLite which don’t experience any ill effects when this happens.
I have been hacking on various systems and operating systems since the 70’s. I have worked in the computer industry for over 40 years. I started cracking copy protection schemes on Apple II’s and went on to developing software in assembly language and a variety of higher level languages. I worked at Apple for a decade. I ran my own company with hundreds of servers running various Unix-based operation systems in a data center. I’ve run storage pools using a variety of RAID and ZFS implementations. Now that we have our résumés (or CVs) out of the way…
Do you know more about Synology than the folks at Synology? Are they saying the problem is with Roon or is the problem the hard crashes and restarts? I’ll tell you as a data center professional, Synology was not held in high regard in that market segment.
Is this type of problem happening with Roon on all Linux-based systems? Or just the Synology systems? Does it happen on Windows or macOS-based systems? If not, why is Roon reliable on these systems but not Synology systems? What’s different?
Following up here, we have just released Roon Hotfix build 1232 which may help with this behavior. Please try to update your Roon Core to the latest version and let us know if it helps resolve the issue on your end, thanks!
Just a quick update… I haven’t experienced the NAS problem since upgrading. Honestly, no idea why. ¯_(ツ)_/¯. So haven’t been able to test the fix provided in build 1232.
I can just say I’m still working with Synology support and I’ve had the issue twice in the last month and both times I ended up having to replace the empty files in the backup directories before Roon would detect the previous backups. So sadly, whatever fix was put in place didn’t work.
If at all possible, sharing a more specific date and time around when the issue occurs would be helpful in regards to our team reviewing your core diagnostic.
With that, can you confirm that you’re not using a VPN with your current setup?
Should have backed up Saturday (the 15th) morning at 2am, but didn’t. Later that morning I noticed the system was behaving badly and had to do a forced-power off. Up until I powered off, it was semi-operational.
Why it didn’t do a backup on the 14th??? I’m not sure I can say. Was the system showing some issues on the 14th and I didn’t notice? That is possible.
I should also mention that after the system crashes/forced power off, Roon ends up in an odd state. Not only are those backup files truncated (always) but sometimes it seems like the Roon DB is corrupted. Things like my audio endpoints are often missing. Backup schedule is always missing. Audio track storage/history is often wrong. Stuff like that. So I always end up having to restore from the last backup.
Definitely the last few backups were corrupted but I was able to successfully restore the 7/13 backup? At least as far as I can tell… nothing in the UI indicates any issues. I have local backups for 30 days (6/17)… how would I know if a backup is good?
A few dozen skipped files. most due to “tag extraction failed”
So Synology just found something interesting which might explain everything. Just going to copy & paste their message:
Hello,
Our developers took a closer look, and they observed that the NAS had exhausted its inotify resources:
tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling
Inotify watches is an API that the Linux kernel uses to notify applications of filesystem events without forcing the program to physically read the file to check for changes, dramatically reducing I/O load. The drawback is that inotify watches need to be stored in RAM, so to avoid causing out-of-memory conditions the kernel places a limit on the number of inotify watches that can be registered. If the inotify watch limit is reached, it can cause a wide variety of unexpected behaviors, including DSM not recording logs properly, and could also potentially be a contributing factor to containers spawning runaway runc init processes, as well as the other issues you’re observing.
With that in mind, we need to address this issue before anything else. While it’s not 100% certain that this is the root cause of your problem, the exhausted inotify resources are preventing DSM from properly recording logs, blocking further investigation.
Our developers looked into this, and it appears that the Roon container is primarily responsible for the inotify exhaustion:
Thank you.
