Lengthy Recurring re-scan of main library since build 262 & 269 [Resolved]

Dear @support:
Once again I am experiencing aberrant (or so it seems) behavior with the latest roon build. Since becoming a life member early on, roon has in previous builds only scanned my main library after I added new albums/tracks which are 95% AIFF. And the spinning circle in the upper right column only needed to spend a relatively short time “adding” the new music.
Several weeks ago with build 262, I discovered 6700+ files in my clean up section. The number of albums dropped gradually by one thousand and the tracks by close to 30K which it is doing presently.
My main library which I add to from time to time is essentially no different than it has been in the early years with roon builds, except to increase in size.
The spinning circle which I gather represents roon scanning my library again which it did those several weeks ago and I don’t know what triggered the long scan resulted in putting the numbers back up to where they had been and the clean library section was reduced from 6700+ to 10 files.
Only recently did I add several (ten or twelve new albums) and roon which watches only my main music library stored on a Promise Pegasus 18TB R6 Raid 5 Thunderbolt HD I have had for years and is stable. I regularly maintain a backup of my database using roon’s backup operations but do it manually.
Once again, roon is behaving in a very similar way, and I can’t understand what has triggered the program which had been so stable to now repeat a highly lengthy scan given only a small change in added albums.
Albums are doubling with metadata supplied by roon’s program and a first one with my naming the album. I am presently 40K tracks and climbing less than I should have and down a thousand albums and the scan of my one watched folder my main library is taking hours. I don’t know what triggered the program to become troublesome when for so long it was stable with hardly any issues. This is the second incident where a lengthy scan is taking place. And it’s both interfering with my enjoyment of the program’s previously exemplary behavior and interfering with my enjoyment of my music library while I wait for a scan that is taking hours.
I would greatly appreciate discovering what the impediment or trigger is and fix it.
Last time the scan eventually terminated and my album and track numbers returned. So I left it at that. But a second incident is just unacceptable. So i would greatly appreciate being the recepient of the excellent support furnished to others over the years.
PLEASE HELP ME SORT OUT THIS PROBLEM!!!
Best regards
Richard

@support with no space…

Dear @support
The scanning referred to above has stopped. I now have 8520 files for deletion in my Clean up section. My album and track numbers are reduced by thousands. Of course, I closed roon and replied. Apparently my first post has @ support with a space. I hope support sees this please I believe it’s too late to edit the topic in the Support section starting with Lengthy Scan etc.
I am terribly concerned about this circumstance and would greatly appreciate the attention this condition requires.
Thank you.
Best,
Richard

Thank you for the help. I hope support finds it as it is too late to edit.
Best,
Richard

1 Like

What would you like it to say? Looks like I can edit the title of the thread…
They have been flagged by adding the @ support without the space. :grinning:

Sorry for my inexactness. I meant to correct the spacing issue with @ support so the support team is made aware of my problem. The rest of the post wordy as I made is remains the same.
Again, thank you for your attention and for calling my error to my attention.
Best,
Richard

Dear @support,
I hope my topic which incorrectly refers to signaling you as @ support with a space which Bob called my error to my attention. In any case, I will appreciate getting help with 8520 files in my clean up section, which don’t belong there as my album count used to be 8700+ and tracks 112K now my albums are at 4619 and tracks are at 57760 and snapshottingh Databases for Backup has appeared on its own twice in the last 20 minutes. I don’t even recognize the manner of operations roon is performing. I really could use sorting this out a restoring my library. I have not deleted anything or even know how to recover the 8520 files in Clean Up library.
Best,
Richard

Hey @REShaman – we’re going to enable some diagnostics on your account, and then we should be able to pull a report with logs. That will give us a clearer sense of what’s happening here, and then we can get this stable for you.

It may take a day for us to full analyze the logs, so thanks in advance for your patience here. @Eric or I will be in touch soon!

Dear Mike,
I always backup manually my main library located on a Promise Pegasus 18TB R6 Raid 5 Thunderbolt drive at P2R618/Music Drive P/iTunes Msuic which contains approx. 8700+ albums and 112K tracks and backup to PR408/Roon Backup Database. There is also on PR408 an abbreviated old main library that I don’t use with roon. It’s located at PR408/Music Drive P/iTunes Music. AND,
That abbreviated library corresponds to what roon is showing me at the number of albums and tracks.
When I looked at roon/build 269/Settings/Backups, I found scheduled backups which appear to be backing up the iTunes Music (library) located on PR408 instead of iTunes Music located on PR2618 which contains the 8700+ albums and 112+ tracks.
I gather then that roon is displaying the abbreviated library as my main library located on the PR408 drive instead of my main music library located on the PR2618 drive. And has re-scan the abbreviated library so I don’t know how to recover my main library located on PR2618 even though that is the watched folder.
Also, I don’t know how to protect those 8520 files in the Clean Up section. I have lost control over roon and my main music library. I have no idea how it got this way or how to restore my main library.
Also, I don’t know how to cancel the scheduled backups which if I may have scheduled to protect my main library on PR2618 because now there are three scheduled backups.
I should have stuck to the manual backups from PR2618 to PR408/Roon/Roon Backup Database.
Totally confused.
Best,
Richard

Dear Mike,
Thank you. I will have to be patient as I do not wish to cause any damage to my library. I may have replaced my main library with the abbreviated library which would explain that reduced number of albums and tracks. I gather that roon build 269 is launching with the PR408 abbreviated library and not the full main library located on PR2618/Music P/iTunes Music.
When backups were accomplished by selected the directory and going to the Library and finding the roon backup folder, I was not having these issues.
I gather I have confused the program into displaying the wrong library. And I will not go near the Clean Up library of those 8520 files that don’t belong there. Last time this happened, roon re-scanned and recover the missing albums and tracks, but this time the numbers stayed reduced.
Sorry for repeating myself ad nauseam as I am so concerned about losing music that can not be replaced when so many are downloads. Hence my behavior.
Best,
Richard

I wouldn’t worry too much. I took a quick look at the diagnostics, and it looks like we’re getting hung up on some kind of image file somewhere in your library.

Once we get the information @Eric requested, we’ll get some feedback from the developers and then we should be able to identify the file that’s causing this unexpected behavior.

To be clear, there shouldn’t be any risk of lost data here, whether in Roon or in your media library. Just sit tight and get Eric that information he requested, and we’ll be on our way. Fear not! :slight_smile:

Dear Mike,
Thank you that’s good to know. I’ll just let the process take its course and make the necessary cures once I am advised as to what needs to be done.
With appreciation,
Richard

Hi @REShaman ---- Thank you for you feedback and more importantly, thank you for your patience.

I wanted to touch base with you to see if you had a chance to review the PM I sent you. The PM can be found by clicking the the link @mike was referencing in his post: “Requested”

-Eric

Dear Eric,
Very sorry to have discovered your missive after leaving for the City (New York). When I return home will comply with the Request. Sorry for any delays my schedule today may be causing.
With appreciation for the excellent support,
Richard

No apologies necessary, @REShaman :wink: The support team will be on the look out for your follow up!

-Eric

Hi @REShaman ---- Thank you for you patience here why we have been discussing what the next steps should be in addressing this behavior you’ve reported to us with our techs.

As per our discussion, our tech team has asked if you could please provide the following:

  1. A screenshot of your “storage” tab in Roon.

  2. A description of your iTunes, meaning…

  • Where does iTunes run?

    • If I am not mistaken you are running a MacMini as your core, is this device also hosting iTunes, please verify.
  • Where is the XML that iTunes is generating located?

    • Please verify.
  • Does the XML update when you update iTunes (like creating a new playlist, etc)?

    • I believe you may have confirmed this already in one of your PMs to me, but I want to be certain how changes to your iTunes library are being picked up Roon.
  • The two iTunes XMLs you uploaded to our servers, where are they stored?

    • Please verify.

-Eric

Dear Eric,
Before I respond directly to you queries, allow me to update you and the support staff. I have decided to make some changes that may or may not be pertinent to the issues I have reported. And/or may cure some of what might be triggering roon, and may be useful in resolving recent problems. Let me explain:

  1. I searched my Mac Mini and all external HDs Thunderbolt Raid 5, external SSD, Thunderbolt XTB single drives (i.e., Seagate, 3TB for example for Pure Music’s pmbmf files which Rob of Pure Music required for playing DSD-DSF files several years ago and which may be triggering problems especially with High Sierra Supplemental. I equivocate because in the past my present experiences were not apparent in earlier builds of roon. And I have no intentions of disparaging Pure Music’s methods for playing DSD-DSF if those pmbmf are still required. In any case, I have removed all of those files.
  2. I have decided to disable the iTunes Library which has been enabled and appears under the heading in bold in Settings/Storage/ below the title 1.2 and older builds of roon. I did so because ALL albums/files reside in my main music library on the external 18TB Raid 5 HD. So there’s no real reason to also watch the iTunes Library, i.e., the library that the Apple program iTunes is comprised of and where every album save those formats, i.e., DSD-DSF and formats that iTunes does not work appear given the main music library contains every album/track/file, hence what reason to require roon to also watch the iTunes Library. I really don’t know if that is actually a different folder to watch that matters for roon to include all albums or whether it essentially merely duplicates those albums already in the main music library located on the 18TB Raid 5 Thunderbolt HD

Consequently, my album count has been reduced considerably as well as the track count. And I gather that means albums are not being counted twice. What the reason that those thousands of files ended up in the Clean Up section confuses me.

Now to answer you questions.

  1. I will provide a screenshot of my storage tab.
  2. I believe the answer to your question re iTunes is yes, meaning the iTunes program resides on my Mac Mini as the core but the library files are located on the external 18TB Raid 5 Thunderbolt HD at PR2618 (Promise Pegasus, 2nd edition) 18TB R6 (6 HD drives of 3TB each) Raid 5 at Music Drive P/iTunes Music (folder). ITunes Music folder is the main directory which stores some 4000+ albums added to this directory AND for formats that can be played in the iTunes program is then added to the iTunes program using the Add To in the Menu/File of iTunes. With roon, I no longer use iTunes for critical listening as well as formats iTunes is not enabled to playback. AND,
    Yes, the iTunesLibrary.xml and iTunes Music Library.xml are located at PR2618/Music Drive P (Given there are two xml files does that mean I should re-enable the iTunes Library in addition to the PR2618/Music Drive P/iTunes Music which contains ALL my music albums in various folders.
    I hope the following is a reply that is acceptable to you re update XML and roon etc: If I add an album or track to my PR2618/Music Drive P/iTunes Music folder, roon watches this folder and when launched the spinning circle in the upper right corner indicates it detects a change to the watch folder and adds to roon display. When I also watched iTunes Library, adding to the iTunes program (using Add to Library command) roon saw the addition to the iTunes program and this appeared in the roon display. If I only add an album to the PR2618/Music Drive P/iTunes Music and not “Add To Library” within the iTunes program, iTunes when launched does not show the album as present in my library. Hence the reason I concluded that it was a waste of roon’s operations to have it watch both my Music Drive P/iTunes Music folder and the iTunes Library which as you will see from the screen shot I just disabled.
    Again, the t two XMLs I uploaded are stored at PR2618/Music Drive P.
    I am doing my best to help you help me.
    Now that I have disabled Settings/Storage/iTunes Library, in roon my albums are reduced from 8700+ albums and 112K tracks to 4621 albums and 57771 tracks. The Clean Up has 11 files in it. So apparently I misjudged the actual size of my library thinking I had 8700+ albums when the difference between the two numbers is probably roon counting albums in both locations. Or do I have this wrong?

I hope I have answered your inquiries satisfactorily.

Best,
Richard

Hi @REShaman ----- Thank you for the follow up and taking the time to answer my questions. The insight is very appreciated and my apologies for the wait, I have been out of the office for a few days and I am getting caught up on all of follow ups.

Moving forward, based on your latest this all sounds reasonable to me. After this re-adjustment:

  • Is the rescan still occurring?

  • Does all you music import as you expect it to?

  • Are your playlists present?

Furthermore, I must say, great work with disabling the “old” iTunes + Roon integration method :thumbsup: When we released 1.3 it was left in for convenience purposes but I believe that your experience will be more stable functioning this way.

-Eric

Dear Eric,
Thank you for responding to my typically long posts. Sorry, as I am trying to clarify so there will be as little confusion as possible which may be mostly on my part given that there is probably a preferred methodology for working with roon that supports its architecture, design and intention in the best way. And with other programs, which I own (most of them save JR Music) it’s a matter of choosing a location for the main music library which as my screen shot illustrates I have defaulted to. The resultant calibration of numbers of albums and tracks are reduced reliably and reflected by the same numbers in other programs without duplications of the same album with only partial tracks requiring either mending (amending, partial pun intended) or hiding albums.
So to answer your queries, the following is accurate:

  1. No, there are no rescans still occurring other than when actually adding new albums. And the rescans are for short periods typical when adding a small quantity of new music as opposed to the lengthy rescanning and the thousands of files listed in the Clean Up section. That is over with and now only eleven (11) files reside in that Clean section which I have not cleaned (yet).
  2. So far, the last addition of a new download from HDtracks.com was added correctly and in short time.
  3. Perhaps not typical of other users, I don’t employ Playlists, other than those “recommended” and made available through TIDAL HiFi as I listen to whole albums.
  4. Your conclusions are borne out by my own experience and a willingness to change old behaviors that accompanied earlier editions of roon. The stability I sought by thinking I was being thorough by watching the iTunes Library has improved by doing the counter intuitive, i.e., NOT watching the iTunes Library any further as it is duplicative and is/was probably clashing with the methods employed by roon.

So the old adage, “Less is more”, is apt in my experience with roon though not in numbers of albums/tracks but demonstrated by a stable, reliable operation of roon and by counting the correct number of albums and tracks and ending the forced and induced errors by adding a redundancy that serves no advantage.

Sorry to have taken up so much of your time and other Support Staff’s time. And thank you for helping me confront my practices in error that are reformed. Build 1.3 and its progeny while allowing us to add to for convenience for some of us have caused the addition of an unstable recokoning of two libraries when only one is needed.

With appreciatioan for the exemplary support practically unmatched by any other support staff anywhere, though honorable mention goes to certain other innovators not connected with roon.

Best,
Richard

1 Like

Hi @REShaman ----- Thanks for the follow up, and no apologies necessary at all :wink: Your feedback has been appreciated!

Glad to hear things are back in stable state, happy listening!

-Eric