Ridding the database of duplicates

Hello Eric:

You have helped in the past. So you are the likely candidate for some input. Take a look at this post.

Would welcome any further thoughts. Not really making any progress with the existing discussion.

No idea why there are so many duplicates or how to efficiently get rid of them. Also, many albums show with just a few tracks instead of the entire album.

My intention is to clean up the database and then add to it. But do not want to do so until this duplicate issue is under control. Would like to get back to actually listening instead of just doing maintenance.

Thanks
Dan

Hi @Daniel_Greenberg ----- Thank you for reaching out. I have made your PM a public topic in the support thread on the community site so we can address this behavior with you :microscope:

Continuing forward, please verify that you are still making use of the same Roon setup (see below) and furthermore can you verify if your library location has changed at any point recently.

“I use Roon on a Mac Mini running OS 10.105. I am running Roon 1.3 (build 269). The music library is attached via USB external drive to Mac Mini. I am using my iPad Mini 3 and phone iPhone 6 as remotes. I am also running a remote on an office iMac running 10.12.6. (mid-2010)”

Additionally, can you please tell me if you are seeing anything in “library cleanup”?

-Eric

Thanks Eric.

The configuration is the same with this exception. iMac (used as a Roon remote) is now running MacOS 10.13.2. The macMini (Roon Core) is running MacOS 10.13.13. I still use iPad Mini 3 and iPhone as additional remotes.

Running Roon 1.4 build 300.

(As an aside, subscribed to Tidal through Roon. Wow, what a nice surprise in terms of additonal selections for my favorite artists. And the integration with Roon is very nice.)

If I solve the duplicate issues, will be a very happy camper.

Not sure if this applies or not. Any input of what these numbers in the boxes are? See below

D

Where am I looking for “Library Cleanup” to see what you are alluding to?
D

typo above. Mac mini is running High Sierra 10.13.3

Not 10.13.13

D

@Daniel_Greenberg ---- Sorry as I was a touch unclear in my wording. Please see below.

Edited: "Additionally, can you please tell me if you are seeing anything in “clean up library”?

Path: “settings” > “Library” > "Library maintenance

-Eric

ok-thanks

You can see the results of the Library Cleanup.

That is only a portion of the duplicate items. I have been manually removing duplicate albums when they are identical to other albums listed. Hours and hours. Why are there so many (15,000+ more duplicate tracks)?

But there are still lots of albums where there are only a few songs with the remainder appearing on others. Or the track lists and timing do not match up.

Just a thought, but what does your Storage page in the Settings look like? You haven’t, by accident, got an overlap of your storage locations perhaps?

Thanks Geoff got your input.

I have a external drive to core (Mac mini) and another drive off an iMac acting as a remote. Originally (before Roon) these were iTunes libraries.

Tracks are added from both, as one has a DVD player (iMac)

and the other (core) will only accept flash drives to add songs.

Still have lots of duplicates from same Core/external drive.

See here.

Thanks Daniel. I’m not an iTunes user, so I’m not familiar with the intricacies of using it within Roon, but I note that there have been changes since Roon version 1.3. See here. Is it possible that Roon is counting albums twice - once in your directly connected storage locations, and once in your iTunes library?

1 Like

Thank you for the follow up(s) @Daniel_Greenberg, very appreciated! @Geoff_Coupe, you beat me to the punch here, great observation!

Daniel, it appears that you are watching your iTunes library “twice”, please confirm.

-Eric

Gentlemen:

There are two (2) iTunes libraries. One is on the Core machine (USB drive off of it). The other is off the iMac currently (different USB Drive in other part of the house ) used as a Roon remote.

For various reasons, these two locations contain similar but not identical tracks. (One was originally a backup of the other, but over time there were differences. )

So when I set up Roon, I wanted it to watch both as new tracks could be added to one or the other depending on the format and source. So, lets say I downloaded tracks-those would be on a flash drive that would be added to the Core library (Mac Mini). The Core does not have a CD/DVD drive on it. But if I borrowed CDs or they were from my own collection, those would be added on the iMac and its attached library.

My understanding was that Roon was watching each drive and adding new tracks as they were added. When a new addition on one drive were the same ones as already in the database, it would see they were in the library and not add them. Is that a false assumption? Was that the way it was supposed to work?

I am not sure how Roon 1.2 versus Roon 1.3 handles this, and if there are any issues. I read Geoff’s reference. Maybe I am one of those people that had these issues. Maybe that changed when Roon went to 1.3.

I originally thought you set up separate locations for Roon to watch so it could see each library and only add what is new.

Somewhere I must have gone wrong. Appreciate your input as to how to fix this.

Dan

Roon will add any content in your watched folders to your library – it’s not going to prevent you from importing a second copy of a track just because it exists elsewhere (on a compilation, greatest hits, mix cd, etc).

It will identify duplicates in your collection and group them, but everything in the folders Roon is watching will be added to your library.

Mike:

thanks for your comment.

Two questions:

  1. how do I prevent Roon from adding duplicates if they are from the same albums when watching two different, but similar drives (one library and one backup which is not identical)? My assumption (which may not be correct based on your comments) is that Roon should not add identical tracks/albums, but rather just one copy, or alternatively, give you the easy ability to filter out the duplicates. Hypothetically, If I am watching three or four drives, is Roon adding the same tracks over and over?

I do not want to delete “watching a drive” because the other drive is not identical. Sometimes I add albums on one drive or the other depending on the format that is easiest. One drive accepts CD’s, the other downloaded USB files on a flash drive.

  1. how do I get rid of the many duplicates (see my earlier post screenshots) I have in some sort of efficient way? Remember there are thousands of duplicate tracks. When I click the 3 dots to the right of an entry (Album or Track), and I can edit, I select delete album or delete track as the case may be. For the most part, but not always, there is no effect on the Roon listings. And the duplicates remain listed.

Many times after going through multiple steps, I am told it cannot be deleted in Roon, but must be done in iTunes. There are 2 iTunes drives. How am I editing in Roon and deleting what Roon has added from iTunes? Or can I delete the album/tracks in Roon versus editing in both Roon and iTunes? I want to delete one place not in multiple programs. The reason for the duplicates is how Roon is importing, not duplicates in the within the iTunes libraries.

I guess I am not really seeing how I can easily delete an album or tracks in Roon using the tools present. Perhaps I am missing the effective use of the built in tools.

Your thoughts are appreciated.

Thanks

Dan

Thank you for touching base with with us @Daniel_Greenberg , the feedback is very appreciated! A few things moving forward:

From your previous:

“1. how do I prevent Roon from adding duplicates if they are from the same albums when watching two different, but similar drives (one library and one backup which is not identical)? My assumption (which may not be correct based on your comments) is that Roon should not add identical tracks/albums, but rather just one copy, or alternatively, give you the easy ability to filter out the duplicates. Hypothetically, If I am watching three or four drives, is Roon adding the same tracks over and over?”

“2. how do I get rid of the many duplicates (see my earlier post screenshots) I have in some sort of efficient way? Remember there are thousands of duplicate tracks. When I click the 3 dots to the right of an entry (Album or Track), and I can edit, I select delete album or delete track as the case may be. For the most part, but not always, there is no effect on the Roon listings. And the duplicates remain listed.”

  • As Mike specified in his last, Roon is going to import any content it has access to and furthermore if the same files are being added in multiple location being watched, there is no way to know which is the one “you want” it is going to just add it to your library

  • If you want to be able to “bulk” delete duplicates I would recommend using our “focus” feature to zero in on a specific storage location an the duplicates that exist there. After you have used the focus filters you can either tag the items as duplicates (using tags) or you can delete them. Remember that deleting content in Roon is a permanent action in the sense that the content is not going to be just removed from Roon it will be deleted from their storage location. I would recommend backing up your musical library first before performing this action, just to have a fall back in the event that something happens.

"Focus" example:

OK-Not crazy about the answers, but it is what it is. Maybe some future release will allow the exclusion of iTunes duplicates that are seen on different drives. But for now, ok-I got it.

So now to avoid the problem going forward, I should only specify that Roon looks at one drive? Subject to further comment, I am going to make that the backup drive (USB off the office iMac used as a remote). The reason is that this machine has a CD/DVD drive. I have only added 65% of my library of actual CD’s this way. The same iMac can also accept, with some supplementary hardware attachments, a traditional USB Flash drive if I download something to be added.

Assuming Roon is just watching this drive (iMac), new tracks will be added to the library managed by the Core machine (where Roon is installed as the Core), but from the iMac. Is that correct?

Still not sure where I am deleting tracks/albums (as duplicates) that are indicated as iTunes tracks. Am I deleting in iTunes off the iMac (acting as Roon remote), with Roon seeing those changes. Or am I deleting them in the Core drive (USB off the MacMini). If I delete them in iTunes on the (then) single drive being watched by Roon, will the tracks disappear (deleted in Roon)? Or do I have to delete them on the other drive (MacMini) first in iTunes, so the duplicates are not present and then change the drive Roon is watching

I am assuming that if I only specify only one drive to watch, as long as I delete the duplicate tracks on the other drive, and in Roon, there should be no issue. Correct? Or I eliminating the duplicates within iTunes and in Roon? In other words is any of this self-correcting with the existing database once I remove the duplicates in either Roon or iTunes? Or do I have to remove the duplicates in each on the drive currently being watched? Or the other drive currently connected but “not watched” in the future?

I am going to address the 2nd comment about deleting in “bulk” today. I can spend some time on it. I will do backups first.

Will report back once I try your suggestions. In the meantime any further comments you may have relative to duplicate removal of the existing library are welcome. I will try the Bulk removal using Focus and see how it goes. My experience so far has not really worked well but I did not “tag” item as duplicated, only delete the album with no material results in Roon. My conclusion so far is that I have to delete iTunes tracks according to Roon in both iTunes and any action in Roon does not really delete or hide anything. Is that correct?

Thanks again for your patience.

Dan