Roon/TIDAL connection seriously messing with my Library!

Jeff_Bellune · April 28, 2018, 2:20pm

TL;DR: Library operations in Roon (merging, deleting, and sometimes adding) will cause “Favorite” songs in TIDAL to move from one album version to another version of the same album, and from one album by an artist to a different album by the same artist!

Recent examples: .38 Special, Phil Collins, 10,000 Maniacs

I was browsing my albums the other day and I noticed that it showed my CD version of In My Tribe by 10,000 Maniacs as expected, but that I also had a TIDAL album called Campfire Songs right next to it. I’ve never favorited that album or any tracks from it in TIDAL because i didn’t even know it existed. I did favorite some tracks in the TIDAL version of the Tribe album, and I grouped the TIDAL version with my CD version. So I checked my TIDAL favorites in the TIDAL desktop app and sure enough, the track “Verdi Cries” in the Campfire album was starred!

The corresponding track in the Tribe album was no longer starred. I know for a fact that I favorited that in TIDAL, along with most of the other tracks from the Tribe album.

Similar things have been happenening with several other artists for a while now, but today I got to watch it happen in real time with .38 Special.

I starred several tracks in their Anthology album in TIDAL, but when I went to look at the album in Roon, only some of the tracks appeared. The ones that didn’t appear under the Anthology album were actually in my Roon Library, only they were listed under two other greatest hits type albums. So the missing tracks from Anthology were not missing due to any update delays between Roon and TIDAL. They had to be missing for other reasons.

I went back to TIDAL and the tracks of the Anthology album that were missing in Roon (the very ones I had just starred under the Anthology album in the TIDAL desktop app) were no longer starred in the TIDAL app!

I was able to duplicate the same issue with some Phil Collins tracks that I starred in TIDAL, except that the starred tracks moved from one version of an album to another version of the same album (Remastered to Deluxe).

This is a serious bug in communication between Roon and TIDAL. Roon should never be able to change what I physically “favorite” in TIDAL, unless I explicitly add or delete a TIDAL track in Roon. It is completely messing up my carefully curated favorites in TIDAL, which I use for mobile listening, and it is adding redundant flotsam to my carefully curated Roon Library.

I’ve tried cleaning my Roon Library, but it didn’t help.

Please note the “New” banners on the random .38 Special albums. It also looks like I’m going to have to re- curate the TIDAL versions of my ABBA albums – note the two versions of The Visitors!

EDIT: Add screenshot and note.

BrianW · April 29, 2018, 8:31am

I’ve never experienced your problem, and find that

favouriting/unfavouriting a track in Roon has no effect in Tidal
adding a track to my collection in Tidal app only affects Roon if the exact track/album combination doesn’t already exist in Roon, otherwise a new, single track album appears.

I also find it’s important that you ensure they have synced - I use the little sync icon in the Tidal part of Roon to avoid waiting.

It sounds as though you’ve found a bug which breaks this - if you can describe a set of steps that @support can reproduce then they will have something to work on. (And I’ve just flagged them)

Brian

Jeff_Bellune · April 29, 2018, 1:26pm

Thanks for flagging support, Brian. To your points:

That is exactly the behavior I expect. Unfortunately, that’s not happening for me.

I use Settings/Services/TIDAL Edit/Sync Library Now. I can’t tell you how many hours I’ve spent watching that little white circle go round and round and round and round waiting for the sync to finish, and it frequently never does. I have to quit Roon and restart it to get the sync to finish on restart. I didn’t know about the sync button in the TIDAL tab; I’ll have to use that from now on. I think the spinning blue circle that appears occasionally in the top right corner of the main Roon UI indicates sync progress, but I’m not sure because it’s spinning and the spinning of the white circle in the Sync Library Now dialog rarely coincide.

I’ll see if I can find reliable, reproducible steps to cause the issue, but so far to me it appears completely random as to when and where this issue will manifest itself.

Jeff_Bellune · April 29, 2018, 3:14pm

This post is my attempt at reproducible steps.

Specific baseline conditions:

Artist has at least one album with multiple versions.
Artist has at least one Greatest Hits collection.
You have had several tracks from one album starred in TIDAL for a long time, with at least one of those tracks currently in a Greatest Hits collection.
TIDAL added/changed another version (MQA, Remastered, Deluxe) of the album with the starred tracks later on.
TIDAL added/changed a Greatest Hits collection later on.

What to do next:

Check all the TIDAL albums by that artist in your Roon Library. Any surprises? If so, you’ve been bitten by the bug. Proceed to step 7.
No surprises? Go into the TIDAL desktop app and move some or all of the tracks from your existing version of the album to a new version, say, the MQA one. Do this by starring the tracks in the new version and then removing the starred tracks from the old version.
Go back into Roon and sync TIDAL. Your changes should appear shortly, and be correct.
Wait. Go get a cup of coffee or go for a walk.
Repeat steps 2, 3 and 4 for another artist in the TIDAL desktop app. Sync and wait again.
Is everything still correct in Roon for the changes you made in TIDAL? If so, come back tomorrow and check again. If everything is still correct, then you are probably immune to the bug. If not, continue on.
Check the TIDAL desktop app. You’ll find the tracks you starred there have mysteriously been removed. The same tracks will now mysteriously be starred in a different album.
Try to consolidate all the wayward tracks into the album version you want in your Roon Library. For me, I usually try to consolidate into the new MQA version of the original album.
After consolidating in TIDAL, go back to Roon and sync. Everything should appear as expected.
Wait.
After some time you may find that your consolidation has failed, and the tracks are scattered randomly among albums and versions once again. I have no solution for this, and need help from @support.

eric · April 29, 2018, 5:56pm

Thank you for the very detailed reported @Jeff_Bellune , your diligence is greatly appreciated!

I have added the feedback and steps provided above to a ticket which I am going to schedule for testing by our tech team to see if we can replicate this behavior in house. Once the team has completed their testing and the mentioned ticket has been updated I will be sure to share their thoughts/findings with you asap.

-Eric

Jeff_Bellune · May 3, 2018, 2:42pm

In case anyone who wants to try to duplicate this issue would like some artists to test, here’s a partial list of artists affected by the random music selection bug between TIDAL and Roon:

10,000 Maniacs
.38 Special
ABBA
AC/DC
Aerosmith
Airbourne
Alice Cooper
America
Amy Grant
Bachman-Turner Overdrive
Bad Company
The Bangles
Bastille…
…and on…
…and on

[whine]It’s taken me almost 3 hours to sort out which tracks are the ones I want, select them and then delete the randomly selected tracks for this list just down to Amy Grant. I will have to go through more than a hundred artists to get my library back in shape, and even then it’s no guarantee that it won’t happen again.[/whine]

I hope the release of 1.5 and it’s improvements to how Roon communicates with and displays information from TIDAL will squash this bug on my system. I say “my system” because it’s possible it’s specific only to me. If that is true, however, it would be most disheartening.

Jeff_Bellune · May 6, 2018, 5:36pm

Question:

Could this particular issue be caused by an incomplete or partially corrupt Roon database?

I ask because I just discovered I have a failing Windows 10 system drive on a Samsung SSD. A disk repair utility (chkdsk) found and marked some bad clusters. None of the bad clusters existed in the Roon folder or in any of Roon’s files, but its parent, my User\AppData\Local folder, had many file entries that did.

I have manually copied the Roon database over to a separate HDD, as well as making a backup of it.

Jeff_Bellune · May 8, 2018, 4:23pm

Hi @eric,

Bare metal install of Windows 10 and fresh install of Roon with a virgin database.

Roon found all my local music files and then grabbed my TIDAL tracks. As one might expect, the TIDAL tracks ended up in the same state that they were in the last time I logged out of Roon using my old database in my old Windows installation. Two artists, ABBA and .38 Special, had their tracks distributed exactly the way I wanted them due to the work I did in the old Roon database. That was encouraging.

I started working on getting the rest of my new Roon database in a similar happy state, adding and deleting TIDAL tracks as necessary. All was going well until I forced a TIDAL refresh from Roon. I forced the refresh because some tracks would not delete. They remained in the Roon database, even after cleaning the Library from Settings. When I tried to delete them again, I got a warning that the file I was trying to delete was not there. The deleted-but-still-visible tracks would not play, either.

As soon as I forced the refresh, Roon and TIDAL started randomly moving ABBA and .38 Special tracks between greatest hits and alternate album versions again. I got to sit and watch this: every time the little blue wheel in the top right of the UI would spin, another track would disappear from one of my desired albums and reappear in some greatest hits album somewhere. A couple tracks ended up in various artist compilations with names like Top Hits Of The 80’s and 70’s Solid Gold Pop (or some such nonsense).

At this point I think it’s safe to say that possible file corruption in my old database was not the issue. It’s unlikely to be a Windows issue, either.

What does that leave? Anti-virus interference? I use Webroot. Roon/TIDAL syncing issues?

I hope the tech team can help me get this sorted.

Thanks,
Jeff

Geoff_Coupe · May 8, 2018, 4:27pm

And did you replace the failing SSD?

Jeff_Bellune · May 8, 2018, 5:15pm

Yes. The failing SSD was my Windows system drive. The new SSD necessitated the bare metal Windows installation.

eric · May 8, 2018, 7:27pm

Hi @Jeff_Bellune — Thank you for the follow up from our previous discussion. Sorry to hear of the troubles.

Continuing forward, I am going to enable diagnostics on your accounts so our techs can try to determine what could be causing this behavior to occur. What this action will do is the next time the application is active on your core device a diagnostics report containing a set of your Roon logs will automatically be generated/uploaded directly to our servers.

Once the mentioned reports have been received I will be sure to update this thread so you know that we have it and then I will swiftly pass them over to the team for further analysis.

Lastly, while it may seem a touch elementary, since noticing this behavior with the mentioned TIDAL content in your library have you tried rebooting the core machine?

-Eric

Jeff_Bellune · May 8, 2018, 8:20pm

Hi @eric,

I have rebooted many times over the last couple of days, BUT I don’t think I’ve rebooted since noticing the problem in the new database. I’ll do that immediately and report back.

EDIT: This Windows 10 machine is the core, and main control point.

Cheers,
Jeff

Jeff_Bellune · May 8, 2018, 11:04pm

Hi @eric,

I rebooted the core. The albums and artists I worked on this morning remain intact, so far. We’ll see by tomorrow if they still stick.

eric · May 9, 2018, 4:52pm

Hi @Jeff_Bellune ---- Just checking in on you, let me know how things are going and we can continue from there.

Thanks!
-Eric

Jeff_Bellune · May 9, 2018, 7:12pm

Hi @eric,

It seems if I add an entire TIDAL album to my Roon library, then none of the album’s tracks will get deleted if the “magic shuffle” occurs. However, I still get additional tracks from greatest hits and other album versions added to my library.

If I only add some tracks from an album, then the magic shuffle adds new, undesired tracks from undesired albums AND deletes the corresponding track(s) from the desired album(s).

Once I’ve corrected an artist’s collection of tracks 2 or more times, the changes seem to stick around longer even if other artists are having issues. But I can’t be sure they’ll stick forever.

To be honest, the thought of having to go through my whole Roon library and correct almost every artist’s entire collection of starred TIDAL tracks multiple times is enough to drive me to drink!

Here’s some perspective:

I’ve stopped listening to TIDAL music on my mobile devices altogether. I’ve given up the superior sound quality of TIDAL for the predictable behavior and convenience of Apple Music on my iPhone and with CarPlay. (As I go through my Roon library trying to correct all the scattered tracks, I add the songs I want to my Apple Music library.) Sadly, being able to listen to CD-quality sound from a streaming service on my iPhone was what prompted me to subscribe to TIDAL in the first place.

As long as this situation exists for me, something has to go. I could give up Roon and just use TIDAL everywhere, but that means giving up all of Roon’s playback flexibility and metadata goodness. I could give up TIDAL, but that costs me MQA and streaming lossless goodness at home and on the road with my laptop. My decision to give up TIDAL only on the road seems the lesser of the evils.

Cheers,
Jeff

mike · May 10, 2018, 1:54am

Hey Jeff,

First, I’d like to give you a little background here.

So, when content is in your Roon library, that means it’s “starred” on TIDAL. There are two ways TIDAL content can get into your Roon library – you can star the track, or you can star the album. Both result in tracks being added to your Roon library, but on the TIDAL side you might have tracks, or albums, or both that are starred.

The behavior can get a bit confusing if, for example, you add an album to your library, but then you delete a single track in Roon – when you delete something in Roon we remove the star, but if the star was at the album level, we can’t actually just delete some of the music – the star you’ve told us to remove is at the album level, so that’s what will be removed.

I’m not sure how closely that behavior relates to what you’re describing here, but it’s good background for how you’re using Roon and TIDAL.

As for the issues you’re describing, I can tell you this is not something we’ve heard about before, and clearly something has gone very wrong here. I’m not yet 100% confident there’s a bug here, but it’s definitely possible you’ve found something that’s broken here, perhaps because most Roon users are adding albums and from what I can tell you’re mixing albums and tracks more than most.

I think the next step here is to separate what’s going on in your Roon database from what’s going with your TIDAL library. There may be problems in both, but it will be easier for us to understand what’s going on here if we separate the two.

First things I’d like you to do is find the Roon database on your Core machine (instructions here), and rename the folder Roon_old. Then, reinstall Roon, leave your local files aside for now, and just login to TIDAL.

Let us know how things work out – if things look sane with a fresh database we will know a lot more here Jeff, so thanks in advance for your patience.

Jeff_Bellune · May 10, 2018, 4:23am

Hi @mike

Thanks for the info! I worked all that out on my own quite a while ago, mostly because of my use case, which you have pegged. I’ll try to keep its description short and to the point.

TIDAL desktop app: manage my starred tracks and albums, and create playlists. iOS app not as good for that, and until 1.5, neither was Roon.

TIDAL iOS app: curate playlists and play music on the go. Track shuffle is preferred listening mode.

TIDAL-Roon integration: seamlessly add TIDAL tracks and albums to mix in with my local music file library for play in various zones around the house. Focused track shuffle is preferred listening mode.

Add entire TIDAL album or just selected tracks: favorite albums from my past and new ones that I want to audition get added entirely. Play album tracks in order preferred for these. Best tracks from all albums get a star. Track shuffle preferred listening mode.

I’ll follow your instructions and see what happens.

I’ve already put several hours into correcting my fresh new database from my Windows and Roon reinstall, so I’d like to know if getting back to where I am now is simply a matter of renaming/moving the Roon TIDAL-only folder and then renaming Roon_old back to Roon. Will that work?

Cheers,
Jeff

Jeff_Bellune · May 10, 2018, 4:27pm

Hi @mike @eric

Step-by-step we go.

Renamed Roon folder Roon_old
Re-installed Roon from the Roon download page and updated
Disabled all local files in library
Signed in to TIDAL
Enabled various Audio zones
Waited while Roon loaded all my TIDAL tracks and albums
Opened the album browser
Curating work that has been performed at least two times and up to 4 or 5 times intact. Both album and track favorites point to the correct TIDAL item.
Many artists whose tracks and albums have been curated only once or not at all still have starred items scattered in all the wrong places. Assume that’s leftovers from earlier Roon/TIDAL interactions.

Immediately observed wonky TIDAL behavior (with screenshots):

Roon album browser shows only 2 Bob Dylan albums with starred tracks, either at the album or track level

TIDAL shows more than that:

Roon lists a “Various Artists” album as a Bob album. It doesn’t do that for Kim Carnes or Kelly Clarkson (the only other 2 I checked for sure):

Couldn’t delete the 2 Bob Dylan albums shown in the album browser:

But those tracks really do exist in TIDAL:

Again, all this is with a fresh install of Roon and only TIDAL albums and tracks in the library.

NB: The two tracks on the greatest hits albums? I never starred them in TIDAL and I never chose them in Roon. Whatever is going on here randomly selected those tracks for me.

Hope this helps,
Jeff

eric · May 10, 2018, 5:45pm

@Jeff_Bellune ---- Letting you know that I have followed up with you via PM, thanks!

-Eric