MP3 tracks don't show up since build 1193, album length is 0 minutes

Roon Core Machine

$ lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal

$ uname -a
Linux hpserver 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

// From RoonServer.txt 
01/29 20:36:07 Info: Starting RoonServer v2.0 (build 1193) production on linuxx64

$ ffmpeg -version
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

Networking Gear & Setup Details

(Not applicable)

Connected Audio Devices

(Not applicable)

Number of Tracks in Library

Just 5000 local tracks, I mostly use streaming. A small local library, where a fraction of them are lossy files.

Description of Issue

All albums that consist of MP3, AAC or mixed formats no longer show any tracks.

Everything was fine prior Roon Core update (I was on the version before current build 1193).

More symptoms:

  • The album length is 0 minutes, there are no tracks visible
  • The metadata is still there and the albums show up in results

I confirmed that my system’s ffmpeg has the correct support for MP3 (it includes the --enable-libmp3lame flag, and grep finds mp3 in the codecs available):

$ ffmpeg -codecs | grep mp3
ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
 DEA.L. mp3                  MP3 (MPEG audio layer 3) (decoders: mp3float mp3 ) (encoders: libmp3lame libshine )
 D.A.L. mp3adu               ADU (Application Data Unit) MP3 (MPEG audio layer 3) (decoders: mp3adufloat mp3adu )
 D.A.L. mp3on4               MP3onMP4 (decoders: mp3on4float mp3on4 )

Any ideas on what’s going on and what else I can try? Is there a way to downgrade the Core to the build prior 1193?

There’s something going on here and I wonder how I can reach out to Roon Support team to investigate further. I present my findings below. The bottom line is that MP3s show up when I re-add a folder to Library, only to disappear later on. Until they’ve disappeared I can play them, indicating that everything’s fine with my ffmpeg MP3 codecs.

A quick look into logs show a lot of lines like this:

02/07 12:02:56 Debug: [music/storage] processing reshuffle and delete for track 67:0:Music/{foo}/{bar}.mp3 from /mnt/storage/media/iTunes/iTunes Media/Music

I have no idea what triggers this removal from library, but I try to explain how it happens.

To verify that my system can play files encoded with MP3, I created a test folder and copied some MP3s there. Next, I added them as another folder (Settings->Storage->Folders->Add folder). This works as intended.

So I tried to force a rescan on my mail library. It did nothing regarding the MP3s.

Next I changed the path of the folder one step up in the hierarchy (../). This triggered some real rescan and my MP3s started to show up. I saw how my library was growing in size (turns out I have more MP3s than expected). I started one MP3-only album while Roon was indexing the folder.

Suddenly the album stopped and the queue was emptied. There was no sign of what I’ve listened to what so ever. And tracks count was back on the same, lower, level as before. All MP3s disappeared.

I repeated the process and took some screenshots.

  1. Test folder and Main folder. Test folder contains mixed formats (ALAC, FLAC, MP3), all the tracks are visible and playable in the Library. The Main folder contains mixed formats too but the MP3s do not show up (they do show up when copied to Test folder though!)

  2. I go with the default import settings:

  3. After I change the path of Main folder, a rescan is initiated and the track count starts ticking up. That’s because my MP3 tracks are added temporarily. In the screenshot below, the library has more than 6000 tracks.

  4. But suddenly, the track count starts ticking down. The animation below shows how it counts down, supposedly because it’s removing all MP3s.
    Skärminspelning 2023-02-07 kl. 12.03.02

  5. Finally, I’m back on the initial track count and the only MP3s that are visible in my library are from the Test folder:

What is going on here? Any idea on what triggers the removal that shows in log?

Hey @jacobwod,

Thanks for your patience as we work through each thread. If possible, could you head into Roon Settings> Library > Clean up the library (do not actually clean up the library) and take a screenshot of the Library maintenance screen?

With that, could you share some of the MP3’s with the team and upload here if possible.

Lastly, try disabling your regular watched folder in Roon Settings. Then, copy and paste tracks into a new folder and enable / route to the new folder, is it stable with Roon matching tracks?

I’ll be on standby for your reply :+1:

Hi @benjamin!

I did Clean up the library step both before and after the Force rescan. It showed up zeros on all over the place (nothing to clean up) but I did press the Clean up library button anyway.

What’s interesting is that I settled on finding all my MP3s and moving them to another location, outside my folder (in case anyone is wondering how: it’s easy with Bash or ZSH - when standing at the top of your library run mv -a --parents **/*.mp3 /new/dest).

After that I did a force rescan on my main library and added the new (mp3s-only) directory as a separate folder in Roon (see screenshot):

What’s really interesting here is that I did not have that many mp3s after all. So I have no idea how the run-away scanning (see my first post) could sum up to over 6000 tracks, when there only are 4545+306 in total.

I will gladly share some of the MP3s with dev team, but I no idea which one were causing the problem, nor why (as they seem to scan fine when in separate library).

File permissions are exactly the same in both locations.

Anything special I can look out for in the logs, perhaps better sharing that with you?

Hey @jacobwod,

Thanks for the additional information! Out of curiosity, do the problem mp3s have ‘CD’ in the title, or are associated in the file in any way? If you rename the file name, does the same issue happen?

We’re seeing some interesting errors when enabling diagnostics on your account, but not yet seeing a clear indication as to why your tracks are getting removed. A good next step would be to reproduce the issue and share the specific date and time when the issue occurs.

I realize this is a bit tedious to do, and I appreciate your patience while working through this! :pray:

Thanks for your reply @Benjamin.

While I can not say if none of the files contain CD in title, I’m quite certain that all of them do not.

Anyway, further debug follows.

I tried a Force rescan on the library that points to my main folder (the one that contains all files, including MP3s). The operation is quick and nothing new shows up:

02/22 14:42:00 Trace: [storage] [directory] force rescan requested for /mnt/storage/media/iTunes/iTunes Media/Music
02/22 14:42:00 Trace: [storage] [directory] initial scan of /mnt/storage/media/iTunes/iTunes Media/Music took: 173 ms

To really trigger the run-away scan I can change the path to my folder (simply by going one step up). You can see it starts here, where I remove the old path:

02/22 14:46:26 Info: [music/storage] Unloading storage backend /mnt/storage/media/iTunes/iTunes Media/Music
02/22 14:46:26 Debug: [storage] dispose deleting: /var/roon/RoonServer/Temp/43a0282a5b8f467caf80ec57c622fb36

You see the timestamp above, and I understand that you can see my logs remotely.

This scan goes on for a while and seems to think that I have over 8000 songs, suddenly:

For a while it was over 9000 songs, but then it starts ticking down again. In my log, it happens around 14:51.

Finally, it completes and my library is back on some 4000 tracks:

I took a copy of the logs generated today (in case they get rotated before you get a chance to see them), let me know if you want me to send them over in some secure manner.

Hey @jacobwod,

Reviewing logs would be a good next step. If you could reproduce the issue and use the directions found here and send over a set of logs to our File Uploader.

Thanks, Jacob!

Hi! Perhaps I wasn’t clear but I’ve already reproduced the issue. I used screenshots and timestamps from log files in my last post.

Anyway, I’ve packed up the logs and sent them using your log upload form. The tarball is marked with my first and last name.

Looking forward to hear more from you!

Hey @jacobwod,

Thanks again for your patience here! After bringing your issue to the team, we have a next step for you to test, but, you may lose track data around import time and play history in the process. It’ll be important to create a fresh backup before moving forward. :+1:

After creating a backup:

  1. Head into your Roon Settings>Storeage and disable your watched folder

  2. Perform a ‘Clean up library’ and delete all MP3 entries as well as any files associated with the disabled watched folder.

  3. Re-enable your watched folder, and let me know the results.

Hi @jacobwod,

I’m the developer on the Roon team responsible for the file tracking subsystem. @benjamin asked me about this issue, and I want to explain at least the track count behavior you’re seeing here.

Roon has a list of files it knows about in each storage location, identified by relative path from the watched folder to the file. We also use a hash based “signature” mechanism to identify files, so we can detect that the same audio data is present in more than one place or has moved or whatever.

When the watched folder path is changed like this, the storage location starts scanning again at the new path. It still has the old list of files by relative path, and it also finds what appear to be new files at different relative paths. That causes the reported number of tracks imported to increase, theoretically peaking at twice the actual track count.

The original count was about 4550, so over 9000 lines up perfectly.

After the storage system has finished scanning, it can finally discover the files from the old path are missing, and start issuing delete events. When that happens, Roon uses the file signature system to detect that we’re getting a delete event for an older file with the same signature as a newer file, and replaces stored information (things like import date) for the newer one with the older one. That’s what’s going on with the the processing reshuffle and delete for track messages in the logs.

This is a lot of words to say that the track count behavior when changing watched folder path is weird and confusing, but also the expected result of the rest of our design decisions, and not causing the problem here. Thanks for the question, I enjoyed figuring this out and writing the explanaion :slight_smile:

Hi @ben. Sorry about the late response.

I really appreciate your response and as a developer myself I can completely understand the design choice made for the re-scan algorithm, creating the unique hash for each track and as a final step comparing and removing duplicates of that hash. Sounds reasonable and well thought out.

As you’ve mention, this does explain the seemingly weird count during update but does not explain the issue itself (i.e. why the same MP3 tracks are “invisible” when mixed in the same directory structure as other files, but show up when I move them to another location and add that as a new Folder).

Nonetheless, I now consider this a minor issue, due to the work-around I mentioned (placing the files outside the main library Folder). And I truly understand if you – and the dev team – focus your effort on more important fixes and adding new features (e.g. this would be a really nice one, in my opinion :slight_smile:).

Thank you!

1 Like

This topic was automatically closed 36 hours after the last reply. New replies are no longer allowed.