Skipped files during scan

Support
1

Core Machine

Nucleus/Rock
OS: Version 1.0 (build 227) stable
Server: Version 1.8 (build 790) stable
HW: Intel NUC10i7FNH, 32 G RAM, 1 TB SSD

Network Details

CAT 6 cabling, linksys/intel 100/1000 switches
NFS Server: pi4, Rasbian, 12 TB USB3 dis

Audio Devices

Irrelevant to problem: Benchmark DAC-3

Description of Issue

I have a large library. Roon successfully imported about 227,000 tracks. 267 tracks were listed in the roon “skipped files” list as corrupt. Another 372 tracks were listed as “i/o failure”. These appear to be separate issues.

  1. corrupt files: 224 of these are digitized 78 tracks: low bit rate CBR mp3s, mostly from archive.org. I have over 1000 more 78 tracks that were succesfully imported. When I checked the files roon lists as corrupt by playing them with foobar, directly off the server, they play fine, and using foobar properties/details, I can read the tags. I found no errors in the dozen or so tracks that roon reports as corrupt that I tested, either in playback or tags. One possibility is that roon does not handle low rate mp3 well: some of these are 40 kbps historic recordings.
  2. i/o failure: this is a more serious problem. The 372 tracks listed as i/o failure all appear fine (again checking by playing them off the fileserver using foobar). They tend to be mostly entire albums (but not entirely). It appears that the failures can be repeated (limited testing) with repeatable results, and that the failures are mostly (but not entirely) complete albums. If the problem were network or disk, I would not expect repeatable failures. The fileserver logs have no disk errors. I’m not sure how your rescan works, but I tried “touching” an album and tracks to change its date, then rescanning in roon, and the error appeared to repeat. (in my opinion user visibility and control over the scanning process is a roon weakness).

Another observation is that before migrating to my current ROCK setup, while testing the core on a windows laptop, I’m almost certain that the skipped files from the windows scan was a very small number, under 20 instead of nearly 700. Perhaps there is a difference in how the scan performs in the stripped down linux version, and how it works on windows.
Please advise, thanks.

2

Hi @jim_hamilton

Can you upload an album of both types here and we’ll take a look on our end? Thanks!

3

Thanks. I uploaded a tar file to the requested location with 2 tracks that roon skipped as well as the spreadsheet of all skipped files. The README in the tar file has additional details.

4

ok. I partially solved my skipped files problem. Like many problems, it was obvious in retrospect. I’m using a linux fileserver, and routinely when adding music do recursive chmod and chown to set user/group ownership to myself, and permissions to 755. (its a personal system, so world execute is not an issue). the files which were skipped with roon error “io_failure” had permissions 600. I apparently forgot to set the permissions when I added those files. .

but, its not all me here. I had no list of causes which generate an “io_failure” message, so just had to notice the permissions problem. If it had said “read error” the issue would have been transparent, and I would not have spent roon resources contacting tech support. Or if there was a list of causes of “io_failure” in the knowledge base article on skipped files, I’d know what to check.

and there’s more. when I fixed the permissions (simple chmod -R) and did a force rescan, the “fixed” files still did not show up in roon, and the “skipped files” list appears unchanged. The documentation for skipped files and scanning in the knowledge base are way too cursory. There appears to be no way for a user to reset the skipped files list. There is no description of how scanning works, and what is checked on rescan. In this case, I found that by renaming one of the albums that had been skipped because of permissions, I was able to successfully import it into roon and play it. So apparently once something is on the “skipped files” list, its not retried on rescan, and I can’t reset the skipped files list. This needs to be rethought.

I was able to fix the permissions in a few seconds with “chmod -R”, but since the fixed files apparently won’t be rescanned, I’m going to have to go through all the albums that were skipped, rename them at least temporarily, and then scan. If there is a way to reset the “skipped files” list, please let me know. I could move the album directories, clean the library, put the directory back, and rescan (it appears). This is awkward to say the least. I could throw up my hands, delete the entire database, and rescan after fixing all errors. ugh.

The other error I reported seeing on some files that were skipped was “corrupt_file”. I haven’t figured that one out yet. It would help if roon can tell me what error types can cause this message. Please provide this information. So far all the “corrupt” tracks I have looked at can be played in other music systems, and have readable metadata. (haven’t checked all of them).

I think I have some more general comments on the “appliance model” for rock at the current stage of development, without decent user accessible error messages and control capabilities (such as resetting skipped files, but this may be the tip of the iceberg). I’ll save these for later. Roon has plenty of features: improving this sort of usability and diagnois should be very high on priorities.

now, back to the skipped “corrupt_file” items, which so far all play properly and have readable metadata…I’m hoping to hear from roon regarding what errors cause roon to generate this message.

5

Thanks for the update, @jim_hamilton.

I’ve made an edit to our Help Center to clarify that the I/O Failure message can be related to permissions. Thanks for that feedback!

Force Rescan should trigger Roon to try re-importing this content. I’m checking with the team on other options here.

For corrupt files — In the past, we’ve seen this be caused by bad headers in the files. Some apps ignore these headers, but Roon does not, which could explain why other apps are playing them anyway. Often times converting to the same file type using something like DBPoweramp can help.

Edit — Just to verify, did the Modification Date on the files change after you updated permissions?

Edit 2 — Our team did a test and it looks like rebooting the Core machine helped. Can you give that a try!

6

feedback to @dylan

  1. from my tests, roon does not attempt to re-import files that are skipped… until a reboot. when I rebooted, all my files that had “io_failure” (due to permissions that I had fixed) were successfully imported. I still think letting users reset skipped files, or changing the behavior to retry during rescan would be better, but the reboot worked. Another note to your help center: have the user reboot before investigating skipped files.
  2. will try reconverting some of the “corrupt” files and report back.
    update: I reconverted about 120 of the old 78 record mp3s. There were no errors reported in the conversion. I then rebooted the roon machine. they are still skipped. I played parts of about 30 of these with foobar, and they all played just fine: I never heard an error. mp3tag reads them all with no errors. If there is someplace in the roon logs I can look to get better insight, let me know. Also, these are public domain. If you want a bunch of these for testing, I can tar them up for you. admittedly this is now a corner case, perhaps of interest to few, but it might behoove roon to at least understand why it won’t import files I can play, reconvert, and examine the tags for, even if these are antique mp3s…
  3. changing permissions does not affect modification date, at least in debian and probably all *nix.
7

This problem seems to have been forgotten after some good initial progress. I think it deserves resolution.
After fixing initial problems with permissions on some files in my library, I’m left with 224 tracks that roon lists as “file is corrupt”. this is out of 225,000 total tracks. All are mp3s from archive.org for tracks off old 78 records. I have a couple thousand other tracks from the same source that roon ingested fine, and that play with roon. All these files reside in the same music library on a linux fileserver.
I’ve done the following:

  1. played many of these rejected files using foobar on windows and vlc media player on linux. I found no errors from any of the ones I tried, all played fine.
  2. looked at the tags using 3 different tag editors (Easytag on linux; mp3tag and ezmetag editor, both on windows). all read the file tags fine, none found errors.
  3. recoded the files as CBR mp3s using EZ CD Audio converter on windows, same program I have used for thousands of rips and conversion. There were no errors in reading the files or converting them
  4. ran checkmate mp3 checker against the recoded files. the checker found no errors.

The easiest next step would be to examine roon logs to see if a better error message is in the logs (what led to “file is corrupt” message?). I’ve looked at the knowledge base articles for logs and for skipped files, and there is not enough information there. so… In the roon server logs, I searched for corrupt file messages. there they are. typical one:

05/10 11:17:55 Trace: [storage] [directory] extracting /roon/sys/storage/smbmounts/RoonStorage_a11e35d07478eb587f892a75e09667c8661a3ec6/music/78s/jazz/20sJazz/kentuckb_64kb.mp3
05/10 11:17:55 Warn: [storage] [directory] Failed to extract audio format from ‘/roon/sys/storage/smbmounts/RoonStorage_a11e35d07478eb587f892a75e09667c8661a3ec6/music/78s/jazz/20sJazz/kentuckb_64kb.mp3’: CorruptFile

played this file all the way through with vlc media player. its fine (great for an old 78. nice remaster).
vlc reports the standard metadata (title, artist, album, genre) correctly, and shows no extra metadata. For the codec, vlc has:
stream 0
codec MPEG audio layer 1/2 (mpga)
type audio
channels mono
sample rate 22050 hz
bits per sample 32
bitrate 64kb/s

so, 7 audio tools on 2 platforms can read and play these files fine, but roon can’t. the key message in the logs seems to be “Failed to extract audio format” why?

8

Maybe this? Can you convert a track to “2 channel”-mono (same format) and/or mono (single channel) FLAC as a test to see if things change?

9

thanks for the suggestion from @BlackJack . I solved the puzzle. I took an mp3 that roon wouldn’t read and tried various combination of encodings, including those he suggested. To my surprise, they all worked: roon imported the files. Comparing an original, “corrupt” file to one that roon imported, I find the common feature is that the files roon doesn’t like are reported as “mpeg-2 layer 3 (lame 3.96r)”, while other, successfully imported files are mpeg1-layer 3, lame 3.0 or lame 3.1.
When I originally tested re-encoding, I left some settings as “automatic”. I guess my encoder kept the mpeg 2 format in that case, whereas for the final tests I set everything by hand.
mpeg-2 does not import.
mpeg-1, lame 3.0 and 3.1 both import for the following tests performed on an identical mpeg 2 file after it is re-encoded in multiple ways:

  1. mpeg-1 layer 3 lame 3.0, 44.1, 64 kbit/sec, joint stereo (monox2)
  2. mpeg-1 layer 3 lame 3.1, 44.1, 64 kbit/sec, stereo
  3. flac 1.3.3 16 bit mono 245 Kbit
  4. flac 1.3.3 16 bit stereo 245 kbit

testing my skipped files, it appears all the 224 files files skipped were mpeg-2 and play on other systems (at least after testing a couple dozen). is it documented somewhere that roon does not support mpeg-2? guess I’ll convert my files. This thread can be closed from my viewpoint, though I’d be interested in a response from roon. At the very least, it would be an improvement if the roon log (at least, and preferably the user skipped file interface) reported a better error than just “corrupt file”. As I have indicated previously, its also awkward for troubleshooting not to have any user control to clear skipped files (except a reboot) and to scan designated folders instead of an entire large library.

10

I have had problems importing some albums. They had all been ripped from CDs using XLD for macOS with no errors declared in the log report. The file format was .flac. Forced rescan of the storage location did not resolve the problem but rebooting the core (MacMini) did. Only after finding this conversation did it occur to me to do the reboot so, yes, it would be helpful if this advice was more prominent in the user guidance built into Roon.
Incidentally none of the skipped files was identified to me by Roon.