Library "endmutation" degrades over uptime (130ms → 28s), causing 25-30s track transitions — root cause: SentryCache backlog from failing ARC

Roon Software Discussion
1

Content you’re reporting an issue with

Have you made any edits to this content in Roon?

Is the album identified in Roon?

Is this content from local files, TIDAL, or Qobuz?

Screenshot of import settings

Description of the issue

Roon Server version: 2.67 (build 1661) production
Server OS: GentooPlayer (Linux 6.12.77 x86_64, PREEMPT_RT), .NET 10 runtime
Hardware: Intel NUC, i5-8259U, 16 GB RAM, NVMe SSD
Library size: ~22,400 tracks / ~3,900 albums / 91 playlists
Audio chain: RoonServer → RAAT → RoonBridge host (RPi4) → Diretta → Diretta Target (RPi4, Allo DigiOne Signature) → SPDIF → DAC

Summary

After a fresh RoonServer restart, track-to-track transitions take ~1 second.
Over the following days of uptime they get progressively worse, reaching
25–30 seconds by day 3–5. A restart fixes it, then it degrades again.

Root cause I traced

The delay maps exactly to the library mutation step. From the RoonServer log:

[library] endmutation in 28791ms

I tracked endmutation timing across 5 days of uptime:

Day 0 (after restart): ~130 ms
Day 1:                 ~5,000–11,000 ms
Day 2:                 ~11,000 ms
Day 3:                 15,000 → 28,000 ms
Day 5:                 28,000–30,000 ms

During the stall the CPU is idle and RAM is free (8+ GB free, no swap,
no THP, NVMe disk, the DB flush itself is <100 ms), so the thread is
blocked/waiting, not working.

The actual culprit is the Sentry telemetry cache.
~/.RoonServer/SentryCache had accumulated 1,477 .envelope files (42 MB).
It appears the Sentry .NET SDK enumerates/manages this directory during
library operations, so endmutation scales with the number of cached
envelopes.

Proof: with the server still running (no restart), I deleted the
*.envelope files. endmutation dropped from 28,791 ms to ~130 ms
immediately, and transitions went back to ~1 second. A ~200x improvement
from a single rm, no restart.

Why the cache fills up

The envelopes are almost all failed network captures:

651 × System.Net.WebException: The request was aborted: The request was canceled.
526 × System.Net.WebException: (504) Gateway Timeout
128 × System.IndexOutOfRangeException: Index was outside the bounds of the array
 16 × System.AggregateException: A Task's exception(s) were not observed
 (+ assorted SocketException / IOException)

The 504 / canceled errors are Roon ARC connectivity checks failing
~150 times per hour, 24/7 (multinat Failed, UPnP NotFound, NAT-PMP NotFound,
Port Check 504 from api.roonlabs.net). I do not use ARC and there is no way
to disable it, so it retries forever and Sentry can never upload the events
(network is the thing failing), so they pile up on disk indefinitely.

The 128 × IndexOutOfRangeException is a separate, genuine bug. Stack trace:

Dictionary<TKey,TValue>.TryInsert
RemotingServerV2._PutObject
Sooloos_Broker_Api_PerformerLite_Adapter.Serialize
RemotingServerV2.NotifyStateChange
ServerConnectionV2.OnBeforeEntry
SynchronizationContextThread.OnBeforeEntry

IndexOutOfRangeException inside Dictionary.TryInsert is the classic signature
of a Dictionary being accessed from multiple threads without locking.

Three issues for you to consider

  1. SentryCache should be bounded. A growing offline cache should never be
    able to degrade core library performance 200x. Please cap the cache size /
    file count, or stop touching it on the library hot path.

  2. ARC needs a real off switch. When ARC can’t establish connectivity it
    should back off, not retry every ~25–30 s forever and generate an
    un-uploadable exception each time. Users without port forwarding pay this
    cost permanently. (The “port 0” workaround resets on every restart.)

  3. Thread-safety bug: concurrent unlocked access to a Dictionary in
    PerformerLite serialization (stack above).

My current workaround

Hourly cron that deletes SentryCache/*.envelope older than 5 minutes.
Keeps transitions at ~1 s with no restarts. Happy to provide full logs.

2

UPDATE — SOLVED. Root cause found (and it’s a Core design issue worth fixing)

Following up on my own thread with the resolution, in case it helps others and the dev team.

Short version: the 25-second track-transition stalls were Roon Core blocking the transport thread for ~25 s while trying to push a state update to a single unresponsive remote control client. One slow/sleeping client was enough to freeze playback for every zone and every other client — including the desktop remote I was actually issuing commands from.

How I proved it

  • The stalls had a tell-tale constant duration of ~25,000 ms every time → a fixed timeout, not variable work.

  • During a stall the CPU was idle, disk idle, RAM free, no swap, no THP, no GC pressure. The thread was simply blocked waiting, not working.

  • Every stall lined up with my phone (a Galaxy S24 Ultra running Roon Remote) re-establishing its broker connection (Initialized Fresh Session on port 9332), and the stack of the related captured exception was always in the client-serialization path:

    Sooloos_Broker_Api_PerformerLite_Adapter.SerializeRemotingServerV2._PutObjectRemotingServerV2.NotifyStateChangeServerConnectionV2.OnBeforeEntry

  • Decisive test: I put the phone fully offline. After two more ~25 s timeouts (the Core draining the now-dead socket), the stalls stopped completely — I then skipped through dozens of tracks with consistent ~130–300 ms transitions.

What was destabilizing the phone

The phone had AdGuard running in its Android VPN mode, with Roon included in the filtered set. That VPN was intercepting Roon’s LAN traffic to the Core and breaking/stalling the connection — especially when the phone dozed with the screen off and the VPN service paused. Excluding the Roon app from AdGuard’s filtering made the phone connection stable immediately, and the Core stalls disappeared while still using the phone normally. (My desktop remote never caused this, because AdGuard on desktop runs as a local proxy, not a traffic-capturing VPN.)

The part that’s on Roon (please fix)

The flaky client was my problem to fix — but a single slow client should never be able to freeze the transport for 25 seconds. The state-change serialization to remote clients appears to run synchronously on the playback/transport path with a ~25 s per-client timeout. That conflates two things that must be independent:

  • playing music (must never wait on a UI client), and

  • refreshing a client’s screen (best-effort).

Suggested fix: push client state changes asynchronously / fire-and-forget, with a short per-client timeout (1–2 s) and per-client isolation, so an unresponsive or sleeping remote can’t block the Core’s transport or other clients.

Secondary note (from my original post)

The SentryCache backlog I described earlier (1,477 undeliverable .envelope files from constantly-failing ARC connectivity checks) was a real, separate drag on baseline library endmutation times — clearing it helped baseline responsiveness. But the headline 25 s stalls were the client-blocking issue above, not the cache. The ARC checks failing every ~25–30 s and piling up un-uploadable Sentry events 24/7 is still worth addressing on its own.

Happy to provide full logs/timestamps if useful.

3

Hi @Zeljko_Naumovic1,

Please don’t post directly in Support. Instead, please follow this link to provide the details of your case to Technical Support: Technical Support Request.

In this instance, I have moved the thread to Roon Software Discussion. I think you have uncovered a symptom, not the underlying cause. Did you use an LLM to analyse the logs?

Since Roon does not monitor this category, use the following to open a technical support request.

Respond to the prompts there to ensure that you’ve performed basic troubleshooting and to ensure Technical Support has the full details necessary to expedite Technical Support’s investigation into the case.

Your responses will create a ticket in Roon Lab’s support tracking system and auto-generate a Community thread in the appropriate section.

Because this Community thread will be in a public forum, please do not include any personal information, such as your email address, postal address or telephone number in your submission.

Thank you.

4

I am a paying Roon subscriber and I need help.

My track-to-track transitions take 25-30 seconds. This happens consistently after a few days of uptime and a server restart fixes it temporarily, but the problem comes back.

I am not able to find the cause myself and I need a Roon engineer to help me understand why this is happening.

Why am I paying for a subscription if I can’t get basic support?

5

Which is why you have been asked to open a technical support request using the correct method. Once this is done, this thread can be linked (typically, I would have converted your message to a PM.)