Synology DSM + Roon Server: IPv6/DHCPv6 broken because Roon exhausted inotify watches

I am posting this in case it helps anyone else running Roon Server on Synology with IPv6 (whether in concert with IPv4 or not).

I recently tracked down a strange IPv6 issue on my Synology NAS running Roon Server. At first blush, the symptom looked like a pfSense, firewall, Router Advertisement, or DHCPv6 problem, but the actual cause turned out to be a combination of Synology IPv6 behavior and Roon consuming nearly the entire default inotify watch budget.

And yes, ChatGPT helped me sleuth this issue.

My Environment

  • Synology NAS running DSM
  • Roon Server installed in Synology Container Manager
  • pfSense router/firewall
  • IPv4 + IPv6 enabled on the LAN
  • DHCPv6 reservation configured for the Synology (I am allowing SLAAC and DHCPv6)
  • Router Advertisements enabled on the LAN
  • Synology interface: bond0

FYI I am using obfuscated IPv6 addresses below.

Symptom

After refreshing the Synology’s network configuration (even after a reboot), bond0 would come up with only a link-local IPv6 address:

inet6 fe80::xxxx:xxxx:xxxx:xxxx/64 scope link

It would not get either of the expected global IPv6 addresses (DHCPv6 or SLAAC):

2001:db8:1234:5678::5/128
2001:db8:1234:5678:xxxx:xxxx:xxxx:xxxx/64

In my setup:

  • 2001:db8:1234:5678::5/128 represents my DHCPv6 reservation from pfSense.
  • 2001:db8:1234:5678:xxxx:xxxx:xxxx:xxxx/64 represents the SLAAC address.

My other two Synology boxes — with identical network configurations — on the same LAN were working perfectly. Only the Synology running Roon Server had the problem.

First clue: Router Advertisements were not being accepted

After reboot, this command:

cat /proc/sys/net/ipv6/conf/bond0/accept_ra

returned:

0

That meant the NAS was not accepting IPv6 Router Advertisements on bond0.

When I manually ran:

sudo sysctl -w net.ipv6.conf.bond0.accept_ra=2

the NAS immediately processed the Router Advertisement, created the SLAAC address, and DSM’s own networking daemon, synonetd, automatically started the DHCPv6 client.

So DHCPv6 was not fundamentally broken. DSM was waiting to see/process the RA before starting the DHCPv6 client.

Second clue: DHCPv6 failed because systemd could not watch the PID file

When I tried to manually start the DHCPv6 client, it failed with:

Failed to set a watch for dhclient6@bond0-client.service's PID file /tmp/dhclient6-bond0-client.pid: No space left on device

So, this was not a disk-space problem; disk space and inode usage were fine. As it turns out, the problem was inotify watch exhaustion. The default Synology limit was:

fs.inotify.max_user_watches = 8192

Checking which processes were using inotify watches showed that Roon was consuming almost all of them:

8154  pid XXXXX  /Roon/app/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll ...

That left almost no inotify headroom for DSM/systemd. Roon was using nearly the entire default watch budget. As a result, systemd could not manage the DHCPv6 service properly.

FYI this is the command sequence that showed which processes were using the most inotify watches:

sudo sh -c '
for p in /proc/[0-9]*; do
  n=$(grep -hs "^inotify" "$p"/fdinfo/* 2>/dev/null | wc -l)
  [ "$n" -gt 0 ] || continue
  printf "%7d  pid %-6s  %s\n" \
    "$n" "${p##*/}" "$(tr "\0" " " < "$p/cmdline" 2>/dev/null)"
done | sort -nr | head -25
'

The fix

I made two sysctl settings persistent.

First, I backed up /etc/sysctl.conf:

sudo cp -a /etc/sysctl.conf /etc/sysctl.conf.before-roon-ipv6

Then I added these lines to /etc/sysctl.conf:

# Headroom for Roon plus DSM services
fs.inotify.max_user_watches = 65536

# Allow DSM-created interfaces to receive IPv6 Router Advertisements
net.ipv6.conf.default.accept_ra = 2

I applied the settings immediately:

sudo sysctl -p /etc/sysctl.conf

Then I rebooted the Synology.

Verification after reboot

After reboot, I checked:

sysctl fs.inotify.max_user_watches
cat /proc/sys/net/ipv6/conf/default/accept_ra
cat /proc/sys/net/ipv6/conf/bond0/accept_ra
systemctl is-active dhclient6@bond0-client.service
ip -6 addr show dev bond0

The good result was:

fs.inotify.max_user_watches = 65536
default.accept_ra = 2
bond0.accept_ra = 2
dhclient6@bond0-client.service = active

And bond0 had both expected global IPv6 addresses:

inet6 2001:db8:1234:5678::5/128 scope global dynamic
inet6 2001:db8:1234:5678:xxxx:xxxx:xxxx:xxxx/64 scope global mngtmpaddr dynamic
inet6 fe80::xxxx:xxxx:xxxx:xxxx/64 scope link

In summary…

This looked like a pfSense or DHCPv6 problem, but it wasn’t. In my case, there were two separate issues:

  1. Roon Server on Synology was consuming nearly the entire default inotify watch limit.
  2. DSM was bringing up bond0 without accepting Router Advertisements unless I set net.ipv6.conf.default.accept_ra = 2.

Once I increased the inotify watch limit and made RA acceptance persistent, IPv6 survived reboot cleanly. No router/pfSense changes were needed.

For anyone running Roon Server on Synology and seeing strange IPv6/DHCPv6 behavior, I recommend checking both of these:

sysctl fs.inotify.max_user_watches
cat /proc/sys/net/ipv6/conf/bond0/accept_ra

Just in case a non-Synology user stumbles here, it should be noted that modern kernels (5.x) shipped with, for example, Ubuntu or Fedora, adjust the value of 8192 automatically depending on available memory. Typically up to 1 million.

Point well taken. I suspect Synology might as well, because it seems to be able to track changes automatically now with my ~65,000 song collection once fully booted, but I am not sure where this capability starts happening in the boot sequence (at kernel level, one would assume it’s early) vis-a-vis Router Advertisement responses and Container Manager booting Roon. I know it’s not 1 million because this didn’t work as consistently when I had 370,000 files in my music directory due to Synology’s sidecar (@eaDir) files…

I think Synology and SNAP ship with legacy kernels intentionally, but I can never understand why 8192 is the default as any software that accesses lots of files is going to hit that limit.

The last time I experienced this was on my HP Proliant Mini server running Ubuntu 16.04. Both Squeeze Server and my cloud backup needed more watches. A sensible staring point nowadays would be 128k, which consumes a massive 1MB!

This is news to me. Thanks, @mjw!

@DDPS - I thought you were using the approach I recommended for modifying inotify on Synology. I think it might be better for most people than modifying sysctl.conf.

It’s here:

@gTunes no, frankly, yours is a great reference, but I just never cared about the limit in my installation before. For this purpose, I prefer modifying sysctl.conf rather than relying on a script to run, especially since sysctl’s work will happen at early boot when the system is initially trying to negotiate the router advertisements. FWIW, early on in my discovery process, I had considered simply scripting sysctl -w net.ipv6.conf.bond0.accept_ra=2 but that seemed wonky to me :slight_smile:.

Got it.

I prefer the task approach because I tend to lose track of modifications I make to system files. The task is easy to find, modify, delete, and it has the benefit of sending a notification to you via email every time it runs. But both approaches get you to the same end state :blush: