Search fails or is extremely slow (search service failed: Result[Status=NetworkTimeout])

Roon Core Machine

# uname -a
Linux zfs 5.13.19-6-pve #1 SMP PVE 5.13.19-14 (Thu, 10 Mar 2022 16:24:52 +0100) x86_64 GNU/Linux
# lscpu 
Architecture:                    x86_64
CPU(s):                          8
Model name:                      Intel(R) Atom(TM) CPU C3758 @ 2.20GHz
...

Networking Gear & Setup Details

Gigabit Ethernet

Description of Issue

For the past several weeks, I have found that the Roon interface on my iphone is extremely slow. I have now taken a look at the logs in ~/.RoonServer/Logs/RoonServer_log.txt and observe a number of extremely slow responses from roonlabs.net servers. For example:

07/01 01:21:15 Debug: [easyhttp] [244] GET to https://search.roonlabs.net/1/search?q=nancy%20wilson%20heart&count=50&trace=true&c=qobuz-us returned after 14177 ms, status code: 200
07/01 01:21:15 Debug: [easyhttp] [245] GET to https://internetradio.roonlabs.net/2/api/stations/search?query=nancy wilson heart&count=50&format=msgpack& returned after 14200 ms, status code: 200
07/01 01:22:46 Debug: [easyhttp] [270] GET to https://search.roonlabs.net/1/search?q=nancy%20wilson%20daughter&count=50&trace=true&c=qobuz-us timed out after 15000 ms
07/01 01:22:46 Error: search service failed: Result[Status=NetworkTimeout]
07/01 01:22:46 Debug: [easyhttp] [271] GET to https://internetradio.roonlabs.net/2/api/stations/search?query=nancy wilson daughter&count=50&format=msgpack& returned after 15220 ms, status code: 200

I do not believe that these issues arise from my own network connectivity. For example, while simple requests to these same severs without authentication fail, the response from the endpoint occurs within 73ms:

$ time curl search.roonlabs.net
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

real	0m0.073s
user	0m0.013s
sys	0m0.005s

Thanks.
~Jacob

In other instances, albums have considerable delay before starting and sometimes fail to begin whatsoever. A current instance yields the following log:

07/01 01:44:59 Debug: FTMSI-B-OE qo/578A1030 created new req 10 for block 0 p 0; active requests 1
07/01 01:45:04 Debug: FTMSI-B-OE qo/578A1030 rid:10 interrupt requested; reason: should remove
07/01 01:45:04 Debug: [easyhttp] [433] GET to https://streaming-qobuz-std.akamaized.net/file?uid=1122726&eid=45053511&fmt=6&profile=raw&app_id=188245549&cid=1300350&etsp=1656643454&hmac=TotpkhQ-6vUHCP9_5jbaHulOsO0 canceled after 5002 ms
07/01 01:45:04 Debug: FTMSI-B-OE qo/578A1030 interrupted req 10; missing block 0
07/01 01:45:04 Debug: FTMSI-B-OE qo/578A1030 created new req 11 for block 0 p 0; active requests 1
07/01 01:45:04 Debug: FTMSI-B-OE qo/578A1030 rid:10 request ended -- first block: 0 blocks read: 0 download speed: -256kbps response time: -1ms

what are your DNS settings? suggest using 1.1.1.1 or 8.8.8.8 on the core

intel atom CPU is well under spec too…

1 Like

Hello.
Thank you for your note.

I’ve spent a bit more time looking into this and observe that these issues arise only when the Roon core has IPv6 connectivity to the internet. I cannot yet rationalize how this affects the various roonlabs.net endpoints as I do not see AAAA DNS records for any of the logged ^.*.roonlabs.net/ hostnames. I have not narrowed down whether an HTTP redirect occurs to another hostname that does have a AAAA record.

The Qobuz endpoints do have AAAA records but requests go unreplied. For example, the following packet received no response.

21:20:30.266867 IP6 (flowlabel 0x24356, hlim 64, next-header TCP (6) payload length: 40) 2001:470:X:X:X:X:X:X.56024 > g2600-141b-e800-000c-0000-0000-172c-8599.deploy.static.akamaitechnologies.com.http: Flags [S], cksum 0x9e01 (incorrect -> 0x2b4d), seq 3945294784, win 64800, options [mss 1440,sackOK,TS val 1099107902 ecr 0,nop,wscale 7], length 0

I am using 8.8.8.8 for DNS.

FWIW, the Atom CPU line ranges widely in performance. This system is not CPU-bound.

Thanks again.
~Jacob

PS: How does one disable markup on this forum? It has made a terrible mess of my verbatim command line output.

Welcome to the community, @Jacob_Joseph!

This specification falls way below the minimum requirement to run Roon Server. You’ll need an i3 or equivalent to satisfactorily experience Roon.

Also, running ZFS on a low-spec machine isn’t ideal; typically you should allocate 1 GB memory for every TB of storage over and above normal system requirements. So, a 4 TB mirror would need 8 GB plus 2-4 GB for Linux.

Sluggish response times is an indicator of an underpowered core.

Whilst you may want to check DNS as @wizardofoz suggests, I’d look for other bottlenecks, especially if your library has grown since starting out with Roon.

1 Like

four spaces at the beginning of the paragraph achieve the same.

Example: CommonMark Spec

Martin,
Thank you for your note.

I do suspect, however, that the line of reasoning so far here is tilting at windmills.

I believe I already addressed aspects of this CPU in my prior response. Similarly, 15 second network responses have little to do with local CPU speed. To reiterate, the Intel Atom range varies widely in performance. This is, of course, true of the Intel Core i3 as well.

As for ZFS, you are correct to intuit from the hostname that I use ZFS. I disagree with your memory calculations and do not follow your position. FWIW, this host has 128GB of RAM.

Thanks again.
~Jacob

Maybe, but single core performance is not great, and much less than that of an older i3.

Roon works best with high single thread performance.

Turn off ipv6 and work from there

Hi @Jacob_Joseph,

Just checking in to inquire whether disabling IPv6 helped at all? As @wizardofoz and @Martin_Webster have mentioned you’re running your Core on an underpowered Core. From checking your diagnostics if appears that it only has 2gb of RAM as well.

If disabling IPv6 on your router doesn’t help I’m afraid you’re going to have to consider migrating your Core to another machine. I’m moving your thread to Tinkering, as choosing to continue on an underpowered machine qualifies as such. Thanks.

Hello.
I can confirm that disabling IPv6 on this host (via /proc/sys/net/ipv6/conf/eth0/disable_ipv6) resolves this issue. While this is a “fix,” having IPv6 connectivity is not particularly unusual and should not cause Roon to fail to function.

I’m happy to help debug the network aspect of this by providing additional network traces.

Thanks. ~Jacob

This topic was automatically closed 45 days after the last reply. New replies are no longer allowed.