Linux Roon server not playing high res files to Squeezebox [Fixed in 1.3]

ok @Eric and @Carl_Seibert I officially give up. Reinstalled my server, exactly the same as I did with my VM earlier this week, and boom there’s the noise again :frowning:

Investigated memory, investigated CPU throttling and performance in general. Investigated networking/infrastructure… everything ok.

For now I revert back to 96K max.

Actually, I think that is exactly where @spockfish is at the moment. What Motherboard/CPU/Memory are you using?

Just shoot me now.

I installed a second hard drive in the machine that hosts my files. I installed Ubuntu Server 16.0.4 on it Installed RoonServer and practically nothing else. Then I mounted the original hard drive as a data drive. It’s at the same mount point in both OSes, so I didn’t have to reconfigure a ton of Samba stuff.

And it fails. But a little differently now. Now I’m seeing failure to noise followed by an interval of music, failure to noise for a while, then music again for a while. Seconds, a minute or so tops.

Now this machine is not very powerful, compared to the Fedora machine on which I’ve been trying to run RoonServer.

[root@asok ~]# lscpu
Architecture: i686
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
Vendor ID: AuthenticAMD
CPU family: 15
Model: 75
Model name: AMD Athlon™ 64 X2 Dual Core Processor 3800+
Stepping: 2
CPU MHz: 2008.927
BogoMIPS: 4017.99
Virtualization: AMD-V
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K

It has only two gigs of RAM.

At first, I was getting noise on both 96K and 192K files. I had the data mounted via CIFS. I changed that to a local file system mount and things improved a little, to where 96K files play alright, but no change at 192K. This felt like lack of resources to me, so I tried lessening processor and memory load a little by turning off background processing and reducing memory reserved for pictures in Roon. Might have helped a little, but didn’t resolve the issue.

I tried setting bit rate limiting. If I limited the bit rate to 48K, my 192K file played perfectly. Despite the increased processor load, go figure, right? Limiting bit rate to 96K didn’t help - same pattern of failure and music in intervals.

Whether playing a 192K file, a 96K file, with bit rate conversion on or off, system load looked like this:

root@asok:/home/carl# ps -aux | grep RoonServer
root 2003 0.0 0.0 12540 1900 ? Ss 16:35 0:00 /bin/bash /optRoonServer/start.sh
root 2009 0.0 0.4 544288 13920 ? Sl 16:35 0:01 /opt/RoonServe/Mono/bin/RoonServer --debug --gc=sgen --server RoonServer.exe
root 2020 80.5 33.6 1762752 1036668 ? Sl 16:35 80:05 /opt/RoonServe/Mono/bin/RoonAppliance --debug --gc=sgen --server RoonAppliance.exe -watchdogport=33452
root 2027 0.0 0.0 7328 780 ? S 16:35 0:00 /opt/RoonServe/Server/processreaper 2020
root 2050 0.0 0.2 830992 8836 ? Sl 16:36 0:03 /opt/RoonServe/Mono/bin/RAATServer --debug --gc=sgen --server RAATServer.exe
root 4262 0.0 0.0 14228 980 pts/0 S+ 18:15 0:00 grep --color=auto RoonServer

That’s CPU hovering just under 80% and memory at about 30%, give or take a point or two.

Now as I write this, I’m playing to an Oppo HA-2, through my laptop as a Roon endpoint, and a 192K file plays perfectly, with a nice purple “lossless” signal path. Which suggests to me that the meager power of this server machine isn’t really the issue. UPDATE: I just checked and indeed, streaming to the Roon endpoint does take less CPU - 56.6%, compared to ~80%. Memory use is actually up a little, at 33.4%. Hmmm. But before I get excited, I remember the Fedora machine has ample resources.

I suppose I could go back to one of the virtual machines and cut back memory and CPU capacity in increments until it fails to see if I can make the same failure mode.

EDIT: If I could make Roon fail on a VM, I’d be happy to send you guys the whole VM, so you’d have it. But so far, it seems bulletproof on VMs. At least I suppose I’ve shown that it is possible that Harry’s machine and my original one are not the only two in the world with this problem. Maybe you will be able to replicate.

Yeah, I thought “network” again. But there’s nothing in really common between Harry’s environment and mine and both fail. Everything in the network where I’m getting failures is also there (and more) when I put a VM on that same network. So that thought exercise didn’t go anywhere.

Hopefully, I’ve at least injected a few new data points into the mix and maybe something will ring a bell with somebody. Hopefully. To tell the truth, I did this so I might be able to just listen to music for a while without having to mess with IT work every time I sat down to listen. But that was not to be, I guess.

And in case… here are all the log files from RoonServer on the new machine:

And another promising thought leads nowhere…

@brian Your post got me thinking.

Based on what we’ve seen, especially the fact that it doesn’t appear that I’m having resource issues on the server end, and that we’ve eliminated the network as a suspect (although a 192K PCM stream stresses my setup right to the max) I figured it would be a reasonable guess that we overwhelming the I/O capabilities of the Squeezebox. Then the least little thing happens and kerplewy.

So the least little thing appears to be happening after RoonServer and before the NIC of the server’s hardware.

So I thought “buffer”. It turns out that there are buffer settings on the client side, exposed in the EDO UI. I couldn’t find any documentation as to exactly WHAT buffer, but what the heck. So I tried all of the available settings. And no joy.

So then I set the transmit buffers on the server box to about 8MB (About the double the old max value and some 40x the default, which was 212K-ish. And. Still no joy.

Now what I knowing about buffers wouldn’t fill one, so I could be on the right track but not going in the right direction. So this line of thought might still be of value.

And the lower level reason may yet provide a clue: We know that we’ve never made a failure on a virtual machine, even when the virtual machine is the same OS version as the host it’s running on, and RoonServer directly on the host fails. So, our signal path, so to speak, is going from RoonServer through the virtual machine’s kernel and network stack, through some virtual machine magic that functions as a router, then from the virtualization software to the host machine and out its network stack, and on through the same NIC as when the system fails. And despite all that complexity and latency, it works. In other words, passing through the OS once fails, but twice succeeds! Thus my thought that doubling the size of the transmit buffer might help. It didn’t, at least the way I did it. But what else could there be, through which two passes instead of one might be doing us some good?

BTW, I think that streaming as FLAC may be of great help. We know that the little processor in the Touch can decode FLAC at 192K. But I/O-wise, 24/192 PCM is four times more than the device was designed for, and about ten times more than was probably routinely sent through it. (At least at my house. I stopped sending PCM from the server when the Touch came out. It was a benefit with the old SB-3s, but with the more powerful Touch, the only effect seemed to be network load.)

I hope maybe something here rings a bell.

In the meantime, back to cleaning up. One of my cats barfed on my favorite headphones.

I agree regarding the FLAC streaming.

My impression is, without FLAC streaming, the SB Touch is very near “the edge” of its networking capabilities at 192k. When it goes over the edge, the symptoms happen. Moving to VM, and other things that you guys are testing is pushing it closer or further from the edge a little bit, but they’re not central to the issue.

From our perspective, the real next step here is to test again once FLAC streaming support rolls out with 1.3.

That sounds sensible. Agreed about the Touch working at the edge of its capabilities.

I would love to know why on earth we’re OK with virtual machines and not real ones. But if it’s to become a moot point, it’s just one of those things. We could put our effort to better use elsewhere in the interim. (Like adding Focus to Radio :slight_smile:

Any timeframe for the release of 1.3?

I don’t know for sure, but…

There’s a phenomenon that happens when you take a machine with a fast network port and ask it to spray packets at a machine with a slower one. Sometimes, you end up with lots of packets dropped on the floor–usually by some node (switch or wifi router) in between the two that’s mediating the speed difference, but that doesn’t have enough buffer to smooth things out.

At minimum, this creates extra work for the people on both sides doing flow control (the TCP stacks in the two kernels, in this case). It may mean more packets need to be resent, and it may cause the connection to be throttled more intensively by the flow control mechanism.

VM’s virtualize networking and almost certainly slow things down a bit. Maybe it’s enough to make it “ok”.

If you were really curious, the way to investigate would probably be to look at what’s going on from the Touch’s perspective. As I recall, it’s pretty easy to get a shell on the machine. The normal linux kernel stuff is going to be there, they have logs, etc.

I’m not brave enough to hazard a guess as to when 1.3 will ship. We’re working hard on it…it will be out when it’s done.

@brian, interesting thought. That’s just what I did. I’ve logged in to the Squeezebox and looked around. And of course I played with a few (kernel) settings as well. So far no luck, but I’m not gonna give up.

Actually I was wondering too what the difference could be in the setup. My VM experiment (and my MBP for that matter) are all accessible via WiFi and my SB Touch is wired on 100MB.

But my Linux RoonServer is on a wired 1GB network. So coming weekend I’ll dive into this further, and do an simple experiment by putting my RoonServer also on 100MB.

Looked around, played around. Weird thing: when I set my Linux RoonServer to 100 MB, full duplex, I had dropouts. Which is weird, as my Linux RoonServer VM is accessible only via WiFi.

Anyhow, no further luck in playing 192K content.

However, snooping around on the Squeezebox Touch, it made me think. It’s accessible via SSH, and rather open. It would, theoretically, be possible to run different software on the Squeezebox Touch.

So I could see this solution where you install a ‘Roon Applet’ on the device (that must be written in Lua) which pulls in additional stuff.

Gonna see if I can cross compile something and check if this runs.

Guys, I think I’ve got it working!

Need to be careful, but I’m already hallways Supertramps’ Crime of the Century in 192K.

Bottom line: the SBT is running out of steam with it’s default settings, so it needs to be tuned to get things properly working.

So what have I done: I started with increasing the default Linux kernel network stack buffer sizes. They are rather small. However that was not enough.

Then I found on the internet some info on the TT plugin. Digging further I found a set of kernel parameters that I’ve tried and that did the trick.

Now, this is all rather ‘hackish’, but it would be really cool if @Carl_Seibert can test this as well.
@Carl_Seibert If you’re up to this then ping me, and I’ll provide you with a script that takes care of things.

update: played 3 albums without any problem.

3 Likes

Great you got it working, still mightily puzzling that it works when using VM. All said though I’d flog the Touch, get an ODROID or something similar and be done with it. If you need the DAC capability you can add a Hi-Fi shield. The ODROID’s hi-fi shield is a step up from the Touch and you should have quite a bit of change left after selling the Touch.

This was not about ditching the Touch. It will be ditched in the (near) future, but I could not stand that it would not play 192K.

Yip, it’s puzzling because a number of us have no issue playing 24/192 content with EDO installed.

Private message sent.

Hi. I am new to roon (on a test license currently) and have a pretty similar setup to the ones discussed here - unfortunately with pretty similar problems:

Setup:
Server: Linux Debian Stretch x64, Core i3, 16GB memory, wired per GBit to router, 115,000 songs on a local internal hard drive (ext4)

Audio endpoints:
Logitech Transporter (currently on WIFI)
MS Surface 3 pro (currently on WIFI)

Symptoms:
The MS Surface plays fine with all resolutions (tested up to 24/192).
The Logitech Transporter playes fine with 16/44 content, however, with 24/44 and 24/96 and 24/192 (downsampled to 24/96) it starts playing ok but then switches to white noise, sometimes goes back to music and then back to noise.
I tried switching Audio Analysis Speed (incl. off) with no change. I did powercycling with all equipment involved with no change. Wifi connection is good (it worked for years with LMS) - I may later change to wired connection but given the reports here I am not confident.
I also tried downsampling with no change.

Can I try further options? Are logfiles of any help?

Update: I now also tried a wired ethernet connection to the transporter with no change. In addition I also tried a Mac as an Audio endpoint. The Mac works flawlessly.

So the problem remains the combination of Roon server and Logitech Transporter with anything above 16/44.

I’m not sure how experimental you’re willing to be, but i’ve got a plugin over here, home backed, that at least makes it work on my SB Touch. It’s an update on the existing EDO plugin + some kernel enhancements. If you’re willing to try it out just say the word.

Does the Transporter even run the EDO plugin?