Linux Roon server not playing high res files to Squeezebox [Fixed in 1.3]

Audio Gear Talk Squeezebox
101

So I created a VM, based on openSUSE Tumbleweed, and ran Roon.
Guess what: this works beautifully with 192K content.

So this means we can take a distribution specific issue of the table as well.

This also means that unfortunately I’m running out of options on what could be the issue, besides real hardware problems. Which, I still can not explain why they would kick in only when playing 192K and not with other content. And believe me, my Roon setup made quite the hours over the last year :wink:

I’m still gonna follow up on my remark about processor throttling. See if I can create a reproduction scenario on my VM.

On a side note: @brian while setting up I remembered you mentioning sometime ago working on a Roon specific Linux distro. And I was thinking when doing all this stuff…

  1. how about a Docker container? Not because it makes sense, but purely because it’s fun :slight_smile: Do you know if someone already did this? because otherwise I’m gonna give it a go.
  2. the tiny Roon distro… is this not something we can turn into a simple community driven project?

Regards,

102

This is why I’m suggesting a memtest to verify your ram is good throughout its addressable range.

103

You’ve read my remark the issue only appears when playing content besides 192K right?
If it would be memory then it seems logical that the issue would appear ‘all over the place’.

But, in general, you’re right and I need to test it to rule it out.

104

@all,

sorry to have been absent. Was fighting another battle. Synology took 4 days to incorporate 2 new 3TB disks, everything went dark with dashboard Volume at 100%.
Roon was inaccessible, switched it off eventually.
After all is up again, the NAS dashboard is totally calm, no excessive use of anything, and still the whitenoise persists.
I need to re-read and chew on the previous posts, all way too technical for me.

105

ok @Eric and @Carl_Seibert I officially give up. Reinstalled my server, exactly the same as I did with my VM earlier this week, and boom there’s the noise again :frowning:

Investigated memory, investigated CPU throttling and performance in general. Investigated networking/infrastructure… everything ok.

For now I revert back to 96K max.

106

Actually, I think that is exactly where @spockfish is at the moment. What Motherboard/CPU/Memory are you using?

107

Just shoot me now.

I installed a second hard drive in the machine that hosts my files. I installed Ubuntu Server 16.0.4 on it Installed RoonServer and practically nothing else. Then I mounted the original hard drive as a data drive. It’s at the same mount point in both OSes, so I didn’t have to reconfigure a ton of Samba stuff.

And it fails. But a little differently now. Now I’m seeing failure to noise followed by an interval of music, failure to noise for a while, then music again for a while. Seconds, a minute or so tops.

Now this machine is not very powerful, compared to the Fedora machine on which I’ve been trying to run RoonServer.

[root@asok ~]# lscpu
Architecture: i686
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
Vendor ID: AuthenticAMD
CPU family: 15
Model: 75
Model name: AMD Athlon™ 64 X2 Dual Core Processor 3800+
Stepping: 2
CPU MHz: 2008.927
BogoMIPS: 4017.99
Virtualization: AMD-V
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K

It has only two gigs of RAM.

At first, I was getting noise on both 96K and 192K files. I had the data mounted via CIFS. I changed that to a local file system mount and things improved a little, to where 96K files play alright, but no change at 192K. This felt like lack of resources to me, so I tried lessening processor and memory load a little by turning off background processing and reducing memory reserved for pictures in Roon. Might have helped a little, but didn’t resolve the issue.

I tried setting bit rate limiting. If I limited the bit rate to 48K, my 192K file played perfectly. Despite the increased processor load, go figure, right? Limiting bit rate to 96K didn’t help - same pattern of failure and music in intervals.

Whether playing a 192K file, a 96K file, with bit rate conversion on or off, system load looked like this:

root@asok:/home/carl# ps -aux | grep RoonServer
root 2003 0.0 0.0 12540 1900 ? Ss 16:35 0:00 /bin/bash /optRoonServer/start.sh
root 2009 0.0 0.4 544288 13920 ? Sl 16:35 0:01 /opt/RoonServe/Mono/bin/RoonServer --debug --gc=sgen --server RoonServer.exe
root 2020 80.5 33.6 1762752 1036668 ? Sl 16:35 80:05 /opt/RoonServe/Mono/bin/RoonAppliance --debug --gc=sgen --server RoonAppliance.exe -watchdogport=33452
root 2027 0.0 0.0 7328 780 ? S 16:35 0:00 /opt/RoonServe/Server/processreaper 2020
root 2050 0.0 0.2 830992 8836 ? Sl 16:36 0:03 /opt/RoonServe/Mono/bin/RAATServer --debug --gc=sgen --server RAATServer.exe
root 4262 0.0 0.0 14228 980 pts/0 S+ 18:15 0:00 grep --color=auto RoonServer

That’s CPU hovering just under 80% and memory at about 30%, give or take a point or two.

Now as I write this, I’m playing to an Oppo HA-2, through my laptop as a Roon endpoint, and a 192K file plays perfectly, with a nice purple “lossless” signal path. Which suggests to me that the meager power of this server machine isn’t really the issue. UPDATE: I just checked and indeed, streaming to the Roon endpoint does take less CPU - 56.6%, compared to ~80%. Memory use is actually up a little, at 33.4%. Hmmm. But before I get excited, I remember the Fedora machine has ample resources.

I suppose I could go back to one of the virtual machines and cut back memory and CPU capacity in increments until it fails to see if I can make the same failure mode.

EDIT: If I could make Roon fail on a VM, I’d be happy to send you guys the whole VM, so you’d have it. But so far, it seems bulletproof on VMs. At least I suppose I’ve shown that it is possible that Harry’s machine and my original one are not the only two in the world with this problem. Maybe you will be able to replicate.

Yeah, I thought “network” again. But there’s nothing in really common between Harry’s environment and mine and both fail. Everything in the network where I’m getting failures is also there (and more) when I put a VM on that same network. So that thought exercise didn’t go anywhere.

Hopefully, I’ve at least injected a few new data points into the mix and maybe something will ring a bell with somebody. Hopefully. To tell the truth, I did this so I might be able to just listen to music for a while without having to mess with IT work every time I sat down to listen. But that was not to be, I guess.

108

And in case… here are all the log files from RoonServer on the new machine:

109

And another promising thought leads nowhere…

@brian Your post got me thinking.

Based on what we’ve seen, especially the fact that it doesn’t appear that I’m having resource issues on the server end, and that we’ve eliminated the network as a suspect (although a 192K PCM stream stresses my setup right to the max) I figured it would be a reasonable guess that we overwhelming the I/O capabilities of the Squeezebox. Then the least little thing happens and kerplewy.

So the least little thing appears to be happening after RoonServer and before the NIC of the server’s hardware.

So I thought “buffer”. It turns out that there are buffer settings on the client side, exposed in the EDO UI. I couldn’t find any documentation as to exactly WHAT buffer, but what the heck. So I tried all of the available settings. And no joy.

So then I set the transmit buffers on the server box to about 8MB (About the double the old max value and some 40x the default, which was 212K-ish. And. Still no joy.

Now what I knowing about buffers wouldn’t fill one, so I could be on the right track but not going in the right direction. So this line of thought might still be of value.

And the lower level reason may yet provide a clue: We know that we’ve never made a failure on a virtual machine, even when the virtual machine is the same OS version as the host it’s running on, and RoonServer directly on the host fails. So, our signal path, so to speak, is going from RoonServer through the virtual machine’s kernel and network stack, through some virtual machine magic that functions as a router, then from the virtualization software to the host machine and out its network stack, and on through the same NIC as when the system fails. And despite all that complexity and latency, it works. In other words, passing through the OS once fails, but twice succeeds! Thus my thought that doubling the size of the transmit buffer might help. It didn’t, at least the way I did it. But what else could there be, through which two passes instead of one might be doing us some good?

BTW, I think that streaming as FLAC may be of great help. We know that the little processor in the Touch can decode FLAC at 192K. But I/O-wise, 24/192 PCM is four times more than the device was designed for, and about ten times more than was probably routinely sent through it. (At least at my house. I stopped sending PCM from the server when the Touch came out. It was a benefit with the old SB-3s, but with the more powerful Touch, the only effect seemed to be network load.)

I hope maybe something here rings a bell.

In the meantime, back to cleaning up. One of my cats barfed on my favorite headphones.

110

I agree regarding the FLAC streaming.

My impression is, without FLAC streaming, the SB Touch is very near “the edge” of its networking capabilities at 192k. When it goes over the edge, the symptoms happen. Moving to VM, and other things that you guys are testing is pushing it closer or further from the edge a little bit, but they’re not central to the issue.

From our perspective, the real next step here is to test again once FLAC streaming support rolls out with 1.3.

111

That sounds sensible. Agreed about the Touch working at the edge of its capabilities.

I would love to know why on earth we’re OK with virtual machines and not real ones. But if it’s to become a moot point, it’s just one of those things. We could put our effort to better use elsewhere in the interim. (Like adding Focus to Radio :slight_smile:

Any timeframe for the release of 1.3?

112

I don’t know for sure, but…

There’s a phenomenon that happens when you take a machine with a fast network port and ask it to spray packets at a machine with a slower one. Sometimes, you end up with lots of packets dropped on the floor–usually by some node (switch or wifi router) in between the two that’s mediating the speed difference, but that doesn’t have enough buffer to smooth things out.

At minimum, this creates extra work for the people on both sides doing flow control (the TCP stacks in the two kernels, in this case). It may mean more packets need to be resent, and it may cause the connection to be throttled more intensively by the flow control mechanism.

VM’s virtualize networking and almost certainly slow things down a bit. Maybe it’s enough to make it “ok”.

If you were really curious, the way to investigate would probably be to look at what’s going on from the Touch’s perspective. As I recall, it’s pretty easy to get a shell on the machine. The normal linux kernel stuff is going to be there, they have logs, etc.

I’m not brave enough to hazard a guess as to when 1.3 will ship. We’re working hard on it…it will be out when it’s done.

113

@brian, interesting thought. That’s just what I did. I’ve logged in to the Squeezebox and looked around. And of course I played with a few (kernel) settings as well. So far no luck, but I’m not gonna give up.

Actually I was wondering too what the difference could be in the setup. My VM experiment (and my MBP for that matter) are all accessible via WiFi and my SB Touch is wired on 100MB.

But my Linux RoonServer is on a wired 1GB network. So coming weekend I’ll dive into this further, and do an simple experiment by putting my RoonServer also on 100MB.

114

Looked around, played around. Weird thing: when I set my Linux RoonServer to 100 MB, full duplex, I had dropouts. Which is weird, as my Linux RoonServer VM is accessible only via WiFi.

Anyhow, no further luck in playing 192K content.

However, snooping around on the Squeezebox Touch, it made me think. It’s accessible via SSH, and rather open. It would, theoretically, be possible to run different software on the Squeezebox Touch.

So I could see this solution where you install a ‘Roon Applet’ on the device (that must be written in Lua) which pulls in additional stuff.

Gonna see if I can cross compile something and check if this runs.

115

Guys, I think I’ve got it working!

Need to be careful, but I’m already hallways Supertramps’ Crime of the Century in 192K.

Bottom line: the SBT is running out of steam with it’s default settings, so it needs to be tuned to get things properly working.

So what have I done: I started with increasing the default Linux kernel network stack buffer sizes. They are rather small. However that was not enough.

Then I found on the internet some info on the TT plugin. Digging further I found a set of kernel parameters that I’ve tried and that did the trick.

Now, this is all rather ‘hackish’, but it would be really cool if @Carl_Seibert can test this as well.
@Carl_Seibert If you’re up to this then ping me, and I’ll provide you with a script that takes care of things.

116

update: played 3 albums without any problem.

3 Likes
117

Great you got it working, still mightily puzzling that it works when using VM. All said though I’d flog the Touch, get an ODROID or something similar and be done with it. If you need the DAC capability you can add a Hi-Fi shield. The ODROID’s hi-fi shield is a step up from the Touch and you should have quite a bit of change left after selling the Touch.

118

This was not about ditching the Touch. It will be ditched in the (near) future, but I could not stand that it would not play 192K.

119

Yip, it’s puzzling because a number of us have no issue playing 24/192 content with EDO installed.

120

Private message sent.