FreeNAS/TrueNAS changes afoot at iX Systems

[David_Hamby]

Am I the only crazy who runs his Roon Core in a FreeNAS VM?

iX Systems, makers of FreeNAS software and enterprise grade storage appliances is in the process of rebranding FreeNAS as TrueNAS and rearchitecting the product distribution into small, mid-scale, and enterprise versions all running the same code base. The current products have three separate code bases with three documentation sets. TrueNAS is looking to simplify its development and test workflow by consolidating the code base and documentation set.

The small and mid-scale products will continue to run under the FreeBSD operating system. The enterprise system is designed to run on a large scale Linux cluster.

Version 12 is the revised architecture. Product roll out will be in the late fall. The small and mid-scale products should look familiar. There is no talk of changing the host OS or virtualization subsystems of these products. The Enterprise product is new and allows TrueNAS to scale past the limits of current storage area network based TrueNAS systems.

According to Wikipedia, FreeBSD implements the Linux ABI in the FreeBSD kernel. This allows the Enterprise feature set Linux image to run directly on FreeBSD. FreeBSD 12 continues with BHYVE virtualization and no change of UI is expected so VM rebuilds should not be needed.

Base functionality is available without a license key. Unlocking of mid-scale and large-scale features will require a paid license.

Advantages of running Roon Core on TrueNAS:
FreeNAS provides very robust file service to networks of any scale from home and office to enterprise work groups and maybe larger.

On small scale systems the store can be single or dual failure protected (Raid Z2) and is expandable. The store can be backed up by replication to a commercial service like Backblaze B2, a second FreeNAS server, or to an internal disk provided for that purpose. USB external disks didn’t work well as a replication target. The USB disk firmware would take too long to wake the disk from hibernation when a replication transfer started.

Roon Core runs in a Virtual Machine. The VM mounts the music library storage as a VM device. A second VM device mounts the backup directory. Running Roon Core in the VM ensures that Roon starts after the file server is ready to serve files.

Issues I’ve encountered:
The only fly in the ointment is that the VM can become confused by power glitches from windstorm flashover clearing. The controllers will hang. A restart of the Roon VM corrects the issue.

From time to time iX Systems changes the FreeNAS/TrueNAS subsystem to track developments in the underlying FreeBSD and Linux operating systems. A system update can force a Roon Core re-installation but this is relatively easy and is a relatively rare and announced occurrence.

Separate Roon Core Experience
I’ve tried Roon Core on a 2009 Mac Mini both under MacOS and Linux. The issue was that Mac OS was not designed to run network services like Roon Core. I had to leave a user logged in on the console so the shares could be mounted. It won’t mount them at boot but it will start Roon Core at boot. Doh.

With Linux, Roon Core would run as a service and Linux would mount the shares at boot up but I had to take care that the file server was ready before booting the Linux Roon Core machine. A NUC or Nucleus would have this same issue with network storage. Roon’s installation instructions did not cover automounter configuration for Roon.

[Staffan_Jonson]

I have my library on a FreeNAS and I run Roon Server on a TrueNAS with a Bhyve VM running an Ubuntu Server for Roon Server. Works decent, I have some problems at reboots, where I might have to reboot Ubuntu and sometimes the Roon Server even if I have the Ubuntu VM to autostart with TrueNAS. It might have to do with that i have my primary DC on a Win Server VM on the same host. I also somehow have old albums show as newly added all the time. I havent figured that one out yet. It might have to do with that I ran the Roon Server on a NUC before on my profile, and it cleans out old saved once. Dunno. I would love if Roon and Free/TrueNAS would cooperate and make an official plugin directly on FreeBSD instead.

[David_Hamby]

Ahoy, Staffan. You have a domain controller? Does it authenticate users mounting your media shares, Roon Database, etc? You definitely could have a race going on as VM boot order is unspecified and concurrent vs sequential start is unspecified.

I’ve had relatively little trouble. Here’s what I think I’ve done.

Roon VM is resident in a ZVOL in the server’s main pool – VM creation requires ZVOL for system volume.

Roon library, Roon Database, and Roon Backups are resident in tier one filesystems (were created as datasets) in the server’s main pool. I created these as datasets in the storage editor.

Datasets storage resources can grow as needed within the pool’s free space. A ZVOL or volume is used to create an upper bound on the size of a storage resource and the space is reserved in the pool at creation. Other than that, the two have similar, maybe identical, behavior.

The VM mounts Library, Database, and Backups as either filesystems or volumes. Either appears to work. This design bypasses Samba races and file I/O is native fast rather than CIFS slow. The key here is that the VM Add Device Dialog needs to have visibility of the pool filesystem or volume.

The big realization was that the library needed to be a CIFS share for adding media from MacOS but that the Roon Core could mount the ZFS ZVOL or Dataset as a VM device. This solved all of my issues with races and boot order.

In your environment, your media management client would CIFS mount the Roon Library using DM to determine access rights. Your TrueNAS server would also use the domain controller to authenticate users and determine access rights. I have to take some pains to synchronize a user name and staff to use for media management.

There is some terminology confusion in the UI still. Storage view of the pool uses different terminology than the VM Add Device dialog. TrueNAS folks are slowly running this to earth.

The other thing to watch is that the file server network address rather than its hostname should be used configuring storage. Roon appears to be uneven in its use of MDNS to map host names to host addresses. It appears to miss dismal.local (my TrueNAS host) lookups some of the time.

Thoughts on NUCS and Rocks and such: The VM with mounted devices is a superior solution in terms of stability and reliability as all data is kept in redundant ZFS file systems. If you had Roon Core on a NUC or similar, the NUC would have to start after TrueNAS was up and CIFS was up. When I had my Roon Core on a Mac MINI I continually had the problem of out of order starts and hangs when the Roon Core got ahead of FreeNAS. With Roon Core in a VM, this race problem is engineered out.

[Scott_Muckleroy]

I have been running Roon Core in the native QNAP app from the beginning and has worked great. Now I am considering moving from QNAP to TrueNAS. Do you have all spinning disks or do you have an SSD pool for things that need the extra speed? Do you know if USB devices can be presented to TrueNAS VMs(Not related to Roon env)?

[Harrison_Murchison (Harrison M)]

Nice thread. I’m thinking about building a TrueNas system with a Silverstone CS381 12-Bay case

If Roon Core is stable in a VM then that would be ideal since I need to get some experience with them.

[Scott_Muckleroy]

I’ve read good things about that case. I think the review said that it can be a little tight routing cables and case can be hard to find? I was going to build my own also but the new TrueNAS Mini’s are a nice package at a decent price. Was there a particular reason you didn’t want to go with a TrueNAS Mini?

[Harrison_Murchison (Harrison M)]

Yes, thankfully there are good reviews out there that prepare you for the smaller PS, the tight routing and most importantly how to properly plan your CPU choose given that cooling will need to be addressed. I feel confident that it’s a case I can grow with. I also was considering QNAP but I figured I’ve have a go at setting up a ZFS system

[David_Hamby]

I’ve been running TrueNAS Core 12 just fine. If you like building machines, go ahead. Consider AMD Zen 2 hardware from Gigabit. Just confirm that the MB supports ECC (has the traces and bios bits). Gigabit is good about this in the specs but not the marketing slicks. Use a low end APU (with graphics) IPMI is optional if you have a display about for maintenance. I use the TV (a pain).

If not, you’ll be hard pressed to beat iXsystems TrueNAS Mini systems. I use Raid Z2 based on reports of a second disk failing while a failed disk replacement was being rebuilt in the Raid Z1 configuration. Either way memory and storage will drive the cost. You’ll be paying retail while iXsystems pays OEM volume schedule prices.

Are you a Dune fan? Good to enrich House Vernius! Appropriate with the new movie expected in 2021.

The TrueNAS core systems are modestly spec’d processor wise but one is targeted at media serving with a bit more processor. All are ECC and all use the WD Red+ file server grade disks. Note that you’ll need a 6 slot enclosure for Raid Z2.

The SSD cache devices are optional at low home workloads.

If you need more storage down the road, you can add an external disk shelf. iXsystems offers the proper interfaces and shelves. Today, I would buy from iXsystems rather than home brewing.

Do spring for ECC memory as the machine will run continuously and will have the occasional soft memory error from background radiation. ECC will correct all 1bit errors of this nature and will detect most multibit errors. Radium and daughters in the ground and building materials is the root cause.

I have 12 TB usable (way more than I need but there is a Time Machine spool on the beast also) with 4TB or so used. I replicate to an 8TB 7th internal disk for local backups.

What many appear to do is to make their older server the replication target when they build a new one. So far, I’m not feeling the need to do that.

I really like it that I can mount the media library as a VM disk rather than pulling media in via Samba/CIFS. This avoids all the race issues where Roon becomes ready for CIFS mount before CIFS is ready.

[Yethal]

@David_Hamby How do you mount a dataset inside VM in TrueNAS? When I go to Devices menu of my Roon VM the only disk device I can choose are zvols, not datasets. Do I need to create a zvol and move all my music there in order to be able to mount it in the VM?

[David_Hamby]

It appears I was mistaken. Three+ years of tinkering have become scrambled in memory. The screen shots here apply to MacOS BigSur, the current Roon, and TrueNAS 12 Core on an E3 Xeon.

Reviewing what I have running, I have a zvol that is the VM boot device. That is the only disk in the VM. Disks can be a pool or a volume, but not a filesystem to use TrueNAS 12 terminology.

The file server’s served storage pool, FreeNAS, is organized as shown below. The pool contains filesystems for Media, Photos, and Roon Backups.

This pool has the following SMB shares for media plus another couple for Time Machine. The Roon interface lets you mount local disks or SMB shares. At the time of the original build, I put everything in a single pool.

In Roon I’m SMB mounting the shares to ZFS filesystems in the served pool.

I have a second pool used for backups. This pool has a single 8TB disk that is a replication target for the main pool.

I have my media split into 3 groups, iTunes (ALAC) stuff, HD Audio (FLAC), and Qobuz (FLAC) purchased from Qobuz. Each is in a separate share. Not strictly needed as I could share the common parent. This makes it easy to list what came from where using file system utilities. This is not strictly necessary as Roon can find stuff by location in the UI.

When I was setting this up, everything was in flux UI wise and documentation wise. Things are now stable and naming more like that in general computing.

The nice thing is that the server is sorted before the VM starts and Roon mounts the shared media properly.

Do watch user names and user IDs as big system access control is present and can be awakened unintentionally. Some documentation refinement is needed here.

Roon runs as root so has global read write permissions to the VM accessible files. Keep other treasures like photos in a separate directory tree within the share.

[Yethal]

Thanks for the detailed answer, I kinda suspected this would be the case since documentation doesn’t mention that possibility. I managed to configure everything but I now ran into another issue. I created separate users for all services running in jails (plex, syncthing etc) and for the roon user. I then created a group called services and added all of those users to that group.
Then in Pool permissions settings I gave that group permissions for all actions except Delete and Delete children. Now all services running in jails work just fine so the permissions are okay (they might be too loose but I’ll fix that later).
Now when i try mounting the SMB share as the roon user (both from within Roon and from my own machine) it mounts fine but it reports that the directory is empty. I can see the files just fine when I mount the share as my own user who is a member of the dataset owner group. Here’s what my settings look like:

Add images

Sorry for asking about such basic things, I recently migrated from a Linux server + Synology nas setup so a lot of TrueNAS stuff is still arcane knowledge to me.

EDIT: NVM I figured it out. For posterity: if you run into the same issue as me (user has access to share but can’t see files) make sure you tick the apply ACLs recursively checkbox

[David_Hamby]

Check permissions!

First, PC user must be a server user with the same user number! Its the number that is in the directory entries, not the name.

Second, ACLs are for the pros. The filesystem edit dialog slips into ACL mode pretty easily. The control to put it back into basic mode is usually off the bottom of the browser viewport. Scroll down and return to Basic mode.

Third, this may be review, ensure directory nodes are RWX, RWX, RWX where the execute bit is taken as “traverse” (show the directory). Read to see what’s in the directory, W to delete, rename, and add files. It is usually the execute bit that trips people up.

Unless you get a domain controller up (Samba), you have to keep all the login stuff in sync by hand.

[Yethal]

Thanks! I already fixed that by the time you replied but you were absolutely correct, it was a permissions issue.

[Daniel_Lu1 (Daniel Lu)]

I don’t know if I am the only person have trouble in running Roon server on Freenas ubuntu virtual machine.
I am new to Roon, and I really like the software. Recently it is upgraded to 1.8 and I have no issue with that. It’s only became an issue when I wanted it on my Freenas.
I probably tried Roon server on every platform , the Roon server works great on windows, Mac, Linux. and the Rock can run on any PC I tried. To me it’s really great if Roon server can work on Freenas.
I am new to Linux, I haven’t used Linux for over 20 years. I installed ubuntu and Roon server on different test PC, it works great! but I would prefer Roon server to run on Linux wihin NAS.
The same ubuntu installations inside Freenas or Truenas virtual machine were successful, but the Roon server doesn’t work as expected. The Roon will work immediately after setting up, until rebooting the system. the symptoms are, after rebooting, the Roon will stop working, on the remote application will show these information when you try to play music:
“Playback was interrupted because a track fail to load…Too many failures… Stopping play back”.
(In fact, above error message also can occur in other occasions during normal use. for example, you moved or deleted a file in the library and then you tried to play them)
Naturally I tried to check the possibility of sharing issues, permissions etc, but it didn’t seem to be the problem. I can copy some songs to ubuntu VM local folders, still the same problem.
I do find a few solutions to this, such as , A. don’t ever shut down the computer; B. when it’s not working, re-install Roon server in the linux terminal, very simple, just copy and paste: sudo ./roonserver-installer-linuxx64.sh, and you are good to go before rebooting it. C. block the internet access of the virtual machine (ubuntu), Lan has to be working , since you need remote app to control it. To achieve this , I just set a wrong gateway in ubuntu then everything seemed to be fine and you can reboot the computer whenever you like. The obvious problem is the Roon server does need internet. (to download album info etc)
I probably installed dozens times of ubuntu on virtual machine , almost every time was same failure. by saying ‘almost’ , because I realised sometimes I would have good luck. It turned out to be some update of the Roon server builds are good!
For example, I didn’t have this ‘fail to play’ trouble in Roon server software (in Linux) build 753, 756, and 763, now it updated to build 764, the Roon server in VM machine would not work properly again (and I still can use above tricks to get around)
Anyway I think this is not something I can control. I finally decided use a PC ubuntu server, it never has such issues. yes, I managed to learn have to mount a second hard drive for my music library and sharing folders on home network.
Overall I like Roon very much, I can see lots of people complained about it after upgraded to 1.8. Consider I am a superficial user, I don’t have any of their troubles. I hesitated to purchase a NAD M33 because it’s not Roon certified, I found the Roon file transfer protocol is superior to airplay, Before I get a proper Roon certified endpoint, such as a streamer, I found it’s a good idea to install Rock on a old fanless minipc, for example I have several useless Dell WYSE mini PC with 16G SSD, it’s perfect to run Rock software and work as a Roon certified endpoint (and connect to DAC by USB). Also , a volumio on Pi is very good option to work as Roon certified endpoint too! Unfortunately , volumio on PC (such as on WYSE) is poorly supported, the Roon bridge plugin doesn’t work on x86, have to use airplay if you have to.

[David_Hamby]

There’s a lot in here. I’m not sure where to start. Maybe with your TrueNAS hardware choice.

TrueNAS CORE Hardware Guide

It is important to use server grade motherboards like those from Supermicro or ASROCK Rack. Xeon processors are recommended because they have appropriate reliability, availability, and serviceability features. ECC memory is recommended as machine is expected to run continuously. And storage array rated disks are recommended.

Second, update Ubuntu using apt update and apt upgrade commands in your server’s shell. Get Ubuntu up to date with security fixes. Use Ubuntu Server LTS rather than the yearly. No need to install the window system and desktop applications.

Third, follow Roon’s instructions carefully. Take notes! All design decisions, All packages added, etc so you can review against Roon checklists.

Fourth, move discussion to DIY/Tinkering. You’re tagged onto the end of a very old topic.

[Daniel_Lu1 (Daniel Lu)]

Thank you David.
While I am using my Ubuntu PC for Roon Server, I was still research solutions for it to run on Freenas virtual machine.
I don’t think there is anything to do with the hardware. I am using a i7 Pc with 32G ram. It’s a very stable PC, 24/7 working.
I did read from some where it was said I should allocate 8G memory for VM, and set 2 cores and 4 threads. I tried but it didn’t work, the Roon server stopped working just same as before.
It must be something to do with the network, somehow the Roon server might fail to talk to it’s remote server or fail to receive the ‘talk’ back.
Accidentally, when I reinstall the ubuntu on virtual machine, in the configuring this Linux machine (at the beginning) , there is a setting for ‘bind’ network card. This time I put in my freenas IP address (I have fixed address),otherwise , by default was ’ 0.0.0.0’, I never changed before, this time I changed it to something like’ 192.168.1.100’ (my Freenas IP address) .
Installing Ubuntu was the same and it took 10mins, after this, everything works fine. I can reboot the Freenas and it still works after restarting.
I tried on another backup NAS, which is a fresh installed Truenas. The result was same good. It seems finally a solution was found.

[David_Hamby]

TrueNAS Hardware Recommendations Guide

Were I commissioning a new TrueNAS server, I would forego the pleasure of DIY and buy a TrueNAS Mini in the larger case and provision for ZRaid 2. The iX Systems hardware is competitive with new from Amazon apples to apples.

If you must DIY, I strongly suggest running TrueNAS on server hardware meeting the baseline recommended in the hardware guide. No regular desktop PC meets the ECC requirement. ECC is error correcting memory that will correct all single bit errors and detect all 2 bit errors, and all are logged. Because storage servers have long running processes, ECC memory is a very important Reliability, Availability, and Servicability feature.

Similarly, you need a robust server motherboard from SuperMicro or ASROCK Rack. Again this issue is RAS. Sound, fancy video, overclocking, etc are all useless to TrueNAS.

My installation here runs TrueNAS on SuperMicro metal. iX Systems encourages this choice with Roon in a VM as a Linux Guest. That’s what I have going here. You’re putting the cart before the horse with Ubuntu and Roon on the metal and TrueNAS in a VM. iX Systems discourages this arrangement.

When I round up money for a make over, I’ll pick up a 2021 Apple Silicon Mac Mini for $700 and put Roon and the Music library in it and use Time Machine to back up my bits to TrueNAS. This works very very well. Time Machine data storage is a high priority TrueNAS use case.

[Gabriel_Schwartz]

I have the following TrueNAS hardware:
SuperMicro X9SCM-F, LGA 1155, Micro ATX Server board
Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
32GB RAM ECC
8TB x 4 Seagate IronWolf NA
SuperMicro AOC-S3008L-L8E 12Gb/s SAS 8 Port Internal PCIe HBA Controller Low Pro]
Silver Stone Technologies CS380B ATX tower case

Just started with Roon Server on Debian VM
I have allocated 2GB RAM and it works fine so far as I don’t do any DSP (prefer bit perfect output to my DAC).
Do you allocate more than 2GB RAM?
My remote is on iPad Pro which also transports the audio via USB 3 Camera adapter.
Any suggestions are welcome?

P.S.
I have FLAC, ALAC, HiRes FLAC and DSD (no Tidal or something like that as I have iTunes music subscription).