Thunderstruck at Dismal Manor

Tinkering
1

What happened

We had a round of thunderstorms at Dismal Manor. Not terribly bad but there were some flashover clearing events. When the dust had settled and the rain had stopped, the singleton boot device on my FreeNAS box was corrupt and FreeNAS would not start.

This is not a big deal as the storage pool is on separate media and all of the metadata to reconstitute the storage pool is in multiple spots in the array devices. FreeNAS would try to start, find system files borked, and fail to start. It clearly diagnosed the problem to the boot media.

How FreeNAS helped

FreeNAS is designed to allow replacement of the boot media and easy recovery of the array. If you make a system configuration save and squirrel it away safely, complete recovery is as easy as replacing the boot media, installing from a thumb drive, and restoring the configuration snapshot. Your shares, jails, VMs, etc are restored.

FreeNAS boot media use the ZFS file system and may be a RaidZ0 device (redundant pair). if one fails, it is likely the other is good. In my case, I had a singleton boot device and the power drama caused a write to go bad, probably stomping on a file not being updated. When the machine restarted, the process using the file trapped

ZFS is one of a small number of file systems that checksum both the payload and the file metadata. This made it pretty easy for ZFS to catch the bad bit and call a crash without creating collateral damage or a visit to the illegal instruction or bus error trap handlers. It knew what was wrong and could say rather than leaving clues to a puzzle.

So, FreeNAS users, visit System -> General and save a configuration. Squirrel away a copy on your cloud storage (iCloud, Google, or MS) that you can reload should recovery become necessary. Update your saved configuration each time you change the system configuration.

You did have a configuration save?

I didn’t have a configuration save so I had to recreate the shares and the VM running Roon Server. This was easy when I first did it in 2017. Well, it is 2018 and Debian and FreeNAS have both been updated. There was a glitch in the FreeNAS VM stuff. It now required hand creation of the guest install root directory. Once past that, Debian 9.4 networking would not come up. Ubuntu had similar problems. CentOS 7 had different problems. I couldn’t get a current VM to run.

The old Mac Mini gathering dust on the shelf is now a Ubuntu 18-04 LTS box running my Roon Server and life is again good.

If you’re not an OS tinkerer

If you’re not a tinkerer, buy the recommended NUC or Nucleus. These products have two big advantages. First, Roon worries about the software, all of it including a Linux distribution tailored to Roon. No apt-getting the dependencies.

The second is that Nucleus is designed to be part of a music player. The premium cost is in the power supplies. Effort goes into keeping both power and ground clean so ground grunge doesn’t creep out to affect attached DACs or receiver/preamps. Those of us home brewing compensate by transformer coupling the SPDIF coax or using TOSlink optical SPDIF.

2

This sounds like an advert.

3

Interesting post. Thanks. ZFS user here, running on Ubuntu server. This is also where Roon core runs. Been using the same box for four years; have replaced one failed disk in that time, which ZFS recovered gracefully (have redundancy in the array).

In short, +1 for ZFS.

4

It is. FreeBSD and FreeNAS are da bomb. :slight_smile:

5

Glad your disk transplant was successful. That’s why I’m a fan of FreeNAS

Did you put in the same size or market catch of the day best value size? Wondering if the pool can be enlarged as bigger disks become available. Smallest disk in the array determines size but a rolling replace is possible.

6

Did you put in the same size or market catch of the day best value size?

Hi,

Same size. I’m running RAID-Z2 with 7 x 4TB disks. It formats out to about 18TB usable; I am thinking about how I could add more storage (it would mean shuffling everything off to temporary storage, adding a couple of disks, and doing a rebuild). 4K HDR movies are killing me.

In reality, I think I’ll wait for the prices on 8 or 10TB disks to come down some more, and do a new machine (4 years of always on with consumer grade hardware is pretty good, I think; asking another 4 of the mobo might be a bit much).

I thought this http://blog.fosketts.net/2017/07/10/zfs-best-filesystem-now/ was a pretty good write up. He talks about how enlarging a pool is technically possible, but not necessarily straightforward.

7

Before my FreeNAS box, I had a couple of Drobo Gen 2 DAS that the FreeNAS box replaced. When a disk failed, I put in what every was sweet in the market, always bigger than what was replaced. When all four disks finally grew to the new smallest size, the array was magically larger. I believe FreeNAS will do the same. Consider replacing disks with the largest sanely priced disk currently available. The smaller ones tend to get pricy when they go out of manufacture.

8

That makes sense. My back up box is a bit like that; has a bunch of whatever-was-best-value stuffed into it, running as JBOD on ZFS. I figure I’d have to be very unlucky to lose an server with two disk redundancy and a completely separate machine as well.

Now that I’ve typed that, it will probably happen tomorrow … (but I think my USB disk copy is only three months old… :slight_smile: )

Cheers.

9

I have a replication process going to an 8TB USB disk on the same box. Not doing backups in the usual sense, just an up to date image. Thinking is that a failure during a backup disk replacement is unlikely.

10

Right – same philosophy here, more or less.

11

David, congrats on getting your file server and VM up & running again.

I suggest monthly cold storage backups of all configurations, file server, firewall, etc. there is also a freenas script around that automatically saves & timestamps configurations nightly to the main pool.

12

I have a replication to a USB disk large enough to hold the active part of the array. I’ve been saving the configuration when I make changes and archiving in a folder on the array and on iCloud.

A common mode failure like the house burning down would be bad. To provide some mitigation, I have photos saved to Google Photos and MACOS Documents and ~/Library setting saved in iCloud. Not fully covered but passwords (1Password), books (Banktivity) etc are saved. Music collection is not archived off site. Something to follow up on since I have some out of print LPs and about 600 or so CDs.