ROCK on an Intel NUC booting straight into UEFI Shell – and the rather painful fix

I thought I’d share this because it took far more messing around than expected to get an existing ROCK installation booting on an Intel NUC. :sweat_smile:

In the end the actual ROCK installation was perfectly fine. The problem turned out to be the tiny EFI system partition and, more specifically, the way it had been formatted.

Hopefully this saves somebody else several hours of head scratching.

The hardware

The NUC has a 256 GB Western Digital SN520 NVMe SSD:

WDC PC SN520 SDAPNUW-256G-1006

The SSD already contained what looked like a complete ROCK installation. I therefore didn’t want to simply wipe the SSD and reinstall everything before finding out what was actually wrong.

The symptom was simple:

The NUC would not boot ROCK.

Instead, after powering it on it dropped straight into:

UEFI Interactive Shell v2.2

with:

EDK II
UEFI v2.70 (American Megatrends)

So at first sight it looked like either the SSD wasn’t bootable, the EFI bootloader was missing/corrupt, or the NUC simply wasn’t seeing the correct boot device.

First suspicion: wrong M.2 slot or SSD not detected?

Because the NUC went directly to the UEFI shell, one of the first thoughts was whether the NVMe might have been installed in the wrong slot or whether the BIOS/UEFI didn’t properly recognise it.

That turned out not to be the case.

The UEFI shell showed the NVMe device as several BLK devices, but there was something odd:

The internal SSD did not get a usable filesystem mapping such as FS0:.

The only proper filesystem mapping was the USB stick we were using.

Commands such as:

map

map -r

connect -r

map -r

showed the NVMe hardware and its partitions as block devices, but the EFI partition itself wasn’t being exposed by the firmware as a filesystem that we could browse.

That was the first useful clue.

Booting Arch Linux to inspect the SSD

At this point I booted an Arch Linux live USB so we could inspect the SSD without modifying the ROCK installation.

Running:

lsblk -f

showed something very encouraging.

The NVMe contained four partitions:

/dev/nvme0n1p1   100 MB    EFI

/dev/nvme0n1p2 16 GB ext4 ROONBOOT

/dev/nvme0n1p3 16 GB ext4 ROONAPP

/dev/nvme0n1p4 ~206 GB ext4 ROONDATA

So the ROCK installation was clearly there.

ROONBOOT, ROONAPP and ROONDATA all looked normal.

That immediately changed the diagnosis: there was little reason to reinstall ROCK.

The problem was almost certainly isolated to the 100 MB EFI partition.

Checking the EFI partition

We mounted it manually:

mkdir -p /mnt/roon

mount /dev/nvme0n1p1 /mnt/roon

and checked its contents.

The expected EFI bootloader was actually present:

/EFI/BOOT/BOOTX64.EFI

Even more importantly, checking it with:

file /mnt/roon/EFI/BOOT/BOOTX64.EFI

reported:

PE32+ executable for EFI (application), x86-64

So we had:

  • a valid GPT disk;

  • an EFI System Partition;

  • an apparently valid x86-64 EFI executable;

  • the complete ROCK partitions;

  • an NVMe SSD which both Linux and the UEFI could physically see.

And yet the NUC firmware still refused to treat that EFI partition as a bootable filesystem.

This was the point where things became interesting.

Running fsck on the EFI partition

We then checked the FAT filesystem:

fsck.fat -v /dev/nvme0n1p1

And this finally produced the clue that led us to the solution.

The EFI partition was formatted as FAT32, but fsck.fat reported that it only contained:

25544 clusters

while the required minimum for FAT32 is:

65525 clusters

The warning was essentially:

Filesystem is FAT32 according to fat_length and fat32_length fields,

but has only 25544 clusters, less than the required minimum of 65525.

This may lead to problems on some systems.

That last sentence turned out to be rather important:

“This may lead to problems on some systems.”

Apparently this NUC’s UEFI was one of those systems.

There was also an invalid volume-label entry for ROONEFI.

We first let fsck.fat remove/fix that invalid label entry and checked the filesystem again.

No luck.

The NUC still dropped into the UEFI shell.

At this point the theory became pretty clear

Linux could mount the EFI partition without any real problem.

The EFI executable itself was valid.

But the NUC firmware did not create a filesystem mapping for the partition.

So the most plausible explanation was that the firmware’s FAT driver simply didn’t like this technically dubious 100 MB FAT32 filesystem with too few clusters.

Instead of reinstalling the whole thing, we decided to rebuild only the EFI filesystem.

Obviously, before doing that, the existing EFI executable had to be saved.

Backing up the ROCK EFI bootloader

From the Arch environment we copied the existing bootloader to the live environment:

cp /mnt/roon/EFI/BOOT/BOOTX64.EFI /root/BOOTX64.EFI

and verified it:

ls -l /root/BOOTX64.EFI

The file was approximately:

2,977,792 bytes

So now we could safely recreate the filesystem without losing the ROCK bootloader.

The actual fix: FAT16 instead of FAT32

We unmounted the partition:

umount /mnt/roon

and reformatted only the 100 MB EFI partition as FAT16:

mkfs.fat -F 16 -n ROONEFI /dev/nvme0n1p1

There was a harmless warning about lowercase labels/DOS compatibility, but the filesystem was created successfully.

Then we mounted it again:

mount /dev/nvme0n1p1 /mnt/roon

recreated the standard EFI directory structure:

mkdir -p /mnt/roon/EFI/BOOT

and restored the original ROCK bootloader:

cp /root/BOOTX64.EFI /mnt/roon/EFI/BOOT/BOOTX64.EFI

followed by:

sync

and finally:

ls -l /mnt/roon/EFI/BOOT/

The result showed our original:

BOOTX64.EFI

back in place, with the expected size of approximately 2.98 MB.

Moment of truth…

We then unmounted it:

umount /mnt/roon

and rebooted:

reboot

The Arch USB stick was removed during reboot.

I was fully expecting to see that bloody UEFI shell again. :grinning_face_with_smiling_eyes:

Instead:

ROCK booted immediately.

No UEFI shell.

No manual boot command.

No reinstall.

Straight into:

Manufacturer: Roon Labs

Model: Roon Optimized Core Kit

OS: RoonOS 2.1 (build 271) production

Hostname: ROCK

and it obtained an IP address normally.

Opening the ROCK web interface showed:

Operating System

OK

Version 2.1 (build 271) production

and:

Roon Database & Settings

OK

with roughly 205 GB available, exactly as expected from the existing ROONDATA partition.

The only remaining message was Missing Codecs, which is a completely separate/easy issue and has nothing to do with the boot problem.

So what was actually wrong?

As far as I can determine, the ROCK installation itself was never broken.

The NVMe was fine.

The M.2 slot was fine.

The GPT partition table was fine.

BOOTX64.EFI was present and valid.

The problem was the 100 MB EFI System Partition formatted as FAT32 with only 25,544 clusters.

Linux was perfectly happy to access it, but the NUC’s UEFI apparently wasn’t.

That explains the confusing behaviour: the firmware could see the NVMe and its block devices, but it wouldn’t expose the EFI partition as a usable filesystem and therefore never found:

\EFI\BOOT\BOOTX64.EFI

Recreating that same 100 MB EFI partition as FAT16, while leaving the rest of the SSD completely untouched, fixed the problem immediately.

For anyone finding this through Google

If your ROCK NUC suddenly/consistently boots into the EDK II UEFI Interactive Shell, while the SSD itself is still detected, I would definitely not immediately wipe the SSD and reinstall ROCK.

First boot a Linux live USB and check:

lsblk -f

If you still have partitions resembling:

ROONBOOT

ROONAPP

ROONDATA

your actual ROCK installation may be perfectly intact.

Then inspect the EFI System Partition and check whether:

EFI/BOOT/BOOTX64.EFI

exists.

And particularly on a small EFI partition, run:

fsck.fat -v /dev/nvme0n1p1

Obviously substitute the correct device name for your system.

If you get the warning that the filesystem claims to be FAT32 but contains fewer than the minimum 65,525 clusters, that may be the reason some UEFI implementations refuse to recognise it.

In my particular case the solution was:

backup BOOTX64.EFI → reformat only the EFI partition as FAT16 → recreate /EFI/BOOT/ → restore BOOTX64.EFI → reboot.

And after many hours of screwing around with UEFI shells, USB sticks, mappings, BLK devices, fsck, bootloaders and wondering whether the NVMe was even in the correct slot…

ROCK finally just booted as if absolutely nothing had ever been wrong. :joy:

Hopefully this helps somebody else avoid the same adventure.

3 Likes

Your report and findings are very useful for someone experiencing troubles! Highly appreciated!

But, i don’t understand why you simply didn’t re-image RoonOS?
It takes all of 3 minutes or so, and leaves you with an identical “appliance”?

Fair point — with hindsight, simply re-imaging Roon OS would probably have been the quickest solution. :grinning_face_with_smiling_eyes:

But at that point we didn’t know that the EFI boot partition was the actual problem.

I had only bought this NUC second-hand the day before, including the NVMe drive and RAM, so I had no idea yet whether the hardware itself was completely reliable. On top of that, it initially had BIOS 0098 installed, and while troubleshooting we found quite a few reports from other NUC owners having UEFI/boot-related problems with that BIOS version.

So when the NUC suddenly booted straight into the UEFI Shell, there were quite a few unknowns: the NUC itself, RAM, NVMe drive, BIOS settings/version, the ROCK installation, partitioning, or the EFI bootloader.

Only after going through all of that did we establish that the hardware was fine, ROCK itself was actually present on the NVMe, and the real problem was the EFI boot partition/bootloader.

So yes, knowing what we know now, a fresh image would almost certainly have saved us many hours. But as we say in Dutch: “Achteraf kijk je een koe in z’n reet.” :joy:
Roughly translated: hindsight is always easy.

At least after this rather painful exercise I now know the NUC, RAM and NVMe are working properly, the BIOS is sorted, and exactly what went wrong with the boot process. And hopefully the detailed troubleshooting in this thread will help somebody else who ends up staring at the same UEFI Shell someday. :grin:

3 Likes