Roon OS woes, build 1.0.221 [resolved]

What happened

We released a new version of Roon OS (for ROCK and Nucleus), and while the majority of installs were fine, a small number were left not-booting.

With Roon OS 1.0.221, our primary goal was to implement support for NUC10 devices with ROCK, which included the requirement of updating the Linux kernel. In recent months we’ve seen an increase in requests for this support, and @danny stated last month that this build was in more advanced testing outside our company’s lab:

This Roon OS build was with our alpha testers for nearly a month, and was running without issue on 50 to 60 devices of various models and configuration. In addition to this alpha testing, our 5 person QA team performed extensive systematic testing on a wide range of hardware, as is our standard for these types of releases. Our non-QA staff had also been running this at home on our Nucleus and ROCK devices during this time without any problem.

Ultimately our testing resulted in no sign of any issues. We released Roon OS 1.0.221 in the hopes of fulfilling the numerous requests we’ve received about NUC10 support.

Within 2 hours of release about 1,000 users updated to Build 221 and, at that time, we’d just seen 3 reports of problems coming in that we began to investigate. After 12 hours the number of users updating was over 5,000, and we’d received about 20 reports of problems.

Our development, QA, and support team spent the entirety of the day investigating the reports for this issue that appeared to be affecting roughly 0.4% of Roon OS users to understand what was causing this problem.

After hours of investigating and gathering data, our development team began working with a few users who were kind enough to try some beta builds of Roon OS to help us resolve the problem. After several beta builds were tested today, we received unanimous reports of success from one build, which has been released to stable.

The Issue

The maintainers of the Linux Kernel made some changes in this version to be more strict about standards compliance for NVMe drives. It appears that a tiny fraction of SSD units were “dinged” by this change, and the kernel was refusing to interoperate with them. This prevented Roon OS from seeing the drive during installation and startup.

Why didn’t we test with this hardware that was affected?

We did test it, but our hardware didn’t exhibit this issue. This makes sense given the relative rarity of this issue. In our QA lab, there are devices that match the exact RAM, SSD, and NUC as affected users, and none of them exhibited the issue. Even after receiving reports from users, we were never able to reproduce in-house. It’s hard to say why this is, possibly a revision of the chips, or just slight performance or firmware-version differences between SSDs. It is clear that there are variations even among the same hardware that were affected while others were not.

Next steps for those affected

ROCK

For those running ROCK, you’ll want to download the latest version here and run the recovery function. Note that if you install instead of recover, your database and settings will be lost, so it’s important you choose the correct option.

After running the recovery, everything should function properly again.

Nucleus

Nucleus can be resolved using the same method outlined above. The primary difference is that with Nucleus you want to take extra care to use the recovery options and not install. Installing RoonOS over the current Nucleus firmware can lead to further issues.

For step by step instructions, please see this article. The only difference is that, instead of installing, just choose the recovery option and you’ll be all set!

Important Note - If you have an internal SSD for storage, please disconnect this drive before recovering

If you have any issues please let us know!

Final Note

If you didn’t already take this update, or if you updated previously with no issues, you don’t have to take any of the steps outlined above and all should be functioning properly.

25 Likes

Kudos for recognizing and addressing this so quickly.

4 Likes

Damn fun business to be in, ain’t it?

3 Likes

@xxx totally ruined my day… but I’m glad to have the NUC10 support out the door!

sheesh

11 Likes

Thanks for the prompt attention to this issue. I’ll be anxiously awaiting the fix for my Nucleus (can’t live without it!).

Good that you acknowledge the issue and explain the current situation. I’m a new Roon user since a month or so and seeing all the posts with people complaining about their ROCK not booting didn’t leave me with a very good impression. I guess this situation is exceptional and not every update turns out like this :laughing: Luckily I had a hunch this morning and didn’t update immediately. I’ll wait another day, just to be safe :+1:

1 Like

Thank you.
Recovered painlessly with this new version. BTW updated BIOS in my NUC.

Cheers, Kuba [edit: Ok, Kuba is my third name ;)]

It’s times like these I’m glad I have a standard sata SSD in my MOCK. What a nightmare to sort out. Well done for mailing it so soon.

Thanks for the quick fix. I’m back up and running

This is what worked for me

"Next steps for those affected

ROCK

For those running ROCK, you’ll want to download the latest version here and run the recovery function . Note that if you install instead of recover, your database and settings will be lost, so it’s important you choose the correct option.
[/quote]

Option 1: Windows / Mac OS / Linux

  • Plug in your USB flash drive (it must be larger than the factory reset image size – 1GB is more than enough).
  • Download and install Etcher from https://etcher.io/
  • Start Etcher
  • From the Etcher application interface, select your .img.gz file as “the image”
  • From the Etcher application interface, select your USB flash drive as “the drive”
  • Click on “Flash!”

Installing

  • Turn off your NUC.
  • Unplug any USB storage you might have plugged into it. You can plug it back in after your install.
  • Insert the USB flash drive you prepared earlier into the front of the NUC
  • Turn on the NUC and press the F10 key on your keyboard. Continue to hold the F10 key down until you appear at a menu that is asking you which device you’d like to boot from.
  • Select the USB flash drive.
  • Follow the instructions to install Roon OS. This procedure can take a few minutes, depending on the performance of your USB flash drive and the SSD to which you are installing.
  • Once it is done installing, unplug the USB flash drive and hit ENTER to reboot.
  • On next boot, don’t hold F10. Your NUC should boot from the SSD, and a few seconds later, you should see a Roon message displaying the IP address of the NUC. This IP address should be entered in a web browser (on another computer) in order to complete ROCK setup (installing Codecs).
1 Like

A post was split to a new topic: Can’t fix nucleus after 1.0.211

4 posts were split to a new topic: Having trouble booting ROCK installer

A post was split to a new topic: Trouble recovering ROCK

I have recovered Rock with the latest image file, and afterwards I could do the update without losing connection. Thanks for the fast workaround!

This new image works well.

Thank you!

Thanks to Dylan and the Roon team for the fast turnaround on the fix and Rodrigo for these essential detailed steps, I’m also up and running again!

However, I strongly urge that the Roon team acquire a Nucleus that is susceptible to bugs like this for testing of any future software builds. This was not fun. I purposely bought the Nucleus to avoid wonky problems like this and just enjoy music.

Ha! Good luck with that. If you read the analysis report, it is stated that the Roon Labs team did have the same hardware as the 0.4% of Roon users who were struck by this issue - and the Roon Labs team did not see the issue manifest itself on their hardware.

As they say, it is possible that certain SSD manufacturers have changed chips internally on certain of their SSD models…

Why didn’t we test with this hardware that was affected?

We did test it, but our hardware didn’t exhibit this issue. This makes sense given the relative rarity of this issue. In our QA lab, there are devices that match the exact RAM, SSD, and NUC as affected users, and none of them exhibited the issue. Even after receiving reports from users, we were never able to reproduce in-house. It’s hard to say why this is, possibly a revision of the chips, or just slight performance or firmware-version differences between SSDs. It is clear that there are variations even among the same hardware that were affected while others were not.

1 Like

Although I was not impacted by this issue (I have never had any issues with Roon), I did watch it unfold.
All at Roon need to commended for their quick response and hard work.
Roon found a needle in a haystack issue quickly, fixed it, tested it and released the fix within hours.

Well done all at Roon. You all need to give yourselves a pat on the back for an incredible job!

9 Likes

2 posts were split to a new topic: Failing to flash Roon OS image to USB stick

Agreed. Everyone needs to remember that issues are amplified on the forum. I don’t need 100% perfection, just 100% responsiveness and transparency, which the Roon team delivers.

5 Likes