What happened
We released a new version of Roon OS (for ROCK and Nucleus), and while the majority of installs were fine, a small number were left not-booting.
With Roon OS 1.0.221, our primary goal was to implement support for NUC10 devices with ROCK, which included the requirement of updating the Linux kernel. In recent months we’ve seen an increase in requests for this support, and @danny stated last month that this build was in more advanced testing outside our company’s lab:
This Roon OS build was with our alpha testers for nearly a month, and was running without issue on 50 to 60 devices of various models and configuration. In addition to this alpha testing, our 5 person QA team performed extensive systematic testing on a wide range of hardware, as is our standard for these types of releases. Our non-QA staff had also been running this at home on our Nucleus and ROCK devices during this time without any problem.
Ultimately our testing resulted in no sign of any issues. We released Roon OS 1.0.221 in the hopes of fulfilling the numerous requests we’ve received about NUC10 support.
Within 2 hours of release about 1,000 users updated to Build 221 and, at that time, we’d just seen 3 reports of problems coming in that we began to investigate. After 12 hours the number of users updating was over 5,000, and we’d received about 20 reports of problems.
Our development, QA, and support team spent the entirety of the day investigating the reports for this issue that appeared to be affecting roughly 0.4% of Roon OS users to understand what was causing this problem.
After hours of investigating and gathering data, our development team began working with a few users who were kind enough to try some beta builds of Roon OS to help us resolve the problem. After several beta builds were tested today, we received unanimous reports of success from one build, which has been released to stable.
The Issue
The maintainers of the Linux Kernel made some changes in this version to be more strict about standards compliance for NVMe drives. It appears that a tiny fraction of SSD units were “dinged” by this change, and the kernel was refusing to interoperate with them. This prevented Roon OS from seeing the drive during installation and startup.
Why didn’t we test with this hardware that was affected?
We did test it, but our hardware didn’t exhibit this issue. This makes sense given the relative rarity of this issue. In our QA lab, there are devices that match the exact RAM, SSD, and NUC as affected users, and none of them exhibited the issue. Even after receiving reports from users, we were never able to reproduce in-house. It’s hard to say why this is, possibly a revision of the chips, or just slight performance or firmware-version differences between SSDs. It is clear that there are variations even among the same hardware that were affected while others were not.
Next steps for those affected
ROCK
For those running ROCK, you’ll want to download the latest version here and run the recovery function. Note that if you install instead of recover, your database and settings will be lost, so it’s important you choose the correct option.
After running the recovery, everything should function properly again.
Nucleus
Nucleus can be resolved using the same method outlined above. The primary difference is that with Nucleus you want to take extra care to use the recovery options and not install. Installing RoonOS over the current Nucleus firmware can lead to further issues.
For step by step instructions, please see this article. The only difference is that, instead of installing, just choose the recovery option and you’ll be all set!
Important Note - If you have an internal SSD for storage, please disconnect this drive before recovering
If you have any issues please let us know!
Final Note
If you didn’t already take this update, or if you updated previously with no issues, you don’t have to take any of the steps outlined above and all should be functioning properly.