Some Words regarding Support & Service in connection with Build 880/882/884

I am one of the probably many “corrupted core database” victims of the last Build 880/882/884 update. With more than 10,000 well-maintained albums including changed metadata, I am not willing to just start with a fresh database. I am not a programmer, when I buy a database solution I expect the software manufacturer to understand his job and take it seriously. I don’t need every design update. For me it is important that the software works reliably. As a user, I also have duties, e.g. to take care of system security and backups. I do that continuously.

Now that the roles are clear, I will record the current status: We all have received an automatic update. On my mobile phone I have got used to installing updates 3 weeks later. But not with Roon. Since the update I have to reboot my NUC-Roon every 45 minutes because the system tells me, “There was an issue loading your database”. Unfortunately, importing one of the previous backups so that the loss of data is limited does not help. For me that means: Either the file format of Build 880 is completely different and does not go well with the old one, or the database has been inconsistent for a longer time.

It is unacceptable for Roonlabs to shift responsibility to its paying users in such a situation. In this situation, it is unfair, as has happened in some cases, to accuse your customers of having caused this problem with their hardware themselves. And it is unprofessional not to implement a clear communication strategy in such a situation.

I am optimistic that Roon Labs will act more professionally in the future, but I have a clear demand: A follow-up update should eliminate the problems of Build 880/882/884 without anyone losing any data. In cases where this is not possible, a tool should be provided with which you can export the metadata from your old database and read it into the new database. Speaking of which: That would be a useful feature, which would also be of help to users in other situations.

Last but not least: I would recommend not to close or delete this discussion. I would appreciate it if this discussion takes place exclusively within the Roon community and does not publicly discourage new users of this, in my opinion, fantastic music software.

40 Likes

Well said freejazz but I think the damage is already done and Roonlabs is in damage control mode. Perhaps the release of a major upgrade before the Christmas Holiday was with good intentions, however it’s turned into a nightmare in my opinion. My guess is available Staff heading into the holiday is limited.

Personally, my support topic that has been posted well over 48 hours has not even been acknowledged by RoonLabs support staff. Very disappointing.

For a young Company that needs word of mouth support from their customers this will have a negative impact.

Mason

6 Likes

Roon have released a fix for an issue that was causing corrupted databases to go undiscovered until they failed catastrophically. The remedy was to kill the database and allow it to rebuild if you did not have an uncorrupted backup. The fix means the vast majority of us with uncorrupted databases will now not necessarily suffer that fate. Sadly it means that those with still functional but corrupted databases have to rebuild that database if they don’t have an uncorrupted backup. The unemotional response to all of that is that it was a necessary course of action that was going to negatively impact a small but not insignificant number of people. That is the long and short of this. Except for that previously mentioned small but not insignificant number of people, this is an overwhelmingly positive move by Roon. It is just a shame it seems that the ‘400’ can’t be offered a less disruptive way of restoring their builds.

2 Likes

Henry,

Will have to agree to disagree. Pardon me what is your relationship to RoonLabs?

My DB running very smoothly until the 882 upgrade. My hardware is all new, 2021 vintage.

This arbitrary “400” number this forum keeps throwing around is undocumented and for the moment convenient for the purposes saving face for RoonLabs.

I glad your feeling all warm and fuzzy about your experience. But for the rest of the Peon’s who are experiencing less than favorable results excuses don’t cut it.

I have been working on computer, networks and sever based computing for over 20yrs, as far back as Novell. There is more going on here than a DB correction.
Mason

11 Likes

Very well stated freejazz!

2 Likes

Having never had occasion to visit this forum in the 4 years I’ve been using Roon - without issue - but now having been returned to year zero, I’d be interested to know the extent of the problem they were trying to fix with 880. Did the forum echo daily to people complaining of “corrupted databases failing catastrophically”? Is the fix worse than the problem ?

The way this has been handled has been comically inept, but for all I know, Roon is run by half a dozen guys n gals from a garden shed with no budget for schmoozing. For £120, it’s still fantastic value to me, given I’m long retired, and play music all day long.

My son - who keeps telling me he is a whizz with all this stuff - will try to get Roon up at running when he’s here at Xmas. A nation holds it’s breath !

4 Likes

Is the fix worse than the problem ?

That is the million dollar question.

Send my regards to East Tytherley.

Mason

I’m just a customer since 2017 and someone who tries to contribute here and on one of the Facebook groups. And someone who has a little bit of experience of large scale roll out of firmware updates into people’s homes enabling me to take a more impartial view of this. I know this is of absolutely no consolation to anyone affected but this was Roon doing their job. Finding an issue and fixing it but at the expense of some who were unknowingly impacted by the issue they were trying to fix. I fully understand people’s upset but it doesn’t change the fact that this problem needed to be addressed.

3 Likes

The company I worked for implemented a Change Freeze at key times of the year to prevent this…

Xmas, Easter, financial year end etc

6 Likes

Your support thread has been updated and your dB issues are probably related to the underpowered server you are using 300k tracks on a celeron.

2 Likes

Hi Henry,

Thank you for your comment. I am fully aware that with every update there are always individual cases in which some extremely rare boundary condition causes problems. But, I find it a bit irritating how confidently a number “400” is repeated here in many places. At the moment, Roon Labs is obviously not in a position to describe the specific causes and effects in connection with the destroyed databases. So, this number can only be very arbitrary.

It can be assumed that a large number of customers do not know the forum or do not express themselves here (e.g. lack of language skills and technical understanding). Also, Obviously, many users gave up and reinstalled very early. In this respect, this number is very doubtful and I am very curious to see how many people really are, or at least were.

Anyone here have more believable assumptions?

3 Likes

To be honest, percentages of customers having issues doesn’t mean anything to me. For the one who has trouble it’s just 100% of missing out the fun.
I have a running ticket of the function “Downmix as needed” inside of Roon which is broken longer than half a year ago and they just don’t solve this problem. It’s just plain broken. See this post https://community.roonlabs.com/t/re-quad-files-no-longer-recoginzed-as-4-0-but-as-other/160033
Instead there was a complete overhaul from 1.7 to 1.8 with more “goodies”. That might be the choice from a commercial point of view. Hoe cares about one customer. That it is an issue in my use of case doesn’t seem to matter at all. And that’s the most frustrating point - at least in my case.

5 Likes

The heading of my forum post could have been supplemented by the “Build 884” published today, because there is neither a repair nor a way to restore existing data (lists, reviews, history, etc.). I’m afraid that Roon Labs will no longer deliver here. As I laboriously read through some more technical threads, I also slowly begin to understand the problem.

However, I think that a sign of at least a halfway professional attitude is still necessary. Not here, fragmented and hidden as an answer in some forum posts. It would be better to email the customer that something has gone wrong (for a long time), which has now been corrected with the 880 update and unfortunately leads to some users losing their data.

5 Likes

The figure of 400 is incorrect - see the final group of posts in this thread:

I’m not sure how anyone can calculate even an approximate number, however. As has been pointed out there are many Roon users who don’t post in these forums or contact Roon support.

As I understand it there have been two principal sets of issues: dealing with corrupt databases (the subject of the current complaint) and missing albums. For many, such as myself, the latter was fixed by the build 882 update.

1 Like

Ouch. Sorry to hear that a number of you have discovered database corruption during the recent Christmas eve (doh) update. I’d like to offer a theory that Linus Torvalds (Father LINUX) would likely endorse.

Ars Technica on Linus on ECC ram mandatory

Most of us are running our Roon Core on equipment without memory error detection or single bit error correction. We expect Core to continuously run correctly on equipment not capable of doing so. Soft and hard error rates in solid state memory and solid state storage are well known and well documented. Sooner or later, a bit will flip in a place where it matters. That error will go undetected. If it is in data to be written to disk, the kernel happily writes it with parity and forward error correction calculated for the erroneous data. Disk error checking can’t fix main memory issues.

I’m fortunate to be running Roon Core on an Intel Xeon part having ECC and an operating system that supports ECC (TrueNAS FreeBSD distribution). I’ve had relatively little trouble with Roon core, mostly service fades while access points and switches are updated. I’ve been careful to ensure that Roon is actually saving backups.

1 Like

I concur with your point Mike - many co’s/projects I have worked on over the years have change freezes in the lead up to a main public holiday to avoid the holiday depleted support team have to deal with a major problem.
I would.also point out that Issue triage (and I mean an iTIL ‘Issue’ - where a root cause causes multiple tickets) often takes a back seat in a depleted support team, leading to poor resolution processes and user comms.

1 Like

Roon Database Integrity Checking
Like some others on this thread, I have a background in delivering major business software solutions in industries (e.g. finance) where system failure is “expensive” - it costs real money. We didn’t rely on hardware (or software) working reliably and put in place software to check that any database corruption was detected early enough so that it could be fixed. This included:

  1. Storing counts and/or hash totals of data in the database, e.g counts of the number of albums; the hash value of track titles in each album, etc.
  2. Checking the integrity of links in the database, e.g. for Roon, that every record track has an album or playlist, or the tracks in a playlist actually exist, etc.
  3. Providing a text i.e. human readable, export of the data from the database from which, in extremis, the database could be recreated. This allowed corruptions to be corrected if needed
  4. Developing “Integrity Checking Software” to carry out these checks at the same time as developing the system - then keeping it up to date and using it before each new release. This often helped fix problems with software before a new system or new release went live.
  5. Running these checks frequently to identify a problem and before your older backups were deleted and the problem couldn’t be fixed.

I don’t know if Roon does these kind of checks, but if they did it would mean:

  1. Errors would be identified even if backups aren’t being run
  2. Memory errors (as @David_Hamby suggested) causing database corruption would be identified by checking hash totals.
  3. No impact on performance by running the checks. They could be run every night, when Roon is not being used to play music.

Now I can understand why Roon Labs might not want to export a text version of their database as it would make it much easier to reverse engineer Roon but it is not that hard to do the rest.

However, the benefits of following such an approach are not “visible” - there’s no obvious benefit that the end user can see - at least until something goes wrong as it seems it recently has. So developing such a “belt and braces” approach to checking database integrity may not be a high priority.

7 Likes

Hello David,

you mentioned the point that you understand, why Roon Labs does not want to export text of the database. Here, I completely agree with you. I also would not like to see others using things for free while I am financing these things with my license fee.
But, I am only thinking of metadata the user wrote/managed by themselves: Ratings, playlists, own artist descriptions, notes etc.
Not that exhausting work if you use a traditional database management system. With LevelDB that seems to be a challenge.

Best regards
Toralf

1 Like

I’ve been away for Xmas and I’m a bit confused what this is all about. I only picked this up from the community summary email. I bought Roon for music not to spend hours on forums.

I would expect an email from Roon explaining the risk of upgrading etc. I’m reluctant to start my system now if this is still an issue.

Communicate!!

4 Likes

Either the cat is dead, or it’s not, but there’s no way of knowing until you check. Chances are you’ll be fine.

2 Likes