Isn’t this what beta testing is supposed to be about? There has been comment that the problems with 1.8 are because roon did not do enough beta-testing with enough people. (Though given Danny’s comments elsewhere one has to wonder if the problem wasn’t that roon simply didn’t want to listen to its beta testers.)
(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)
Roon 1.8 was tested by both Alpha and Beta groups. I know because I Moderated the feedback from both groups.
2 Likes
They must have been in deep sleep mode…Or not well compiled (echo chamber?)… So the auggestion above still makes a lot of sense. Have participated in many beta programs. Small fry always slips by. But this?..
Is it that much fun to demean people you don’t even know?
5 Likes
No I dont know them, but they obviously didnt do the greatest of all jobs.
1 Like
And you are not doing the greatest of jobs at probability. There are around 250K Roon systems according to Roon Labs. Each of those systems is essentially unique because no two current computers or LANs are exactly the same. Let’s suppose there’s a bug that affects 100 of the 250K, and let’s suppose that there are around 100 testers. What’s the probability that a tester will experience that bug? ~1.6e-07 QED.
It is one thing to say that in your opinion testing was inadequate. It is quite another to say it never took place.
2 Likes
Agreed. Good to know it took place. Pity that it still missed so much. Nough said.
I’ve shifted this discussion into its own thread as it should not derail the Feature Request where it started.
People involved in testing for Roon have agreed to confidentiality obligations. I can tell you that neither Alpha nor Beta were echo chambers. Alpha, in particular, takes a weird pride in breaking Roon.
It is difficult to have a discussion where one side has no knowledge of the facts and the other cannot reveal them. Suffice it to say that a testing program that is guaranteed to find every bug is not a realistic aim.
I imagine they did a good a job as you would have done, i.e. reported any bugs that they discovered, so dissing the beta team won’t help. However, this does beg the question as to how representative the beta testers are of the Roon population as a whole. For example, how did the tagging issue slip by them? Don’t they use tags? How is it that none of them spotted the issue regarding labelling tracks that have been added to playlists? Don’t they use playlists? Again, I’m sure they did a good job, but maybe Roon need to think a bit more about who they ask to test new releases.
1 Like
I was in the beta period for a short time. Roon worked without a problem and most of the feedback involved minor bugs and some difficulties compared to 1.7 regarding the UI. However the major bugs reported in the commnity after the 1.8 release were either not reported as feedback during the beta period or they were fixed. The major crashes and bugs reported in the community seem to be affecting a minority of the users which i hope and am sure will be fixed by Roon. Other than most of the feedback regarding UI falls in the feature request section. I hope the pitfalls of 1.8 compared to 1.7 willl be improved by roon
1 Like
Do you consider that a bug though? I completely agree that it makes no sense to us why they were removed, but to me it sounds like it was an intentional move to get rid of those, just like they got rid of the the album star ratings in the Album view and in discographies. It was said somewhere they did that on purpose because they already organized the albums by best ratings in the discography. I see it more as a design choice that backfired. Or do you think this stuff was just forgotten about?
1 Like
This sentence is a tautology.
But in fact, it is not a question of whether computers are identical, but whether they are properly programmed and configured - according to strict and unambiguous rules.
Because indeed the point is, the clutter on on any malfunctioning computer is fully different…
AND/OR breaking with tags and UI things like adding artists requiring to type the name twice, or tab sequence totally broken in recording date edit are not system dependent and would be immediately apparent to anyone trying the feature just once.
1 Like
No, strictly speaking, neither are bugs, they’re a deliberate change, but neither make much sense. The change to tags broke a whole bunch of bookmarks, and the whole tagging process is now a convoluted nightmare in comparison to 1.7. Same for playlists: it’s now impossible to tell if a specific track is part of a playlist, without checking your playlists.
So, either the beta team didn’t care about the change, or didn’t notice because they don’t use either feature, or they did notice and report it but Roon went ahead and changed them anyway. I hope it’s not the latter.
2 Likes
I think it is important to distinguish between pre-release testing which looked for bugs/crashes and UX/functionality appraisal.
The former will never have a testing population able to represent every remote/core/endpoint combination, and the number of users effected by serious issues seems small compared to a userbase of almost 250k.
However, that the latter did not expose quite important changes in functionality or that Roon simply ignored those reports - tagging, focus, shuffle limit, playlists etc… is a clear indication that Roon must reappraise this aspect of their development cycle.
I have every confidence that they will do so.
I was a member of the preview “beta”, which cannot be considered part of pre-release testing, as access was given just a few days before release.
Yeah, that’s a good point. If changing or removing a feature causes other features to break or to become useless I would consider that a bug. It is hard to imagine that no one noticed this though through the whole process of development and testing. It is a strange thing to do as well.
1 Like
The beta testers may or may not have picked up on the issues where bugs were listed. Are you trying to make a specific point, [Moderated]
1 Like
We can all second-guess their testing strategy, but such Monday-morning quarterbacking is really an exercise in futility since we don’t have anywhere near the enough information to judge.
It’s entirely possible (I’d say likely) that Roon knew this would be a difficult release: whenever you change UI dramatically, it’s always challenging. Beta testing can have many uses, and it varies significantly depending on the dev team and company policies. Some use it as PR with a little debugging. Some run long, tightly-controlled alpha and beta cycles that dovetail with the in-house testing to eliminate inevitable real-world surprises (and require large internal teams to manage). Most are somewhere between. In all cases, how beta fits into the overall testing plan is a strategic decision for every product team.
It’s helpful to remember that a team of real humans is doing real work in the face of real uncertainty and real constraints. They do this with financial constraints, staffing constraints, and time constraints that might seem wrong or illogical to outsiders (and even some insiders). Software companies fiddle with those constraints to deal with their own realities. I for one believe Roon will continue to do this.
There were a couple of head-slapping errors on this release (changing the behavior of bookmarks, messing up tag removal on tracks, and crashing international versions strike me as some of the worst). They fixed the crash bug right away, and they’ve promised to address a number of other issues. Cool by me.
It’s impossible for us to do anything but GUESS whether this release was better or worse than expected simply by reading posts here. Only Roon has enough data for that assessment. But from the viewpoint of any customer who can’t listen to music after the release, it’s “obvious” that testing was inadequate. For those of us where things still work, the opposite seems true.
3 Likes