Some updates–
We rolled out a few changes in the past 72hrs, and have been monitoring intensively this week. As I mentioned above, we have more than one angle of attack here.
The really long searches–30 seconds or more–are operational problems that happen when an individual backend server becomes overloaded temporarily. We’ve made changes that have reduced the frequency of this failure mode by about 97%, but we are still seeing a short window each day–a few minutes right around peak usage for the day, where one or two of our backend servers gets “behind” on processing requests.
One more change is rolling out tomorrow morning to hopefully get the last of this problem squashed, but if it doesn’t play out that way we’ll obviously keep at it. It’s much quicker to iterate on a problem when it happens a few times an hour (like earlier in the week) than once per day (the situation we are in now), because to do a proper controlled experiment at this point, we need roll something out and see how it does through the peak hours.
The “2-5 second” slow search experience is caused mostly by the amount of work done when processing search results + formatting them for display. Yesterday, we released a performance optimization that cuts the per-search processing time in our servers by about 60%. That went live late in the day yesterday (in the US) and has made a substantial difference. Our “end to end” benchmark, which looks at the time it takes the Roon Core to generate a full screen of search results across all data types has dropped from an avg of 3.2s/search to 1.2s/search.
Obviously I’d like that to be even smaller, but this change has really made things feel noticeably better.
That last optimization also improves load times for the artist screen, the TIDAL/Qobuz screens, the playlist screen (when loading TIDAL/Qobuz playlists), TIDAL/Qobuz library sync speed, and the “versions” tab on the album pages.
Finally, a crash that contributed “no search results when there should be” problems for people logged into both TIDAL and Qobuz at once is also fixed as of yesterday evening.