Embarrassing Search Capability

Roon Software Discussion
42

I agree that a cloud-scale, social network-powered search function would be desirable. I have gone to Amazon to look up a book that I own and remember only the title of, so I can find it on my author-organized library shelves. But this is not fair:

Consider the practicalities of Roon providing such search for Tidal content. I have no inside knowledge of the Roon-Tidal contract, or what business decisions Tidal might make, but we can make informed guesses. I can see three possible solutions.

  1. Roon imports the entire Tidal catalog, adds it to your catalog, and processes it like it does now. It wouldn’t solve the problem, it wouldn’t know about Ed Sheeran’s current popularity. And since Tidal has 60 million tracks (Wikipedia), about 1,000 times a typical (large) private library, the catalog would require terabytes of disk and a lot of RAM and CPU to process. And finally, I believe Tidal would not want to share their entire catalog, including daily updates, with the world.

  2. Roon could import the Tidal catalog and build a cloud service of their own, and connect to it during searches. Tidal may still not want to do it — it doesn’t spread their sensitive data as far and wide, but it means they outsource the protection of their proprietary data to Roon, I would not do that in their shoes, nor would my board allow me. And building such a cloud-scale service is not a small project for Roon, and operating it is not inexpensive.

  3. Tidal could extend the API they provide to Roon so it includes such intelligent, social network-powered queries. Again, I imagine they would be disinclined to share this data. And it’s effort — I don’t think Roon is sufficiently important to Tidal to justify opening up internal data mining services to external partners, especially as they likely evolve rapidly. This is their Crown Jewels.

I think Roon’s future must involve cloud services and data mining, and @Brian has confirmed this. But it is not trivial. And the current implementation does not deserve the opprobrium served up here.

3 Likes
43

Good post. There seem to be a lot of armchair experts on this thread.

44

I agree. Our current search is terrible. It’s a Lucene index over the textual information in the music metadata done in the simplest way possible.

At the time when we built it (prior to our launch in 2015) we had no usage data on which to base a smarter search feature. And of course like @AndersVinberg said, people do not give that kind of data away–even “not for free” and definitely not to partners in the context of a relationship like the one we have with TIDAL. These sorts of data assets are considered extremely valuable and in need of protection.

So we had to build our own–this means acquiring enough users, and waiting enough time for them to create data. Somewhere in 2017, I started to judge our pile of data as “likely to be big enough to do interesting things” and we kicked off projects to begin warehousing it properly + building data/learning pipelines to make use of it.

Along the way, we had to totally re-invent our approach to running/hosting cloud services, carve out real, modern places to run regular batch processing, come up with new processes for development, deployment, release management, etc for all of the data processing bits + cloud services that we will have to build.

We also had to build basic expertise in the team about machine learning–both casual expertise in the hands of our product design team, and also deeper domain/implementation expertise to get the work done. I spent months, personally, studying this field + getting practical experience under my belt just to be able to properly manage this sort of transition. We also employ outside experts for advice (in a consulting capacity) to make sure that we are not missing anything–since this is both a new domain of expertise for us, and also a very fast-moving field.

Recently, we moved 95% of our cloud infrastructure to a new cloud provider, and transitioned most of our services into a container-based architecture–which has opened up significant efficiencies in how we develop/roll cloud services–efficiencies that we will need to deliver on our goals in this area. We “flipped the switch” on that just a few weeks ago after months of running old/new systems in parallel and iterating on the processes, operations management, etc aspects of the “new world”. Like most operational transitions, if it happens silently, it was a success and few people noticed this one…getting that done moved a large amount of backend work done over the past year into the “shippable” column.

All of this is groundwork required to start deploying product improvements based our new data-related capabilities. Whether that be a better search feature that uses usage data to form a model of content relevance + use that to improve results, recommendation systems, smarter navigation paths in-app, improvements to “radio”…there is a lot of potential to be extracted from all of this.

Contrary to the snide remarks about “junior coders”, doing this stuff well is actually a lot of work. The good news is…we’ve been putting effort into this for a long time, and we’ll be able to start releasing the fruits of that labor soon.

22 Likes
45

The question I have, @brian is how and/or if Roon has decided that this infrastructure project is “worth it”. Your major releases last year would have been relatively easy to justify in two words: more revenue. MQA and Chromecast were both bound to expand your base. Easy calls.

It is much harder to internally justify mega-infrastructure projects where there are multiple quarters of new costs followed by…well, what? Very hard to project inflows with accuracy. Will there be net new users attracted by these “hidden” improvements?

Then again, one could justify these expenditures on defensive grounds alone, i.e., “if we don’t improve, we’ll die.” But one can really fool themselves with this tricky logic.

Final comment: there’s a series of projects near and dear to me: the proverbial classical library upgrades. From a financial point though, nailing the “classical issue” in order to appease and appeal to a mere 5% of the market seems like a tough sell for Roon internally.

Just some thoughts: I don’t expect this is a topic for public conversation. Just empathizing with your difficult decision processes.

Good luck, and remember this: people only comment about apps that they really like and want to succeed.

46

I just did a search for The Great Gig in the Sky, and I got it right away. I can’t understand all this kvetching. I’ve never had any of the problems you and others are complaining about (that’s what “kvetching” means, in case you haven’t been around much). This whole thread strikes me as a lot of malarkey. You can use Google to figure that out, too.

1 Like
47

@brian . It’s really great to hear about the work you guys are doing in the background. Please do keep making these updates as it really invigorates us users who’s belief in Roon’s ongoing development may be fading :slight_smile: , and who may be tempting to stray. When you speak, we do listen!

1 Like
48

It works fine if the track is part of your libary (and if the search string is specific enough). In case you have to switch to the “Tidal track results”, it gets very unpleasant. First and foremost, because this results can not be further filtered (or sorted) as it is possible in menu “Tracks”.

1 Like
49

Thank you, Frank. Your post saved me from myself. I was gearing up for a reply using words like “obtuse”, “blinkered” an so on… I might even have quoted Jacob Cats :joy:

50

@brian
Thank you for the very detailed and informative answer.

Are you sure you’re not overshooting the target regarding the simple search function? I have no working knowledge of Lucene, but a quick look at the documentation reveals that Lucene offers a tagged search function in the style of artist: “Singer” AND title:“some song”.

After reading the Roon knowledge base article about the Metadata model, it seems like these data categories are present since track title can be linked to an artist trough the album field.

So although I am excitedly looking forward to the kind of new capabilities you put forward, I sincerely hope you’ll implement a simpler search by artist and/or track name and/or album alongside the more advanced features.

51

The community and Roon is in agreement the search is a problem.
Field searchable forms need not apply to fix this problem.

End of discussion.

52

@brian Thanks for sharing this information and I’m glad to see you and your team are putting significant effort in modernizing Roon’s software architecture. This is important though not glamorous work–certainly needed to not incur too much technical debt and stay competitive.

Now that you have gone to containers and to a more data-centric architecture (not sure if you rearchitected services to be smaller when you did this) I wonder if your development plans and release schedules will change to be more frequent/continuous. I understand if you can’t say… :zipper_mouth_face:

53

Yeah, and I had to walk miles/kilometres to school in the snow. Make them stand outside in the freezing cold with their headphones, searching for music. That’ll learn 'em, I say.

2 Likes
54

We had to make our headphones out of cow patties. Decent midrange, lousy bass. Smelled.

1 Like
55

This occurred to me as I was going to sleep, that people having these problems aren’t cognizant of the kind of software Roon really is: a music library management system that has chosen to append Tidal, and not a universal database search system. Any expectations along the latter lines are misplaced.

56

I don’t think the request here is misplaced, and would suggest that if the Tidal integration were merely “appended” to Roon, then other streaming options would exist as well. The reason there are limitations in the number of streaming partners is directly related to the depth of integration Roon requires, to make streaming sources of music first class citizens in the Roon interface.

@brian has acknowledged above that search could use some polish, which would indicate to me that the requests aren’t unreasonable.

As someone who has used Roon + Tidal to bridge the gap between an existing local library, to now dependence on streaming sources for music (Tidal, in Roon), I don’t think I’m alone as a user who depends fulsomely on the deep, almost invisible integration Roon + Tidal provides. An integration that could use some search improvements, which Roon staff now have acknowledged are coming.

I think the power of Roon is that it caters to a variety of different use cases. It’s your dismissal of our divergent needs that are perhaps misplaced.

1 Like
57

On the backend, sure, we release freely and frequently already…but most of that is invisible and it’s almost always unannounced.

For the clients, we’ve substantially picked up our release pace over the past year already. We’re hoping to manage 3 major releases in ~13 months (counting the upcoming one). The prior three major releases came 9-14 months apart–each. And this doesn’t count the “mid cycle” releases that introduced Chromecast support and the Displays feature in the summer and fall. I don’t think this will get much faster from here–a certain amount of non-negotiable time is needed to build + iterate on product changes/improvements. Speeding up that process (in my experience) leads to releasing half-baked product designs.

Yes…the calculus for this set of improvements is pretty clear. It helps Roon be useful to everyone in the household, and makes Roon sellable to people who are not file hoarders–particularly in a dealer-sales environment where Roon + audio gear are being sold together.

Field-based search is obsolete–fine for a line-of-business app or something like that, but not what people expect for music search.

It’s reasonable to expect search to not show nonsense that is textually similar to the very popular thing that they searched for, like our current search does. This is why Google’s PageRank was so revolutionary–it figured out which content was important independent of its relevance to the search terms, then combined those two ideas to produce the results. The main thing that will make our search feature better is incorporating a content relevancy model (thankfully, music is a lot simpler than web pages in this regard…).

That’s more a description of the past than the future.

5 Likes
58

Personally, I have no problems with the current search function. It fulfills my requirements. But I can understand the objection of the other users very well: a simple smart search has a big impact on the user experience. Roon seems to be aware of this.

1 Like
59

Music to @AndersVinberg 's ears, I expect, and I look forward to seeing what Roon Labs will do…

1 Like
60

It is.

I didn’t want to be dogmatic, but I have never been tempted to do field-based search in Google or Amazon or HDTracks.

Outlook has field-based search, but I have never used it.

61

“Time present and time past
Are both perhaps present in time future,
And time future contained in time past.
If all time is eternally present
All time is unredeemable.
What might have been is an abstraction
Remaining a perpetual possibility
Only in a world of speculation.
What might have been and what has been
Point to one end, which is always present.”

—T. S. Eliot, “Burnt Norton”

1 Like