Author: Brian Luczkiewicz

Roon Search Overhauled – Dec 2022 Update

The Roon team has been working on improving the search engine throughout 2022, and last week, we rolled out a whole new architecture for search.

It’s been just about a year since I last wrote about search, so I figured it was a good time to provide an update on what’s been going on behind the scenes.

The Roon team has been working on improving the search engine throughout 2022, and last week, we rolled out a new architecture for search. If you are running Roon 2.0, you have been using the new stuff for several days.

Previously, the cloud performed a search of TIDAL/Qobuz libraries and the Core performed a search of your personal library, then merged the results.

Now, the Core gathers potential matches for the search query and submits them to a cloud service. The cloud service then searches the TIDAL/Qobuz libraries if needed, ranks and merges the results, and returns the final list to the app.

This change will allow us to deliver better results and improve the search engine more quickly. For more information, keep on reading.

Automated Testing Tools

One of the challenges of improving a search engine is understanding the effects of each change that we make. A change that improves one search might accidentally make other searches worse. In order to make progress, we need to be able to test our search system and understand the intended and unintended consequences of each change.

We introduced an automated testing system in 2022 to help us with that. This system allows us to test thousands of searches in the cloud without using the Roon Core. Our test infrastructure will help us make sure we’re not making things worse as we continue to improve the search engine.

Shorter Cycle Time for Improvements

By moving the search engine to the cloud, we can release improvements without shipping new versions of the Roon Core and apps. This reduces the time it takes to deliver improvements to our users, and allows us to iterate quickly on specific issues when they are reported.

Metrics

While our primary goal is to make our users happy, it’s also helpful to have objective signals that show we are making things better. The new search system includes an array of metrics that allow us to monitor search results quality on an ongoing basis. This helps us see if our changes are helping, and also helps us catch any accidental issues that might degrade the quality of search results for our users.

As an example, the “Average Click Position” metric measures the average position where a user clicks when they select an instant search result. For example, a result of ‘1’ means that the user selected the first item in the list.

These are the results from last week’s rollout. You can see that the number dropped quickly around the time of our rollout, which shows that our changes were an improvement.

Smarter Ranking

One of the challenges with the old search system was merging lists that had been ranked separately in the Core and in the cloud. It was difficult to compare scores from different ranking systems, so we relied on heuristics that tried to balance the ranks, popularity information, and text match accuracy.

This often led to less-than-ideal results, and troubleshooting issues was time-consuming because we had to replicate each user’s library in-house in order to see the issues. Since search ranking is now performed in the cloud in a unified way, this class of tricky issues has disappeared.

More Powerful Algorithms

Today, state-of-the-art search systems use Machine Learning to deliver results. It was difficult to use these techniques in the old Roon Core, so one of the main goals of the re-architecture was to enable this.

In the future, we will be able to tailor results to individual users. For example, a Beatles fan typing “John” into the search box probably expects “John Lennon” to be the top result, while a Jazz listener might expect “John Coltrane”.

This will also enable the implementation of modern search techniques like spelling corrections, search suggestions, semantic search, and gradient boosting, none of which are practical within the Roon Core.

Instant Search Improvements

After deploying the new architecture internally in September, we started the process of tuning it to perform better than the old system. We focused on improving the instant search dropdown, making sure that fewer characters are required to reach the desired result, and that the results continue to feel sensible as more characters are entered.

In addition to automated testing, we have a weekly review process where the product and search teams come together to examine a set of 50 representative searches and discuss how the results changed because of the previous week’s work. This helps us understand the tradeoffs and make decisions that prioritize the user experience.

Dozens of Smaller Improvements

As part of this work, we made many of smaller improvements to the search system. Shorter queries like “john” or “pink” should now return more coherent results. The system is not directly auto-correcting misspellings, but it is more tolerant of misspelled terms in multi-word searches. It is also better at prioritizing exact matches, deduplicating similar results, and choosing the best version of an album or track when there are multiple versions available.

While things have definitely gotten better in the past year, 2022 was mostly about laying the groundwork. We plan to make faster and more visible progress in 2023 and beyond.

–

I want to thank everyone for their patience as we work on improving the search engine, and for all the feedback in last year’s thread. It has been very helpful in understanding our users’ perspectives, and we hope you will continue to use our product and provide feedback in the future.

Improving how you search for music in Roon

I’d like to fill you all in on what’s been happening with search in Roon, what we have done in our latest release to make things better, and a little bit of what we have planned for the future.

Before I get started, one piece of background: we feel strongly that it’s best for the product to present a single, clear answer to a search query that blends both library and streaming content without putting them into separate “silos” as some other products do. This allows Roon to give a single set of answers to a query without forcing the user to pick apart, disambiguate, or dig deeper based on where the results are coming from. You’re not searching your library OR searching TIDAL, you’re just searching and getting results. It’s a simpler and better experience.

Thus, Roon has to independently search your library, held within the Roon Core, and streaming content, held within cloud services, and then merge the results together. This merging problem is a tricky one. You don’t often see interleaved results from multiple search engines, and there’s a good reason for that; the fact that we hadn’t completely cracked it had left Roon’s search experience in an unacceptable state.

A bit over a year ago, we decided that not only did this have to be fixed, but that search was a “forever problem” – not something that we could fix once and forget about. It requires continual care and feeding and dedicated staff who think about search and only search, so we hired a search specialist and about a year ago and we got to work tackling search with fresh eyes.

We released the auto-complete feature earlier this year, and in building that, gained a detailed understanding of exactly how and why our existing search engine was getting things wrong. That allowed us to kick off the “big project”: an overhaul of Roon’s search infrastructure end to end.

We began by analyzing hundreds of complaints and reports from the Roon community to understand what the problems were. We used your feedback to build test cases and validate our work. Separately, we analyzed anonymized data from our servers to understand what real-world search queries looked like.

As we dug deeper, we figured out that one of the major problems is that the search engine used for the Roon library just worked too differently from the search engine used for streaming content. The two search engines computed and scored results according to different principles, each established during different eras of Roon’s product development.

The library algorithm generally returned results that were too noisy and numerous, and in a significant number of cases, noise from the library drowned out more accurate streaming results. This was especially painful for people with large libraries.

Another problem that we found is that queries for classical music just look different from queries for other content, and Roon’s search engine was behaving particularly badly with some of these queries.

We decided that in general, our approach to cloud-based search was sane (if in need of some tweaks), and the approach to library search was, quite simply, wrong.

Thus, the library search engine required a complete, ground-up rewrite. Since the most mature search technology is cloud-based and Roon’s library is not, we ended up building an embedded search engine that implements the same ideas as cloud-based engines like ElasticSearch, but in a way that lets it run inside of the Roon Core.

We also built a model that can distinguish classical and non-classical search queries prior to performing a search, so that we can tweak various parts of the search process to produce more appropriate results for classical or non-classical queries. Alongside this, we updated the user interface to give more priority to composers and compositions when a classical search is detected, which should save classical users a bit of scrolling.

Then, we had to come up with a new approach to merging library and cloud results. This required a fair amount of consideration, but we ended up landing on a really neat (and as far as we know, novel) approach for making consistent scores for search results that came from different search engines, and we’ve implemented it in Roon.

Finally, we spent months testing this stuff amongst ourselves, then with increasingly larger groups of users, until it was clear that people were feeling improvement. During this process, we iterated on all parts of the system.

I’m confident that the major and structural issues with Roon’s search engine have been addressed. I’m also sure that for the foreseeable future, people will sometimes run into searches that they don’t feel are working right. Search is a “forever problem”, right?

Now that the bulky work is done, we will be able to iterate with the Roon community more rapidly as feedback comes in, and we intend to continue improving search indefinitely.