Recommendation systems: how Spotify knows what you want to hear

This article opens a new series here on the blog, and it has a very practical destination: building, from scratch, a rock music recommendation API, similar to what Spotify and Deezer do, with everything running on .NET. But the real subject of this series is not just "building a recommender that works". It is understanding, in depth, how to know if it works well. That second part is the one almost nobody explains, and it is exactly where this series is going to spend the most time.

Before we get there, we need to understand something simpler: how does a machine decide which song to play next? As always here on the blog, in the easiest way I can manage. If you have never programmed, you can come along.

And let me answer a question that always comes up: we are going to use ML.NET, the machine learning path maintained by Microsoft itself. The Python ecosystem is the standard of the ML market and deserves all the respect, but .NET has its own mature path for this, and that is the path this whole series will walk.

Recommendation by content: recommending something similar to what you already like

The simplest way to recommend something is to compare characteristics. If you tell me you like Iron Maiden and Metallica, I can look at my catalog and find other bands with heavy metal, heavy guitar and a strong vocal, and suggest them to you. I didn't need to know anything about other people, I just needed to know my catalog well and what you said you like.

This is called content-based recommendation. It looks at the characteristics of what you already liked (genre, subgenre, era, instrumentation) and looks for similar things. It's simple, easy to explain to the user ("we recommended this because you like a similar band") and it works well from the very first second, even with no history from the person at all.

The problem is its limit. Content-based recommendation tends to trap you in a bubble. If you only said you like heavy metal, it will never suggest that alternative rock band you would probably like a lot, because it doesn't know what you didn't mention. It's great to get started, but on its own it doesn't evolve.

Collaborative recommendation: learning from everyone's taste

Now think differently. Instead of looking at the characteristics of the songs, I look at people's behavior. If you and I liked the same ten bands, and I also like a new band you have never heard, it's quite likely you will like it too. I don't know anything about that band's sound, guitar, drums, none of that. I just know that people similar to you liked it.

This is called collaborative recommendation, and it is the engine behind the big streaming services. It doesn't need to understand music, it needs to understand patterns of taste among thousands of people. The more people using the system and giving feedback (liked, didn't like, listened all the way through, skipped), the richer this network of patterns gets, and the better the recommendation.

The classic technique that pulls off this kind of magic is called matrix factorization. Without going into the formula, the idea is: the system learns, on its own, a set of hidden characteristics for each song and for each person, in a way that "combining" these characteristics predicts how much that person is going to like that song. Nobody wrote these characteristics by hand, they emerge from the data, and this is exactly the kind of model we are going to train further ahead in the series, using ML.NET.

And to place this properly on the map: this is supervised learning, the same category used to detect spam or fraud, the model learns from known examples (this person liked this song, this person did not) and uses that to guess the labels it hasn't seen yet. It's worth saying what it is not, too: it's not a transformer, it's not an LSTM, and it's not a deep neural network either. Those are architectures built for sequences, where the order of what comes before matters a lot, like a sentence. Matrix factorization doesn't care about order, each interaction is just an independent fact to it. It's older and simpler than a transformer, and it remains one of the most used techniques in the recommendation industry to this day, precisely because it works so well for this specific problem.

The problem of the empty start: cold start

Here is the trap of collaborative recommendation: it needs data to work. A user who just arrived hasn't liked anything yet, hasn't skipped anything, has no behavior pattern at all. For this system, that person is a complete mystery. This problem even has a name, cold start, and it happens on both sides: with a new user, who hasn't given any signal yet, and with a new song, which hasn't been listened to by enough people yet to have a reliable pattern.

That is why, in practice, the big services mix both approaches. When you enter a music app for the first time, it's quite common for it to ask which artists or genres you like. That is not just a nice welcome screen, it is the solution to cold start: while the system doesn't know you by behavior, it knows you by what you told it yourself, and it uses content-based recommendation to give you the first guesses. As you like or dislike the suggestions, the system starts building up real signal, and collaborative recommendation takes the wheel.

This is exactly the path our rock recommender is going to follow further ahead in the series: the person picks a few bands they like to solve the cold start, and the system learns from the like and the dislike of each suggested song.

And where does model evaluation come into this story?

Here is the point almost every recommendation tutorial skips. It's easy to train a model and see that it is "recommending something". It is a lot harder to answer the question that really matters: is this model recommending well, or is it just recommending?

A recommendation model can look great on the numbers and be terrible in practice, or the other way around. The metrics we use to measure "correctness" in a common prediction (like predicting whether a customer will cancel a service) don't tell the whole story when the subject is recommendation, because here what matters is not just predicting right, it's getting the order right: out of the hundred songs in the catalog, are the ten the model put at the top really the ones the person would most want to hear?

That is the question the next article in the series is going to answer, calmly, with a numeric example of every metric involved. It is the true heart of this series.

Conclusion

Recapping:

Content-based recommendation compares the characteristics of what you already like with the catalog, works from the first second, but traps you in a bubble
Collaborative recommendation learns from the taste pattern among thousands of people, and usually uses matrix factorization under the hood
Cold start is the problem of the empty beginning, both for a new user and for a new song, and the practical solution is mixing both approaches
In our rock recommender, the bands picked at the start solve the cold start, and the like/dislike on each song feeds the collaborative part
Knowing that a model "is recommending" is not the same as knowing that it is recommending well, and that is exactly the difference this series is going to explore starting from the next article

Blog