How to evaluate a recommendation model

In the previous article of this series, we saw the two ways of recommending something (content-based and collaborative) and the cold start problem. At the end, I left a question open: knowing that a model "is recommending" is not the same as knowing that it is recommending well. This article exists to answer exactly that.

As always, calmly and with a practical example, because a metric without an example is just a memorized formula.

Why the usual metrics are not enough here

If you have seen anything about machine learning, you have probably heard of accuracy, that is, how many times the model got it right. That works well when the question is simple, like "is this email spam or not". But recommendation is not a yes-or-no question, it's a question of order.

Think about it this way: our rock recommender is not going to give you a single answer, it's going to hand you a list, something like "here are the next ten songs I think you are going to like, in this order". Two lists can have exactly the same ten correct songs, only one of them puts the song you would like the most in first place, and the other leaves it in ninth. Both "got the songs right", but one delivered a much better experience. That kind of difference is what plain accuracy can't see, and that is why recommendation has its own set of metrics.

Precision@K: of the suggestions I gave, how many did I get right

This is the most intuitive metric. Precision@K (precision at the top K) answers: of the first K songs I recommended, how many did the person actually like?

Example. I recommend 5 songs to you (K = 5). You like 3 of them and don't like 2. The math is simple:

Precision@5 = liked songs / recommended songs
Precision@5 = 3 / 5 = 0.6 (or 60%)

Sixty percent accuracy at the top of the list. It looks like a great metric, and it is, but it has a limitation: it doesn't know how much is still left to discover. If your catalog has 50 songs you would probably like, and I only showed you 3 of them in those 5 recommendations, Precision@K doesn't penalize that. To see this other half of the story, we need another metric.

Recall@K: of what the person would like, how many did I find

Recall@K (recall at the top K) looks the other way around: of everything the person would actually like, how much did I manage to place in my K recommendations?

Continuing the example. Let's say there are, in total, 10 songs in the catalog that you would like to hear (we find this out by observing their history, or in a test, by hiding some songs on purpose that we know they liked, more on that shortly). Of the 5 I recommended, 3 were among those 10. The math:

Recall@5 = relevant songs found / total relevant songs
Recall@5 = 3 / 10 = 0.3 (or 30%)

I only found 30% of what existed that was good for you. Now notice the natural tension between the two metrics: if I raise K, recommending 20 songs instead of 5, my chance of catching more of the 10 relevant ones goes up (recall improves), but the chance of filling that bigger list with songs you don't like also goes up (precision drops). There is no magic here, there is balance, and that is why the two metrics almost always show up together, never alone.

NDCG: when the spot in the list matters

Neither Precision@K nor Recall@K cares about where, within the top K, the relevant song landed. To them, hitting the first position is worth exactly the same as hitting the last one. In practice, this matters a lot: almost nobody scrolls down to the tenth suggestion, most people only look at the first ones.

To solve this, there is the NDCG (Normalized Discounted Cumulative Gain, a complicated name for a simple idea: hits at the top of the list are worth more than hits at the bottom).

A lean example. Imagine two lists of 3 recommendations, marking with 1 when the person liked it and 0 when they didn't:

List A: [1, 0, 0]  (hit the first position)
List B: [0, 0, 1]  (hit the third position)

Both lists have exactly one hit in three attempts, so Precision@3 is the same for both (0.33). But List A delivered the hit right away, and List B made the person go through two wrong suggestions before reaching the good one. NDCG applies a "discount" the further down the hit is (usually dividing the value by the position, or by something that grows with the position), sums everything up, and then normalizes that value by comparing it to the best possible list, which gives a number between 0 and 1. In the end, List A gets a higher NDCG score than List B, and that is exactly the difference Precision@K would never show. That is why NDCG is considered the most complete metric to measure the real quality of a ranking.

RMSE and MAE: the classic metric, and why it alone is misleading in a recommender

If you have read anything about recommendation systems, you have certainly seen these two acronyms: RMSE (root mean squared error) and MAE (mean absolute error). They come from an older way of thinking about recommendation, as a problem of predicting a rating. The model tries to predict "this person would give this song a 4.2", and then we compare it to the real rating they gave.

A simple example with MAE. The model predicted the ratings 4.0 and 2.0 for two songs, and the real ratings the person gave were 4.5 and 3.0:

Error 1 = |4.0 - 4.5| = 0.5
Error 2 = |2.0 - 3.0| = 1.0
MAE = (0.5 + 1.0) / 2 = 0.75

On average, the model missed the rating by 0.75 points. That looks useful, and it is, but here is its limitation: this metric measures how close the predicted rating landed to the real rating, not whether the order of the recommendations turned out good. A model can have a great MAE and still put the wrong song in first place, if the predicted ratings are all close to each other. Since in our recommender the signal we have is like and dislike (not a rating from 1 to 5), we will even use an adapted version of these error metrics further ahead in the series, but the real leading role stays with Precision@K, Recall@K and NDCG, because they are the ones that answer the question that really matters: was the list I delivered good?

Splitting training and testing in a recommender

A specific catch in recommendation, worth getting ahead of here. In common machine learning, we split off part of the data for testing at random. In a recommender, doing this carelessly can leak information: if I accidentally leave a song the person liked last year mixed into training and another one from last month into testing, both from the same person, the model might be "peeking" at a piece of their behavior that should have stayed hidden.

The most common technique to avoid this is called leave-one-out: for each person, we deliberately hide one (or a few) of the songs they liked, train the model without that information, and then test whether the model can "guess" that hidden song among the recommendations it generates. That is exactly the trick we used earlier to figure out the "total relevant songs" in the Recall@K example.

Conclusion

Recapping:

Recommendation is a problem of order, not yes or no, and that is why it uses its own metrics
Precision@K measures how much of the recommendations was correct, but ignores how much is still left to discover
Recall@K measures how much of what was good I managed to find, but encourages lists that are too big if used alone
NDCG solves the limitation of the two previous ones, giving more value to a hit at the top of the list
RMSE and MAE come from the older way of thinking about recommendation (predicting a rating), and they remain useful, but they don't replace ranking metrics
Training and testing in a recommender usually uses the leave-one-out technique, deliberately hiding part of what the person liked

With these metrics in hand, we already have a way to honestly answer whether our rock recommender is good or not. In the next article, we get our hands dirty: we build the dataset of rock bands and songs, and train the first model with ML.NET.

Blog