Evaluating the model in practice

In the previous article we trained the first recommendation model with ML.NET. But a trained model is not the same thing as a good model, and that is exactly the difference I explained in article 2. Time to apply those metrics for real, with code.

Splitting the test with leave-one-out, in practice

In article 2 I explained the leave-one-out technique: for each person, we deliberately hide a song they liked, train without that information, and then see if the model can "guess" it. Let's actually do this, splitting the interactions by user:

var interactionsByUser = interactions
    .Where(i => i.Liked)
    .GroupBy(i => i.UserId)
    .Where(g => g.Count() >= 2); // only enters the test if they liked more than one song

var trainingSet = new List<Interaction>();
var heldOutByUser = new Dictionary<Guid, Guid>(); // UserId -> hidden SongId

foreach (var userGroup in interactionsByUser)
{
    var songsLiked = userGroup.ToList();
    var heldOut = songsLiked[0]; // hides the first one, the rest goes to training

    heldOutByUser[userGroup.Key] = heldOut.SongId;
    trainingSet.AddRange(songsLiked.Skip(1));
}

// The dislikes go straight into training, they are not part of what we hide
trainingSet.AddRange(interactions.Where(i => !i.Liked));

The model is trained with trainingSet, the same way we already saw in the previous article. The question we want to answer: for each user, does the song stored in heldOutByUser show up among the first recommendations the model generates?

Generating each user's ranking

With the model trained, we create a PredictionEngine, the ML.NET piece that makes one prediction at a time, and we use it to score every song in the catalog the user hasn't seen yet:

public class SongRatingPrediction
{
    public float Score { get; set; }
}

var predictionEngine = mlContext.Model
    .CreatePredictionEngine<SongRatingSample, SongRatingPrediction>(trainedModel);

public static class Ranker
{
    public static List<Guid> RankSongsForUser(
        Guid userId,
        IEnumerable<Guid> alreadySeenSongIds,
        List<Song> songs,
        PredictionEngine<SongRatingSample, SongRatingPrediction> predictionEngine)
    {
        var candidates = songs
            .Select(s => s.Id)
            .Except(alreadySeenSongIds);

        return candidates
            .Select(songId => new
            {
                SongId = songId,
                Score = predictionEngine.Predict(new SongRatingSample { UserId = userId.ToString(), SongId = songId.ToString() }).Score,
            })
            .OrderByDescending(x => x.Score)
            .Select(x => x.SongId)
            .ToList();
    }
}

This ordered list is our ranking, exactly the kind of list the metrics from article 2 evaluate.

Implementing Precision@K and Recall@K

Straight from the formula we already saw, just in code:

public static class RankingMetrics
{
    public static double PrecisionAtK(List<Guid> rankedSongIds, HashSet<Guid> relevantSongIds, int k)
    {
        var hits = rankedSongIds.Take(k).Count(relevantSongIds.Contains);
        return (double)hits / k;
    }

    public static double RecallAtK(List<Guid> rankedSongIds, HashSet<Guid> relevantSongIds, int k)
    {
        var hits = rankedSongIds.Take(k).Count(relevantSongIds.Contains);
        return (double)hits / relevantSongIds.Count;
    }
}

An important detail of our specific case: since leave-one-out here hides only one song per person, relevantSongIds is going to have size 1 almost always. This turns Recall@K, in practice, into a simpler question: "did the hidden song show up in the top K, yes or no?". This simplified version even has its own name in the literature, hit rate, and it is one of the most used ways of evaluating a recommender in practice.

Implementing NDCG

Also straight from what I explained in article 2, with the discount by position:

public static class RankingMetrics
{
    public static double NdcgAtK(List<Guid> rankedSongIds, HashSet<Guid> relevantSongIds, int k)
    {
        double dcg = 0;
        var topK = rankedSongIds.Take(k).ToList();

        for (var position = 0; position < topK.Count; position++)
        {
            if (relevantSongIds.Contains(topK[position]))
                dcg += 1.0 / Math.Log2(position + 2); // +2 because the position starts at 0 here

        }

        // IDCG: the DCG of the perfect list, with every hit already at the top
        var idealHits = Math.Min(relevantSongIds.Count, k);
        double idcg = 0;

        for (var position = 0; position < idealHits; position++)
            idcg += 1.0 / Math.Log2(position + 2);

        return idcg == 0 ? 0 : dcg / idcg;
    }
}

Running the evaluation

Putting the pieces together, we calculate the average of each metric across all the users in the test:

const int k = 5;
var precisionScores = new List<double>();
var recallScores = new List<double>();
var ndcgScores = new List<double>();

foreach (var (userId, heldOutSongId) in heldOutByUser)
{
    var alreadySeen = trainingSet.Where(i => i.UserId == userId).Select(i => i.SongId);
    var ranking = Ranker.RankSongsForUser(userId, alreadySeen, songs, predictionEngine);
    var relevant = new HashSet<Guid> { heldOutSongId };

    precisionScores.Add(RankingMetrics.PrecisionAtK(ranking, relevant, k));
    recallScores.Add(RankingMetrics.RecallAtK(ranking, relevant, k));
    ndcgScores.Add(RankingMetrics.NdcgAtK(ranking, relevant, k));
}

Console.WriteLine($"Precision@{k}: {precisionScores.Average():P0}");
Console.WriteLine($"Recall@{k} (hit rate): {recallScores.Average():P0}");
Console.WriteLine($"NDCG@{k}: {ndcgScores.Average():F2}");

Running this against the real 280-song catalog and 60 synthetic users, this is what actually comes out:

Precision@5: 8%
Recall@5 (hit rate): 38%
NDCG@5: 0.25

In other words: a bit more than one out of three times, the song the person would actually like showed up among the top 5 suggestions, out of 280 possible songs, well above the roughly 2% a random guess would get. And when it does show up, it is not always sitting right at the top, hence the NDCG being noticeably lower than the hit rate. It is this kind of reading, combining the three metrics, that separates "the model recommends something" from "the model recommends well".

Is this model actually good?

Numbers alone do not answer that, they need a comparison. A random guess among the candidates would land a hit about 5 out of 280 times, roughly 2%. Landing a hit 38% of the time is about 21 times better than that, which is a clear sign the model picked up a real taste pattern from the synthetic likes and dislikes, it is not just shuffling the catalog and getting lucky.

The Precision@5 of 8% deserves a second look too, because in this specific leave-one-out setup there is only one relevant song per person, so the highest Precision@5 could ever be is 1/5, 20%. An 8% score is 40% of that ceiling, exactly what the 38% hit rate divided by 5 gives, which is a good sign the metric is being computed correctly, not that the model is doing poorly.

The one number that does point at a real weakness is NDCG (0.25) sitting well below the hit rate (0.38): when the hidden song does show up among the top 5, it is usually not sitting right at the top. In other words, the model is reasonably good at narrowing the catalog down to a short list, but still noisy about which one of those five to lead with.

Two caveats matter here. The synthetic listeners are more predictable than real people, an 85%/15% like probability is a much cleaner signal than actual human taste. And the training settings (ApproximationRank = 32, NumberOfIterations = 20) were chosen to train fast for this series, not tuned for the best possible score. So this is not a production-ready, optimized model, it is exactly what this series set out to build: proof that the whole pipeline, from cold start to the collaborative model to the metrics that evaluate it, works end to end and learns something real.

Now that we know how to evaluate the model with confidence, one piece is missing: putting it to work for real, inside an API. That is the subject of the next and last article of the series.

Blog