In the previous article, the pipeline gained the right data source. But training a new model isn't, by itself, a reason to put it into production. It has to beat what's already there.
IsBetterThan: comparing three metrics without collapsing them into one
The leave-one-out evaluation (from the evaluation article) calculates Precision@K, Recall@K and NDCG@K separately. Adding the three into a single score sounds convenient, but this test holds out exactly one relevant song per user, which means Precision@K is always Recall@K divided by K, the same hit or miss, just on a different scale. Summing them would count that one signal twice and let it drown out NDCG, the only one of the three that also weighs where in the ranking the hit landed.
So the decision keeps the three metrics separate, and compares them one by one:
public sealed record LeaveOneOutResult(double PrecisionAtK, double RecallAtK, double NdcgAtK, int EvaluatedUserCount)
{
public bool IsBetterThan(LeaveOneOutResult other) =>
PrecisionAtK >= other.PrecisionAtK &&
RecallAtK >= other.RecallAtK &&
NdcgAtK >= other.NdcgAtK &&
(PrecisionAtK > other.PrecisionAtK || RecallAtK > other.RecallAtK || NdcgAtK > other.NdcgAtK);
}A model only counts as better if it never loses on any of the three metrics, and wins on at least one. If the candidate improves NDCG but quietly loses on Recall, it doesn't get promoted, exactly the kind of trade-off a single summed score would have hidden.
Evaluating the model that's already live
The original leave-one-out evaluation only knew how to evaluate a model it had just finished training. To really compare, the pipeline also needs to load the model that's serving the Api today, and evaluate it against the same data split used for the candidate:
public static class TrainedModelLoader
{
public static PredictionEngine<SongRatingSample, SongRatingPrediction>? TryLoad(MLContext mlContext, string modelPath)
{
if (!File.Exists(modelPath))
return null;
using var stream = File.OpenRead(modelPath);
var model = mlContext.Model.Load(stream, out _);
return mlContext.Model.CreatePredictionEngine<SongRatingSample, SongRatingPrediction>(model);
}
}If the active model's file doesn't exist yet (the first time the pipeline runs in a new environment), there's nothing to compare against, and the candidate gets promoted with no competition.
Writing to the candidate first, never straight to the active one
The decision uses the same evaluation split (the same train/test division) for both the candidate and the active model, so the comparison is fair:
var split = LeaveOneOutEvaluator.Split(trainingInteractions.Interactions);
var candidateResult = EvaluateCandidate(mlContext, split, songs, settings.EvaluationK);
var activeResult = EvaluateActiveModel(mlContext, split, songs, settings.EvaluationK, settings.ModelPath);
var promoted = activeResult is null || candidateResult.IsBetterThan(activeResult);Up to this point, the pipeline has only read the active model's file, it hasn't written anything to it. Only after the decision is made does it train the final model (now on 100% of the data, not just the slice used for evaluation) and write it out:
private static void TrainAndPromoteFinalModel(MLContext mlContext, List<Interaction> interactions, string candidateModelPath, string activeModelPath)
{
var samples = interactions.Select(SongRatingSampleMapper.ToSample);
var trainedModel = RecommenderModelTrainer.Train(mlContext, samples);
mlContext.Model.Save(trainedModel.Model, trainedModel.InputSchema, candidateModelPath);
File.Copy(candidateModelPath, activeModelPath, overwrite: true);
}Notice the order: the final model is saved first to a separate, candidate-only path, and only then copied over the active path. If that final training run had failed halfway through, or if the decision had been "don't promote", the model serving the Api would have stayed exactly as it was.
A real case that only showed up testing with real users
This comparison was designed with synthetic users in mind. Testing it with real users, created through actual HTTP requests, uncovered a problem that hadn't shown up before: the active model, trained only on synthetic data, scored a perfect result against real users it had never seen, and the candidate, trained on the real data, scored zero, an impossible result on both ends.
The cause: ML.NET returns NaN as a score whenever the UserId (or the SongId) is unknown to the model. Since the active model had never seen any of these users, every prediction for them came back NaN. And since C# treats every NaN as "equal" to another NaN when sorting, the tiebreak falls back to the original list order, not to any real prediction, a result that looks good or bad purely by chance.
The fix lives in the ranking: if the song that was held out for testing (the one that decides a hit or a miss) scores NaN for the model, that user doesn't count toward that model's evaluation, because it genuinely has no opinion about that person. Any other candidate scoring NaN (a song nobody interacted with enough to have a learned profile) just drops out of the ranking, without discarding the whole user:
var scoredSongs = candidateSongs
.Select(song => new ScoredSong(song.Id, predictionEngine.Predict(new SongRatingSample { UserId = userId.ToString(), SongId = song.Id.ToString() }).Score))
.ToList();
var heldOutScore = scoredSongs.Single(scoredSong => scoredSong.SongId == heldOutSongId).Score;
if (float.IsNaN(heldOutScore))
return null;
return [.. scoredSongs
.Where(scoredSong => !float.IsNaN(scoredSong.Score))
.OrderByDescending(scoredSong => scoredSong.Score)
.Select(scoredSong => scoredSong.SongId)];After this fix, comparing the active model against real users it had never seen started showing exactly what's true: "Evaluated users: 0", because it genuinely has no information about them at all. And the candidate, which learned from those same users, shows a real result, neither perfect nor zeroed out.
The next article covers when this whole cycle runs: not on every feedback event, and not only when someone remembers to run it by hand.