Closing the feedback loop in RockRecommender
A six-part series that finishes RockRecommender's training pipeline. It learns to decide automatically between synthetic and real feedback, compares a candidate model against production before promoting it, runs the retraining on a schedule, and reloads the model live, with a real NaN bug found and fixed along the way.
What you'll learn
- How to decide automatically between synthetic data and real feedback by comparing volume
- How to compare a candidate model against the one in production before promoting it
- How to automate a periodic retraining job with BackgroundService and PeriodicTimer, with no scheduling library
- How to reload an ML.NET model in production without restarting the application
- How to defend a model comparison against ML.NET returning NaN scores for users it has never seen
Articles in this series
RockRecommender: closing the loop between feedback and training
First article of a new series. RockRecommender has been collecting real like and dislike feedback since the API article, but the model is still trained only on synthetic data. This series closes that loop: the pipeline learns to decide on its own when real data is enough, compare the new model against the current one before switching, and reload everything without restarting anything.
RockRecommender: reading real feedback straight from Mongo
Second article in the series. The training pipeline gets its first new piece: a rule that decides on its own, by comparing volumes, when there's already enough real feedback to set the synthetic data aside.
RockRecommender: comparing models before promoting
Third article in the series. After picking the data source, the pipeline avoids the most obvious mistake of any automated retraining: replacing the production model just because the new one is newer, not because it's better.
RockRecommender: automating retraining with a scheduled job
Fourth article in the series. Turning the pipeline into a service that runs on its own ran into a classic dependency injection trap: a long-lived service can't just inject a short-lived one.
RockRecommender: reloading the production model without restarting the Api
Fifth article in the series. The last piece sits on the Api side: noticing on its own that a new model was promoted, with no restart and no manual endpoint needed.
RockRecommender: the feedback loop closed, from a like to the model in production
Sixth and last article in the series. An end-to-end walkthrough joining the five previous pieces, and an honest close about what was left out, on purpose.