The previous two articles settled the what: which data and which model. What's missing is the when. RockRecommender.Training stops being a console app that runs once and exits, and becomes a long-lived host that wakes up on a configurable interval.
A BackgroundService with PeriodicTimer
PeriodicTimer is a native .NET feature for "do something every X time", with no scheduling library needed:
public sealed class RetrainingBackgroundService(IServiceScopeFactory scopeFactory, IOptions<TrainingOptions> options, ILogger<RetrainingBackgroundService> logger) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var timer = new PeriodicTimer(options.Value.RetrainInterval);
do await RunPipelineAsync();
while (await timer.WaitForNextTickAsync(stoppingToken));
}
private async Task RunPipelineAsync()
{
try
{
using var scope = scopeFactory.CreateScope();
var pipeline = scope.ServiceProvider.GetRequiredService<TrainingPipeline>();
await pipeline.RunAsync();
}
catch (Exception exception)
{
logger.LogError(exception, "Retraining run failed");
}
}
}The do/while is deliberate: the first run happens as soon as the host comes up, without waiting for the first interval to pass, and the following ones only happen on each tick of the timer.
Why an IServiceScopeFactory, and not the pipeline directly
The first version of this service injected TrainingPipeline straight into the constructor, and the host refused to start. The reason is a lifetime rule: a BackgroundService is a single instance for the whole life of the application (a singleton), but TrainingPipeline depends, underneath, on things like the feedback repository, which have a shorter lifetime, one per operation. A singleton can't directly depend on something with a shorter lifetime, the container refuses that mix at runtime.
The way out is to request a new scope on every run, and resolve the pipeline inside it, exactly what this service does in RunPipelineAsync. That also has a nice side effect: every retraining run gets its own instances, with nothing left over from one run to the next.
One bad run doesn't take down the host
The try/catch around each run exists because this is a long-lived process: if a retraining run fails (Mongo was briefly unreachable, for example), the host stays up and tries again at the next interval, just logging the error.
The interval between runs is just one more setting, alongside the ones that already existed (SyntheticUserCount, EvaluationK), so switching from once a day to once an hour, for example, doesn't require any code change.
One piece is still missing: what's the point of automatically promoting a new model if someone still needs to restart the Api for it to take effect? The next article closes exactly that gap.