MLOps

From four runs to a champion model: experiment tracking with MLflow

Tracking a forecasting experiment in MLflow: four runs, metrics side by side, and a registry alias that serving reads. Real screenshots from our run.

"Which model is in production, and how was it trained?" If answering that means searching notebooks and chat history, the model is a liability. MLflow handles the boring part: every run records its parameters, metrics and artifacts, and the registry records which version is live.

To show the workflow end to end, we ran a small demand-forecasting experiment on synthetic daily sales data (two years, weekly and yearly seasonality, a gentle trend and noise) and tracked it in MLflow 3.17.

Four candidates, one experiment

We trained four models on the same features (sales lagged by 1, 7, 14 and 28 days, plus day of week and month) and held out the last 90 days for testing: two random forests and two gradient-boosting models with different settings.

mlflow_forecast.pyTracked experiment and registry promotion
"""Demand-forecasting experiment tracked in MLflow (synthetic daily sales data), used for the vithupro.in MLflow page.
Logs params, metrics and the model for several runs, then registers the best one with the 'champion' alias."""
import numpy as np
import pandas as pd
import mlflow
from mlflow import MlflowClient
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.metrics import mean_absolute_error, mean_absolute_percentage_error

rng = np.random.default_rng(7)
days = pd.date_range("2024-01-01", periods=730, freq="D")
t = np.arange(len(days))
sales = 200 + 0.15 * t + 40 * np.sin(2 * np.pi * t / 7) + 60 * np.sin(2 * np.pi * t / 365) + rng.normal(0, 18, len(t))
df = pd.DataFrame({"ds": days, "y": sales})
for lag in (1, 7, 14, 28):
    df[f"lag_{lag}"] = df["y"].shift(lag)
df["dow"], df["month"] = df["ds"].dt.dayofweek, df["ds"].dt.month
df = df.dropna()
features = [c for c in df.columns if c not in ("ds", "y")]
train, test = df.iloc[:-90], df.iloc[-90:]

mlflow.set_tracking_uri("sqlite:///runs/mlflow.db")
mlflow.set_experiment("demand-forecast")
candidates = [
    ("rf-200-d8", RandomForestRegressor, {"n_estimators": 200, "max_depth": 8}),
    ("rf-400-d12", RandomForestRegressor, {"n_estimators": 400, "max_depth": 12}),
    ("gbr-300-lr0.05", GradientBoostingRegressor, {"n_estimators": 300, "learning_rate": 0.05, "max_depth": 3}),
    ("gbr-600-lr0.03", GradientBoostingRegressor, {"n_estimators": 600, "learning_rate": 0.03, "max_depth": 4}),
]
best = None
for name, Model, params in candidates:
    with mlflow.start_run(run_name=name) as run:
        model = Model(random_state=7, **params).fit(train[features], train["y"])
        pred = model.predict(test[features])
        mae, mape = mean_absolute_error(test["y"], pred), mean_absolute_percentage_error(test["y"], pred)
        mlflow.log_params({"model": Model.__name__, **params, "horizon_days": 90, "features": len(features)})
        mlflow.log_metrics({"mae": mae, "mape": mape})
        mlflow.set_tags({"team": "vithupro", "dataset": "synthetic-daily-sales-v1"})
        mlflow.sklearn.log_model(model, name="model", input_example=test[features].head(3), skops_trusted_types=["sklearn.tree._tree.Tree"])
        print(f"{name:16s} MAE={mae:6.2f}  MAPE={mape:.3%}")
        if best is None or mae < best[1]:
            best = (run.info.run_id, mae, name)

mv = mlflow.register_model(f"runs:/{best[0]}/model", "demand-forecaster")
MlflowClient().set_registered_model_alias("demand-forecaster", "champion", mv.version)
print("champion:", best[2], "version", mv.version)
Output of this run
rf-200-d8        MAE= 19.79  MAPE=7.315%
rf-400-d12       MAE= 20.00  MAPE=7.391%
gbr-300-lr0.05   MAE= 17.56  MAPE=6.539%
gbr-600-lr0.03   MAE= 18.59  MAPE=6.874%
champion: gbr-300-lr0.05 version 1

Each run logs its parameters, MAE and MAPE, a few tags and the fitted model. The output under the code is from that run.

Comparing runs without a spreadsheet

MLflow UI comparing MAE and MAPE across four forecasting runs
Real MLflow UI: four tracked forecasting runs compared on MAE and MAPE (synthetic sales data).

The comparison view puts every run's metrics side by side. Gradient boosting with 300 trees and a 0.05 learning rate wins, with an MAE of 17.56 and a MAPE of 6.5%. The bigger model with 600 trees did worse. That is exactly the kind of result people forget when it only exists in a notebook cell.

Promotion through the registry

The best run is registered as demand-forecaster version 1 and given the alias champion.

MLflow model registry showing version 1 with the champion alias
Model registry: version 1 carries the champion alias that serving reads.

Serving code loads models:/demand-forecaster@champion, never a file path. To ship a better model, you register version 2, compare it with the champion, and move the alias. Rolling back means moving the alias back.

A safety detail we hit on the way

MLflow 3.17 refused to save our scikit-learn models until we explicitly trusted one type, sklearn.tree._tree.Tree. Newer releases serialise scikit-learn models with skops and block types that could be abused when a file is loaded. We trusted only that one type, because we trained the models ourselves. The lesson generalises: treat model files like code, and load only what you produced or verified.

What we add in production

  • A tracking server with a real database and object storage, behind authentication.
  • A retraining pipeline (we use Airflow) that logs every run to MLflow.
  • An automated check that compares a candidate with the champion before the alias moves.
  • Drift monitoring linked to the model version that is serving.

Start small: one experiment, consistent run names, and a registry alias for what is in production. That alone answers "what's live, and how was it built?"

Vithupro Engineering

The engineering team at Vithupro Infotech, Pune. We build agents, GraphRAG and ML systems for production, and write up what we learn along the way.

Have an AI system to build?

Tell us what it needs to do. We reply within one business day with questions, a rough plan and the next step.

Book a call
  • Free 30-minute architecture call
  • NDA before you share details
  • You own all code and IP
  • Reply within one business day