Apache Airflow

Airflow pipelines for ML and AI

Scheduled data, training and batch inference pipelines that are visible, retryable and easy to change.

Airflow pipeline: asset trigger, extract, quality check, four parallel training tasks, promote
The pipeline defined in the code below: asset-triggered, mapped training per region, guarded promotion.

How we use Apache Airflow

Airflow schedules and monitors pipelines written as Python DAGs. We use it for the data side of AI: ingestion, feature preparation, retraining, batch scoring, embedding refreshes for RAG, and data quality checks between steps.

What we build with it

  • Nightly or event-driven model retraining
  • Batch inference and scoring jobs
  • Document ingestion and embedding refresh for RAG and GraphRAG
  • Data quality and reporting pipelines
See it working

Apache Airflow, in code

The pipeline definition, written for Airflow 3's TaskFlow API and checked against the Airflow Task SDK. The graph above is drawn from it.

airflow_retrain_dag.pyAsset-triggered retraining DAG (Airflow 3 TaskFlow API)
"""Weekly retraining pipeline (Airflow 3 TaskFlow API) used on the vithupro.in Airflow page.
Pulls sales, checks data quality, trains one model per region in parallel (dynamic task mapping),
logs to MLflow and promotes the best model only if it beats the current champion."""
from datetime import datetime

from airflow.sdk import Asset, dag, task

SALES = Asset("s3://vithupro-demo/sales/daily.parquet")


@dag(schedule=[SALES], start_date=datetime(2026, 1, 1), catchup=False, tags=["mlops", "forecast"])
def retrain_demand_forecaster():
    @task(retries=3)
    def extract() -> list[str]:
        return ["north", "south", "east", "west"]  # regions with fresh data

    @task
    def check_quality(regions: list[str]) -> list[str]:
        # row counts, null rates, value ranges; fail the run instead of training on bad data
        return regions

    @task
    def train(region: str) -> dict:
        # fits the model and logs params/metrics/artifact to MLflow (mlflow.start_run)
        return {"region": region, "mae": 17.6}

    @task
    def promote(results: list[dict]) -> str:
        # compares against the 'champion' alias in the MLflow registry before switching
        return "champion unchanged" if not results else "promoted v2"

    regions = check_quality(extract())
    promote(train.expand(region=regions))


retrain_demand_forecaster()

Patterns we rely on

TaskFlow API

Pipelines written as plain Python functions with data passed between tasks.

Dynamic task mapping

One task fans out over files, customers or models at run time.

Asset-driven scheduling

Downstream pipelines start when the data they depend on is updated, not on a guess of timing.

Isolated execution

Heavy steps run in their own containers on Kubernetes, so dependencies do not collide.

Our production checklist

What we put in place before a Apache Airflow system goes live.

  • DAGs in version control with CI checks
  • Retries and alerts per task
  • Data quality checks between steps
  • Secrets in a secrets backend
  • Logs and metrics shipped to monitoring
  • Backfill plan for historical runs

Questions

Managed Airflow or self-hosted?

Managed services (for example Amazon MWAA or Google Cloud Composer) reduce operations. Self-hosting on Kubernetes gives more control. We work with both.

Airflow or Temporal?

Airflow for scheduled data and ML pipelines; Temporal for long-running application workflows and agents.

Often used together with:

Need Apache Airflow engineers?

Tell us what it needs to do. We reply within one business day with questions, a rough plan and the next step.

Book a call
  • Free 30-minute architecture call
  • NDA before you share details
  • You own all code and IP
  • Reply within one business day