Airflow pipelines for ML and AI
Scheduled data, training and batch inference pipelines that are visible, retryable and easy to change.

How we use Apache Airflow
Airflow schedules and monitors pipelines written as Python DAGs. We use it for the data side of AI: ingestion, feature preparation, retraining, batch scoring, embedding refreshes for RAG, and data quality checks between steps.
What we build with it
- Nightly or event-driven model retraining
- Batch inference and scoring jobs
- Document ingestion and embedding refresh for RAG and GraphRAG
- Data quality and reporting pipelines
Apache Airflow, in code
The pipeline definition, written for Airflow 3's TaskFlow API and checked against the Airflow Task SDK. The graph above is drawn from it.
"""Weekly retraining pipeline (Airflow 3 TaskFlow API) used on the vithupro.in Airflow page.
Pulls sales, checks data quality, trains one model per region in parallel (dynamic task mapping),
logs to MLflow and promotes the best model only if it beats the current champion."""
from datetime import datetime
from airflow.sdk import Asset, dag, task
SALES = Asset("s3://vithupro-demo/sales/daily.parquet")
@dag(schedule=[SALES], start_date=datetime(2026, 1, 1), catchup=False, tags=["mlops", "forecast"])
def retrain_demand_forecaster():
@task(retries=3)
def extract() -> list[str]:
return ["north", "south", "east", "west"] # regions with fresh data
@task
def check_quality(regions: list[str]) -> list[str]:
# row counts, null rates, value ranges; fail the run instead of training on bad data
return regions
@task
def train(region: str) -> dict:
# fits the model and logs params/metrics/artifact to MLflow (mlflow.start_run)
return {"region": region, "mae": 17.6}
@task
def promote(results: list[dict]) -> str:
# compares against the 'champion' alias in the MLflow registry before switching
return "champion unchanged" if not results else "promoted v2"
regions = check_quality(extract())
promote(train.expand(region=regions))
retrain_demand_forecaster()Patterns we rely on
TaskFlow API
Pipelines written as plain Python functions with data passed between tasks.
Dynamic task mapping
One task fans out over files, customers or models at run time.
Asset-driven scheduling
Downstream pipelines start when the data they depend on is updated, not on a guess of timing.
Isolated execution
Heavy steps run in their own containers on Kubernetes, so dependencies do not collide.
Our production checklist
What we put in place before a Apache Airflow system goes live.
- DAGs in version control with CI checks
- Retries and alerts per task
- Data quality checks between steps
- Secrets in a secrets backend
- Logs and metrics shipped to monitoring
- Backfill plan for historical runs
Questions
Managed Airflow or self-hosted?
Managed services (for example Amazon MWAA or Google Cloud Composer) reduce operations. Self-hosting on Kubernetes gives more control. We work with both.
Airflow or Temporal?
Airflow for scheduled data and ML pipelines; Temporal for long-running application workflows and agents.
Often used together with:
Need Apache Airflow engineers?
Tell us what it needs to do. We reply within one business day with questions, a rough plan and the next step.
- Free 30-minute architecture call
- NDA before you share details
- You own all code and IP
- Reply within one business day