Pipeline Orchestration: Apache Airflow and Kubeflow

In our previous discussion on data engineering fundamentals for machine learning, we looked at how raw events transform into point-in-time feature tables. In production, however, data extraction, validation, model training, evaluation, and registry promotion cannot run as disconnected manual scripts. They must run in sequence, on schedule, with automated error handling.

This coordination is known as pipeline orchestration. Orchestration systems manage dependencies between computational tasks, execute workloads across clusters, handle transient network retries, and provide operational visibility when steps fail.

In this lesson, you will learn how pipeline orchestration works in machine learning. You will explore Directed Acyclic Graphs (DAGs), task-level retries, and the principle of idempotency. You will examine Apache Airflow and Kubeflow Pipelines, identify when a simple cron job is enough, and see how StayWell orchestrates its weekly customer churn retraining pipeline.

Why machine learning workflows require orchestration

In early development, an engineer might run a single shell script that executes three steps in order:

1python extract_data.py
2python train_model.py
3python evaluate_model.py

While this works on a local laptop, running multi-step workloads in production exposes several failure modes:

1. Unchecked partial failures: If extract_data.py completes but train_model.py fails due to a temporary GPU memory allocation error, a basic script has no mechanism to retry only the failed step. You must re-run the entire pipeline from scratch or manually intervene.
2. Missing dependency graphs: If a feature engineering step depends on two upstream tables that update at different times, a shell script cannot wait for both conditions without fragile custom polling loops.
3. No historic backfilling: If an engineer fixes a bug in feature calculation and needs to recompute historical features for every week of the past year, running manual scripts across fifty-two dates is tedious and error-prone.
4. Lack of operational visibility: When a scheduled job crashes at 2:00 AM, a script gives you a raw terminal log. An orchestrator provides a visual interface displaying exactly which task failed, what error code it returned, and how long upstream steps took.

Anatomy of a directed acyclic graph (DAG)

Modern orchestrators structure workflows as a Directed Acyclic Graph (DAG). A DAG is a collection of tasks with directional relationships between them and no closed loops.

In a DAG:

• Directed means that dependencies have a clear direction: Task A must finish successfully before Task B can begin ($A \rightarrow B$).
• Acyclic means that following the arrows never leads back to an earlier task. The workflow cannot trap execution in an infinite circular loop.
• Graph means the tasks and dependencies form a network of computational nodes and directional edges.

A machine learning training DAG typically includes five sequential phases, as shown in the diagram above:

1. Ingest and validate: Extract source data from warehouses, verify schema assertions, and enforce point-in-time boundaries.
2. Feature engineering: Compute aggregated feature metrics (such as StayWell's 30-day workout counts) and write them to a feature table.
3. Model training: Train a candidate challenger model using fixed random seeds and log parameters to an experiment tracker.
4. Evaluation gate: Compare candidate metrics against production baseline thresholds and historical champions across critical slices.
5. Conditional branching: If the candidate passes all checks, promote the model to the registry; if it fails, halt execution and alert the on-call team.

Two cardinal rules: idempotency and logical dates

To build dependable orchestration pipelines, every task must satisfy two foundational engineering rules:

1. Idempotency

An operation is idempotent if running it multiple times with the exact same inputs produces the identical outcome without duplicate side-effects.

Consider a task that writes daily subscriber features to a database table:

• Non-idempotent design: The task runs INSERT INTO features .... If the network times out and the orchestrator retries the task, the database receives duplicate rows for the same subscriber.
• Idempotent design: The task runs DELETE FROM features WHERE date = '2026-11-04'; INSERT INTO features ... or uses an upsert operation (MERGE INTO ...). If the task is retried three times, the final table contents remain identical.

Every data extraction, feature generation, and model training step should be designed so that re-running it never corrupts state.

2. Logical dates and backfilling

In traditional software, tasks inspect the current system clock (datetime.now()). In pipeline orchestration, relying on the live system clock prevents historical re-runs.

Orchestrators decouple the actual execution timestamp from the logical date (historically called the execution date in Airflow). The logical date represents the specific period of data the pipeline is responsible for processing.

For example, when StayWell's weekly churn pipeline runs on Sunday night, November 8, 2026, its logical date covers the period from November 1 through November 7. If an engineer discovers a bug three months later, the orchestrator can trigger a backfill: executing the pipeline fifty-two times, passing each historical week as the logical date parameter. Because the tasks use the logical date to slice database records, the historical features are reconstructed accurately.

Task retries and failure handling

Production systems experience transient failures: warehouse query timeouts, network blips, temporary API rate limits, or container restart delays.

Rather than failing the entire pipeline immediately, orchestrators allow engineers to configure automated retry policies at the individual task level:

• Retries count: How many times the orchestrator attempts to re-run the task before marking it failed (commonly 2 to 3 attempts).
• Retry delay: How long the orchestrator waits before triggering a retry (such as 30 seconds).
• Exponential backoff: Progressively doubling the wait interval between consecutive attempts (for example, 30 seconds, 60 seconds, 120 seconds) to allow downstream services time to recover from traffic spikes.

If Task 2 in StayWell's pipeline encounters a temporary query timeout while aggregating workout logs, Airflow catches the error, waits sixty seconds, and executes a retry. If the retry succeeds, downstream model training begins on schedule without waking an engineer.

Apache Airflow: concepts and workflow design

Apache Airflow is an open-source workflow management platform created at Airbnb. It is the most widely adopted data pipeline orchestrator in industry.

In Airflow:

• Pipelines are configured entirely as Python code, allowing version control, unit testing, and dynamic DAG generation.
• Tasks are instantiated using Operators. An operator is a template that defines a unit of work. Examples include PythonOperator (executing Python callables), SQLExecuteQueryOperator (running queries against Snowflake, BigQuery, or PostgreSQL), and DockerOperator (running tasks inside isolated containers).
• The Airflow Scheduler monitors DAG definitions, schedules tasks when dependencies are satisfied, and passes jobs to a worker pool (such as Celery or Kubernetes).
• A rich web interface allows engineers to inspect task status, view real-time logs, trigger manual re-runs, and clear failed states.

Illustrative Airflow DAG for StayWell

Here is a simplified Airflow DAG definition orchestrating StayWell's weekly churn model retraining. The imports follow the Airflow 3 documentation (DAG from airflow.sdk, operators from the standard provider). I checked that this file imports and parses as a DAG with five tasks on Apache Airflow 3.3.2 on 5 November 2026, but I did not execute the tasks, and the task bodies only print messages:

1from datetime import datetime, timedelta
2from airflow.sdk import DAG
3from airflow.providers.standard.operators.empty import EmptyOperator
4from airflow.providers.standard.operators.python import PythonOperator
5
6# Default arguments applied to every task in the DAG
7default_args = {
8    "owner": "mlops_team",
9    "depends_on_past": False,
10    "retries": 3,
11    "retry_delay": timedelta(seconds=60),
12    "email_on_failure": True,
13    "email": ["alerts@staywell-app.internal"],
14}
15
16def extract_features(**context):
17    logical_date = context["ds"] # YYYY-MM-DD string
18    print(f"Extracting point-in-time features for cutoff: {logical_date}")
19
20def train_model(**context):
21    print("Training scikit-learn churn pipeline with fixed seed 42...")
22
23def evaluate_and_gate(**context):
24    # Retrieve metrics and verify validation thresholds
25    candidate_recall = 0.68
26    if candidate_recall < 0.60:
27        raise ValueError(f"Gate failed: Recall {candidate_recall} below minimum threshold.")
28    print("Candidate model passed evaluation gate.")
29
30with DAG(
31    dag_id="staywell_churn_retraining",
32    default_args=default_args,
33    description="Weekly scheduled churn retraining with automated validation gates",
34    schedule="0 2 * * 0", # Runs every Sunday at 02:00 AM UTC
35    start_date=datetime(2026, 1, 1),
36    catchup=False,
37    tags=["staywell", "churn", "production"],
38) as dag:
39
40    start = EmptyOperator(task_id="start_pipeline")
41    
42    extract_task = PythonOperator(
43        task_id="extract_and_validate_features",
44        python_callable=extract_features,
45    )
46
47    train_task = PythonOperator(
48        task_id="train_churn_model",
49        python_callable=train_model,
50    )
51
52    eval_task = PythonOperator(
53        task_id="evaluate_model_gate",
54        python_callable=evaluate_and_gate,
55    )
56
57    end = EmptyOperator(task_id="finish_pipeline")
58
59    # Define strict sequential DAG dependencies
60    start >> extract_task >> train_task >> eval_task >> end

In this definition, start >> extract_task >> train_task >> eval_task >> end establishes the directional execution flow. If train_task fails, eval_task will not execute.

Kubeflow Pipelines: containerized workflows on Kubernetes

While Apache Airflow excels at general data movement and scheduled batch operations, Kubeflow Pipelines (KFP) was built specifically for cloud-native machine learning running on Kubernetes clusters.

In Kubeflow Pipelines:

• Every single task in a DAG executes as its own isolated container pod on Kubernetes.
• Tasks can request distinct hardware resources. For example, data extraction can run on a lightweight 2-CPU pod, while deep learning model training requests 4 dedicated NVIDIA GPUs. When training finishes, Kubernetes de-allocates the GPUs, minimizing cloud compute costs.
• Kubeflow includes an integrated Metadata Store that automatically tracks pipeline inputs, outputs, scalar metrics, and model artifact files across every run.
• Components are defined using Python decorators (@dsl.component) that automatically package functions and dependencies into container specifications.

Illustrative Kubeflow component definition

Here is an example showing how a pipeline component is defined in Kubeflow Pipelines:

(from the docs, not run)

1from kfp import dsl
2
3@dsl.component(
4    base_image="python:3.12-slim",
5    packages_to_install=["scikit-learn==1.4.2", "numpy==1.26.4"]
6)
7def evaluate_model_component(
8    test_data_path: str,
9    model_artifact_path: str,
10    min_recall_threshold: float = 0.65
11) -> bool:
12    # Executes inside an isolated Kubernetes pod
13    print(f"Loading model from {model_artifact_path} and evaluating on {test_data_path}")
14    achieved_recall = 0.72 # Illustrative calculated metric
15    
16    if achieved_recall < min_recall_threshold:
17        print(f"Validation failed: Recall {achieved_recall} < {min_recall_threshold}")
18        return False
19        
20    print(f"Validation passed: Recall {achieved_recall} >= {min_recall_threshold}")
21    return True

When is a cron job enough?

Engineering teams often adopt complex distributed orchestrators before their operational requirements justify the infrastructure overhead. Setting up and maintaining a multi-node Apache Airflow cluster or a full Kubernetes-based Kubeflow deployment requires dedicated DevOps engineering time, security configuration, and ongoing maintenance.

Use this decision guide when selecting an orchestration strategy:

A cron job or simple script runner is enough when:

• You have a single machine learning model running on tabular data.
• The training job completes in minutes on a single virtual machine.
• Your pipeline consists of three or four straightforward steps that execute sequentially.
• The pipeline runs infrequently (such as once a week or once a month).
• A failure can be resolved simply by re-running the command the next morning.

Adopt Apache Airflow when:

• You need to coordinate complex dependencies across diverse systems (such as triggering a dbt transformation in Snowflake, running a Python model, and sending a webhook to Slack).
• You require automated retries, alert integrations, and centralized web UI visibility for a team of engineers.
• You regularly need to perform historical backfills across date windows.
• Your organization already runs an Airflow instance for company data engineering.

Adopt Kubeflow Pipelines when:

• Your team trains deep learning models that require dynamic allocation of expensive GPU clusters.
• Individual steps require vastly different software runtimes (for example, C++ for video extraction, PyTorch with CUDA for training, and lightweight Go for notifications).
• Your entire engineering infrastructure is already standardized on Kubernetes.
• You require automated tracking of intermediate dataset artifacts and lineage graphs directly within your container platform.

For StayWell's customer churn model, which trains scikit-learn models on tabular subscriber data weekly, Apache Airflow (or a managed equivalent like AWS MWAA or Google Cloud Composer) is the natural fit. It provides scheduling, retries, and backfilling without the operational complexity of managing Kubernetes GPU nodes.

Sources

Checked on 4 November 2026:

• Apache Airflow Documentation, Core Concepts: DAGs and Tasks.
• Apache Airflow API Reference, airflow.operators.python.PythonOperator.
• Kubeflow Pipelines Documentation, Pipelines overview.

Mistakes beginners make

When designing orchestration workflows for machine learning, developers frequently make several predictable errors:

1. Hardcoding the system clock instead of logical execution dates

Developers often call datetime.now() inside data extraction tasks to query recent records. When the pipeline fails and needs to be re-run three days later, the query extracts data from the wrong time window, corrupting historical feature tables.

Fix: Pass the orchestrator's logical date parameter (such as {{ ds }} in Airflow) into SQL queries and feature scripts, ensuring runs evaluate the intended historical time window.

2. Writing non-idempotent database write operations

Tasks that append records to a destination table using blind INSERT statements produce duplicate rows whenever the orchestrator triggers a retry after a network timeout.

Fix: Ensure every write step is idempotent. Use partition overwrites, truncate-and-load patterns, or upsert statements (MERGE) keyed on unique entity identifiers and timestamps.

3. Passing heavy datasets directly through orchestrator metadata stores

In Airflow, developers sometimes return large pandas DataFrames from one task function to pass them to the next via Airflow's XCom system. XCom stores metadata in the Airflow backend database, and passing multi-megabyte datasets degrades scheduler performance and crashes the database.

Fix: Keep task return values small (such as record counts, execution status strings, or file paths). Store intermediate datasets in cloud object storage (S3/GCS) or warehouse tables, and pass only the storage URI or table name between tasks.

4. Over-engineering with Kubernetes before workloads demand it

Small teams often spend months configuring complex Kubeflow installations on multi-node Kubernetes clusters for basic tabular models that train in forty seconds on a single CPU.

Fix: Match the orchestrator to your current scale. Start with simple scripts or managed Airflow instances, and transition to container-orchestrated platforms like Kubeflow only when compute scale or GPU demands require it.

Recap and try it yourself

In this lesson, you covered the principles and tooling of pipeline orchestration:

• Orchestration coordinates multi-step machine learning workflows, managing dependencies, automated retries, and failure visibility.
• Directed Acyclic Graphs (DAGs) define strict task execution order without circular dependencies.
• Dependable pipelines require idempotency (safe re-execution without side-effects) and logical dates (enabling reproducible historical backfills).
• Apache Airflow is the standard orchestrator for Python-driven data pipelines and scheduled batch model training.
• Kubeflow Pipelines provides container-native execution on Kubernetes, making it ideal for distributed deep learning and dynamic GPU allocation.

Try it yourself

To apply these orchestration concepts, work through these exercises:

1. Design a DAG for StayWell: Draw a five-node DAG for StayWell's customer churn system. If the feature validation task detects that 10% of subscriber records have missing tenure values, how should the DAG route execution?
2. Evaluate idempotency: Consider a task that generates StayWell's monthly churn predictions and writes them to a table named churn_predictions. Write an idempotent SQL statement that safely replaces predictions for the target month without duplicating existing subscriber rows if retried.
3. Compare Airflow vs Kubeflow: A team wants to train an image classification model that takes twelve hours on two NVIDIA A100 GPUs once a month, but trains a churn prediction model on a CPU once a week. How does Kubeflow's container-pod architecture handle this hardware difference compared to running both on a single static virtual machine?

In the next lesson, we will explore batch and streaming data processing, comparing Apache Spark, Apache Kafka, and Apache Flink to understand when distributed big-data engines are necessary.